Diff: data-log-text.txt

Revision:

4:fa5bbe31a039

Parent:

3:5550814cc21c

Child:

5:286459acee56

--- a/data-log-text.txt	Wed Mar 30 21:44:22 2016 +0000
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,117 +0,0 @@
-To make the data log format somewhat flexible and 
-some what easy to debug while remaining somewhat
-consise while remaining human readable.  The format
-will be a custom format but is designed to be relatively
-easy to parse while accomodating the following caveates:
-   * Fields logged over time will change as firmware is reved.
-   * Old log entries for the same record type will have different fileds.
-   * storage in fram is realtive expensive so self describing data 
-     formats like JSON would not be cost effective. 
-   * When things mess up we want to be able to parse it and 
-     analyze it and have some chance of recovery 
-   * We need fast access to pont in time for at least 1 day 
-     of data. 
-     
-will use a custom format.  
-
-Each record type will have record type label which will be
-composed of 4 characters separated from the data by a The known
-record types are:
-
-Records will be appended to the data log in sequential order. Each
-record will be terminated with a \n.  Some records will contain fixed
-length data but will still be terminated with \n and will never contain
-a \n in the stored data. unless the first number after the record type 
-is 16 bit number storing the number 
-
-The general assumption is that all data will be stored for the short
-term in high speed FRAM or SRAM chips and copied to EPROM or Micro 
-SD chips in the background once sufficnet data has accumulated to 
-minimize over-write fatigue in the long term storage.  I used FRAM 
-specifically to allow after power loss data retention without having 
-to worry about overrite fatigue for specific segments that would need
-rapid updates.    The other assumption is that data is stored and logged
-then will be accessed generally starting at a given date reading forward 
-
-Short term Simplification Notes:
-  Each time the CPU reboots it will record current record header field 
-  names for all record types active at that time.  This will create some
-  duplicates but save some overhead in firmware.
-  
-  The firmware will only record next log position internally.  It will
-  simply write next log item at end of prior one.
-  
-  Firmware will not record day start indexes.
-  
-  Firmware will provide simple dump feature to send all data to serial
-  IO line. 
-  
-  TODO:  How to read serial input in non blocking
-     fashion. 
-
-
-Beginning Address.  Since the log is presumably stored in a FRAM chip
-without support for a file system driver we must know the offset of where
-the log begins.  Space before the log is generally used for system configuration
-data. 
-
-Pre-Log Values: 
-  DATOFNDX:
-    Date Offset Index.  Array of 100 string formated number pairs
-    containing dateOffset records. Each pair contains
-    one number which is string containing the date of 
-    the first record in that date offset in the format
-    ccyymmd,offset the assumption is we can quickly scan
-    or binary search this array to find the first offset
-    record that describes the date range we are seeking then
-    jump and read that offset record.  Each time we add a new 
-    date offset log item we update this value. 
-    
-
-Record Types:
-
-date: 
-  Date types will be composed of the format ccyy-mm-dd hh:mm:ss
-  a new date record will be recorded whenever the system detects 
-  a new date when preparing to generate a new log item.  At items
-  that occur after this date are presumed to occur on that date. 
-  
-Record Type active defenition:
-  A space at first of system that stores the iformation about the
-  currently active record types. 
-
-Record Type defenition: 
-  Prefix RTDE,RecordType,List of Field Names This is written into
-  the log whenever the system reads a record type defenition from the 
-  current record types that is different than the once currently used
-  in the firmware.  When ths occurs it will write a new record type 
-  defenition  
-  
-
-next_write_offset:  NWOF 
-  Next write offset record stores the byte postion of the next
-  postion in the log byte offset where the next 
-
-date_offset: 
-  Fixed Length Binary Data - 
-  Preix: DATOF 
-         NumRec
-         Offset Of Next DATOF 32 bit number
-         Offsets (array of 32 bit numbers)
-         RecordOffset records are store in a region of  storage
-  after the before the actual data logs are written.  These are
-  fixed length records where the starting byte offset for each
-  days of logging are stored as 32 bit unsigned integer numbers
-  as an array. The first number contains the number of spaces 
-  pre-allocated. The last number written will be the offset to 
-  the next date_offset record.  In general we will allocate 365
-  365 days worht of space so the total row length will be 
-  32 + (365 * 32) + 32 bits = 11,744 bits
-  X number of days with a a
-  
-  
-reading: 
-
-System state:
-  
-