Revision 4:fa5bbe31a039, committed 2016-03-31

Comitter:

joeata2wh

Date:

Thu Mar 31 05:02:28 2016 +0000

Parent:

3:5550814cc21c

Child:

5:286459acee56

Commit message:

write functions basically working still working on testing the read chunked version.

Changed in this revision

--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/data-log-read.me.txt	Thu Mar 31 05:02:28 2016 +0000
@@ -0,0 +1,117 @@
+To make the data log format somewhat flexible and 
+some what easy to debug while remaining somewhat
+consise while remaining human readable.  The format
+will be a custom format but is designed to be relatively
+easy to parse while accomodating the following caveates:
+   * Fields logged over time will change as firmware is reved.
+   * Old log entries for the same record type will have different fileds.
+   * storage in fram is realtive expensive so self describing data 
+     formats like JSON would not be cost effective. 
+   * When things mess up we want to be able to parse it and 
+     analyze it and have some chance of recovery 
+   * We need fast access to pont in time for at least 1 day 
+     of data. 
+     
+will use a custom format.  
+
+Each record type will have record type label which will be
+composed of 4 characters separated from the data by a The known
+record types are:
+
+Records will be appended to the data log in sequential order. Each
+record will be terminated with a \n.  Some records will contain fixed
+length data but will still be terminated with \n and will never contain
+a \n in the stored data. unless the first number after the record type 
+is 16 bit number storing the number 
+
+The general assumption is that all data will be stored for the short
+term in high speed FRAM or SRAM chips and copied to EPROM or Micro 
+SD chips in the background once sufficnet data has accumulated to 
+minimize over-write fatigue in the long term storage.  I used FRAM 
+specifically to allow after power loss data retention without having 
+to worry about overrite fatigue for specific segments that would need
+rapid updates.    The other assumption is that data is stored and logged
+then will be accessed generally starting at a given date reading forward 
+
+Short term Simplification Notes:
+  Each time the CPU reboots it will record current record header field 
+  names for all record types active at that time.  This will create some
+  duplicates but save some overhead in firmware.
+  
+  The firmware will only record next log position internally.  It will
+  simply write next log item at end of prior one.
+  
+  Firmware will not record day start indexes.
+  
+  Firmware will provide simple dump feature to send all data to serial
+  IO line. 
+  
+  TODO:  How to read serial input in non blocking
+     fashion. 
+
+
+Beginning Address.  Since the log is presumably stored in a FRAM chip
+without support for a file system driver we must know the offset of where
+the log begins.  Space before the log is generally used for system configuration
+data. 
+
+Pre-Log Values: 
+  DATOFNDX:
+    Date Offset Index.  Array of 100 string formated number pairs
+    containing dateOffset records. Each pair contains
+    one number which is string containing the date of 
+    the first record in that date offset in the format
+    ccyymmd,offset the assumption is we can quickly scan
+    or binary search this array to find the first offset
+    record that describes the date range we are seeking then
+    jump and read that offset record.  Each time we add a new 
+    date offset log item we update this value. 
+    
+
+Record Types:
+
+date: 
+  Date types will be composed of the format ccyy-mm-dd hh:mm:ss
+  a new date record will be recorded whenever the system detects 
+  a new date when preparing to generate a new log item.  At items
+  that occur after this date are presumed to occur on that date. 
+  
+Record Type active defenition:
+  A space at first of system that stores the iformation about the
+  currently active record types. 
+
+Record Type defenition: 
+  Prefix RTDE,RecordType,List of Field Names This is written into
+  the log whenever the system reads a record type defenition from the 
+  current record types that is different than the once currently used
+  in the firmware.  When ths occurs it will write a new record type 
+  defenition  
+  
+
+next_write_offset:  NWOF 
+  Next write offset record stores the byte postion of the next
+  postion in the log byte offset where the next 
+
+date_offset: 
+  Fixed Length Binary Data - 
+  Preix: DATOF 
+         NumRec
+         Offset Of Next DATOF 32 bit number
+         Offsets (array of 32 bit numbers)
+         RecordOffset records are store in a region of  storage
+  after the before the actual data logs are written.  These are
+  fixed length records where the starting byte offset for each
+  days of logging are stored as 32 bit unsigned integer numbers
+  as an array. The first number contains the number of spaces 
+  pre-allocated. The last number written will be the offset to 
+  the next date_offset record.  In general we will allocate 365
+  365 days worht of space so the total row length will be 
+  32 + (365 * 32) + 32 bits = 11,744 bits
+  X number of days with a a
+  
+  
+reading: 
+
+System state:
+  
+

--- a/data-log-text.txt	Wed Mar 30 21:44:22 2016 +0000
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,117 +0,0 @@
-To make the data log format somewhat flexible and 
-some what easy to debug while remaining somewhat
-consise while remaining human readable.  The format
-will be a custom format but is designed to be relatively
-easy to parse while accomodating the following caveates:
-   * Fields logged over time will change as firmware is reved.
-   * Old log entries for the same record type will have different fileds.
-   * storage in fram is realtive expensive so self describing data 
-     formats like JSON would not be cost effective. 
-   * When things mess up we want to be able to parse it and 
-     analyze it and have some chance of recovery 
-   * We need fast access to pont in time for at least 1 day 
-     of data. 
-     
-will use a custom format.  
-
-Each record type will have record type label which will be
-composed of 4 characters separated from the data by a The known
-record types are:
-
-Records will be appended to the data log in sequential order. Each
-record will be terminated with a \n.  Some records will contain fixed
-length data but will still be terminated with \n and will never contain
-a \n in the stored data. unless the first number after the record type 
-is 16 bit number storing the number 
-
-The general assumption is that all data will be stored for the short
-term in high speed FRAM or SRAM chips and copied to EPROM or Micro 
-SD chips in the background once sufficnet data has accumulated to 
-minimize over-write fatigue in the long term storage.  I used FRAM 
-specifically to allow after power loss data retention without having 
-to worry about overrite fatigue for specific segments that would need
-rapid updates.    The other assumption is that data is stored and logged
-then will be accessed generally starting at a given date reading forward 
-
-Short term Simplification Notes:
-  Each time the CPU reboots it will record current record header field 
-  names for all record types active at that time.  This will create some
-  duplicates but save some overhead in firmware.
-  
-  The firmware will only record next log position internally.  It will
-  simply write next log item at end of prior one.
-  
-  Firmware will not record day start indexes.
-  
-  Firmware will provide simple dump feature to send all data to serial
-  IO line. 
-  
-  TODO:  How to read serial input in non blocking
-     fashion. 
-
-
-Beginning Address.  Since the log is presumably stored in a FRAM chip
-without support for a file system driver we must know the offset of where
-the log begins.  Space before the log is generally used for system configuration
-data. 
-
-Pre-Log Values: 
-  DATOFNDX:
-    Date Offset Index.  Array of 100 string formated number pairs
-    containing dateOffset records. Each pair contains
-    one number which is string containing the date of 
-    the first record in that date offset in the format
-    ccyymmd,offset the assumption is we can quickly scan
-    or binary search this array to find the first offset
-    record that describes the date range we are seeking then
-    jump and read that offset record.  Each time we add a new 
-    date offset log item we update this value. 
-    
-
-Record Types:
-
-date: 
-  Date types will be composed of the format ccyy-mm-dd hh:mm:ss
-  a new date record will be recorded whenever the system detects 
-  a new date when preparing to generate a new log item.  At items
-  that occur after this date are presumed to occur on that date. 
-  
-Record Type active defenition:
-  A space at first of system that stores the iformation about the
-  currently active record types. 
-
-Record Type defenition: 
-  Prefix RTDE,RecordType,List of Field Names This is written into
-  the log whenever the system reads a record type defenition from the 
-  current record types that is different than the once currently used
-  in the firmware.  When ths occurs it will write a new record type 
-  defenition  
-  
-
-next_write_offset:  NWOF 
-  Next write offset record stores the byte postion of the next
-  postion in the log byte offset where the next 
-
-date_offset: 
-  Fixed Length Binary Data - 
-  Preix: DATOF 
-         NumRec
-         Offset Of Next DATOF 32 bit number
-         Offsets (array of 32 bit numbers)
-         RecordOffset records are store in a region of  storage
-  after the before the actual data logs are written.  These are
-  fixed length records where the starting byte offset for each
-  days of logging are stored as 32 bit unsigned integer numbers
-  as an array. The first number contains the number of spaces 
-  pre-allocated. The last number written will be the offset to 
-  the next date_offset record.  In general we will allocate 365
-  365 days worht of space so the total row length will be 
-  32 + (365 * 32) + 32 bits = 11,744 bits
-  X number of days with a a
-  
-  
-reading: 
-
-System state:
-  
-

--- a/dataLog.h	Wed Mar 30 21:44:22 2016 +0000
+++ b/dataLog.h	Thu Mar 31 05:02:28 2016 +0000
@@ -5,7 +5,8 @@
   
   See data-log-text.txt for detailed design and layout notes
   See: xj-data-log-test-and-example.c for example use.
-  
+       https://developer.mbed.org/users/joeata2wh/code/xj-data-log-test-and-example/wiki/Homepage
+
   By Joseph Ellsworth CTO of A2WH
   Take a look at A2WH.com Producing Water from Air using Solar Energy
   March-2016 License: https://developer.mbed.org/handbook/MIT-Licence 
@@ -47,18 +48,17 @@
 #define dlMaxOperationSize -4;
 // Chip DLog Chip Memory Layout
 const long dlAddrInitByte  = 1500; // data before this is assumed to be used for system config variables
-const char dlInitByteValue = 213;
+const char dlInitByteValue = 214;
 const int dlMaxReadWriteSize = 32000; // limit imposed by streaming interface for the chip
 const long dlAddrNextWritePos = dlInitByteValue + 1;
-const long dlAddrNextWritePosSize = 4;
-const long dlAddrCurrYDay = dlAddrNextWritePos + dlAddrNextWritePosSize;
-const long dlAddrCurrYDaySize = 2;
-const long dlAddrHeaders =   dlAddrCurrYDay + dlAddrCurrYDaySize ;
+const long dlNextWritePosSize = 4;
+const long dlAddrCurrYDay = dlAddrNextWritePos + dlNextWritePosSize;
+const long dlCurrYDaySize = 2;
+const long dlAddrHeaders =   dlAddrCurrYDay + dlCurrYDaySize ;
 const long dlHeadersLen  =   256;
 const long dlDateIndex   =   dlAddrHeaders + dlHeadersLen + 1;
 const long dlDateIndexLen=   1000;
 const long dlFirstLogEntry=  dlDateIndexLen + dlDateIndexLen + 1;
-const long dlBuffLen = 256;
 const char dlEmpty[] = {0,0,0,0,0,0,0};
 const long dlMaxLogSize = dlChipMaxAddr - dlFirstLogEntry;
 #define MIN(X,Y) X <? Y
@@ -76,49 +76,65 @@
 // save the current nextWritePos to the chip so we hav it
 // just in case of a reboot
 void dlSaveNextWritePos(struct DLOG *wrk) {
-   wrk->chip->writeStream(dlAddrNextWritePos,(char *) &wrk->nextWritePos,dlAddrNextWritePosSize);   // write next write postion
+   wrk->chip->writeStream(dlAddrNextWritePos,(char *) &wrk->nextWritePos,dlNextWritePosSize);   // write next write postion
 }
 
 long dlReadNextWritePos(struct DLOG *wrk) {
-  wrk->chip->readStream(dlAddrNextWritePos, (char *) &wrk->nextWritePos, dlAddrNextWritePosSize);
+  wrk->chip->readStream(dlAddrNextWritePos, (char *) &wrk->nextWritePos, dlNextWritePosSize);
+  //printf("dlReadNextWritePos wrk->nextWritePos=%ld\n\r",wrk->nextWritePos);
   return wrk->nextWritePos;
 }
 
 int dlReadCurrYDay(struct DLOG *wrk) {
-  wrk->chip->readStream(dlAddrNextWritePos, (char *) &wrk->currYDay, dlAddrCurrYDaySize);
+  wrk->chip->readStream(dlAddrNextWritePos, (char *) &wrk->currYDay, dlCurrYDaySize);
+  //printf("dlReadCurrYDay wrk->currYDay=%d\r\n", wrk->currYDay);
   return wrk->currYDay;  
 }
 
 void dlUpdateCurrDate(struct DLOG *wrk, int newYDay) {
   wrk->currYDay = newYDay;
-  wrk->chip->writeStream(dlAddrCurrYDay,(char *) &wrk->currYDay, dlAddrCurrYDaySize);   // write next write postion
+  wrk->chip->writeStream(dlAddrCurrYDay,(char *) &wrk->currYDay, dlCurrYDaySize);   // write next write postion
+}
+
+// Erase Log from Chip but do not touch 
+// data on chip outside of log space.
+void dlEraseLog(struct DLOG *wrk) {
+   //printf("dlEraseLogStart\r\n");
+   wrk->nextWritePos =  dlFirstLogEntry;
+   wrk->currYDay = -99;
+   wrk->chip->writeStream(dlAddrInitByte, (char *) &dlInitByteValue,1);  // write init byte
+   wrk->chip->writeStream(dlAddrNextWritePos,(char *) &wrk->nextWritePos,dlNextWritePosSize);   // write next write postion   
+   
+   memset(wrk->buff,0,wrk->buffLen);
+   wrk->chip->writeStream(dlAddrNextWritePos, (char *) &wrk->nextWritePos,dlNextWritePosSize);   // null over next write postion
+   wrk->chip->writeStream(dlAddrCurrYDay, (char *) &wrk->currYDay, dlCurrYDaySize); // null over currDay
+   wrk->chip->writeStream(dlAddrHeaders,wrk->buff,MIN(wrk->buffLen,dlHeadersLen));   // nulls over the header region
+   wrk->chip->writeStream(dlDateIndex,wrk->buff,MIN(wrk->buffLen,dlDateIndexLen));   // nulls over the header region   
+   wrk->chip->writeStream(wrk->nextWritePos,wrk->buff,wrk->buffLen);   // nulls first of the log
+   //printf("dlEraseLogDone\r\n");
 }
 
 // New data log chip detected write data to initialize it.
 long dlInitializeChip(struct DLOG *wrk) {
-   wrk->nextWritePos =  dlFirstLogEntry;
-   wrk->chip->writeStream(dlAddrInitByte, (char *) &dlInitByteValue,1);  // write init byte
-   wrk->chip->writeStream(dlAddrNextWritePos,(char *) &wrk->nextWritePos,dlAddrNextWritePosSize);   // write next write postion   
-   
-   memset(wrk->buff,0,wrk->buffLen);
-   wrk->chip->writeStream(dlAddrHeaders,wrk->buff,MIN(wrk->buffLen,dlHeadersLen));   // nulls over the header region
-   wrk->chip->writeStream(dlDateIndex,wrk->buff,MIN(wrk->buffLen,dlDateIndexLen));   // nulls over the header region
-   
-   wrk->chip->writeStream(dlDateIndex,wrk->buff,MIN(wrk->buffLen,dlDateIndexLen));   // nulls over the header region
-   wrk->currYDay = -99;
+   dlEraseLog(wrk);
    return wrk->nextWritePos;
 }
 
 
+
 /* read a initialization byte from chip.  If the byte
 doesn't contain the expected value then write one
 and assume that we are starting our log ad the beginning */
 long dlCheckChipInit(struct DLOG *wrk){
   wrk->buff[0] = 0; 
-  wrk->chip->readStream(dlInitByteValue, wrk->buff, 1);
+  wrk->chip->readStream(dlAddrInitByte, wrk->buff, 1);
   if (wrk->buff[0] != dlInitByteValue) 
+  {
+    //printf("Found empty chip running init");
     return dlInitializeChip(wrk);      
+    }
   else {
+      //printf("Found existing log\r\n");
       dlReadCurrYDay(wrk);
       return dlReadNextWritePos(wrk);
     }
@@ -138,6 +154,7 @@
   return tout;
 }
 
+
 // writes log stream entry to chip and updates the next write 
 // position. Also adds a null terminator to data on chip
 // log entries should not contain null characters because we
@@ -149,29 +166,40 @@
   if ((wrk->nextWritePos + slen) >= dlChipMaxAddr) {
       return dlChipFullErr;
       }
-  wrk->chip->writeStream(wrk->nextWritePos, aStr,slen); 
+  wrk->chip->writeStream(wrk->nextWritePos, aStr,slen);         
   wrk->nextWritePos += slen;
-  wrk->chip->writeStream(wrk->nextWritePos, (char *)dlEmpty, 1);  // add terminating null 
-  
   dlSaveNextWritePos(wrk); // WARNING THIS IS THE LINE THAT WILL KILL EPROM CHIPS 
                            // with over-write fatigue.
   // TODO: add err check read first and last bytes.
   // compare to what was written.
   return wrk->nextWritePos;
 }
-
+char dlLFEmpty[] = "\n\000";
+//record log file entry with time stamp and header
 long dlLog(struct DLOG *wrk, char *recType, char *str) {
-  time_t seconds = time(NULL);
-  tm *ptm = gmtime ( &seconds );
+  
+  time_t seconds;
+  time(&seconds);
+  //printf("dlLog ctime(&seconds)=%s\r\n", ctime(&seconds));
+  struct tm *ptm = localtime( &seconds );
+  printf("dlLog asctime=%s\r\n", asctime(ptm));
+  printf("dlLog ptm->tm_yday=%d\r\n", ptm->tm_yday );
   if (ptm->tm_yday != wrk->currYDay) {
+     int year = 1900 + ptm->tm_year;
+     int month= ptm->tm_mon + 1;
+     
      dlUpdateCurrDate(wrk, ptm->tm_yday);
-     sprintf(wrk->buff,"DATE\t000000\n\000");
+     sprintf(wrk->buff,"\n00:00:00 DATE\t%04d-%02d-%02d", year, month, ptm->tm_mday);
      dlWrite(wrk, wrk->buff);
+     printf("update date dateRec=%s\r\n", wrk->buff);
   }
-  memset(wrk->buff,12,0);
-  sprintf(wrk->buff,"%s\t%2d%2d%2d\t", recType, ptm->tm_hour, ptm->tm_min, ptm->tm_sec);
-  dlWrite(wrk, wrk->buff);   
-  return write(wrk, str);
+  
+  memset(wrk->buff,wrk->buffLen,0);
+  sprintf(wrk->buff,"\n%02d:%02d:%02d %s\t", ptm->tm_hour, ptm->tm_min, ptm->tm_sec, recType);  
+  dlWrite(wrk, wrk->buff);     
+  //dlWrite(wrk, dlLFEmpty);  // add terminating lineFeed
+  printf ("recHead=%s\r\n", wrk->buff);
+  return dlWrite(wrk, str);
 }
 
 // read a block of bytes from log starting at offset
@@ -190,7 +218,7 @@
    long addr = dlFirstLogEntry + offset;
    long maxAddr = MIN(addr + len, dlChipMaxAddr); // no overflow past end of chip
    maxAddr = MIN(maxAddr, wrk->nextWritePos); // no overlow pas end of log
-   int chunkSize = dlBuffLen -1;
+   int chunkSize = wrk->buffLen -1;
    if (addr >= maxAddr)  
      return dlAddressTooLarge;
    long endAdd = MIN(addr + len, maxAddr);