Joseph Ellsworth
/
data_log
Important changes to repositories hosted on mbed.com
Mbed hosted mercurial repositories are deprecated and are due to be permanently deleted in July 2026.
To keep a copy of this software download the repository Zip archive or clone locally using Mercurial.
It is also possible to export all your personal repositories from the account settings page.
Dependencies: W25Q80BV multi-serial-command-listener
Dependents: xj-data-log-test-and-example
data-log-text.txt@0:2c10df524ced, 2016-03-30 (annotated)
- Committer:
- joeata2wh
- Date:
- Wed Mar 30 13:34:34 2016 +0000
- Revision:
- 0:2c10df524ced
WIP
Who changed what in which revision?
| User | Revision | Line number | New contents of line |
|---|---|---|---|
| joeata2wh | 0:2c10df524ced | 1 | To make the data log format somewhat flexible and |
| joeata2wh | 0:2c10df524ced | 2 | some what easy to debug while remaining somewhat |
| joeata2wh | 0:2c10df524ced | 3 | consise while remaining human readable. The format |
| joeata2wh | 0:2c10df524ced | 4 | will be a custom format but is designed to be relatively |
| joeata2wh | 0:2c10df524ced | 5 | easy to parse while accomodating the following caveates: |
| joeata2wh | 0:2c10df524ced | 6 | * Fields logged over time will change as firmware is reved. |
| joeata2wh | 0:2c10df524ced | 7 | * Old log entries for the same record type will have different fileds. |
| joeata2wh | 0:2c10df524ced | 8 | * storage in fram is realtive expensive so self describing data |
| joeata2wh | 0:2c10df524ced | 9 | formats like JSON would not be cost effective. |
| joeata2wh | 0:2c10df524ced | 10 | * When things mess up we want to be able to parse it and |
| joeata2wh | 0:2c10df524ced | 11 | analyze it and have some chance of recovery |
| joeata2wh | 0:2c10df524ced | 12 | * We need fast access to pont in time for at least 1 day |
| joeata2wh | 0:2c10df524ced | 13 | of data. |
| joeata2wh | 0:2c10df524ced | 14 | |
| joeata2wh | 0:2c10df524ced | 15 | will use a custom format. |
| joeata2wh | 0:2c10df524ced | 16 | |
| joeata2wh | 0:2c10df524ced | 17 | Each record type will have record type label which will be |
| joeata2wh | 0:2c10df524ced | 18 | composed of 4 characters separated from the data by a The known |
| joeata2wh | 0:2c10df524ced | 19 | record types are: |
| joeata2wh | 0:2c10df524ced | 20 | |
| joeata2wh | 0:2c10df524ced | 21 | Records will be appended to the data log in sequential order. Each |
| joeata2wh | 0:2c10df524ced | 22 | record will be terminated with a \n. Some records will contain fixed |
| joeata2wh | 0:2c10df524ced | 23 | length data but will still be terminated with \n and will never contain |
| joeata2wh | 0:2c10df524ced | 24 | a \n in the stored data. unless the first number after the record type |
| joeata2wh | 0:2c10df524ced | 25 | is 16 bit number storing the number |
| joeata2wh | 0:2c10df524ced | 26 | |
| joeata2wh | 0:2c10df524ced | 27 | The general assumption is that all data will be stored for the short |
| joeata2wh | 0:2c10df524ced | 28 | term in high speed FRAM or SRAM chips and copied to EPROM or Micro |
| joeata2wh | 0:2c10df524ced | 29 | SD chips in the background once sufficnet data has accumulated to |
| joeata2wh | 0:2c10df524ced | 30 | minimize over-write fatigue in the long term storage. I used FRAM |
| joeata2wh | 0:2c10df524ced | 31 | specifically to allow after power loss data retention without having |
| joeata2wh | 0:2c10df524ced | 32 | to worry about overrite fatigue for specific segments that would need |
| joeata2wh | 0:2c10df524ced | 33 | rapid updates. The other assumption is that data is stored and logged |
| joeata2wh | 0:2c10df524ced | 34 | then will be accessed generally starting at a given date reading forward |
| joeata2wh | 0:2c10df524ced | 35 | |
| joeata2wh | 0:2c10df524ced | 36 | Short term Simplification Notes: |
| joeata2wh | 0:2c10df524ced | 37 | Each time the CPU reboots it will record current record header field |
| joeata2wh | 0:2c10df524ced | 38 | names for all record types active at that time. This will create some |
| joeata2wh | 0:2c10df524ced | 39 | duplicates but save some overhead in firmware. |
| joeata2wh | 0:2c10df524ced | 40 | |
| joeata2wh | 0:2c10df524ced | 41 | The firmware will only record next log position internally. It will |
| joeata2wh | 0:2c10df524ced | 42 | simply write next log item at end of prior one. |
| joeata2wh | 0:2c10df524ced | 43 | |
| joeata2wh | 0:2c10df524ced | 44 | Firmware will not record day start indexes. |
| joeata2wh | 0:2c10df524ced | 45 | |
| joeata2wh | 0:2c10df524ced | 46 | Firmware will provide simple dump feature to send all data to serial |
| joeata2wh | 0:2c10df524ced | 47 | IO line. |
| joeata2wh | 0:2c10df524ced | 48 | |
| joeata2wh | 0:2c10df524ced | 49 | TODO: How to read serial input in non blocking |
| joeata2wh | 0:2c10df524ced | 50 | fashion. |
| joeata2wh | 0:2c10df524ced | 51 | |
| joeata2wh | 0:2c10df524ced | 52 | |
| joeata2wh | 0:2c10df524ced | 53 | Beginning Address. Since the log is presumably stored in a FRAM chip |
| joeata2wh | 0:2c10df524ced | 54 | without support for a file system driver we must know the offset of where |
| joeata2wh | 0:2c10df524ced | 55 | the log begins. Space before the log is generally used for system configuration |
| joeata2wh | 0:2c10df524ced | 56 | data. |
| joeata2wh | 0:2c10df524ced | 57 | |
| joeata2wh | 0:2c10df524ced | 58 | Pre-Log Values: |
| joeata2wh | 0:2c10df524ced | 59 | DATOFNDX: |
| joeata2wh | 0:2c10df524ced | 60 | Date Offset Index. Array of 100 string formated number pairs |
| joeata2wh | 0:2c10df524ced | 61 | containing dateOffset records. Each pair contains |
| joeata2wh | 0:2c10df524ced | 62 | one number which is string containing the date of |
| joeata2wh | 0:2c10df524ced | 63 | the first record in that date offset in the format |
| joeata2wh | 0:2c10df524ced | 64 | ccyymmd,offset the assumption is we can quickly scan |
| joeata2wh | 0:2c10df524ced | 65 | or binary search this array to find the first offset |
| joeata2wh | 0:2c10df524ced | 66 | record that describes the date range we are seeking then |
| joeata2wh | 0:2c10df524ced | 67 | jump and read that offset record. Each time we add a new |
| joeata2wh | 0:2c10df524ced | 68 | date offset log item we update this value. |
| joeata2wh | 0:2c10df524ced | 69 | |
| joeata2wh | 0:2c10df524ced | 70 | |
| joeata2wh | 0:2c10df524ced | 71 | Record Types: |
| joeata2wh | 0:2c10df524ced | 72 | |
| joeata2wh | 0:2c10df524ced | 73 | date: |
| joeata2wh | 0:2c10df524ced | 74 | Date types will be composed of the format ccyy-mm-dd hh:mm:ss |
| joeata2wh | 0:2c10df524ced | 75 | a new date record will be recorded whenever the system detects |
| joeata2wh | 0:2c10df524ced | 76 | a new date when preparing to generate a new log item. At items |
| joeata2wh | 0:2c10df524ced | 77 | that occur after this date are presumed to occur on that date. |
| joeata2wh | 0:2c10df524ced | 78 | |
| joeata2wh | 0:2c10df524ced | 79 | Record Type active defenition: |
| joeata2wh | 0:2c10df524ced | 80 | A space at first of system that stores the iformation about the |
| joeata2wh | 0:2c10df524ced | 81 | currently active record types. |
| joeata2wh | 0:2c10df524ced | 82 | |
| joeata2wh | 0:2c10df524ced | 83 | Record Type defenition: |
| joeata2wh | 0:2c10df524ced | 84 | Prefix RTDE,RecordType,List of Field Names This is written into |
| joeata2wh | 0:2c10df524ced | 85 | the log whenever the system reads a record type defenition from the |
| joeata2wh | 0:2c10df524ced | 86 | current record types that is different than the once currently used |
| joeata2wh | 0:2c10df524ced | 87 | in the firmware. When ths occurs it will write a new record type |
| joeata2wh | 0:2c10df524ced | 88 | defenition |
| joeata2wh | 0:2c10df524ced | 89 | |
| joeata2wh | 0:2c10df524ced | 90 | |
| joeata2wh | 0:2c10df524ced | 91 | next_write_offset: NWOF |
| joeata2wh | 0:2c10df524ced | 92 | Next write offset record stores the byte postion of the next |
| joeata2wh | 0:2c10df524ced | 93 | postion in the log byte offset where the next |
| joeata2wh | 0:2c10df524ced | 94 | |
| joeata2wh | 0:2c10df524ced | 95 | date_offset: |
| joeata2wh | 0:2c10df524ced | 96 | Fixed Length Binary Data - |
| joeata2wh | 0:2c10df524ced | 97 | Preix: DATOF |
| joeata2wh | 0:2c10df524ced | 98 | NumRec |
| joeata2wh | 0:2c10df524ced | 99 | Offset Of Next DATOF 32 bit number |
| joeata2wh | 0:2c10df524ced | 100 | Offsets (array of 32 bit numbers) |
| joeata2wh | 0:2c10df524ced | 101 | RecordOffset records are store in a region of storage |
| joeata2wh | 0:2c10df524ced | 102 | after the before the actual data logs are written. These are |
| joeata2wh | 0:2c10df524ced | 103 | fixed length records where the starting byte offset for each |
| joeata2wh | 0:2c10df524ced | 104 | days of logging are stored as 32 bit unsigned integer numbers |
| joeata2wh | 0:2c10df524ced | 105 | as an array. The first number contains the number of spaces |
| joeata2wh | 0:2c10df524ced | 106 | pre-allocated. The last number written will be the offset to |
| joeata2wh | 0:2c10df524ced | 107 | the next date_offset record. In general we will allocate 365 |
| joeata2wh | 0:2c10df524ced | 108 | 365 days worht of space so the total row length will be |
| joeata2wh | 0:2c10df524ced | 109 | 32 + (365 * 32) + 32 bits = 11,744 bits |
| joeata2wh | 0:2c10df524ced | 110 | X number of days with a a |
| joeata2wh | 0:2c10df524ced | 111 | |
| joeata2wh | 0:2c10df524ced | 112 | |
| joeata2wh | 0:2c10df524ced | 113 | reading: |
| joeata2wh | 0:2c10df524ced | 114 | |
| joeata2wh | 0:2c10df524ced | 115 | System state: |
| joeata2wh | 0:2c10df524ced | 116 | |
| joeata2wh | 0:2c10df524ced | 117 |