Data Logging Data structure¶

This is currently WIP please do not use until this page is changed to indicate it is ready.

Motivation¶

I needed a flexible data log structure that could tolerate evolving data structures as I discovered more things that needed to be measured. I also wanted something that is mostly human readable while remaining suffictly concise to make efficient use of expensive storage resources.

I found it challenging to track everything needed to perform after the fact analysis we need to improve our state machine. In addition what I wanted to measure changed with time and I needed a robust way to log this data so we could analyze it latter. without breaking or conveting all the old data. A self describing data format like JSON or XML would work but FRAM is expensive so I wanted something flexible but still concise.

I am working on A2WH which is a electronic controller for a sophisticated product that balances many sensors, battery charging from photo voltaic panels, controlling speed of many different fans, humidity and enviornmental data. Our main challenge is we never have enough battery power to run everything so we have to make decisions about what to run in an effort to produce the maximum amount of water from the available solar power resource. Our 2nd challenge is that balancing system actions such as increasing or descreasing fan speeds is driven by a complex internal prediction model that attempts balance many competing thermodynamic requirements. To get all this right requires substantial after the fact analysis and that requires logging a large amount of evolving data.

Design Notes¶

See: data-log-read.me.txt in the same project

Sample Use and Basic Test¶

• https://developer.mbed.org/users/joeata2wh/code/xj-data-log-test-and-example/wiki/Homepage

Other Chips¶

For legacy reasons I am using the library for "W25Q80BV.h" simply because I started with it. The actual FRAM chip I am using is 2 MBit FRAM MB85RS2MTPH-G-JNE I also tested it with SRAM 23LCV1024-I/P

Simplifying Design Decision¶

I made a simplifying assumption that everytime we generate a log entry I record the offset of the next write at a specific location in the chip. This works and is fast but it causes lots of updates against a single location. I prefer FRAM because this would rapidly fatigue FLASH chips like the W25Q80BV. Storing this pointer data in the CPU has the same fatigue problem.

Another other option would be to store this offset and our other critical configuration data in the clock chip but it is suseptible to loosing power and loosing this critical data.

One reason I don't log directly to the micro-sd is for the same fatique problem but it is mostly for power managment.

The FRAM chip provides adequate durability and data retention through power outage. The power outage retention is critical because the A2WH systems can be burried under feet of snow in the winter and solar panels do not provide much recharge under that condition.

One design option I have considered but not yet implemented is using a much smaller FRAM chip critical configuration data and rapid update data and then log directly to a larger and less expensive FLASHchip .

Journaling to micro-sd¶

I latter decided to add features to allow after the fact copying of the data to micro-sd cards to obtain larger log storage without soldering in more chips. I found the micro-sd consume quite a lot of power so I still want to log direct to the FRAM then copy to the micro-sd when I have surplus power available. Still thinking about consolidation tactics to allow re-use of FRAM after the data has been copied ot micro-sd.

Future¶

• Support fast indexing by date to only pull back log entries between two dates.
• Record most recent record headers for each record types where they are fast to access so we can send them with the data when only sending back portions of the data.
• Support wrap around use of data log to re-use storage on chip.
• Copy Data to micro sd card and consolidate fram chip for re-use.

License¶

By Joseph Ellsworth CTO of A2WH Take a look at A2WH.com Producing Water from Air using Solar Energy March-2016 License: https://developer.mbed.org/handbook/MIT-Licence Please contact us http://a2wh.com for help with custom design projects.