Hello all,

I'm working on a data logger and would like to write to the SD card. I'm using the LPC4088 Quickstart board with the base board which has the SD card reader built in. I'm using the MCIFilesystem library provided by EA.

I am writing data at 10hz

I am able to do a basic open, write, and close of the card inside the main() function of my program.

What is the best way to hand the opening, writing, an closing of the file? Should i open it once, keep writing, and close at the end? Should i be doing an open, write, close at every data write (10hz)?

I have a 10hz Ticker running and i'm calling a "writeFile()" function form it that has the code to write each additional sample, but i also have the writing of the header of the file in the main() function. Is it ok to declare my fp file handle globally so it can be used globally in the program?

OR (this just occurred to me while thinking about interrupts)

Maybe i should have my 10hz Ticker ISR simple let a flag and test for the flag in my main while loop. If flag is set, then i can do all the data sampling and writing from there...

Any input or advice would be much appreciated. Thanks

I think using a flag for something like SD card writing is a good idea. Although the LPC4088 has a real SD card interface and not SPI (afaik), still writing to an SD card won't be the fastest thing. Although it depends on the rest of your system, if you don't got other critical interrupt that need to be handled it will be fine.

10Hz isn't very fast. It depends then also on how large chunks you are writing, and how large the chance is power goes out for example. At high rates skipping the opening/closing helps, but I wouldn't know how important it is with the MCIFileSystem. You can simply use a timer to check how long both options take, and use that to make your decision.

Setting a file handle globally is allowed in principle, but you really need to take care it isn't changed by different functions at the same time. In general you need to be sure only one process can write to the sd card at a time: For example if your main function is writing to the SD card, and halfway an interrupt occurs which also will write the card, you most likely end up with alot of corrupted data.

I've applications that write 100 bytes to a fairly slow SD card at 100Hz without any issues at all. Generally I structure this so that the data is collected (normally from serial ports) using interrupt based code that puts that data into a buffer.

The main loop then polls that buffer to see if there is data waiting, processes the data, formats the results into text and writes the results to a file.

I keep the file open the whole time however this has a big downside, i you pull the power or SD card then you lose the data.

The best solution is to either have the program know when to stop or use a button to tell the system to close the file.

If that's not practical then you can use multiple files. Either track how much you have written to the file and after a certain amount of data close it and open a new one or on a slow timer close the current file and open a new one. You will lose any data in the file you are currently writing to but all the previous ones will be safe.

Interesting.

So far mine is setup to open the file at startup and at 10hz, i collect/sample all data, crunch it a bit, and write to the SD card. I don't have a buffer. Currently i have a log timer i set for the period of time i want to log for. This isn't practical for my application, but i have it set for 10s and it allows me to work on debugging and formatting the text file, etc.

I was thinking maybe i could create checkpoints by closing and reopening the file every so often. I think i can reopen it in append mode? I'd lose some samples during that time, but that might be ok if i keep the time stamp going.

I don't foresee needing to go above 100hz, but it would be nice do it right and have a foundation for the future.

When using a buffer, when/how do you give the SD card time to catch up to the buffer? Wouldn't the buffer just fill up and eventually overflow? Is there a large dump process at the end of logging the data?

Thanks again

Part of the reason for the buffer is that the time taken for a write to the SD card can vary depending on how log flash erase cycles take and where within the sector it currently is. If you have everything in one long function that is called on a timer then if a flash write takes longer than that time period things can get out of sync.

With a buffer in the path as long as the average speed is fast enough then you are ok even if some individual writes take too long. file writes are done in the main loop so any timer or serial port interrupts will take priority.

I've now managed to crank my system up to 2kHz logging to SD using this structure so it seems reasonably sound.

As a general rule whenever possible you should minimise the amount of work done in interrupt code so that you aren't blocking other interrupts for longer than needed. With a single source like it sounds like you are using this is less important but if you want to start combining multiple data sources then it can matter.

I'm not sure if closing and then appending would work. The data would certainly get written to the card but I'm not sure what state the file system table would be left in. I'd be tempted to play safe and use different files. The code below will close the current file and open the next. A simple refinement you can add if needed is to first try to open the new filename as read only, if that works then close the file and move on to the next name. That way you avoid overwriting logs in the event of a reset.

FILE *myFile = NULL;

void openNextFile() {
  static int fileNumber = 0;
  char fileName[11];

  if (myFile)
    fclose(myFile);
  
  sprintf(fileName,"Log%03d.txt",fileNumber++);

  myFile = fopen(fileName,"w");
}

That's helpful info, thanks for explaining.