Saving Data in Various Formats

I've been working with images saved as arrays and have found a few problems. Mainly that when the arrays get too large (around 22kb) the mbed hangs. So basically I have come to the conclusion that I need to take the image and instead of saving it to an array, save it straight to some external memory and then work with it later on.

This prompts the question "How quickly can I read and write to external memory?" So I thought I would investigate...

Local File System

The LocalFileSystem is well documented in the handbook and allows a easy way to store data in the same flash memory that the mbed programs are stored. The main advantage is that it doesn't require any external components.

So how long does it take to write to and read from? For the purposes of testing I will be working with arrays 19.2kb in size. Why 19200 bytes? - well that is how much data it takes to store an image 160 x 120 pixels in size with a 8-bit greyscale colour resolution.

So onto the test program...

#include "mbed.h"

Serial pc(USBTX,USBRX);
LocalFileSystem local("local");                 // Create the local filesystem under the name "local"

int main() {
    unsigned char data[120*160];                // Test array of data

    // Fill the data with values
    for (int i=0; i<120*160; i++) {
        data[i] = 0xAA;                          // Fill array with data
    }

    // Create timers
    Timer Write_time, Read_time;
    Write_time.start();

    // Write to local file
    FILE *fp = fopen("/local/test.txt", "w");  // Open "test.txt" on the local file system for appending
    for (int i=0; i<120*160; i++) {
        fprintf(fp, "%c",data[i]);
    }
    fclose(fp);
    Write_time.stop();
    pc.printf("\n\rTime taken so write array = %f seconds",Write_time.read());

    Read_time.start();
    // Read from local file
    fopen("/local/test.txt","r");
    // Get data from file and load back into cam
    for (int i=0; i<120*160; i++) {
            data[i] = fgetc(fp);          // Get the next character
    }
    fclose(fp);
    Read_time.stop();
    pc.printf("\n\rTime taken to read array = %f seconds",Read_time.read());
}

The results say that it took 670 ms to write the array to the file using the local file system and 265 ms to read the file back into the array.

This is a little slower than I would have liked (935 ms in total) and I think I can do better...

USB Stick

I can't say that I fully understand the USB host and MCSFileSystem libraries but I did cobble together a program that saved data to and read data from a USB stick.

Here it is...

Import programData_Read_Write

Writing data to and from a USB stick

The results are not very good - probably down to all the other data that is acquired and printed when a file is created. When the serial baud is pushed up to 92160 (note this is the baud to the terminal - not the usb stick!) it takes 460 ms to write the data and 117 ms to read it. that makes a total time of 577 ms

note: the read time is unaffected by the serial baud.

This is quicker than the LocalFileSystem but still not quite quick enough...

SD Card

So the 3rd option is using a SD Card. Modifying the SD card hello world program and running the test reveals a write time of 720 ms and a read time of 123 ms. This gives a total of 846 ms. This is not too good. Looking at the library it works from SPI I can increase the SPI frequency.

I do this and bump it up to 20MHz and the write time falls to 356 ms and the read time to 59 ms giving a total time of 415 ms. This is more impressive and beats the other option thus far. But I still think I can do better...

External SRAM IC

The final option I'm looking at is an external IC. For this I choose the 23k256 IC from Microchip. It is a SPI chip that runs up to 20 Mhz. Also Romilly Cocking and marcel van de Kamp have already done some work and published a library - so my thanks to them for their efforts.

So running with the standard library I get a write time of 175 ms and a read time of 176 ms, pretty good. But I think I can improve on that!

A quick look at the library and it is using the default SPI frequency of 1Mhz. So what happens if I up that to 20MHz? I now complete the read and write cycle in 35 ms for each.

This is not the 20 fold increase that I was expecting but it is still pretty good, and around 10 times quicker than the other methods.

Conclusion

The best times I have for the different formats are as follows...

FormatWrite TimeRead TimeTotal Time
LocalFileSystem670 ms235 ms905 ms
USB Stick460 ms117 ms577 ms
SD Card356 ms59 ms415 ms
External SRAM35 ms35 ms70 ms

I think for my application I will probably go with the SRAM IC. It is easy to work with and only costs £1.23 which is less than the other options (except the LocalFileSystem of course!)

It was interesting exploring the different file systems and I may revisit this investigation later on with other solutions when I get a bit of free time.

Further Work

So it seems that the application I had in mind would benefit from faster read / write access to memory. I've been looking around and found a SPI IC from Ramtron that not only runs at 40MHz but also claims to have zero wait time between writes.

The down side is that it is a bit expensive at £23 EA from Farnell. The up-side is that it is 2 Mb (128k bytes) so it has plenty of memory.

The maths say that it should be able to write a byte in 40 clock cycles or 1us, which is pretty rapid. Anyway I decided to get 1 and check it out. I used the 23k256 library by Romilly Cocking as a starting point because the datasheets read pretty similar.

After a bit of tweaking I'm ready with a library. The results...

Well it now takes around 25ms to send or receive the data. This is a lot longer than the 4ms I was expecting. Clearly some further investigation is needed.

Time for a new notebook page - SPI Investigation


1 comment on Saving Data in Various Formats:

18 Oct 2013

Thank you for all those knowledge. It helped me lot.

Please log in to post comments.