9 years, 1 month ago.

fwrite writing time not consistent?

Hi all,

I am working on an application that requires to write data as fast as possible to a file. I used the fwrite fonction to write the data by blocks of fixed length. However it seems that the time required for one call to fwrite is not consistent: each time the file size is increased by 512 bytes, the writing operation seems to take several ms, while it takes few micro-seconds otherwise, independently of the block size parameter of the fwrite function.

Below is a simple test program to write data in a file by blocks of fixed length, and then display the time required for each write operation.

I made tests on the local file system and with an application board (through the USBHostMSD class and a 2GB memory stick plugged on the USB port), and got similar observations.

Is there a way to have a consistent writing time ?

Thanks in advance

[EDIT] After a closer look, it seems that the data are first buffered before being actually written on the file, which could explain the inconsistency in the writing time. Can someone confirm this ? If it is the case, is there a way to go around the buffering issue ?

include the mbed library with this snippet

#include "mbed.h"

LocalFileSystem local("local");               // Create the local filesystem under the name "local"

DigitalOut led1(LED1);
DigitalOut led2(LED2);

Serial pc(USBTX, USBRX);    // Serial console

#define BUFFER_SIZE 5000    // Buffer to store the data to be written on file

int main() 
{
    // Set the serial baud rate
    pc.baud(921600); 
    
    // Initialize the source buffer
    uint8_t buff[BUFFER_SIZE];
    for(int i = 0; i < BUFFER_SIZE; i++)
        buff[i] = 'a' + i%10;   // abcdefghij
       
    // Create the file
    FILE *pFile = NULL;
    pFile = fopen("/local/a.txt","w");
    
    // Initialize the timer related variables
    Timer timer;
    int nTime_us = 0;
    int nDT_us[BUFFER_SIZE];
    memset((void* )nDT_us, 0, BUFFER_SIZE*sizeof(int));
    
    // Initialize the writing parameters
    int nBlockSize = 1;         // Size of one data block of data to write
    int nBytesToWrite = 513;    // Total number of bytes to write to the file

    int nBytesWritten = 0;      // Number of bytes written in a single write operation
    int nTotalBytesWritten = 0; // Total number of bytes written
    int nNbBlocksToWrite = 1;   // Number of data blocks to write in a single write operation
    int nBlocksWritten = 0;     // Number of data blocks written in a single write operation
    int nNbWriteOp = 0;         // Number of write operations
    
    pc.printf(
            "\r\n\nNB bytes to write: %d (Block size: %d)\r\n", 
            nBytesToWrite,
            nBlockSize
            );

    // Start the data writing
    timer.start();
    while(nBytesToWrite > 0)
        {
        nTime_us = timer.read_us();
        // Write the data block
        nBlocksWritten = fwrite(
                                buff,               // const void *ptr
                                nBlockSize,         // size_t size
                                nNbBlocksToWrite,   // size_t nmemb
                                pFile               // FILE *stream
                                );
        // Store the time required for the write operation
        if(nNbWriteOp < BUFFER_SIZE)
            nDT_us[nNbWriteOp++] = timer.read_us() - nTime_us;
        
        // Update the remaining number of bytes to write
        if(nBlocksWritten > 0)  // Write success
            {
            nBytesWritten = nBlocksWritten * nBlockSize;
            nBytesToWrite -= nBytesWritten;
            nTotalBytesWritten += nBytesWritten;
            led1 = !led1;
            }
        else
            {
            led2 = !led2;       // Write error    
            }
        }
    
    // Display the time required for each write operation
    pc.printf(
                "\r\n%d bytes written\r\n", 
                nTotalBytesWritten
                );
    pc.printf("\r\nWrite time:\r\n");
    for(int i = 0; i < nNbWriteOp; i++)
        {
        if(nDT_us[i] > 10)
            pc.printf("#%d:", i); // Mark the slower write operations
        pc.printf(
                    "%d ", 
                    nDT_us[i]
                    );
        }
    pc.printf("\r\n");
    
    // Close the file
    fclose(pFile);
        
}

1 Answer

9 years, 1 month ago.

Calling fflush(pFile) should do the job (I think). Indeed what you say, by default it is buffering the data. Ideally you would like to do the writing on the background, but I don't think this is possible with LocalFileSystem (since it is one of those things you wonder if it is a smart or a horrible implementation: It uses stuff intended for debugging, not for running a program).

Accepted Answer

Hi Erik,

Thanks for the answer! After I posted the question I was trying to call a fflush after each call to fwrite. Unfortunately, it turns out that the minimum time required for each write operation became around 11ms instead of 6-8us without an explicit call to fflush :(

posted by Harvey N 22 Oct 2015

Well yeah, thats the point :P. Fflush makes it worst case always: It is constant, but it is also long. If thats a problem for your use case, then your options are:

1. Write it somewhere else. For example on an external flash chip. Generally these are a few MB: So it depends on how much data you generate.

2. Write it in the background: If you do it on an SD it is possible to do it on the background (I do have a lib which does that in combination with RTOS, but it is also reasonably old by now)

3. Write on the foreground and make sure your important loop is running as interrupt: This is not possible with LocalFileSystem, it is possible with an SD, it might be possible with USB (depends on your requirements also: USB also uses interrupts and may not be interrupted for too long).

posted by Erik - 22 Oct 2015

Hi Erik, Thanks for the suggestions. Just to make sure, the use of the SD would be through one of the mbed SPI ports (p5-7 or p11-13) with an external card adapter and the USB would be through the mbed USB port (p31-32)?

posted by Harvey N 23 Oct 2015

Yes, correct. There are more options, but those are the main ones I think (you could also get a USB host Arduino shield for example, which you control by Serial/SPI, but I don't think that makes sense if the LPC1768 already has this built-in).

posted by Erik - 23 Oct 2015

Thanks Erik, I will try your suggestions!

posted by Harvey N 26 Oct 2015