Important changes to forums and questions
All forums and questions are now archived. To start a new conversation or read the latest updates go to forums.mbed.com.
9 years, 1 month ago.
fwrite writing time not consistent?
Hi all,
I am working on an application that requires to write data as fast as possible to a file. I used the fwrite fonction to write the data by blocks of fixed length. However it seems that the time required for one call to fwrite is not consistent: each time the file size is increased by 512 bytes, the writing operation seems to take several ms, while it takes few micro-seconds otherwise, independently of the block size parameter of the fwrite function.
Below is a simple test program to write data in a file by blocks of fixed length, and then display the time required for each write operation.
I made tests on the local file system and with an application board (through the USBHostMSD class and a 2GB memory stick plugged on the USB port), and got similar observations.
Is there a way to have a consistent writing time ?
Thanks in advance
[EDIT] After a closer look, it seems that the data are first buffered before being actually written on the file, which could explain the inconsistency in the writing time. Can someone confirm this ? If it is the case, is there a way to go around the buffering issue ?
include the mbed library with this snippet
#include "mbed.h" LocalFileSystem local("local"); // Create the local filesystem under the name "local" DigitalOut led1(LED1); DigitalOut led2(LED2); Serial pc(USBTX, USBRX); // Serial console #define BUFFER_SIZE 5000 // Buffer to store the data to be written on file int main() { // Set the serial baud rate pc.baud(921600); // Initialize the source buffer uint8_t buff[BUFFER_SIZE]; for(int i = 0; i < BUFFER_SIZE; i++) buff[i] = 'a' + i%10; // abcdefghij // Create the file FILE *pFile = NULL; pFile = fopen("/local/a.txt","w"); // Initialize the timer related variables Timer timer; int nTime_us = 0; int nDT_us[BUFFER_SIZE]; memset((void* )nDT_us, 0, BUFFER_SIZE*sizeof(int)); // Initialize the writing parameters int nBlockSize = 1; // Size of one data block of data to write int nBytesToWrite = 513; // Total number of bytes to write to the file int nBytesWritten = 0; // Number of bytes written in a single write operation int nTotalBytesWritten = 0; // Total number of bytes written int nNbBlocksToWrite = 1; // Number of data blocks to write in a single write operation int nBlocksWritten = 0; // Number of data blocks written in a single write operation int nNbWriteOp = 0; // Number of write operations pc.printf( "\r\n\nNB bytes to write: %d (Block size: %d)\r\n", nBytesToWrite, nBlockSize ); // Start the data writing timer.start(); while(nBytesToWrite > 0) { nTime_us = timer.read_us(); // Write the data block nBlocksWritten = fwrite( buff, // const void *ptr nBlockSize, // size_t size nNbBlocksToWrite, // size_t nmemb pFile // FILE *stream ); // Store the time required for the write operation if(nNbWriteOp < BUFFER_SIZE) nDT_us[nNbWriteOp++] = timer.read_us() - nTime_us; // Update the remaining number of bytes to write if(nBlocksWritten > 0) // Write success { nBytesWritten = nBlocksWritten * nBlockSize; nBytesToWrite -= nBytesWritten; nTotalBytesWritten += nBytesWritten; led1 = !led1; } else { led2 = !led2; // Write error } } // Display the time required for each write operation pc.printf( "\r\n%d bytes written\r\n", nTotalBytesWritten ); pc.printf("\r\nWrite time:\r\n"); for(int i = 0; i < nNbWriteOp; i++) { if(nDT_us[i] > 10) pc.printf("#%d:", i); // Mark the slower write operations pc.printf( "%d ", nDT_us[i] ); } pc.printf("\r\n"); // Close the file fclose(pFile); }
1 Answer
9 years, 1 month ago.
Calling fflush(pFile) should do the job (I think). Indeed what you say, by default it is buffering the data. Ideally you would like to do the writing on the background, but I don't think this is possible with LocalFileSystem (since it is one of those things you wonder if it is a smart or a horrible implementation: It uses stuff intended for debugging, not for running a program).
Hi Erik,
Thanks for the answer! After I posted the question I was trying to call a fflush after each call to fwrite. Unfortunately, it turns out that the minimum time required for each write operation became around 11ms instead of 6-8us without an explicit call to fflush :(
posted by 22 Oct 2015Well yeah, thats the point :P. Fflush makes it worst case always: It is constant, but it is also long. If thats a problem for your use case, then your options are:
1. Write it somewhere else. For example on an external flash chip. Generally these are a few MB: So it depends on how much data you generate.
2. Write it in the background: If you do it on an SD it is possible to do it on the background (I do have a lib which does that in combination with RTOS, but it is also reasonably old by now)
3. Write on the foreground and make sure your important loop is running as interrupt: This is not possible with LocalFileSystem, it is possible with an SD, it might be possible with USB (depends on your requirements also: USB also uses interrupts and may not be interrupted for too long).
posted by 22 Oct 2015Hi Erik, Thanks for the suggestions. Just to make sure, the use of the SD would be through one of the mbed SPI ports (p5-7 or p11-13) with an external card adapter and the USB would be through the mbed USB port (p31-32)?
posted by 23 Oct 2015