[ACCEPTED]-Fastest way to create large file in c++?-bulkinsert

Accepted answer
Score: 17

using old style file io

fopen the file for write.

fseek to 2 the desired file size - 1.

fwrite a single byte

fclose 1 the file

Score: 12

The fastest way to create a file of a certain 7 size is to simply create a zero-length file 6 using creat() or open() and then change the size using 5 chsize(). This will simply allocate blocks on the 4 disk for the file, the contents will be 3 whatever happened to be in those blocks. It's 2 very fast since no buffer writing needs 1 to take place.

Score: 2

Not sure I understand the question. Do 4 you want to ensure that every character 3 in the file is a printable ASCII character? If 2 so, what about this? Fills the file with 1 "abcdefghabc...."

#include <stdio.h>
int main ()
{
   const int FILE_SiZE = 50000; //size in KB
   const int BUFFER_SIZE = 1024;
   char buffer [BUFFER_SIZE + 1];
   int i;
   for(i = 0; i < BUFFER_SIZE; i++)
      buffer[i] = (char)(i%8 + 'a');
   buffer[BUFFER_SIZE] = '\0';

   FILE *pFile = fopen ("somefile.txt", "w");
   for (i = 0; i < FILE_SIZE; i++)
     fprintf(pFile, buffer);

   fclose(pFile);

   return 0;
}
Score: 1

You haven't mentioned the OS but I'll assume 13 creat/open/close/write are available.

For 12 truly efficient writing and assuming, say, a 11 4k page and disk block size and a repeated 10 string:

  1. open the file.
  2. allocate 4k * number of chars in your repeated string, ideally aligned to a page boundary.
  3. print repeated string into the memory 4k times, filling the blocks precisely.
  4. Use write() to write out the blocks to disk as many times as necessary. You may wish to write a partial piece for the last block to get the size to come out right.
  5. close the file.

This bypasses the buffering of fopen() and 9 friends, which is good and bad: their buffering 8 means that they're nice and fast, but they 7 are still not going to be as efficient as 6 this, which has no overhead of working with 5 the buffer.

This can easily be written in 4 C++ or C, but does assume that you're going 3 to use POSIX calls rather than iostream 2 or stdio for efficiency's sake, so it's 1 outside the core library specification.

Score: 1

I faced the same problem, creating a ~500MB 6 file on Windows very fast. The larger buffer 5 you pass to fwrite() the fastest you'll 4 be.

int i;
FILE *fp;

fp = fopen(fname,"wb");

if (fp != NULL) {

    // create big block's data
    uint8_t b[278528]; // some big chunk size

    for( i = 0; i < sizeof(b); i++ ) // custom initialization if != 0x00
    {
        b[i] = 0xFF;
    }

    // write all blocks to file
    for( i = 0; i < TOT_BLOCKS; i++ )
        fwrite(&b, sizeof(b), 1, fp);

    fclose (fp);
}

Now at least on my Win7, MinGW, creates 3 file almost instantly. Compared to fwrite() 1 2 byte at time, that will complete in 10 Secs. Passing 1 4k buffer will complete in 2 Secs.

Score: 0

Fastest way to create large file in c++? Ok. I assume fastest way means the one 5 that takes the smallest run time.

Create a flat text file in c++ around 50 - 100 MB with the content 'Added first line' should be inserted in to the file for 4 million times.

preallocate 4 the file using old style file io

fopen the file for write.
fseek to the desired file size - 1.
fwrite a single byte
fclose the file

create a string containing the "Added first line\n" a thousand times.
find it's length.

preallocate 3 the file using old style file io

fopen the file for write.
fseek to the the string length * 4000
fwrite a single byte
fclose the file

open the file for read/write
loop 4000 times, 
    writing the string to the file.
close the file.

That's my 2 best guess. I'm sure there are a lot of 1 ways to do it.

More Related questions