Fastest way to read data from a lot of ASCII files

Posted by Alsenes on Stack Overflow See other posts from Stack Overflow or by Alsenes
Published on 2011-02-12T15:20:11Z Indexed on 2011/02/12 15:25 UTC
Read the original article Hit count: 211

Filed under:
|
|
|

Hi guys, for a college exercise that I've already submitted I needed to read a .txt file wich contained a lot of names of images(1 in each line). Then I needed to open each image as an ascii file, and read their data(images where in ppm format), and do a series of things with them. The things is, I noticed my program was taking 70% of the time in the reading the data from the file part, instead of in the other calculations that I was doing (finding number of repetitions of each pixel with a hash table, finding diferents pixels beetween 2 images etc..), which I found quite odd to say the least.

This is how the ppm format looks like:

P3 //This value can be ignored when reading the file, because all image will be correctly formatted
4 4
255 //This value can be also ignored, will be always 255.
0  0  0    0  0  0    0  0  0   15  0 15
0  0  0    0 15  7    0  0  0    0  0  0
0  0  0    0  0  0    0 15  7    0  0  0
15  0 15    0  0  0    0  0  0    0  0  0

This is how I was reading the data from the files:

ifstream fdatos;
fdatos.open(argv[1]); //Open file with the name of all the images

const int size = 128;
char file[size]; //Where I'll get the image name

Image *img;
while (fdatos >> file) { //While there's still images anmes left, continue
    ifstream fimagen;
fimagen.open(file);  //Open image file
img = new Image(fimagen); //Create new image object with it's data file
    ………
    //Rest of the calculations whith that image
    ………
delete img; //Delete image object after done 
    fimagen.close(); //Close image file after done
}

fdatos.close();

And inside the image object read the data like this:

const int tallafirma = 100;
char firma[tallafirma];
fich_in >> std::setw(100) >> firma; // Read the P3 part, can be ignored

int maxvalue, numpixels;
fich_in >> height >> width >> maxvalue; // Read the next three values
numpixels = height*width;
datos = new Pixel[numpixels];

int r,g,b; //Don't need to be ints, max value is 256, so an unsigned char would be ok.
for (int i=0; i<numpixels; i++) {
   fich_in >> r >> g >> b;
   datos[i] = Pixel( r, g ,b);
}
//This last part is the slow one, 
//I thing I should be able to read all this data in one single read 
//to buffer or something which would be stored in an array of unsigned chars, 
//and  then I'd only need to to do:
//buffer[0] -> //Pixel 1 - Red data
//buffer[1] -> //Pixel 1 - Green data
//buffer[2] -> //Pixel 1 - Blue data

So, any Ideas? I think I can improve it quite a bit reading all to an array in one single call, I just don't know how that is done.

Also, is it posible to know how many images will be in the "index file"? Is it posiible to know the number of lines a file has?(because there's one file name per line..)

Thanks!!

© Stack Overflow or respective owner

Related posts about c++

Related posts about io