Bad performance function in PHP. With large files memory blows up! How can I refactor?
- by André
Hi
I have a function that strips out lines from files. I'm handling with large files(more than 100Mb). I have the PHP Memory with 256MB but the function that handles with the strip out of lines blows up with a 100MB CSV File.
What the function must do is this:
Originally I have the CSV like:
Copyright (c) 2007 MaxMind LLC. All
Rights Reserved.
locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
1,"O1","","","",0.0000,0.0000,,
2,"AP","","","",35.0000,105.0000,,
3,"EU","","","",47.0000,8.0000,,
4,"AD","","","",42.5000,1.5000,,
5,"AE","","","",24.0000,54.0000,,
6,"AF","","","",33.0000,65.0000,,
7,"AG","","","",17.0500,-61.8000,,
8,"AI","","","",18.2500,-63.1667,,
9,"AL","","","",41.0000,20.0000,,
When I pass the CSV file to this function I got:
locId,country,region,city,postalCode,latitude,longitude,metroCode,areaCode
1,"O1","","","",0.0000,0.0000,,
2,"AP","","","",35.0000,105.0000,,
3,"EU","","","",47.0000,8.0000,,
4,"AD","","","",42.5000,1.5000,,
5,"AE","","","",24.0000,54.0000,,
6,"AF","","","",33.0000,65.0000,,
7,"AG","","","",17.0500,-61.8000,,
8,"AI","","","",18.2500,-63.1667,,
9,"AL","","","",41.0000,20.0000,,
It only strips out the first line, nothing more. The problem is the performance of this function with large files, it blows up the memory.
The function is:
public function deleteLine($line_no, $csvFileName) {
// this function strips a specific line from a file
// if a line is stripped, functions returns True else false
//
// e.g.
// deleteLine(-1, xyz.csv); // strip last line
// deleteLine(1, xyz.csv); // strip first line
// Assigna o nome do ficheiro
$filename = $csvFileName;
$strip_return=FALSE;
$data=file($filename);
$pipe=fopen($filename,'w');
$size=count($data);
if($line_no==-1) $skip=$size-1;
else $skip=$line_no-1;
for($line=0;$line<$size;$line++)
if($line!=$skip)
fputs($pipe,$data[$line]);
else
$strip_return=TRUE;
return $strip_return;
}
It is possible to refactor this function to not blow up with the 256MB PHP Memory?
Give me some clues.
Best Regards,
