Tempat Mencatat

Question

I have just found out that my script gives me a fatal error:

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 440 bytes) in C:\process_txt.php on line 109

That line is this:

$lines = count(file($path)) - 1;

So I think it is having difficulty loading the file into memeory and counting the number of lines, is there a more efficient way I can do this without having memory issues?

The text files that I need to count the number of lines for range from 2MB to 500MB. Maybe a Gig sometimes.

Thanks all for any help.

Dominic Rodger · Accepted Answer · 2010-01-29 14:57:40Z

This will use less memory, since it doesn't load the whole file into memory:

$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
  $line = fgets($handle);
  $linecount++;
}

fclose($handle);

echo $linecount;

fgets loads a single line into memory (if the second argument $length is omitted it will keep reading from the stream until it reaches the end of the line, which is what we want). This is still unlikely to be as quick as using something other than PHP, if you care about wall time as well as memory usage.

The only danger with this is if any lines are particularly long (what if you encounter a 2GB file without line breaks?). In which case you're better off doing slurping it in in chunks, and counting end-of-line characters:

$file="largefile.txt";
$linecount = 0;
$handle = fopen($file, "r");
while(!feof($handle)){
  $line = fgets($handle, 4096);
  $linecount = $linecount + substr_count($line, PHP_EOL);
}

fclose($handle);

echo $linecount;

Thanks for the explanation Dominic - that looks good. I had a feeling it had to be done line by line and not letting count of file load the whole thing into memory! — Abs, Jan 29 '10 at 14:38
The only danger of this snippet are huge files without linebreaks as fgets will then try to suck up the whole file. It'd be safer to read 4kB chunks at a time and count line termination characters. — David Schmitt, Jan 29 '10 at 14:51
@David - how does my edit look? I'm not 100% confident about PHP_EOL - does that look right? — Dominic Rodger, Jan 29 '10 at 14:58
not perfect: you could have a unix-style file (\n) being parsed on a windows machine (PHP_EOL == '\r\n') — nickf, Jan 29 '10 at 15:01
@nickf - good point. How would you address it? How does fgets work? — Dominic Rodger, Jan 29 '10 at 15:23

Tempat Mencatat

Sunday, August 2, 2015

Efficiently counting the number of lines of a text file. (200mb+)

11 Answers