grep: memory exhausted

[Robert Watson]

On Wed, 29 Nov 2006, Kris Kennaway wrote:

> On Wed, Nov 29, 2006 at 03:10:33PM -0500, van Osnabrugge, Sean wrote:
>>
>> I am running a fresh install of FreeBSD-6.1-Stable as a guest OS in VMWare 
>> 1.0.1 with 1 GB of RAM.
>>
>> Whenever I try to grep a large text file (400 MB+), grep terminates with 
>> "grep: memory exhausted"
>>
>> I have tried piping grep (cat "file" | grep "search term")
>>
>> I have tried it with -line-buffered
>>
>> ulimit -a show:
>>
>> core file size (blocks, -c) unlimited
>>
>> data seg size (kbytes, -d) 524288
>
> Try increasing this.  I think grep mmaps the file, so the large file could 
> be exceeding your limit.

I regularly grep multi-hundred-gigabyte files without a problem on boxes with 
very little memory, so simple file size is unlikely the cause.  However, it 
would be interesting to know how many lines "wc -l" thinks the text file has, 
and what the length of the longest line is.

Robert N M Watson
Computer Laboratory
University of Cambridge