[svlug] extended grep reg exp

Piotr T Zbiegiel peter at usestrict.org
Tue Aug 12 14:45:34 PDT 2003


I don't understand your comment about grep not opening the file.  It
does open the file.  It reads one line at a time.  Perl can also read it
one line at a time with minimal impact to the system.  In fact, Perl is
MUCH faster than grep when searching large files (in my own tests I was
able to process 2 gigs of logs in <2min versus >15+minutes with grep). 
If it's a Perl script already, use Perl to open the file and read it in
searching for the lines you want.  Just don't slurp it into memory using
the list context of <> or by setting $/ to null and you should be okay.

Later,
Peter

On Tue, 2003-08-12 at 14:14, Robert Khachikyan wrote:
> Indeed it can be done in awk or sed. This file is about 180MB.
> If I awk it, it'll load the file to memory serially and read
> it line by line...not sure if sed does that...
> But it's going to run under a cron every hour in a system where
> the CPU & memory are taken by other higher priority processes. If I
> crash the system, or make it a bit slower, i'm going to piss off
> lots of people...that's why i'm trying to stay away from awk.
> 
> & since grep doesn't open the file, it won't use as much resources
> as awk (i think sed too..but not sure)
> 
> The script is a perl script that calculates some formulas
> and results in 3918404. & I know i want every record until
> the 3928405. But there are duplicates in the middle where
> for each number there are different values of bla bla.
> 
> 
> As Always,
> ...Robert
> 






More information about the svlug mailing list