Optimizing grep (or using AWK) in a shell script

Posted by Ode on Stack Overflow See other posts from Stack Overflow or by Ode
Published on 2010-05-12T17:16:01Z Indexed on 2010/05/12 18:44 UTC
Read the original article Hit count: 110

Filed under:

grep

|

awk

|

shell

Hi -

In my shell script, I am trying to search using terms found in a $sourcefile against the same $targetfile over and over.

My $sourcefile is formatted as such:

pattern1
pattern2
etc...

The inefficient loop I have to search with is:

for line in $(< $sourcefile);do
    fgrep $line $targetfile | fgrep "RID" >> $outputfile
done

I understand it would be possible to improve this by either loading the whole $targetfile into memory, or perhaps by using AWK?

Thanks

© Stack Overflow or respective owner

Related posts about grep

grep is inconsistently defaulting to grep -P?

as seen on Server Fault - Search for 'Server Fault'
I have a script that does some housekeeping that works perfectly well when invoked from an interactive shell, but did nothing when invoked by cron. To troubleshoot this I started a shell with a 'blank' environment with the command: env -i /bin/bash --noprofile --norc Using this blank env I've dug… >>> More
grep pattern interpretted differently in 2 different systems with same grep version

as seen on Server Fault - Search for 'Server Fault'
We manufacture a linux appliance for data centers, and all are running fedora installed from the same kickstart process. There are different hardware versions, some with IDE hard drives and some SCSI, so the filesystems may be at /dev/sdaN or /dev/hdaN. We have a web interface into these appliances… >>> More
grep --exclude/--include syntax (do not grep through certain files)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm looking for the string "foo=" (without quotes) in text files in a directory tree. It's on a common Linux machine, I have bash shell: grep -ircl "foo=" * In the directories are also many binary files which match "foo=". As these results are not relevant and slow down the search, I want grep… >>> More
Grep failing with Emacs (windows), and GnuWin32 Grep

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I've downloaded and installed the GnuWin32 tools, and added the grep executables to the Emacs bin. I've also, for what its worth, added the GnuWin32 bin folder to my Path variable. Problem is though, when I try and run with suggested grep commands, I always get: Grep exited abnormally with… >>> More
How to grep a line start with "*" using grep

as seen on Super User - Search for 'Super User'
Hi, How can I use 'grep' to get lines start with '* ' in my file? I tried grep "" myfile I tried grep " " myfile but returns all the lines of my file. Thank you. >>> More

Related posts about awk

problem with awk script

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, when I call my awk script, I keep getting an error : sam@sam-laptop:~/shell/td4$ awk -f agenda.awk -- -n Robert agenda.txt awk: agenda.awk:6: printf "Hello" awk: agenda.awk:6: ^ syntax error the script contains this : #!/usr/bin/awk BEGIN { } printf "Hello" END { } Thank you >>> More
Awk to grab colo(u)r codes from CSS files aka School me in Awk

as seen on Stack Overflow - Search for 'Stack Overflow'
Nice and (hopefully) easy. I am trying to work out how to grab the variable #XXX from a text file (css file) containing strings like hr { margin: 18px 0 17px; border-color: #ccc; } h1 a:hover, h2 a:hover, h3 a:hover { color: #001100; } Which I would like to return as ccc 777 The… >>> More
Parsing the output of "uptime" with bash

as seen on Super User - Search for 'Super User'
I would like to save the output of the uptime command into a csv file in a Bash script. Since the uptime command has different output formats based on the time since the last reboot I came up with a pretty heavy solution based on case, but there is surely a more elegant way of doing this. uptime… >>> More
AWK scripting :How to remove Field separator using awk

as seen on Stack Overflow - Search for 'Stack Overflow'
Need the following output ONGC044 ONGC043 ONGC042 ONGC041 ONGC046 ONGC047 from this input Medium Label Medium ID Free Blocks =============================================================================== [ONGC044] ECCPRDDB_FS_43 ac100076:4aed9b39:44f0:0001… >>> More
sort associative array in awk - help?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I have an associative array in awk that gets populated like this... chr_count[$3]++ When I try to print my chr_counts I use this: for (i in chr_count) { print i,":",chr_count[i]; } But not surprisingly, the order of i is not sorted in any way. Is there an easy way to iterate over… >>> More