Would shell command join cause out of memory?

Posted by Hancy on Super User See other posts from Super User or by Hancy
Published on 2012-12-19T13:14:55Z Indexed on 2012/12/19 17:05 UTC
Read the original article Hit count: 185

Filed under:
|
|

I have two file to join.

FILE 1:

a  A1
a  A2
a  A3
...
c  C1
c  C2
...

FILE 2:

a  feature1_of_a
a  feature2_of_a
...
a  featureN_of_a
...
...
c  feature1_of_c
c  feature2_of_c
...

after join, i could get File like this:

A1  feature1_of_a
A2  feature1_of_a
A3  feature1_of_a
A1  feature2_of_a
A2  feature2_of_a
A3  feature2_of_a
...
A1  featureN_of_a
A2  featureN_of_a
A3  featureN_of_a
...

In order to do that: i wrote shell command join -11 -21 -o1.2,2.2 file1 file2. But the problem is: number N might be huge. So if join read all feautre of a into memory at once, memory might not be enough.

I don't know how join is implemented. WQould the momery become a problem? If so, is there any way to get what I want?

© Super User or respective owner

Related posts about shell

Related posts about memory