[perl] Efficient processing of large text
- by jesper
I have text file that contains over one million urls. I have to process this file in order to assign urls to groups, based on host address:
{
'http://www.ex1.com' = ['http://www.ex1.com/...', 'http://www.ex1.com/...', ...],
'http://www.ex2.com' = ['http://www.ex2.com/...', 'http://www.ex2.com/...', ...]
}
My current basic solution takes about 600mb of RAM to do this (size of file is about 300mb). Could You provide some more efficient ways?
My current solution simply reads line by line, extracts host address by regex and put url into hash.
