Reading a file N lines at a time in ruby

Posted by Sam on Stack Overflow See other posts from Stack Overflow or by Sam
Published on 2010-03-23T01:58:11Z Indexed on 2010/03/23 2:01 UTC
Read the original article Hit count: 309

Filed under:

ruby

|

file-io

|

fork

I have a large file (hundreds of megs) that consists of filenames, one per line.

I need to loop through the list of filenames, and fork off a process for each filename. I want a maximum of 8 forked processes at a time and I don't want to read the whole filename list into RAM at once.

I'm not even sure where to begin, can anyone help me out?

© Stack Overflow or respective owner

Related posts about ruby

Setting up Rails to work with sqlserver

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok I followed the steps for setting up ruby and rails on my Vista machine and I am having a problem connecting to the database. Contents of database.yml development: adapter: sqlserver database: APPS_SETUP Host: WindowsVT06\SQLEXPRESS Username: se Password: paswd Run rake db:migrate… >>> More
marshal data too short!!!

as seen on Stack Overflow - Search for 'Stack Overflow'
My application requires to keep large data objects in session. There are like 3-4 data objects each created by parsing a csv containing 150 X 20 cells having strings of 3-4 characters. My application shows this error- "marshal data too short". I tried this- Deleting the old session table. Deleting… >>> More
Sinatra and XML POST request

as seen on Stack Overflow - Search for 'Stack Overflow'
I don't know is it my mistake or no. So i have that code: <code> post '/singin/get_token' do content_type :xml puts request.body.read puts xmlRequest xmlRequest = REXML::Document.new(request.body.read) ... </code> And when i post something like that: <code> <?xml… >>> More
how to change ruby path from /usr/bin/ruby to /usr/local/bin/ruby

as seen on Stack Overflow - Search for 'Stack Overflow'
reading around the various ruby install tutorials it's required to change path from /usr/bin/ruby to /usr/local/bin/ruby but i cant seem to be able to do it. Ultimately i want to install Ruby 1.9.2, should i uninstall 1.8.7 or what? i tried to install Ruby 1.9.2 with macports, the installation seemed… >>> More
strange bundler error: tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) o

as seen on Stack Overflow - Search for 'Stack Overflow'
i am getting a strange bundler error when running bundle pack with bundler 0.9.12 any ideas? (see pastie for a better formatted code: http://pastie.org/881328 ) /opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/site_ruby/1.8/rubygems/package/tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) … >>> More

Related posts about file-io

File io error Python

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a program that monitors a folder with word documents for any modifications made on the files. The error -Windows Error[2] The system cannot find the file specified- comes when I run the program, open a .doc within the folder make some changes and save it. Any suggestions on how to fix this… >>> More
Problem with File IO and splitting strings with Environment.NewLine in VB.Net

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I was experimenting with basic VB.Net file read/write and encountered this problem. I don't know whether it has something to do with the File IO or the String splitting. I am writing text to a file like so Dim sWriter As New StreamWriter("Data.txt") sWriter.WriteLine("FirstItem") sWriter.WriteLine("SecondItem") sWriter… >>> More
C# Multithreading File IO (Reading)

as seen on Stack Overflow - Search for 'Stack Overflow'
We have a situation where our application needs to process a series of files and rather than perform this function synchronously, we would like to employ multi-threading to have the workload split amongst different threads. Each item of work is: 1. Open a file for read only 2. Process the data in… >>> More
ASP.Net Application Trust Medium File IO Outside Virtual Directory

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to determine how suicidal this is... I have a hosting environment where a custom ASP.Net CMS application needs to access the files in the root folder of a website even though it is in a virtual folder so it can be shared accross many sites. I can modify the Medium trust on the server… >>> More
On MacOSX, in a C++ program, what guarantees can I have on file IO

as seen on Stack Overflow - Search for 'Stack Overflow'
I am on MacOSX. I am writing a multi threaded program. One thread does logging. The non-logging threads may crash at any time. What conventions should I adopt in the logger / what guarantees can I have? I would prefer a solution where even if I crash during part of a write, previous writes still… >>> More