Processing large recordsets in Rails

Posted by japancheese on Stack Overflow See other posts from Stack Overflow or by japancheese
Published on 2010-04-08T21:08:24Z Indexed on 2010/04/08 21:13 UTC
Read the original article Hit count: 247

Filed under:

ruby-on-rails

|

scaling

|

database

Hello, I'm trying to perform a daily operation on a larger than normal dataset (2m+ records). However, Rails seems to take a very long time performing operations on such a dataset. Operations like

Dataset.all.each do |data|
  ...
end

take a very long time to complete (I assume this is because it can't fit all the items into memory at once, right?).

Does anyone have any strategies on how I could handle this situation? I know SQL would probably speed up the process, but I'm looking to use the Rails environment as I can do many more complicated things to the data than I can with just SQL statements.

© Stack Overflow or respective owner

Related posts about ruby-on-rails

Ruby on Rails - How can I start? [closed]

as seen on Programmers - Search for 'Programmers'
I have misconception in understanding the relationship between Ruby language and Ruby on Rails Framework. Because of 'I am an absolute beginner' in web development I have no idea if I have to grasp the fundamentals of Ruby before I go with Ruby on Rails! I also want to ask who is behind both Ruby… >>> More
Ruby on rails: Image downloads with Authentication/Authorization/Time outs

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm having few doubts on implementing file downloads. I'm creating an app where I use attachment_fu with Amazon s3 to upload files. Things are working pretty well so far on uploading side. Now its the time to start the file downloads. Here is what I need, a logged in user search and browse… >>> More
DES3 decryption in Ruby on Rails

as seen on Stack Overflow - Search for 'Stack Overflow'
My RoR server receives a string, that was encrypted in C++ application using des3 with base64 encoding The cipher object is created so: cipher = OpenSSL::Cipher::Cipher::new("des3") cipher.key = key_str cipher.iv = iv_str key_str and iv_str: are string representations of key and initialization… >>> More
Ruby on Rails deployment, on "thin" server with lot of attachments

as seen on Stack Overflow - Search for 'Stack Overflow'
A lot of PDFs are stored inside MySQL as a BLOB field for each PDF file. The average file size is 500K each. The Rails app will stream the :binary data as file downloads, where there is a user click on the download link. Assume there is a maximum of 5 users downloading 5 PDFs concurrently, what… >>> More
Apply Behavior Driven Development to Ruby on Rails with Rspec

as seen on Internet.com - Search for 'Internet.com'
Applying the Behavior Driven Development (BDD) methodology to your Rails development takes you beyond traditional Test Driven Development. Learn how to do it with the Ruby-based BDD framework RSpec. >>> More

Related posts about scaling

Scaling background without scaling foreground in platformer?

as seen on Game Development - Search for 'Game Development'
I'm currently developing a platform game and I've run into a problem with scaling resolutions. I want a different resolution of the game to still display the foreground unscaled (characters, tiles, etc) but I want the background to be scaled to fit into the window. To explain this better, my viewport… >>> More
The new Auto Scaling Service in Windows Azure

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
One of the key features of the Cloud is the on-demand scalability, which lets the cloud application developers to scale up or scale down the number of compute resources hosted on the Cloud. Auto Scaling provides the capability to dynamically scale up and scale down your compute resources based on… >>> More
WPF performance on scaling a large scene

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a full screen app that I want to be able to zoom in on certain areas. I have the code working fine, but I notice that when I get closer in, the zoom in animation (which animates the ScaleTransform.ScaleX and ScaleTransform.ScaleY properties on a Parent canvas) starts to jerk down a little… >>> More
AWS: setting up auto-scale for EC2 instances

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Originally posted on: http://geekswithblogs.net/EltonStoneman/archive/2013/10/16/aws-setting-up-auto-scale-for-ec2-instances.aspxWith Amazon Web Services, there’s no direct equivalent to Azure Worker Roles – no Elastic Beanstalk-style application for .NET background workers. But you can get the auto-scale… >>> More
java slick2D - problem using ScalableGame class

as seen on Game Development - Search for 'Game Development'
I have problem adjusting the size of the screen, using the ScalableGame class from Slick2D library. So, what I want to achieve, whenever I change display size, background should adjust to screen size, and objects (images, grahpic shapes) should fit (scale). Alright, so this is how state looks by… >>> More