Creating thousands of records in Rails
- by willCosgrove
Let me set the stage: My application deals with gift cards. When we create cards they have to have a unique string that the user can use to redeem it with. So when someone orders our gift cards, like a retailer, we need to make a lot of new card objects and store them in the DB.
With that in mind, I'm trying to see how quickly I can have my application generate 100,000 Cards. Database expert, I am not, so I need someone to explain this little phenomena: When I create 1000 Cards, it takes 5 seconds. When I create 100,000 cards it should take 500 seconds right?
Now I know what you're wanting to see, the card creation method I'm using, because the first assumption would be that it's getting slower because it's checking the uniqueness of a bunch of cards, more as it goes along. But I can show you my rake task
desc "Creates cards for a retailer"
task :order_cards, [:number_of_cards, :value, :retailer_name] => :environment do |t, args|
t = Time.now
puts "Searching for retailer"
@retailer = Retailer.find_by_name(args[:retailer_name])
puts "Retailer found"
puts "Generating codes"
value = args[:value].to_i
number_of_cards = args[:number_of_cards].to_i
codes = []
top_off_codes(codes, number_of_cards)
while codes != codes.uniq
codes.uniq!
top_off_codes(codes, number_of_cards)
end
stored_codes = Card.all.collect do |c|
c.code
end
while codes != (codes - stored_codes)
codes -= stored_codes
top_off_codes(codes, number_of_cards)
end
puts "Codes are unique and generated"
puts "Creating bundle"
@bundle = @retailer.bundles.create!(:value => value)
puts "Bundle created"
puts "Creating cards"
@bundle.transaction do
codes.each do |code|
@bundle.cards.create!(:code => code)
end
end
puts "Cards generated in #{Time.now - t}s"
end
def top_off_codes(codes, intended_number)
(intended_number - codes.size).times do
codes << ReadableRandom.get(CODE_LENGTH)
end
end
I'm using a gem called readable_random for the unique code. So if you read through all of that code, you'll see that it does all of it's uniqueness testing before it ever starts creating cards. It also writes status updates to the screen while it's running, and it always sits for a while at creating. Meanwhile it flies through the uniqueness tests. So my question to the stackoverflow community is: Why is my database slowing down as I add more cards? Why is this not a linear function in regards to time per card? I'm sure the answer is simple and I'm just a moron who knows nothing about data storage. And if anyone has any suggestions, how would you optimize this method, and how fast do you think you could get it to create 100,000 cards?
(When I plotted out my times on a graph and did a quick curve fit to get my line formula, I calculated how long it would take to create 100,000 cards with my current code and it says 5.5 hours. That maybe completely wrong, I'm not sure. But if it stays on the line I curve fitted, it would be right around there.)
