C# : Forcing a clean run in a long running SQL reader loop?
- by Wardy
I have a SQL data reader that reads 2 columns from a sql db table.
once it has done its bit it then starts again selecting another 2 columns.
I would pull the whole lot in one go but that presents a whole other set of challenges.
My problem is that the table contains a large amount of data (some 3 million rows or so) which makes working with the entire set a bit of a problem.
I'm trying to validate the field values so i'm pulling the ID column then one of the other cols and running each value in the column through a validation pipeline where the results are stored in another database.
My problem is that when the reader hits the end of handlin one column I need to force it to immediately clean up every little block of ram used as this process uses about 700MB and it has about 200 columns to go through.
Without a full Garbage Collect I will definately run out of ram.
Anyone got any ideas how I can do this?
I'm using lots of small reusable objects, my thought was that I could just call GC.Collect() on the end of each read cycle and that would flush everything out, unfortunately that isn't happening for some reason.
