C# Process Binary File, Multi-Thread Processing
- by washtik
I have the following code that processes a binary file. I want to split the processing workload by using threads and assigning each line of the binary file to threads in the ThreadPool. Processing time for each line is only small but when dealing with files that might contain hundreds of lines, it makes sense to split the workload.
My question is regarding the BinaryReader and thread safety. First of all, is what I am doing below acceptable. I have a feeling it would be better to pass only the binary for each line to the PROCESS_Binary_Return_lineData method.
Please note the code below is conceptual. I looking for a but of guidance on this as my knowledge of multi-threading is in its infancy. Perhaps there is a better way to achieve the same result, i.e. split processing of each binary line.
var dic = new Dictionary<DateTime, Data>();
var resetEvent = new ManualResetEvent(false);
using (var b = new BinaryReader(File.Open(Constants.dataFile,
FileMode.Open, FileAccess.Read, FileShare.Read)))
{
var lByte = b.BaseStream.Length;
var toProcess = 0;
while (lByte >= DATALENGTH)
{
b.BaseStream.Position = lByte;
lByte = lByte - AB_DATALENGTH;
ThreadPool.QueueUserWorkItem(delegate
{
Interlocked.Increment(ref toProcess);
var lineData = PROCESS_Binary_Return_lineData(b);
lock(dic)
{
if (!dic.ContainsKey(lineData.DateTime))
{
dic.Add(lineData.DateTime, lineData);
}
}
if (Interlocked.Decrement(ref toProcess) == 0) resetEvent.Set();
}, null);
}
}
resetEvent.WaitOne();
