C# Process Binary File, Multi-Thread Processing

Posted by washtik on Stack Overflow See other posts from Stack Overflow or by washtik
Published on 2010-04-21T09:53:43Z Indexed on 2010/04/21 10:13 UTC
Read the original article Hit count: 271

I have the following code that processes a binary file. I want to split the processing workload by using threads and assigning each line of the binary file to threads in the ThreadPool. Processing time for each line is only small but when dealing with files that might contain hundreds of lines, it makes sense to split the workload.

My question is regarding the BinaryReader and thread safety. First of all, is what I am doing below acceptable. I have a feeling it would be better to pass only the binary for each line to the PROCESS_Binary_Return_lineData method.

Please note the code below is conceptual. I looking for a but of guidance on this as my knowledge of multi-threading is in its infancy. Perhaps there is a better way to achieve the same result, i.e. split processing of each binary line.

        var dic = new Dictionary<DateTime, Data>();        
        var resetEvent = new ManualResetEvent(false);

        using (var b = new BinaryReader(File.Open(Constants.dataFile, 
                            FileMode.Open, FileAccess.Read, FileShare.Read)))
        {
        var lByte = b.BaseStream.Length;
        var toProcess = 0;

        while (lByte >= DATALENGTH)
        {
            b.BaseStream.Position = lByte;
            lByte = lByte - AB_DATALENGTH;

            ThreadPool.QueueUserWorkItem(delegate
            {
                Interlocked.Increment(ref toProcess);
                var lineData = PROCESS_Binary_Return_lineData(b);

                lock(dic)
                {
                    if (!dic.ContainsKey(lineData.DateTime))
                    {
                     dic.Add(lineData.DateTime, lineData); 
                    }
                }

                if (Interlocked.Decrement(ref toProcess) == 0) resetEvent.Set();
            }, null);
        }
        }

        resetEvent.WaitOne();

© Stack Overflow or respective owner

Related posts about c#

Related posts about multithreading