.net real time stream processing - needed huge and fast RAM buffer
- by mack369
The application I'm developing communicates with an digital audio device, which is capable of sending 24 different voice streams at the same time.
The device is connected via USB, using FTDI device (serial port emulator) and D2XX Drivers (basic COM driver is to slow to handle transfer of 4.5Mbit).
Basically the application consist of 3 threads:
Main thread - GUI, control, ect.
Bus reader - in this thread data is continuously read from the device and saved to a file buffer (there is no logic in this thread)
Data interpreter - this thread reads the data from file buffer, converts to samples, does simple sample processing and saves the samples to separate wav files.
The reason why I used file buffer is that I wanted to be sure that I won't loose any samples. The application doesn't use recording all the time, so I've chosen this solution because it was safe.
The application works fine, except that buffered wave file generator is pretty slow. For 24 parallel records of 1 minute, it takes about 4 minutes to complete the recording. I'm pretty sure that eliminating the use of hard drive in this process will increase the speed much.
The second problem is that the file buffer is really heavy for long records and I can't clean this up
until the end of data processing (it would slow down the process even more).
For RAM buffer I need at lest 1GB to make it work properly.
What is the best way to allocate such a big amount of memory in .NET? I'm going to use this memory in 2 threads so a fast synchronization mechanism needed. I'm thinking about a cycle buffer: one big array, the Bus Reader saves the data, the Data Interpreter reads it. What do you think about it?
[edit]
Now for buffering I'm using classes BinaryReader and BinaryWriter based on a file.
