Fast distributed filesystem for a large amounts of data with metadata in database
- by undefined hero
My project uses several processing machines and one storage machine. Currently storage organized with a MSSQL filetable shared folder. Every file in storage have some metadata in database.
Processing machines executes tasks for which they needed files from storage and their metadata. After completing task, processing machine puts resulting data back in storage. From there its taken by another processing machine, which also generates some file and put it back in storage. And etc.
Everything was fine, but as number of processing machines increases, I found myself bottlenecked myself with storage machines hard drive performance.
So I want processing machines to put files in distributed FS. to lift load from storage machines, from which they can take data from each other, not only storage machine.
Can You suggest a particular distributed FS which meets my needs? Or there is another way to solve this problem, without it?
Amounts of data in FS in one time are like several terabytes. (storage can handle this, but processors cannot). Data consistence is critical. Read write policy is: once file is written - its constant and may be only removed, but not modified.
My current platform is Windows, but I'm ready to switch it, if there is a substantially more convenient solution on another one.
