Scalable distributed file system for blobs like images and other documents

Posted by Pinnacle on Programmers See other posts from Programmers or by Pinnacle
Published on 2011-10-12T07:18:13Z Indexed on 2012/10/30 23:19 UTC
Read the original article Hit count: 259

Filed under:

distributed-computing

|

file-systems

|

hadoop

|

cassandra

Cassandra & HBase both do not efficiently support storage of blobs like images. Storing directly on HDFS stresses the Namenode because of huge number of files. Facebook uses Haystack for images and attachments storage, but this is not open source. So is Lustre a good choice for distributed blob storage? I have read that Amazon S3 is used by many, but this would cost money and personally, I would not like to rely on third party system. What are other suggestions?

© Programmers or respective owner

Related posts about distributed-computing

Distributed Computing Framework (.NET) - Specifically for CPU Instensive operations

as seen on Stack Overflow - Search for 'Stack Overflow'
I am currently researching the options that are available (both Open Source and Commercial) for developing a distributed application. "A distributed system consists of multiple autonomous computers that communicate through a computer network." Wikipedia The application is focused on distributing… >>> More
Where do I start with distributed computing?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm interested in learning techniques for distributed computing. As a Java developer, I'm probably willing to start with Hadoop. Could you please recommend some books/tutorials/articles to begin with? >>> More
running a python script where dependencies are not avail: distributed computing

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have access to a grid (running condor) that would (potentially) allow to very substantially reduce how long by nltk based nlp tasks take. unfortunately, i dont have root access on the cluster so cannot install new packages, only run whatever is available on the linux boxes. python is of course… >>> More
Java - System design with distributed Queues and Locks

as seen on Programmers - Search for 'Programmers'
Looking for inputs to evaluate a design for a system (java) which would have a distributed queue serving several (but not too many) nodes. These nodes would process objects present in the distributed queue and on occasion require a distributed lock across the cluster on an arbitrary (distributed)… >>> More
Can an issue tracking system be distributed?

as seen on Programmers - Search for 'Programmers'
I was thinking about issue tracking software like Redmine, Trac or even the one that is in Fossil and something hit me: Is there a reason why Redmine and Trac are not possible to be distributed? Or maybe it's possible and I just don't know how it's possible? If it's not possible, why? By distributed… >>> More

Related posts about file-systems

Question regarding filesystems true or false?

as seen on Super User - Search for 'Super User'
Hello all, though I'm familiar with stackoverflow , and loving it , i've actually got a couple of questions myself about something other then programming. Here are my question Is it true that in FAT filesystems the maximum number of files per filesystem equals the number of entries in the FAT table… >>> More
Which file systems support newlines in file names?

as seen on Super User - Search for 'Super User'
For a programming challenge I need a filesystem which supports newline characters in filenames, so a file can be named something like: A filename with newlines I can't find any. Can anyone help me? Thanks >>> More
Distributed file systems

as seen on Server Fault - Search for 'Server Fault'
I need to implement a distributed storage system for a set of nodes(devices) connected in a mesh network. So what basically my design goals are: The storage system should be capable of handling dynamic entry and exit of nodes. Replication (for fault tolerance). For this i am thinking of using… >>> More
Distributed File Systems.

as seen on Server Fault - Search for 'Server Fault'
So, I've been reading several articles around ServerFault as well as google. (For Example, this link) My Requirements are very similar to the link above, however i'd like to also have dynamic or at least resizeable file volumes, so if necessary i can add 4-5 servers to the pool, and then expand the… >>> More
Strange ZFS hidden filesystem problem

as seen on Server Fault - Search for 'Server Fault'
Half of my ZFS filesystems are hidden in ZFS-fuse. Here's my story: So, I love ZFS. I used it for about six months on FreeBSD, but due to it crashing the kernel during heavy inter-filesystem IO load, I tried switching to Solaris 5.10. That was good, but when I attempted to do an import of my Version… >>> More