What is meant by "streaming data access" in HDFS?

Posted by Van Gale on Server Fault See other posts from Server Fault or by Van Gale
Published on 2009-07-14T10:13:54Z Indexed on 2010/05/28 0:22 UTC
Read the original article Hit count: 580

Filed under:

filesystems

|

streaming

|

hadoop

|

hdfs

According to the HDFS Architecture page HDFS was designed for "streaming data access". I'm not sure what that means exactly, but would guess it means an operation like seek is either disabled or has sub-optimal performance. Would this be correct?

I'm interested in using HDFS for storing audio/video files that need to be streamed to browser clients. Most of the streams will be start to finish, but some could have a high number of seeks.

Maybe there is another file system that could do this better?

© Server Fault or respective owner

Related posts about filesystems

Question regarding filesystems true or false?

as seen on Super User - Search for 'Super User'
Hello all, though I'm familiar with stackoverflow , and loving it , i've actually got a couple of questions myself about something other then programming. Here are my question Is it true that in FAT filesystems the maximum number of files per filesystem equals the number of entries in the FAT table… >>> More
Filesystems for webserver with SATA and Solid State disk,

as seen on Server Fault - Search for 'Server Fault'
We have just ordered a new webserver with 120 Gb solid state disk and a SATA disk. I am trying to plan ahead what sort of filesystem to use. This system will be running Linux, Apache/Tomcat to host java services. The main service is a system where people can upload reasonably large files (in the… >>> More
To see virtual filesystems in Ubuntu

as seen on Super User - Search for 'Super User'
How can you see the virtual filesystems at root? They should be /proc and /dev. >>> More
How do filesystems handle concurrent read/write?

as seen on Stack Overflow - Search for 'Stack Overflow'
User A asks the system to read file foo and at the same time user B wants to save his or her data onto the same file. How is this situation handled on the filesystem level? >>> More
Strange ZFS hidden filesystem problem

as seen on Server Fault - Search for 'Server Fault'
Half of my ZFS filesystems are hidden in ZFS-fuse. Here's my story: So, I love ZFS. I used it for about six months on FreeBSD, but due to it crashing the kernel during heavy inter-filesystem IO load, I tried switching to Solaris 5.10. That was good, but when I attempted to do an import of my Version… >>> More

Related posts about streaming

Create NSC-File for multicast streaming with Windows Media Streaming

as seen on Server Fault - Search for 'Server Fault'
For streaming multicast I need to create a NSC-file for the players - this file is pseudo-coded by Microsoft. Now I need to create this file for each proxy-publishingpoint. Is there any way to stream multicast without the NSC-file or can I create the NSC-file on another way? Thanks for any idea ;) >>> More
Intranet Video-streaming

as seen on Super User - Search for 'Super User'
What's a good option for home-video-streaming. For example, somebody may use a laptop webcam to stream their baby's crib allowing them to monitor it from the main home PC. In this case, I wouldn't want to do it across the wire with skype, but instead keep it on my local network. What options exist… >>> More
How to use SQL file streaming win32 API and support WCF streaming

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using Sql server file stream type to store large files in the backend. I'm trying to use WCf to stream the file across to the clients. I'm able to get the handle to the file using SQLFileStream (API). I then try to return this stream. I have implemenetd data chunking on the client side to retrive… >>> More
If Nvidia Shield can stream a game via WiFi (~150-300Mbps), where is the 1-10Gbps wired streaming?

as seen on Super User - Search for 'Super User'
Facts: It is surprising and uncharacteristic that a wireless game streaming solution is the *first to hit the market when a 1000mbps+ Ethernet connection would accomplish the same feat with roughly 6x the available bandwidth. 150-300mbps WiFi is in no way superior to a 1000mbps+ LAN connection aside… >>> More
VLC streaming testbench

as seen on Super User - Search for 'Super User'
Has anyone tried streaming media with VLC as server? I want to deploy VLC as streaming server, but my department didn't had a nice experience with VLC streaming. My question is has anyone tried VLC streaming over LAN with as many as 200 clients? What were the precautions to be taken before going… >>> More