Stream tar.gz file from FTP server

Posted by linker on Server Fault See other posts from Server Fault or by linker
Published on 2012-06-26T19:41:03Z Indexed on 2012/06/26 21:17 UTC
Read the original article Hit count: 161

Filed under:

ftp

|

shell-scripting

|

wget

Here is the situation: I have a tar.gz file on a FTP server which can contain an arbitrary number of files.

Now what I'm trying to accomplish is have this file streamed and uploaded to HDFS through a Hadoop job. The fact that it's Hadoop is not important, in the end what I need to do is write some shell script that would take this file form ftp with wget and write the output to a stream.

The reason why I really need to use streams is that there will be a large number of these files, and each file will be huge.

It's fairly easy to do if I have a gzipped file and I'm doing something like this:

wget -O - "ftp://${user}:${pass}@${host}/$file" | zcat

But I'm not even sure if this is possible for a tar.gz file, especially since there are mutliple files in the archive. I'm a bit confused on what direction to take for this, any help would be greatly appreciated.

© Server Fault or respective owner

Related posts about ftp

sudo apt-get update problem

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a problem with sudo apt-get update that I can't seem to fix and It's causing problems with alot of installations I want to do. I run Ubuntu 12.04. Ign http://ftp.availo.se precise InRelease Ign http://ftp.availo.se precise-updates InRelease Ign http://ftp.availo… >>> More
Why does DEP kill IE when accessing Microsoft FTP?

as seen on Super User - Search for 'Super User'
I start up IE (9.0.8112.16421) with about:blank and I go to ftp://ftp.microsoft.com/ I press Alt, click View and then Open FTP Site in Windows Explorer. At this point IE stops responding and eventually crashes (though the window is still active, sometimes) and I get the usual Windows dialog box saying… >>> More
Shrinking TCP Window Size to 0 on Cisco ASA

as seen on Server Fault - Search for 'Server Fault'
Having an issue with any large file transfer that crosses our Cisco ASA unit come to an eventual pause. Setup Test1: Server A, FileZilla Client <- 1GBPS - Cisco ASA <- 1 GBPS - Server B, FileZilla Server TCP Window size on large transfers will drop to 0 after around 30 seconds of a large… >>> More
Shrinking Windows Size to 0 on Cisco ASA

as seen on Server Fault - Search for 'Server Fault'
Having an issue with any large file transfer that crosses our Cisco ASA unit come to an eventual pause. Setup Test1: Server A, FileZilla Client <- 1GBPS - Cisco ASA <- 1 GBPS - Server B, FileZilla Server TCP Window size on large transfers will drop to 0 after around 30 seconds of a large… >>> More
Weird FTP issue between Unity Express and Windows Server 2008 FTP

as seen on Server Fault - Search for 'Server Fault'
My VOIP specialist complained about not being able to run backups of the Unity Express onto our FTP server (Microsoft FTP on Server 2008). I did a packet trace and observed some weird behavior that I think is even kind of funny in a way. The Unity FTP client is able to initiate both control and data… >>> More

Related posts about shell-scripting

UNIX Shell-scripting: UDV

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a simple unix shell script: #!/bin/bash # abort the script if a command fails set -e # abort the script if an unitialized shell variable is used set -u i = 0; while [$i -l 1] do src/meshpro input/martini.off video/noise/image$i… >>> More
idioms for returning multiple values in shell scripting

as seen on Stack Overflow - Search for 'Stack Overflow'
Are there any idioms for returning multiple values from a bash function within a script? http://tldp.org/LDP/abs/html/assortedtips.html describes how to echo multiple values and process the results (e.g., example 35-17), but that gets tricky if some of the returned values are strings with spaces… >>> More
Shell scripting: Way to parse parameters from command line

as seen on Super User - Search for 'Super User'
myscript [-a a-arg] [-c c-arg] [-b] [-e] somedirectory Given that I want my shell script be invoked at the command line using the above parameters - where [these brackets] denote that they are optional - what is the best method to parse them? >>> More
Best book for learning linux shell scripting?

as seen on Server Fault - Search for 'Server Fault'
I normally works on Windows machines but on some occasions I do switch to development on linux. And my most recent project will be written entirely on a certain linix platforms (not the standard Apache/MySQL/PHP setup). So I thought it would pay to learn to write some linux automation script now. I… >>> More
windows shell scripting through java

as seen on Stack Overflow - Search for 'Stack Overflow'
i am designing a desktop application like nero for removable media storage that also includes features of copyfast. i want to add an option of my own choice in the windows explorer right click menu. how can i do that? am i proceeding in the right direction? >>> More