Ruby zlib deflate massive data

Posted by Bub Bradlee on Stack Overflow See other posts from Stack Overflow or by Bub Bradlee
Published on 2010-04-12T14:59:57Z Indexed on 2010/04/12 15:03 UTC
Read the original article Hit count: 477

Filed under:

ruby

|

zlib

|

compress

|

deflate

I'm trying to use Zlib::Deflate.deflate on a massive file (4 gigs). There are obvious problems with doing that, the first of which being that I can't load the entire file into memory all at once. Zlib::GzipWriter would work, since it works with streams, but it's not zlib compression. Any ideas?

© Stack Overflow or respective owner

Related posts about ruby

Setting up Rails to work with sqlserver

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok I followed the steps for setting up ruby and rails on my Vista machine and I am having a problem connecting to the database. Contents of database.yml development: adapter: sqlserver database: APPS_SETUP Host: WindowsVT06\SQLEXPRESS Username: se Password: paswd Run rake db:migrate… >>> More
marshal data too short!!!

as seen on Stack Overflow - Search for 'Stack Overflow'
My application requires to keep large data objects in session. There are like 3-4 data objects each created by parsing a csv containing 150 X 20 cells having strings of 3-4 characters. My application shows this error- "marshal data too short". I tried this- Deleting the old session table. Deleting… >>> More
Sinatra and XML POST request

as seen on Stack Overflow - Search for 'Stack Overflow'
I don't know is it my mistake or no. So i have that code: <code> post '/singin/get_token' do content_type :xml puts request.body.read puts xmlRequest xmlRequest = REXML::Document.new(request.body.read) ... </code> And when i post something like that: <code> <?xml… >>> More
how to change ruby path from /usr/bin/ruby to /usr/local/bin/ruby

as seen on Stack Overflow - Search for 'Stack Overflow'
reading around the various ruby install tutorials it's required to change path from /usr/bin/ruby to /usr/local/bin/ruby but i cant seem to be able to do it. Ultimately i want to install Ruby 1.9.2, should i uninstall 1.8.7 or what? i tried to install Ruby 1.9.2 with macports, the installation seemed… >>> More
strange bundler error: tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) o

as seen on Stack Overflow - Search for 'Stack Overflow'
i am getting a strange bundler error when running bundle pack with bundler 0.9.12 any ideas? (see pastie for a better formatted code: http://pastie.org/881328 ) /opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/site_ruby/1.8/rubygems/package/tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) … >>> More

Related posts about zlib

Zlib compression in boost::iostreams not compatible with zlib.NET

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I want to send compressed data between my C# to a C++ application in ZLIB format. In C++, I use the zlib_compressor/zlib_decompressor available in boost::iostreams. In C#, I am currently using the ZOutputStream available in the zlib.NET library. First of all, when I compress the same data… >>> More
How to keep g++ from taking header file from /usr/include?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building using zlib.h which I have a local copy to v1.2.5, but in /usr/include/zlib.h there is v1.2.1.2. If I omit adding -I/my/path/to/zlib to my make I get error from using old version which doesn't have Z_FIXED: g++ -g -Werror -Wredundant-decls -D_FILE_OFFSET_BITS=64 -c -o ARCH.linux_26_i86/debug/sysParam… >>> More
How to uncompress the NSData in Php which is compressed using zlib in Iphone

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello Everyone, I am quite new to Iphone development , so please bear me if I ask some some common questions. In my application I have to transfer data from my Iphone app to a PHP server and for this I have to compress the NSdata in my Iphone app and then pass it on to the PHP server and then Uncompress… >>> More
Decompress PNG using zlib

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi All, How can I use zlib library to decompress a PNG file? I have no idea how to read a PNG file. When I opened a PNG file in TextPad(a text editor), the data was not at all understandable. Is it because a PNG file is compressed? Any help is greatly appreciated. I need to read a PNG file using… >>> More
zlib memory usage / performance. With 500kb of data.

as seen on Stack Overflow - Search for 'Stack Overflow'
Is zLib Worth it? Are there other better suited compressors? I am using an embedded system. Frequently, I have only 3MB of RAM or less available to my application. So I am considering using zlib to compress my buffers. I am concerned about overhead however. The buffer's average size will be 30kb… >>> More