Python: slicing a very large binary file

Posted by Duncan Tait on Stack Overflow See other posts from Stack Overflow or by Duncan Tait
Published on 2010-03-02T13:47:17Z Indexed on 2010/03/19 15:51 UTC
Read the original article Hit count: 433

Filed under:

python

|

binary

|

large-files

Say I have a binary file of 12GB and I want to slice 8GB out of the middle of it. I know the position indices I want to cut between.

How do I do this? Obviously 12GB won't fit into memory, that's fine, but 8GB won't either... Which I thought was fine, but it appears binary doesn't seem to like it if you do it in chunks! I was appending 10MB at a time to a new binary file and there are discontinuities on the edges of each 10MB chunk in the new file.

Is there a Pythonic way of doing this easily?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about binary

Binary to strings as binary? C #/.NET

as seen on Stack Overflow - Search for 'Stack Overflow'
Redis keys are binary safe. I'd like to mess around and put binary into redis using C#. My client of choice doesn't support writing binary keys it uses keys and it make sense. However i am just fooling around so tell me how i can do this. How do i convert a raw byte[] into a string? At first i was… >>> More
Finding if a Binary Tree is a Binary Search Tree

as seen on Stack Overflow - Search for 'Stack Overflow'
Today I had an interview where I was asked to write a program which takes a Binary Tree and returns true if it is also a Binary Search Tree otherwise false. My Approach1: Perform an inroder traversal and store the elements in O(n) time. Now scan through the array/list of elements and check if element… >>> More
EXC_BAD_ACCESS and KERN_INVALID_ADDRESS after intiating a print sequence.

as seen on Stack Overflow - Search for 'Stack Overflow'
MAC G4/1.5GHz/2GB/1TB+ OS10.4.11 Start up Volume has been erased/complete reinstall with updated software. Current problem only occurs when printing to an Epson Artisan 800 [USB as well as Ethernet connected] when using Macromedia FreeHand 10.0.1.67. All other apps/printers work fine. Memory has been… >>> More
How to convert binary read/write to non-binary read/write in C++

as seen on Stack Overflow - Search for 'Stack Overflow'
I have some C++ code from somewhere that reads and writes data in binary format. I want to see what it's reading and writing in the file, so I want to convert it's binary read and write to non-binary read and write. Also, when I convert the binary write to non-binary write, I want it to still be… >>> More
division by 2 in Binary Signed digit (Redundant Binary representation)

as seen on Stack Overflow - Search for 'Stack Overflow'
How can I do division by 2 in Binary Signed digit (Redundant Binary representation) ? Shifting won't work right ? >>> More