Best way to save complex Python data structures across program sessions (pickle, json, xml, database

Posted by Malcolm on Stack Overflow See other posts from Stack Overflow or by Malcolm
Published on 2010-01-05T01:57:23Z Indexed on 2010/05/01 18:47 UTC
Read the original article Hit count: 301

Filed under:

python

|

pickle

|

JSON

|

sqlite

Looking for advice on the best technique for saving complex Python data structures across program sessions.

Here's a list of techniques I've come up with so far:

pickle/cpickle
json
jsonpickle
xml
database (like SQLite)

Pickle is the easiest and fastest technique, but my understanding is that there is no guarantee that pickle output will work across various versions of Python 2.x/3.x or across 32 and 64 bit implementations of Python.

Json only works for simple data structures. Jsonpickle seems to correct this AND seems to be written to work across different versions of Python.

Serializing to XML or to a database is possible, but represents extra effort since we would have to do the serialization ourselves manually.

Thank you, Malcolm

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about pickle

pickle on jruby

as seen on Stack Overflow - Search for 'Stack Overflow'
Does anyone know if pickle is compatible with jruby? I just installed cucumber and cucumber-rails. I then tried gem install pickle and it installs, but a script/generate pickle yields Couldn't find 'pickle' generator I did everything according to the readme no dice. Anyone have any experience… >>> More
DjangoUnicodeDecodeError while storing pickle'd data.

as seen on Stack Overflow - Search for 'Stack Overflow'
I've got a simple dict object I'm trying to store in the database after it has been run through pickle. It seems that Django doesn't like trying to encode this error. I've checked with MySQL, and the query isn't even getting there before it is throwing the error, so I don't believe that is the problem… >>> More
How to customize pickle for django model objects

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to pickle a complex object that refers to django model objects. The standard pickling process stores a denormalized object in the pickle. So if the object changes on the database between pickling and unpickling, the model is now out of date. (I know this is true with in-memory objects too… >>> More
Python - How to pickle yourself?

as seen on Stack Overflow - Search for 'Stack Overflow'
I want my class to implement Save and Load functions which simply do a pickle of the class. But apparently you cannot use 'self' in the fashion below. How can you do this? self = cPickle.load(f) cPickle.dump(self,f,2) >>> More
Best way to save complex Python data structures across program sessions (pickle, json, xml, database

as seen on Stack Overflow - Search for 'Stack Overflow'
Looking for advice on the best technique for saving complex Python data structures across program sessions. Here's a list of techniques I've come up with so far: pickle/cpickle json jsonpickle xml database (like SQLite) Pickle is the easiest and fastest technique, but my understanding is that… >>> More