Memory efficient import many data files into panda DataFrame in Python

Posted by richardh on Stack Overflow See other posts from Stack Overflow or by richardh
Published on 2012-09-12T15:36:01Z Indexed on 2012/09/12 15:38 UTC
Read the original article Hit count: 418

Filed under:

python

|

pandas

I import into a panda DataFrame a directory of |-delimited.dat files. The following code works, but I eventually run out of RAM with a MemoryError:.

import pandas as pd
import glob

temp = []
dataDir = 'C:/users/richard/research/data/edgar/masterfiles'
for dataFile in glob.glob(dataDir + '/master_*.dat'):
    print dataFile
    temp.append(pd.read_table(dataFile, delimiter='|', header=0))

masterAll = pd.concat(temp)

Is there a more memory efficient approach? Or should I go whole hog to a database? (I will move to a database eventually, but I am baby stepping my move to pandas.) Thanks!

FWIW, here is the head of an example .dat file:

cik|cname|ftype|date|fileloc
1000032|BINCH JAMES G|4|2011-03-08|edgar/data/1000032/0001181431-11-016512.txt
1000045|NICHOLAS FINANCIAL INC|10-Q|2011-02-11|edgar/data/1000045/0001193125-11-031933.txt
1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-11|edgar/data/1000045/0001193125-11-005531.txt
1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-27|edgar/data/1000045/0001193125-11-015631.txt
1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2011-02-14|edgar/data/1000045/0000929638-11-00151.txt

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about pandas

pandas: complex filter on rows of DataFrame

as seen on Stack Overflow - Search for 'Stack Overflow'
I would like to filter rows by a function of each row, e.g. def f(row): return sin(row['velocity'])/np.prod(['masses']) > 5 df = pandas.DataFrame(...) filtered = df[apply_to_all_rows(df, f)] Or for another more complex, contrived example, def g(row): if row['col1'].method1() == 1: … >>> More
How to sort a boxplot by the median values in pandas

as seen on Stack Overflow - Search for 'Stack Overflow'
I've got a dataframe outcome2 that I generate a grouped boxplot with in the following manner: In [11]: outcome2.boxplot(column='Hospital 30-Day Death (Mortality) Rates from Heart Attack',by='State') plt.ylabel('30 Day Death Rate') plt.title('30 Day Death Rate by State') Out [11]: What… >>> More
Creating a dataframe in pandas by multiplying two series together

as seen on Stack Overflow - Search for 'Stack Overflow'
Say I have two series in pandas, series A and series B. How do I create a dataframe in which all of those values are multiplied together, i.e. with series A down the left hand side and series B along the top. Basically the same concept as this, where series A would be the yellow on the left and series… >>> More
How split a column in two colunms in pandas

as seen on Stack Overflow - Search for 'Stack Overflow'
I have el next dataframe data=read_csv('enero.csv') data Fecha DirViento MagViento 0 2011/07/01 00:00 318 6.6 1 2011/07/01 00:15 342 5.5 2 2011/07/01 00:30 329 6.6 3 2011/07/01 00:45 279 … >>> More
How to find subgroups statistics in pandas?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am grouping a DataFrame using multiple columns (e.g., columns A, B - my_df.groupby(['A','B']) ), is there a better (less lines of code, faster) way of finding how many rows are in each subgroup and how many subgroups are there in total? at the moment I am using: def get_grp_size(grp): grp['size']… >>> More