Python - CSV: Large file with rows of different lengths

Posted by dassouki on Stack Overflow See other posts from Stack Overflow or by dassouki
Published on 2010-03-31T00:56:59Z Indexed on 2010/03/31 1:03 UTC
Read the original article Hit count: 564

Filed under:

python

|

python-csv

|

csv

In short, I have a 20,000,000 line csv file that has different row lengths. This is due to archaic data loggers and proprietary formats. We get the end result as a csv file in the following format. MY goal is to insert this file into a postgres database. How Can I do the following:

Keep the first 8 columns and my last 2 columns, to have a consistent CSV file
Add a Column to the file.

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0, img_id.jpg, -50

1, 2, 3, 4, 5, 0,0,0,0,0,0,0,0,0,0,0 img_id.jpg, -50

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about python-csv

Python - CSV: Large file with rows of different lengths

as seen on Stack Overflow - Search for 'Stack Overflow'
In short, I have a 20,000,000 line csv file that has different row lengths. This is due to archaic data loggers and proprietary formats. We get the end result as a csv file in the following format. MY goal is to insert this file into a postgres database. How Can I do the following: Keep the first… >>> More
Clarification needed about Python CSV file format parsing

as seen on Stack Overflow - Search for 'Stack Overflow'
Format is like: CHINA;2002-06-25 00:00:00.000;5,60 CHINA;2002-06-26 00:00:00.000;5,32 CHINA;2002-06-27 00:00:00.000;5,31 and I try to use Python's CSV tools to parse it but cannot understand the paragraph, source: And while the module doesn’t directly support parsing strings, it can easily be… >>> More
Python csv reader acting weird

as seen on Stack Overflow - Search for 'Stack Overflow'
So OK if I run this wrong code: csvReader1 = csv.reader(file('new_categories.csv', "rU"), delimiter=',') for row1 in csvReader1: print row1[0] print row1[8] category_sku = str(row[8]) if category_sku == sku: classifications["Craft"] = row[0] classifications["Theme"]… >>> More
Python CSV file processing

as seen on Stack Overflow - Search for 'Stack Overflow'
I just got introduced to python, the first language I get to learn, and I have this question below: I have an excel based CSV file with two columns (or rows, Pythonically) that I am working on. What I need to do is to perform some operations so that I can compare the two data entries in each 'row'… >>> More
Upload and parse csv file with "universal newline" in python on Google App Engine

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I'm uploading a csv/tsv file from a form in GAE, and I try to parse the file with python csv module. Like describe here, uploaded files in GAE are strings. So I treat my uploaded string a file-like object : file = self.request.get('catalog') catalog = csv.reader(StringIO.StringIO(file),dialect=csv… >>> More