Reading CSV files in numpy where delimiter is ","

Posted by monch1962 on Stack Overflow See other posts from Stack Overflow or by monch1962
Published on 2010-04-19T01:40:51Z Indexed on 2010/04/19 1:43 UTC
Read the original article Hit count: 387

Filed under:
|
|
|

Hello all,

I've got a CSV file with a format that looks like this:

"FieldName1", "FieldName2", "FieldName3", "FieldName4"
"04/13/2010 14:45:07.008", "7.59484916392", "10", "6.552373"
"04/13/2010 14:45:22.010", "6.55478493312", "9", "3.5378543"
...

Note that there are double quote characters at the start and end of each line in the CSV file, and the "," string is used to delimit fields within each line.

When I try to read this into numpy via:
import numpy as np
data = np.genfromtxt(csvfile, dtype=None, delimiter=',', names=True)
all the data gets read in as string values, surrounded by double-quote characters. Not unreasonable, but not much use to me as I then have to go back and convert every column to its correct type

When I use delimiter='","' instead, everything works as I'd like, except for the 1st and last fields. As the start of line and end of line characters are a single double-quote character, this isn't seen as a valid delimiter for the 1st and last fields, so they get read in as e.g. "04/13/2010 14:45:07.008 and 6.552373" - note the leading and trailing double-quote characters respectively. Because of these redundant characters, numpy assumes the 1st and last fields are both String types; I don't want that to be the case

Is there a way of instructing numpy to read in files formatted in this fashion as I'd like, without having to go back and "fix" the structure of the numpy array after the initial read?

© Stack Overflow or respective owner

Related posts about python

Related posts about numpy