I have a small problem with reading the data from this source correctly. I tried to write:
path = 'http://archive.ics.uci.edu/ml/machine-learning-databases/image/segmentation.data'
df = pd.read_table(path)
And then I got something strange.
Then I wrote:
df = pd.read_table(path, sep=',', header=None)
and got an error: ParserError: Error tokenizing data. C error: Expected 1 fields in line 4, saw 19
Could you, please, help me to find the solution?
The file is basically a csv file so you can use read_csv. Use it in combination with skiprows=2 to skip the first non-relevant rows of the file.
import pandas as pd
path = 'http://archive.ics.uci.edu/ml/machine-learning-databases/image/segmentation.data'
df = pd.read_csv(path, skiprows=2, index_col=False)
Output df.head():
REGION-CENTROID-COL
REGION-CENTROID-ROW
REGION-PIXEL-COUNT
SHORT-LINE-DENSITY-5
SHORT-LINE-DENSITY-2
VEDGE-MEAN
VEDGE-SD
HEDGE-MEAN
HEDGE-SD
INTENSITY-MEAN
RAWRED-MEAN
RAWBLUE-MEAN
RAWGREEN-MEAN
EXRED-MEAN
EXBLUE-MEAN
EXGREEN-MEAN
VALUE-MEAN
SATURATION-MEAN
HUE-MEAN
0
BRICKFACE
140
125
9
0
0
0.277778
0.062963
0.666667
0.311111
6.18518
7.33333
7.66667
3.55556
3.44444
4.44444
-7.88889
7.77778
0.545635
1
BRICKFACE
188
133
9
0
0
0.333333
0.266667
0.5
0.0777777
6.66667
8.33333
7.77778
3.88889
5
3.33333
-8.33333
8.44444
0.53858
2
BRICKFACE
105
139
9
0
0
0.277778
0.107407
0.833333
0.522222
6.11111
7.55556
7.22222
3.55556
4.33333
3.33333
-7.66667
7.55556
0.532628
3
BRICKFACE
34
137
9
0
0
0.5
0.166667
1.11111
0.474074
5.85185
7.77778
6.44444
3.33333
5.77778
1.77778
-7.55556
7.77778
0.573633
4
BRICKFACE
39
111
9
0
0
0.722222
0.374074
0.888889
0.429629
6.03704
7
7.66667
3.44444
2.88889
4.88889
-7.77778
7.88889
0.562919
Can you give encoding like this:
path = 'http://archive.ics.uci.edu/ml/machine-learning-databases/image/segmentation.data'
df = pd.read_csv(path,encoding = 'utf8')
If it does not work, can you try other encodings?
The problem seems to be that the data file contains some meta information that Pandas cannot parse. You need to convert your file to a CSV before it can be read by pandas.
To do this, first download the file to your local machine at some location filepath and remove the lines starting with the ;;; and the empty lines. Then running a pd.read_table(filepath, sep='\t') or a pd.read_csv(filepath) should work as expected.
Note that the header argument does not refer to any generic header information that the file may contain. header lets pandas know whether the first line in your CSV contains the names of the columns (if header is True) or whether the actual data in the file starts from the first line (if header is False).
I have a file (can save as txt, csv, or whatever) that looks like
name num at res x y z
AB 1 TY A 1.0-2.3 3.4
BC 2 ER B 2.1 3.4 -4.5
CD 3 TY A 2.3 32 -5.6
..
I want to be able to just extract all the elements in the 'at' column and then print out a list of the unique elements in that column. The command I used to read in the file was df = pd.read_csv("file.csv").
However, if I try print df['at'].head(), I get an error about hashtables and KeyError: 'at'
How can I fix this?
UPDATE:
you have problems in line 2024 (pandas counts from 0 !):
Expected 12 fields in line 2024, saw 13
so you may want to clean up your data or to ignore bad lines like this:
df = pd.read_csv(..., delim_whitespace=True, error_bad_lines=False, ...)
OLD answer:
most probably, you have read your CSV file incorrectly.
how to check:
print df.columns.tolist()
and pay attention at the leading/trailing spaces in the column names
how to fix that:
df = pd.read_csv(..., sep='\s+', ...)
or
df = pd.read_csv(..., delim_whitespace=True, ...)
I think you need add parameter sep as separator to read_csv
sep="\s+" for arbitrary whitespaces
sep='\t' for tab
sep=r'\s{2,}' two or more whitespaces
Sample:
import pandas as pd
import io
temp=u"""ame num at res x y z
AB 1 TY A 1.0 -2.3 3.4
BC 2 ER B 2.1 3.4 -4.5
CD 3 TY A 2.3 32 -5.6
"""
#after testing replace io.StringIO(temp) to filename
df = pd.read_csv(io.StringIO(temp), sep="\s+")
print df
ame num at res x y z
0 AB 1 TY A 1.0 -2.3 3.4
1 BC 2 ER B 2.1 3.4 -4.5
2 CD 3 TY A 2.3 32.0 -5.6
Let's say I have a text file that looks like this:
Item,Date,Time,Location
1,01/01/2016,13:41,[45.2344:-78.25453]
2,01/03/2016,19:11,[43.3423:-79.23423,41.2342:-81242]
3,01/10/2016,01:27,[51.2344:-86.24432]
What I'd like to be able to do is read that in with pandas.read_csv, but the second row will throw an error. Here is the code I'm currently using:
import pandas as pd
df = pd.read_csv("path/to/file.txt", sep=",", dtype=str)
I've tried to set quotechar to "[", but that obviously just eats up the lines until the next open bracket and adding a closing bracket results in a "string of length 2 found" error. Any insight would be greatly appreciated. Thanks!
Update
There were three primary solutions that were offered: 1) Give a long range of names to the data frame to allow all data to be read in and then post-process the data, 2) Find values in square brackets and put quotes around it, or 3) replace the first n number of commas with semicolons.
Overall, I don't think option 3 is a viable solution in general (albeit just fine for my data) because a) what if I have quoted values in one column that contain commas, and b) what if my column with square brackets is not the last column? That leaves solutions 1 and 2. I think solution 2 is more readable, but solution 1 was more efficient, running in just 1.38 seconds, compared to solution 2, which ran in 3.02 seconds. The tests were run on a text file containing 18 columns and more than 208,000 rows.
We can use simple trick - quote balanced square brackets with double quotes:
import re
import six
import pandas as pd
data = """\
Item,Date,Time,Location,junk
1,01/01/2016,13:41,[45.2344:-78.25453],[aaaa,bbb]
2,01/03/2016,19:11,[43.3423:-79.23423,41.2342:-81242],[0,1,2,3]
3,01/10/2016,01:27,[51.2344:-86.24432],[12,13]
4,01/30/2016,05:55,[51.2344:-86.24432,41.2342:-81242,55.5555:-81242],[45,55,65]"""
print('{0:-^70}'.format('original data'))
print(data)
data = re.sub(r'(\[[^\]]*\])', r'"\1"', data, flags=re.M)
print('{0:-^70}'.format('quoted data'))
print(data)
df = pd.read_csv(six.StringIO(data))
print('{0:-^70}'.format('data frame'))
pd.set_option('display.expand_frame_repr', False)
print(df)
Output:
----------------------------original data-----------------------------
Item,Date,Time,Location,junk
1,01/01/2016,13:41,[45.2344:-78.25453],[aaaa,bbb]
2,01/03/2016,19:11,[43.3423:-79.23423,41.2342:-81242],[0,1,2,3]
3,01/10/2016,01:27,[51.2344:-86.24432],[12,13]
4,01/30/2016,05:55,[51.2344:-86.24432,41.2342:-81242,55.5555:-81242],[45,55,65]
-----------------------------quoted data------------------------------
Item,Date,Time,Location,junk
1,01/01/2016,13:41,"[45.2344:-78.25453]","[aaaa,bbb]"
2,01/03/2016,19:11,"[43.3423:-79.23423,41.2342:-81242]","[0,1,2,3]"
3,01/10/2016,01:27,"[51.2344:-86.24432]","[12,13]"
4,01/30/2016,05:55,"[51.2344:-86.24432,41.2342:-81242,55.5555:-81242]","[45,55,65]"
------------------------------data frame------------------------------
Item Date Time Location junk
0 1 01/01/2016 13:41 [45.2344:-78.25453] [aaaa,bbb]
1 2 01/03/2016 19:11 [43.3423:-79.23423,41.2342:-81242] [0,1,2,3]
2 3 01/10/2016 01:27 [51.2344:-86.24432] [12,13]
3 4 01/30/2016 05:55 [51.2344:-86.24432,41.2342:-81242,55.5555:-81242] [45,55,65]
UPDATE: if you are sure that all square brackets are balances, we don't have to use RegEx's:
import io
import pandas as pd
with open('35948417.csv', 'r') as f:
fo = io.StringIO()
data = f.readlines()
fo.writelines(line.replace('[', '"[').replace(']', ']"') for line in data)
fo.seek(0)
df = pd.read_csv(fo)
print(df)
I can't think of a way to trick the CSV parser into accepting distinct open/close quote characters, but you can get away with a pretty simple preprocessing step:
import pandas as pd
import io
import re
# regular expression to capture contents of balanced brackets
location_regex = re.compile(r'\[([^\[\]]+)\]')
with open('path/to/file.txt', 'r') as fi:
# replaced brackets with quotes, pipe into file-like object
fo = io.StringIO()
fo.writelines(unicode(re.sub(location_regex, r'"\1"', line)) for line in fi)
# rewind file to the beginning
fo.seek(0)
# read transformed CSV into data frame
df = pd.read_csv(fo)
print df
This gives you a result like
Date_Time Item Location
0 2016-01-01 13:41:00 1 [45.2344:-78.25453]
1 2016-01-03 19:11:00 2 [43.3423:-79.23423, 41.2342:-81242]
2 2016-01-10 01:27:00 3 [51.2344:-86.24432]
Edit If memory is not an issue, then you are better off preprocessing the data in bulk rather than line by line, as is done in Max's answer.
# regular expression to capture contents of balanced brackets
location_regex = re.compile(r'\[([^\[\]]+)\]', flags=re.M)
with open('path/to/file.csv', 'r') as fi:
data = unicode(re.sub(location_regex, r'"\1"', fi.read()))
df = pd.read_csv(io.StringIO(data))
If you know ahead of time that the only brackets in the document are those surrounding the location coordinates, and that they are guaranteed to be balanced, then you can simplify it even further (Max suggests a line-by-line version of this, but I think the iteration is unnecessary):
with open('/path/to/file.csv', 'r') as fi:
data = unicode(fi.read().replace('[', '"').replace(']', '"')
df = pd.read_csv(io.StringIO(data))
Below are the timing results I got with a 200k-row by 3-column dataset. Each time is averaged over 10 trials.
data frame post-processing (jezrael's solution): 2.19s
line by line regex: 1.36s
bulk regex: 0.39s
bulk string replace: 0.14s
I think you can replace first 3 occurence of , in each line of file to ; and then use parameter sep=";" in read_csv:
import pandas as pd
import io
with open('file2.csv', 'r') as f:
lines = f.readlines()
fo = io.StringIO()
fo.writelines(u"" + line.replace(',',';', 3) for line in lines)
fo.seek(0)
df = pd.read_csv(fo, sep=';')
print df
Item Date Time Location
0 1 01/01/2016 13:41 [45.2344:-78.25453]
1 2 01/03/2016 19:11 [43.3423:-79.23423,41.2342:-81242]
2 3 01/10/2016 01:27 [51.2344:-86.24432]
Or can try this complicated approach, because main problem is, separator , between values in lists is same as separator of other column values.
So you need post - processing:
import pandas as pd
import io
temp=u"""Item,Date,Time,Location
1,01/01/2016,13:41,[45.2344:-78.25453]
2,01/03/2016,19:11,[43.3423:-79.23423,41.2342:-81242,41.2342:-81242]
3,01/10/2016,01:27,[51.2344:-86.24432]"""
#after testing replace io.StringIO(temp) to filename
#estimated max number of columns
df = pd.read_csv(io.StringIO(temp), names=range(10))
print df
0 1 2 3 4 \
0 Item Date Time Location NaN
1 1 01/01/2016 13:41 [45.2344:-78.25453] NaN
2 2 01/03/2016 19:11 [43.3423:-79.23423 41.2342:-81242
3 3 01/10/2016 01:27 [51.2344:-86.24432] NaN
5 6 7 8 9
0 NaN NaN NaN NaN NaN
1 NaN NaN NaN NaN NaN
2 41.2342:-81242] NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN
#remove column with all NaN
df = df.dropna(how='all', axis=1)
#first row get as columns names
df.columns = df.iloc[0,:]
#remove first row
df = df[1:]
#remove columns name
df.columns.name = None
#get position of column Location
print df.columns.get_loc('Location')
3
#df1 with Location values
df1 = df.iloc[:, df.columns.get_loc('Location'): ]
print df1
Location NaN NaN
1 [45.2344:-78.25453] NaN NaN
2 [43.3423:-79.23423 41.2342:-81242 41.2342:-81242]
3 [51.2344:-86.24432] NaN NaN
#combine values to one column
df['Location'] = df1.apply( lambda x : ', '.join([e for e in x if isinstance(e, basestring)]), axis=1)
#subset of desired columns
print df[['Item','Date','Time','Location']]
Item Date Time Location
1 1 01/01/2016 13:41 [45.2344:-78.25453]
2 2 01/03/2016 19:11 [43.3423:-79.23423, 41.2342:-81242, 41.2342:-8...
3 3 01/10/2016 01:27 [51.2344:-86.24432]
Is there a way to read a file like this in and skip the column index (1-5) like this example? I'm using read_csv.
24.0 1:0.00632 2:18.00 3:2.310 4:0 5:0.5380
21.6 1:0.02731 2:0.00 3:7.070 4:0 5:0.4690
Expected table read:
24.0 0.00632 18.00 2.310 0 0.5380
read_csv won't handle this the way you want because it's not a CSV.
You can do e.g.
pd.DataFrame([[chunk.split(':')[-1] for chunk in line.split()] for line in f])
Your data is oddly structured. Given the colon index separator, you can read the file mostly as text via the usual read_csv. Then, loop through each column in the dataframe (except for the first one), split the string on ':', take the second element which represents your desired value, and convert that value to a float (all done via a list comprehension).
df = pd.read_csv('data.txt', sep=' ', header=None)
>>> df
0 1 2 3 4 5
0 24.0 1:0.00632 2:18.00 3:2.310 4:0 5:0.5380
1 21.6 1:0.02731 2:0.00 3:7.070 4:0 5:0.4690
df.iloc[:, 1:] = df.iloc[:, 1:].applymap(lambda s: float(s.split(':')[1]))
>>> df
0 1 2 3 4 5
0 24.0 0.00632 18 2.31 0 0.538
1 21.6 0.02731 0 7.07 0 0.469
Hoping that this is an allowable SO question but I am hoping to get some advice on how to convert the below code which processes lines in a file to produce a dataframe into one that uses generators and yields because this implementation using list and append is far too slow.
Here is the solution I came up with but I was really hoping to avoid using very slow lists and append operation. I was hoping for cool generator and yield solution instead but not comfortable enough yet working with generators.
Sample lines in file:
"USNC3255","27","US","NC","LANDS END","72305006","KNJM","KNCA","KNKT","T72305006","","","NCC031","NCZ095","","545","28594","America/New_York","34.65266","-77.07661","7","RDU","893727","
"USNC3256","27","US","NC","LANDSDOWN","72314058","KEHO","KAKH","KIPJ","T72314058","","","NCC045","NCZ068","sc007","517","28150","America/New_York","35.29374","-81.46537","797","CLT","317845","
Current Solution:
def parse_file(filename):
newline = []
with open(filename, 'rb') as f:
reader = csv.reader(f, quoting=csv.QUOTE_NONE)
for row in reader:
newline.append([s.strip('"') for s in row[:-1]])
df = pd.DataFrame(newline)
df = df.applymap(lambda x: nan if len(x) == 0 else x).astype(object)
return df
df = parse_file(filename)
Output is just a dataframe with 23 columns and two rows if used against the sample lines above.
The only problem with your file is that each line ends with ,". This confuses the parser. If you can remove the trailing comma and quotation mark, you can use the regular parser.
import pandas as pd
from StringIO import StringIO
with open('example.txt') as myfile:
data = myfile.read().replace(',"\n', '\n')
pd.read_csv(StringIO(data), header=None)
This is what I get:
0 1 2 3 4 5 6 7 8 9 \
0 USNC3255 27 US NC LANDS END 72305006 KNJM KNCA KNKT T72305006
1 USNC3256 27 US NC LANDSDOWN 72314058 KEHO KAKH KIPJ T72314058
... 13 14 15 16 17 18 19 \
0 ... NCZ095 NaN 545 28594 America/New_York 34.65266 -77.07661
1 ... NCZ068 sc007 517 28150 America/New_York 35.29374 -81.46537
20 21 22
0 7 RDU 893727
1 797 CLT 317845
[2 rows x 23 columns]