delimit data on columns to new columns python - python

I want to delimit some data from a txt file into a dataframe, but when I open this file via pandas module, the data just has 1 column. I want to delimit this data into 17 columns. The data from txt file look like:
In python, I have the following code using pandas:
import pandas as pd
count=1
nama = 'Data/'+'%d.txt'%(count)
df = pd.read_table(nama,sep = '\t',header=None)
df_head1=df
df_sta=df
data_sta=df_sta.drop([0,1,2,3,4,5])
print(data_sta)
I need to split into columns like sta, date, time, Latitude, Longitude, and sta time. If i delimit in excel,the data i want look like :
The data i want
ps: i have used delim_whitespace=True, but that's not running and the message is :
pandas.errors.ParserError: Error tokenizing data. C error: Expected 3 fields in line 6, saw 4


if delimit by tab '\t' doesn't work use :
df = pd.read_table(filename,delim_whitespace=True,skiprows = 'number_of_rows_before_the_headers_start')
this is similar to using delimiter=r"\s+" but i feel the above method is faster than the regex.

Related

In Pandas, how can I extract certain value using the key off of a dataframe imported from a csv file?

Using Pandas, I'm trying to extract value using the key but I keep failing to do so. Could you help me with this?
There's a csv file like below:
value
"{""id"":""1234"",""currency"":""USD""}"
"{""id"":""5678"",""currency"":""EUR""}"
I imported this file in Pandas and made a DataFrame out of it:
dataframe from a csv file
However, when I tried to extract the value using a key (e.g. df["id"]), I'm facing an error message.
I'd like to see a value 1234 or 5678 using df["id"]. Which step should I take to get it done? This may be a very basic question but I need your help. Thanks.

The csv file isn't being read in correctly.
You haven't set a delimiter; pandas can automatically detect a delimiter but hasn't done so in your case. See the read_csv documentation for more on this. Because the , the pandas dataframe has a single column, value, which has entire lines from your file as individual cells - the first entry is "{""id"":""1234"",""currency"":""USD""}". So, the file doesn't have a column id, and you can't select data by id.
The data aren't formatted as a pandas df, with row titles and columns of data. One option is to read in this data is to manually process each row, though there may be slicker options.
file = 'test.dat'
f = open(file,'r')
id_vals = []
currency = []
for line in f.readlines()[1:]:
## remove obfuscating characters
for c in '"{}\n':
line = line.replace(c,'')
line = line.split(',')
## extract values to two lists
id_vals.append(line[0][3:])
currency.append(line[1][9:])

You just need to clean up the CSV file a little and you are good. Here is every step:
# open your csv and read as a text string
with open('My_CSV.csv', 'r') as f:
my_csv_text = f.read()
# remove problematic strings
find_str = ['{', '}', '"', 'id:', 'currency:','value']
replace_str = ''
for i in find_str:
my_csv_text = re.sub(i, replace_str, my_csv_text)
# Create new csv file and save cleaned text
new_csv_path = './my_new_csv.csv' # or whatever path and name you want
with open(new_csv_path, 'w') as f:
f.write(my_csv_text)
# Create pandas dataframe
df = pd.read_csv('my_new_csv.csv', sep=',', names=['ID', 'Currency'])
print(df)
Output df:
ID Currency
0 1234 USD
1 5678 EUR

You need to extract each row of your dataframe using json.loads() or eval()
something like this:
import json
for row in df.iteritems():
print(json.loads(row.value)["id"])
# OR
print(eval(row.value)["id"])

how to read data (using pandas?) so that it is correctly formatted?

I have a txt file with following format:
{"results":[{"statement_id":0,"series":[{"name":"datalogger","columns":["time","ActivePower0","CosPhi0","CurrentRms0","DcAnalog5","FiringAngle0","IrTemperature0","Lage 1_Angle1","Lage 1_Angle2","PotentioMeter0","Rotation0","SNR","TNR","Temperature0","Temperature3","Temperature_MAX31855_0","Temperature_MAX31855_3","Vibra0_X","Vibra0_Y","Vibra0_Z","VoltageAccu0","VoltageRms0"],"values":[["2017-10-06T08:50:25.347Z",null,null,null,null,null,null,null,null,null,null,"41762721","Testcustomer",null,null,null,null,-196,196,-196,null,null],["2017-10-06T08:50:25.348Z",null,null,null,null,null,null,346.2964,76.11179,null,null,"41762721","Testcustomer",null,null,null,null,null,null,null,null,null],["2017-10-06T08:50:25.349Z",null,null,2596,null,null,null,null,null,null,null,"41762721","Testkunde",null,null,null,null,null,null,null,null,80700],["2017-10-06T08:50:25.35Z",null,null,null,null,null,null,null,null,null,1956,"41762721","Testkunde",null,null,null,null,null,null,null,null,null],["2017-10-06T09:20:05.742Z",null,null,null,null,null,67.98999,null,null,null,null,"41762721","Testkunde",null,null,null,null,null,null,null,null,null]]}]}]}
...
So in the text file everything is saved in one line. CSV file is not available.
I would like to have it as a data frame in pandas. when I use read.csv:
df = pd.read_csv('time-series-data.txt', sep = ",")
the output of print(df) is someting like [0 rows x 3455.. columns]
So currently everything is read in as one line. However, I would like to have 22 columns (time, activepower0, CosPhi0,..). I ask for tips, thank you very much.
Is a pandas dataframe even suitable for this? the text files are up to 2 GB in size.

Here's an example which can read the file you posted.
Here's the test file, named test.json:
{"results":[{"statement_id":0,"series":[{"name":"datalogger","columns":["time","ActivePower0","CosPhi0","CurrentRms0","DcAnalog5","FiringAngle0","IrTemperature0","Lage 1_Angle1","Lage 1_Angle2","PotentioMeter0","Rotation0","SNR","TNR","Temperature0","Temperature3","Temperature_MAX31855_0","Temperature_MAX31855_3","Vibra0_X","Vibra0_Y","Vibra0_Z","VoltageAccu0","VoltageRms0"],
"values":[
["2017-10-06T08:50:25.347Z",null,null,null,null,null,null,null,null,null,null,"41762721","Test-customer",null,null,null,null,-196,196,-196,null,null],
["2017-10-06T08:50:25.348Z",null,null,null,null,null,null,346.2964,76.11179,null,null,"41762721","Test-customer",null,null,null,null,null,null,null,null,null]]}]}]}
Here's the python code used to read it in:
import json
import pandas as pd
# Read test file.
# This reads the entire file into memory at once. If this is not
# possible for you, you may want to look into something like ijson:
# https://pypi.org/project/ijson/
with open("test.json", "rb") as f
data = json.load(f)
# Get the first element of results list, and first element of series list
# You may need a loop here, if your real data has more than one of these.
subset = data['results'][0]['series'][0]
values = subset['values']
columns = subset['columns']
df = pd.DataFrame(values, columns=columns)
print(df)

Reading a csv file into a Pandads dataframe with more than one separator for the values

I have a a csv-file that has as separator the comma sign and at the same time values are separated by ". The first line is text, the second line is empty and the third line consists of column headings. If I try to import the file into a dataframe using pandas with using the code
IE00B0M62Q58 = pd.read_csv('ETF/sample.csv', sep=',')
I get an error like
ParserError: Error tokenizing data. C error: Expected 2 fields in line 3, saw 13
How can I read the file into a dataframe in Pandas?
I copied and pasted the sample.csv file which looks as follows:
Fondsposition per,"03.Jun.2021"
Emittententicker,Name,Anlageklasse,Gewichtung (%),Kurs,Nominale,Marktwert,Nominalwert,Sektor,ISIN,Börse,Standort,Marktwährung
"AAPL","APPLE INC","Aktien","3,63","123,54","1.722.459","212.792.585","212.792.584,86","IT","US0378331005","NASDAQ","Vereinigte Staaten","USD"
"MSFT","MICROSOFT CORP","Aktien","3,08","245,71","735.512","180.722.654","180.722.653,52","IT","US5949181045","NASDAQ","Vereinigte Staaten","USD"
"AMZN","AMAZON COM INC","Aktien","2,38","3.187,01","43.863","139.791.820","139.791.819,63","Zyklische Konsumgüter ","US0231351067","NASDAQ","Vereinigte Staaten","USD"
"FB","FACEBOOK CLASS A INC","Aktien","1,37","326,04","245.671","80.098.573","80.098.572,84","Kommunikation","US30303M1027","NASDAQ","Vereinigte Staaten","USD"
"GOOG","ALPHABET INC CLASS C","Aktien","1,24","2.404,61","30.223","72.674.528","72.674.528,03","Kommunikation","US02079K1079","NASDAQ","Vereinigte Staaten","USD"

Try to use decimal parameter in your call
IE00B0M62Q58 = pd.read_csv('ETF/sample.csv', sep=',', decimal=',')
Also if . it is a digit separator and 2.404,61 is 2404.61, then you can use the thousands parameter:
IE00B0M62Q58 = pd.read_csv('ETF/sample.csv', sep=',', decimal=',' thousands='.')
Add skiprows if you want to skip reading specific rows at the beginning
IE00B0M62Q58 = pd.read_csv('ETF/sample.csv', sep=',',skiprows=2, thousands='.', decimal=',')

How to split a column into multiple columns? [duplicate]

I a importing a .csv file in python with pandas.
Here is the file format from the .csv :
a1;b1;c1;d1;e1;...
a2;b2;c2;d2;e2;...
.....
here is how get it :
from pandas import *
csv_path = "C:...."
data = read_csv(csv_path)
Now when I print the file I get that :
0 a1;b1;c1;d1;e1;...
1 a2;b2;c2;d2;e2;...
And so on... So I need help to read the file and split the values in columns, with the semi color character ;.

read_csv takes a sep param, in your case just pass sep=';' like so:
data = read_csv(csv_path, sep=';')
The reason it failed in your case is that the default value is ',' so it scrunched up all the columns as a single column entry.

In response to Morris' question above:
"Is there a way to programatically tell if a CSV is separated by , or ; ?"
This will tell you:
import pandas as pd
df_comma = pd.read_csv(your_csv_file_path, nrows=1,sep=",")
df_semi = pd.read_csv(your_csv_file_path, nrows=1, sep=";")
if df_comma.shape[1]>df_semi.shape[1]:
print("comma delimited")
else:
print("semicolon delimited")

How to read a file with a semi colon separator in pandas

I a importing a .csv file in python with pandas.
Here is the file format from the .csv :
a1;b1;c1;d1;e1;...
a2;b2;c2;d2;e2;...
.....
here is how get it :
from pandas import *
csv_path = "C:...."
data = read_csv(csv_path)
Now when I print the file I get that :
0 a1;b1;c1;d1;e1;...
1 a2;b2;c2;d2;e2;...
And so on... So I need help to read the file and split the values in columns, with the semi color character ;.

read_csv takes a sep param, in your case just pass sep=';' like so:
data = read_csv(csv_path, sep=';')
The reason it failed in your case is that the default value is ',' so it scrunched up all the columns as a single column entry.

In response to Morris' question above:
"Is there a way to programatically tell if a CSV is separated by , or ; ?"
This will tell you:
import pandas as pd
df_comma = pd.read_csv(your_csv_file_path, nrows=1,sep=",")
df_semi = pd.read_csv(your_csv_file_path, nrows=1, sep=";")
if df_comma.shape[1]>df_semi.shape[1]:
print("comma delimited")
else:
print("semicolon delimited")

Categories