I have a csv file that has three columns, one called (Age_Groups), one called (Trip_in_min) and the third is called (Start_Station_Name), (actually it comes from a bigger dataset (17 rows and 16845 columns)
Now I need to get the average trip time per age group
Here is the link to the csv file, in dropbox, as I did not know how to paste it properly here
Any help please?
import pandas as pd
file = pd.read_csv(r"file.csv")
# Counting total minutes per age group
trips_summary = (file.Age_Groups.value_counts())
print(("Number of trips per age group"))
print(trips_summary)# per age group
print()
# Finding the most popular 20 stations
popular_stations = (file.Start_Station_Name.value_counts())
print("The most popular 20 stations")
print(popular_stations[:20])
print()
UPDATE
Ok, it worked, I added the line
df.groupby('Age_Groups', as_index=False)['Trip_in_min'].mean()
Thanks #jjj, however as I mentioned, my data has more than 16K row, once I added back the rows, it started to fail and gives me the error below (might be not a real error), with only age groups and not average printed, I can get it only if I have 1890 rows or less, here is the message I am getting for larger number of rows (BTW), other operations work fine with the full DS, just this one):
*D:\Test 1.py:18: FutureWarning: The default value of numeric_only in DataFrameGroupBy.mean is deprecated. In a future version, numeric_only will default to False. Either specify numeric_only or select only columns which should be valid for the function.
avg = df.groupby('Age_Groups', as_index=False)['Trip_in_min'].mean()
Age_Groups*
0 18-24
1 25-34
2 35-44
3 45-54
4 55-64
5 65-74
6 75+
UPDATE 2
Not all columns are numbers, however when I use the code below:
df.apply(pd.to_numeric, errors='ignore').info()
I get the below output(my target is number 12)
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1897 entries, 1 to 1897
Data columns (total 13 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Riverview Park 11 non-null object
1 Riverview Park.1 11 non-null object
2 Riverview Park.2 11 non-null object
3 Start_Station_Name 1897 non-null object
4 3251 98 non-null float64
5 Jersey & 3rd 98 non-null object
6 24443 98 non-null float64
7 Subscriber 98 non-null object
8 1928 98 non-null float64
9 Unnamed: 9 79 non-null float64
10 Age_Groups 1897 non-null object
11 136 98 non-null float64
12 Trip_in_min 1897 non-null object
dtypes: float64(5), object(8)
memory usage: 192.8+ KB
Hope this helps:
import pandas as pd
df= pd.read_csv("test.csv")
df.groupby('Age_Groups', as_index=False)['Trip_in_min'].mean()
Related
I am trying to read in text file from EIA that is zipped. I have been able to get the file downloaded, unzipped, and converted to a string that is I believe JSON formatted but can not seem to convert it into a DataFrame. Help is greatly appreciated.
import pandas as pd
import requests
import io
import zipfile
import json
url_data='https://api.eia.gov/bulk/PET.zip'
r = requests.get(url_data)
with zipfile.ZipFile(io.BytesIO(r.content), mode="r") as archive:
archive.printdir()
text = archive.read("PET.txt") .decode(encoding="utf-8")
To read this file use :
import pandas as pd
df=pd.read_json(path_to_zip,lines=True)
df contains all rows
>>> df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 188297 entries, 0 to 188296
Data columns (total 19 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 series_id 174220 non-null object
1 name 188297 non-null object
2 units 174220 non-null object
3 f 174220 non-null object
4 unitsshort 174220 non-null object
5 description 174220 non-null object
6 copyright 174220 non-null object
7 source 174220 non-null object
8 iso3166 134979 non-null object
9 geography 161968 non-null object
10 start 174220 non-null float64
11 end 174220 non-null float64
12 last_updated 174220 non-null object
13 data 174220 non-null object
14 geography2 105177 non-null object
15 category_id 14077 non-null float64
16 parent_category_id 14077 non-null float64
17 notes 14077 non-null object
18 childseries 14077 non-null object
read_json can already read compressed JSON files. This isn't a JSON file though, it contains one JSON document per line. You can read such files with the lines parameter.
In a JSON document there can be only one root, either an object or array. This means the entire document must be read into memory before it can be parsed. This causes severe problems with large files like this one, or when an application wants to append JSON documents (eg records) to an existing file. The entire file would have to be read and written at once.
To overcome this, it's common to store one unindented JSON document per line. This way, to add a new document all the code has to do is append a new line. To read a subset of the lines, an application only needs to seek to the first newline after an offset and read the next N lines.
read_csv can read a subset of such files when lines = True through the nrows parameter:
>>> df2=pd.read_json(r"C:\Users\pankan\Downloads\PET.zip",lines=True,nrows=100)
>>> df2.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100 entries, 0 to 99
Data columns (total 14 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 series_id 100 non-null object
1 name 100 non-null object
2 units 100 non-null object
3 f 100 non-null object
4 unitsshort 100 non-null object
5 description 100 non-null object
6 copyright 100 non-null object
7 source 100 non-null object
8 iso3166 67 non-null object
9 geography 100 non-null object
10 start 100 non-null int64
11 end 100 non-null int64
12 last_updated 100 non-null object
13 data 100 non-null object
I have this very simple Python Pandas DataFrame calles "sa"
>sa.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7 entries, 0 to 6
Data columns (total 5 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 searchAppearance 7 non-null object
1 clicks 7 non-null int64
2 impressions 7 non-null int64
3 ctr 7 non-null float64
4 position 7 non-null float64
dtypes: float64(2), int64(2), object(1)
memory usage: 408.0+ bytes
with these values
>print(sa)
searchAppearance clicks impressions ctr position
0 AMP_TOP_STORIES 376 376 0.022917 8.108978
1 AMP_BLUE_LINK 55670 55670 0.051522 13.158574
2 PAGE_EXPERIENCE 68446 68446 0.039298 20.056293
3 RECIPE_FEATURE 40175 40175 0.042920 4.186674
4 RECIPE_RICH_SNIPPET 37428 37428 0.069153 18.726152
5 VIDEO 72 72 0.025361 15.896090
6 WEBLITE 1 1 0.001055 51.493671
all is good there.
now I do
sa['ctr-test']=devices['ctr']
this leads to
>print(sa)
searchAppearance clicks impressions ctr position ctr-test
0 AMP_TOP_STORIES 376 376 0.022917 8.108978 0.039522
1 AMP_BLUE_LINK 55670 55670 0.051522 13.158574 0.026543
2 PAGE_EXPERIENCE 68446 68446 0.039298 20.056293 0.051098
3 RECIPE_FEATURE 40175 40175 0.042920 4.186674 NaN
4 RECIPE_RICH_SNIPPET 37428 37428 0.069153 18.726152 NaN
5 VIDEO 72 72 0.025361 15.896090 NaN
6 WEBLITE 1 1 0.001055 51.493671 NaN
do you see all the NaN? but only starting from the 3rd row? it does not make any sense to me.
the dataframe info still looks good
sa.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7 entries, 0 to 6
Data columns (total 6 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 searchAppearance 7 non-null object
1 clicks 7 non-null int64
2 impressions 7 non-null int64
3 ctr 7 non-null float64
4 position 7 non-null float64
5 ctr-test 3 non-null float64
dtypes: float64(3), int64(2), object(1)
memory usage: 464.0+ bytes
i don't get it. what is going wrong? I am using Google Collaborate.
Seems like a bug, but not in my code? Any idea on how to debug this? (If it's not in my code.)
The output of sa.info() includes this line:
5 ctr-test 3 non-null float64
It seems that these three non-null values end up in the first three rows of sa.
sa['ctr-test']=devices['ctr']
stupidity on my side, mix up in dataframes / variables
I'm new to Python and programming, so this is no doubt a newbie question. I want to show the value counts for each unique value of each categorical variable in a data frame, but what I've written isn't working. I'm trying to avoid writing separate lines for each individual column if I can help it.
#
Column
Non-Null Count
Dtype
0
checking_balance
1000 non-null
category
1
months_loan_duration
1000 non-null
int64
2
credit_history
1000 non-null
category
3
purpose
1000 non-null
category
4
amount
1000 non-null
int64
5
savings_balance
1000 non-null
category
6
employment_duration
1000 non-null
category
7
percent_of_income
1000 non-null
int64
8
years_at_residence
1000 non-null
int64
9
age
1000 non-null
int64
10
other_credit
1000 non-null
category
11
housing
1000 non-null
category
12
existing_loans_count
1000 non-null
int64
13
job
1000 non-null
category
14
dependents
1000 non-null
int64
15
phone
1000 non-null
category
16
default
1000 non-null
category
Code I've written:
for col in creditData.columns:
if creditData[col].dtype == 'category':
print(creditData[col].value_counts())
The results:
unknown 394
< 0 DM 274
1 - 200 DM 269
> 200 DM 63
Name: checking_balance, dtype: int64
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-14-6a53236835fc> in <module>
1 for col in creditData.columns: # Loop through all columns in the dataframe
----> 2 if creditData[col].dtype == 'category':
3 print(creditData[col].value_counts())
TypeError: data type 'category' not understood
this works for me
for i in creditData.columns:
if creditData[i].dtype != 'int64':
print(creditData[i].value_counts())
I'm having trouble merging two dataframes in pandas. They are parts of a dataset split between two files, and they share some columns and values, namely 'name' and 'address'. The entries with identical values do not share their index with entries in the other file. I tried variations of the following line:
res = pd.merge(df, df_p, on=['name', 'address'], how="left")
When the how argument was set to 'left', the columns from df_p had no values. 'right' had the opposite effect, with columns from df being empty. 'inner' resulted in an empty dataframe and 'outer' duplicated the number of entries, essentially just appending the results of 'left' and 'right'.
I manually verified that there are identical combinations of 'name' and 'address' values in both files.
Edit: Attempt at merging on a single of those columns appears to be successful, however I want to avoid merging incorrect entries in case 2 people with identical names have different addresses and vice versa
Edit1: Here's some more information on the data-set.
df.info() output:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3983 entries, 0 to 3982
Data columns (total 23 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 3983 non-null int64
1 name 3983 non-null object
2 address 3983 non-null object
3 race 3970 non-null object
4 marital-status 3967 non-null object
5 occupation 3971 non-null object
6 pregnant 3969 non-null object
7 education-num 3965 non-null float64
8 relationship 3968 non-null object
9 skewness_glucose 3972 non-null float64
10 mean_glucose 3572 non-null float64
11 capital-gain 3972 non-null float64
12 kurtosis_glucose 3970 non-null float64
13 education 3968 non-null object
14 fnlwgt 3968 non-null float64
15 class 3969 non-null float64
16 std_glucose 3965 non-null float64
17 income 3974 non-null object
18 medical_info 3968 non-null object
19 native-country 3711 non-null object
20 hours-per-week 3971 non-null float64
21 capital-loss 3969 non-null float64
22 workclass 3968 non-null object
dtypes: float64(10), int64(1), object(12)
memory usage: 715.8+ KB
example entry from df:
0,Curtis Brown,"32266 Byrd Island
Fowlertown, DC 84201", White, Married-civ-spouse, Exec-managerial,f,9.0, Husband,1.904881822,79.484375,15024.0,0.667177618, HS-grad,147707.0,0.0,39.49544760000001, >50K,"{'mean_oxygen':'1.501672241','std_oxygen':'13.33605383','kurtosis_oxygen':'11.36579476','skewness_oxygen':'156.77910559999995'}", United-States,60.0,0.0, Private
df_p.info() output:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3933 entries, 0 to 3932
Data columns (total 6 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Unnamed: 0 3933 non-null int64
1 name 3933 non-null object
2 address 3933 non-null object
3 age 3933 non-null int64
4 sex 3933 non-null object
5 date_of_birth 3933 non-null object
dtypes: int64(2), object(4)
memory usage: 184.5+ KB
sample entry from df_p:
2273,Curtis Brown,"32266 Byrd Island
Fowlertown, DC 84201",44, Male,1975-03-26
As you can see, the chosen samples are for the same person, but their index does not match, which is why I tried using the name and address columns.
Edit2: Changing the order of df and df_p in the merge seems to have solved the issue, though I have no clue why.
I am trying to get the count of unique values in a column using groupby. However I get different results if I just look at groupby results (which are printed for every group) and if I use get_group(). But the results I get is not same for the first method. What is the problem here?
print "Groupby:",bigDF[bigDF.Class == "apple"].groupby('sizeBin').customerId.nunique()
print "Selection:",bigDF[(bigDF.Class == "apple")&(bigDF.sizeBin == 0)].customerId.nunique()
print "Get group:",bigDF[bigDF.Class == "apple"].groupby('sizeBin').get_group(0).customerId.nunique()
Groupby: sizeBin
0 6
1 14
5 26
10 34
20 32
50 3
100 3
200 7
500 0
Name: customerId, dtype: int64
Selection: 34
Get group: 34
I should also note the data types, pd.info() gives me the following, so the sizeBin is a category:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 224903 entries, 0 to 20616
Data columns (total 3 columns):
customerId 224903 non-null int64
Class 224903 non-null object
sizeBin 224903 non-null category
dtypes: category(1), int64(1), object(1)
memory usage: 5.4+ MB