I try and concat or append (neither are working) 2 9-column dataframes together. But, instead of just doing a normal vertical stacking of them, pandas keeps trying to add 9 more empty columns as well. Do you know how to stop this?
output looks like this:
0,1,2,3,4,5,6,7,8,9,10,11,12,13,0,1,10,11,12,13,2,3,4,5,6,7,8,9
10/23/2020,New Castle,DE,Gary,IN,Full,Flatbed,0.00,46,48,0,Dispatch,(800) 488-1860,Meadow Lark Agency ,,,,,,,,,,,,,,
10/22/2020,Wilmington,DE,METHUEN,MA,Full,Flatbed / Step Deck,0.00,48,48,0,Ken,(903) 280-7878,UrTruckBroker ,,,,,,,,,,,,,,
10/23/2020,WILMINGTON,DE,METHUEN,MA,Full,Flatbed w/Tarps,0.00,47,1,0,Dispatch,(912) 748-3801,DSV Road Inc. ,,,,,,,,,,,,,,
10/23/2020,WILMINGTON,DE,METHUEN,MA,Full,Flatbed w/Tarps,0.00,48,1,0,Dispatch,(541) 826-4786,Sureway Transportation Co / Anderson Trucking Serv ,,,,,,,,,,,,,,
10/30/2020,New Castle,DE,Gary,IN,Full,Flatbed,945.00,46,48,0,Dispatch,(800) 488-1860,Meadow Lark Agency ,,,,,,,,,,,,,,
...
,,,,,,,,,,,,,,03/02/2021,Knapp,0.0,Dispatch,(763) 432-3680,Fuze Logistics Services USA ,WI,Jackson,NE,Full,Flatbed / Step Deck,0.0,48.0,48.0
,,,,,,,,,,,,,,03/02/2021,Knapp,0.0,Dispatch,(763) 432-3680,Fuze Logistics Services USA ,WI,Sterling,IL,Full,Flatbed / Step Deck,0.0,48.0,48.0
,,,,,,,,,,,,,,03/02/2021,Milwaukee,0.0,Dispatch,(763) 432-3680,Fuze Logistics Services USA ,WI,Great Falls,MT,Full,Flatbed / Step Deck,0.0,45.0,48.0
,,,,,,,,,,,,,,03/02/2021,Algoma,0.0,Dispatch,(763) 432-3680,Fuze Logistics Services USA ,WI,Pamplico,SC,Full,Flatbed / Step Deck,0.0,48.0,48.0
code is a web request to get data, which I save to dataframe, which is then concat-ed with another dataframe that comes from a CSV. I then save all of this back to that csv:
this_csv = 'freights_trulos.csv'
try:
old_df = pd.read_csv(this_csv)
except BaseException as e:
print(e)
old_df = pd.DataFrame()
state, equip = 'DE', 'Flat'
url = "https://backend-a.trulos.com/load-table/grab_loads.php?state=%s&equipment=%s" % (state, equip)
payload = {}
headers = {
...
}
response = requests.request("GET", url, headers=headers, data=payload)
# print(response.text)
parsed = json.loads(response.content)
data = [r[0:13] + [r[-4].split('<br/>')[-2].split('>')[-1]] for r in parsed]
df = pd.DataFrame(data=data)
if not old_df.empty:
# concatenate old and new and remove duplicates
# df.reset_index(drop=True, inplace=True)
# old_df.reset_index(drop=True, inplace=True)
# df = pd.concat([old_df, df], ignore_index=True) <--- CONCAT HAS SAME ISSUES AS APPEND
df = df.append(old_df, ignore_index=True)
# remove duplicates on cols
df.drop_duplicates()
df.to_csv(this_csv, index=False)
EDIT appended df's have had their types changed
df.dtypes
Out[2]:
0 object
1 object
2 object
3 object
4 object
5 object
6 object
7 object
8 object
9 object
10 object
11 object
12 object
13 object
dtype: object
old_df.dtypes
Out[3]:
0 object
1 object
2 object
3 object
4 object
5 object
6 object
7 float64
8 int64
9 int64
10 int64
11 object
12 object
13 object
dtype: object
old_df to csv
0,1,2,3,4,5,6,7,8,9,10,11,12,13
10/23/2020,New Castle,DE,Gary,IN,Full,Flatbed,0.0,46,48,0,Dispatch,(800) 488-1860,Meadow Lark Agency
10/22/2020,Wilmington,DE,METHUEN,MA,Full,Flatbed / Step Deck,0.0,48,48,0,Ken,(903) 280-7878,UrTruckBroker
10/23/2020,WILMINGTON,DE,METHUEN,MA,Full,Flatbed w/Tarps,0.0,47,1,0,Dispatch,(912) 748-3801,DSV Road Inc.
10/23/2020,WILMINGTON,DE,METHUEN,MA,Full,Flatbed w/Tarps,0.0,48,1,0,Dispatch,(541) 826-4786,Sureway Transportation Co / Anderson Trucking Serv
10/30/2020,New Castle,DE,Gary,IN,Full,Flatbed,945.0,46,48,0,Dispatch,(800) 488-1860,Meadow Lark Agency
new_df to csv
0,1,2,3,4,5,6,7,8,9,10,11,12,13
10/23/2020,New Castle,DE,Gary,IN,Full,Flatbed,0.00,46,48,0,Dispatch,(800) 488-1860,Meadow Lark Agency
10/22/2020,Wilmington,DE,METHUEN,MA,Full,Flatbed / Step Deck,0.00,48,48,0,Ken,(903) 280-7878,UrTruckBroker
10/23/2020,WILMINGTON,DE,METHUEN,MA,Full,Flatbed w/Tarps,0.00,47,1,0,Dispatch,(912) 748-3801,DSV Road Inc.
10/23/2020,WILMINGTON,DE,METHUEN,MA,Full,Flatbed w/Tarps,0.00,48,1,0,Dispatch,(541) 826-4786,Sureway Transportation Co / Anderson Trucking Serv
10/30/2020,New Castle,DE,Gary,IN,Full,Flatbed,945.00,46,48,0,Dispatch,(800) 488-1860,Meadow Lark Agency
I guess the problem could be how you read the data if I copy your sample data to excel and split by comma and then import to pandas, all is fine. Also if I split on comma AND whitespaces, I have +9 additional columns. So you could try debugging by replacing all whitespaces before creating your dataframe.
I also used your sample data and it workend just fine for me if I initialize it like this:
import pandas as pd
df_new = pd.DataFrame({'0': {0: '10/23/2020',
1: '10/22/2020',
2: '10/23/2020',
3: '10/23/2020',
4: '10/30/2020'},
'1': {0: 'New_Castle',
1: 'Wilmington',
2: 'WILMINGTON',
3: 'WILMINGTON',
4: 'New_Castle'},
'2': {0: 'DE', 1: 'DE', 2: 'DE', 3: 'DE', 4: 'DE'},
'3': {0: 'Gary', 1: 'METHUEN', 2: 'METHUEN', 3: 'METHUEN', 4: 'Gary'},
'4': {0: 'IN', 1: 'MA', 2: 'MA', 3: 'MA', 4: 'IN'},
'5': {0: 'Full', 1: 'Full', 2: 'Full', 3: 'Full', 4: 'Full'},
'6': {0: 'Flatbed',
1: 'Flatbed_/_Step_Deck',
2: 'Flatbed_w/Tarps',
3: 'Flatbed_w/Tarps',
4: 'Flatbed'},
'7': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 945.0},
'8': {0: 46, 1: 48, 2: 47, 3: 48, 4: 46},
'9': {0: 48, 1: 48, 2: 1, 3: 1, 4: 48},
'10': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0},
'11': {0: 'Dispatch', 1: 'Ken', 2: 'Dispatch', 3: 'Dispatch', 4: 'Dispatch'},
'12': {0: '(800)_488-1860',
1: '(903)_280-7878',
2: '(912)_748-3801',
3: '(541)_826-4786',
4: '(800)_488-1860'},
'13': {0: 'Meadow_Lark_Agency_',
1: 'UrTruckBroker_',
2: 'DSV_Road_Inc._',
3: 'Sureway_Transportation_Co_/_Anderson_Trucking_Serv_',
4: 'Meadow_Lark_Agency_'}})
df_new = pd.DataFrame({'0': {0: '10/23/2020',
1: '10/22/2020',
2: '10/23/2020',
3: '10/23/2020',
4: '10/30/2020'},
'1': {0: 'New_Castle',
1: 'Wilmington',
2: 'WILMINGTON',
3: 'WILMINGTON',
4: 'New_Castle'},
'2': {0: 'DE', 1: 'DE', 2: 'DE', 3: 'DE', 4: 'DE'},
'3': {0: 'Gary', 1: 'METHUEN', 2: 'METHUEN', 3: 'METHUEN', 4: 'Gary'},
'4': {0: 'IN', 1: 'MA', 2: 'MA', 3: 'MA', 4: 'IN'},
'5': {0: 'Full', 1: 'Full', 2: 'Full', 3: 'Full', 4: 'Full'},
'6': {0: 'Flatbed',
1: 'Flatbed_/_Step_Deck',
2: 'Flatbed_w/Tarps',
3: 'Flatbed_w/Tarps',
4: 'Flatbed'},
'7': {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 945.0},
'8': {0: 46, 1: 48, 2: 47, 3: 48, 4: 46},
'9': {0: 48, 1: 48, 2: 1, 3: 1, 4: 48},
'10': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0},
'11': {0: 'Dispatch', 1: 'Ken', 2: 'Dispatch', 3: 'Dispatch', 4: 'Dispatch'},
'12': {0: '(800)_488-1860',
1: '(903)_280-7878',
2: '(912)_748-3801',
3: '(541)_826-4786',
4: '(800)_488-1860'},
'13': {0: 'Meadow_Lark_Agency_',
1: 'UrTruckBroker_',
2: 'DSV_Road_Inc._',
3: 'Sureway_Transportation_Co_/_Anderson_Trucking_Serv_',
4: 'Meadow_Lark_Agency_'}})
df_new.append(df_old, ignore_index=True)
#OR
pd.concat([df_new, df_old])
All,
My dataframe looks like following. I am trying to convert my Monthly data into Yearly data. I am trying to aggregate my dataframe such that I can add the monthly data-points for the year 1997 and display the sum column. I would like to perform this activity for the years 1997-2018. I have also included dput of my dataset for reference.
Note: Below snapshot only shows few monthly data for the year 1997 and 1998,However,I have entire monthly data for the years 1997 till 2018.
Dput of the dataframe:
{'RegionID': {0: 84654, 1: 91982, 2: 84616, 3: 93144, 4: 91940}, 'RegionName': {0: 60657, 1: 77494, 2: 60614, 3: 79936, 4: 77449}, 'City': {0: 'Chicago', 1: 'Katy', 2: 'Chicago', 3: 'El Paso', 4: 'Katy'}, 'State': {0: 'IL', 1: 'TX', 2: 'IL', 3: 'TX', 4: 'TX'}, 'Metro': {0: 'Chicago-Naperville-Elgin', 1: 'Houston-The Woodlands-Sugar Land', 2: 'Chicago-Naperville-Elgin', 3: 'El Paso', 4: 'Houston-The Woodlands-Sugar Land'}, 'CountyName': {0: 'Cook County', 1: 'Harris County', 2: 'Cook County', 3: 'El Paso County', 4: 'Harris County'}, 'SizeRank': {0: 1, 1: 2, 2: 3, 3: 4, 4: 5}, '1997-01': {0: 344400.0, 1: 197300.0, 2: 503400.0, 3: 77800.0, 4: 96600.0}, '1997-02': {0: 345700.0, 1: 195400.0, 2: 502200.0, 3: 77900.0, 4: 96400.0}, '1997-03': {0: 346700.0, 1: 193000.0, 2: 500000.0, 3: 77900.0, 4: 96200.0}, '1997-04': {0: 347800.0, 1: 191800.0, 2: 497900.0, 3: 77800.0, 4: 96100.0}, '1997-05': {0: 349000.0, 1: 191800.0, 2: 496300.0, 3: 77800.0, 4: 96200.0}, '1997-06': {0: 350400.0, 1: 193000.0, 2: 495200.0, 3: 77800.0, 4: 96300.0}, '1997-07': {0: 352000.0, 1: 195200.0, 2: 494700.0, 3: 77800.0, 4: 96600.0}, '1997-08': {0: 353900.0, 1: 198400.0, 2: 494900.0, 3: 77800.0, 4: 97000.0}, '1997-09': {0: 356200.0, 1: 202800.0, 2: 496200.0, 3: 77900.0, 4: 97500.0}, '1997-10': {0: 358800.0, 1: 208000.0, 2: 498600.0, 3: 78100.0, 4: 98000.0}, '1997-11': {0: 361800.0, 1: 213800.0, 2: 502000.0, 3: 78200.0, 4: 98400.0}, '1997-12': {0: 365700.0, 1: 220700.0, 2: 507600.0, 3: 78400.0, 4: 98800.0}, '1998-01': {0: 370200.0, 1: 227500.0, 2: 514900.0, 3: 78600.0, 4: 99200.0}, '1998-02': {0: 374700.0, 1: 231800.0, 2: 522200.0, 3: 78800.0, 4: 99500.0}, '1998-03': {0: 378900.0, 1: 233400.0, 2: 529500.0, 3: 79000.0, 4: 99700.0}, '1998-04': {0: 383500.0, 1: 233900.0, 2: 537900.0, 3: 79100.0, 4: 100000.0}, '1998-05': {0: 388300.0, 1: 233500.0, 2: 546900.0, 3: 79200.0, 4: 100200.0}, '1998-06': {0: 393300.0, 1: 233300.0, 2: 556400.0, 3: 79300.0, 4: 100400.0}, '1998-07': {0: 398500.0, 1: 234300.0, 2: 566100.0, 3: 79300.0, 4: 100700.0}, '1998-08': {0: 403800.0, 1: 237400.0, 2: 575600.0, 3: 79300.0, 4: 101100.0}, '1998-09': {0: 409100.0, 1: 242800.0, 2: 584800.0, 3: 79400.0, 4: 101800.0}, '1998-10': {0: 414600.0, 1: 250200.0, 2: 593500.0, 3: 79500.0, 4: 102900.0}, '1998-11': {0: 420100.0, 1: 258600.0, 2: 601600.0, 3: 79500.0, 4: 104300.0}, '1998-12': {0: 426200.0, 1: 268000.0, 2: 610100.0, 3: 79600.0, 4: 106200.0}, '1999-01': {0: 432600.0, 1: 277000.0, 2: 618600.0, 3: 79700.0, 4: 108400.0}, '1999-02': {0: 438600.0, 1: 283600.0, 2: 625600.0, 3: 79900.0, 4: 110400.0}, '1999-03': {0: 444200.0, 1: 288500.0, 2: 631100.0, 3: 80100.0, 4: 112100.0}, '1999-04': {0: 450000.0, 1: 293900.0, 2: 636600.0, 3: 80300.0, 4: 113200.0}, '1999-05': {0: 455900.0, 1: 299200.0, 2: 642100.0, 3: 80600.0, 4: 113600.0}, '1999-06': {0: 462100.0, 1: 304300.0, 2: 647600.0, 3: 80900.0, 4: 113500.0}, '1999-07': {0: 468500.0, 1: 308600.0, 2: 653300.0, 3: 81200.0, 4: 113000.0}, '1999-08': {0: 475300.0, 1: 311400.0, 2: 659300.0, 3: 81400.0, 4: 112500.0}, '1999-09': {0: 482500.0, 1: 312300.0, 2: 665800.0, 3: 81700.0, 4: 112200.0}, '1999-10': {0: 490200.0, 1: 311900.0, 2: 672900.0, 3: 82100.0, 4: 112100.0}, '1999-11': {0: 498200.0, 1: 311100.0, 2: 680500.0, 3: 82400.0, 4: 112400.0}, '1999-12': {0: 507200.0, 1: 311700.0, 2: 689600.0, 3: 82600.0, 4: 113100.0}, '2000-01': {0: 516800.0, 1: 313500.0, 2: 699700.0, 3: 82800.0, 4: 114200.0}, '2000-02': {0: 526300.0, 1: 315000.0, 2: 709300.0, 3: 82900.0, 4: 115700.0}, '2000-03': {0: 535300.0, 1: 316700.0, 2: 718300.0, 3: 83000.0, 4: 117800.0}, '2000-04': {0: 544500.0, 1: 319800.0, 2: 727600.0, 3: 83000.0, 4: 120300.0}, '2000-05': {0: 553500.0, 1: 323700.0, 2: 737100.0, 3: 82900.0, 4: 122900.0}, '2000-06': {0: 562400.0, 1: 327500.0, 2: 746600.0, 3: 82800.0, 4: 125600.0}, '2000-07': {0: 571200.0, 1: 329900.0, 2: 756200.0, 3: 82700.0, 4: 128000.0}, '2000-08': {0: 579800.0, 1: 329800.0, 2: 765800.0, 3: 82400.0, 4: 129800.0}, '2000-09': {0: 588100.0, 1: 326400.0, 2: 775100.0, 3: 82100.0, 4: 130800.0}, '2000-10': {0: 596300.0, 1: 320100.0, 2: 784400.0, 3: 81900.0, 4: 130900.0}, '2000-11': {0: 604200.0, 1: 312200.0, 2: 793500.0, 3: 81600.0, 4: 129900.0}, '2000-12': {0: 612200.0, 1: 304700.0, 2: 803000.0, 3: 81300.0, 4: 128000.0}, '2001-01': {0: 620200.0, 1: 298700.0, 2: 812500.0, 3: 81000.0, 4: 125600.0}, '2001-02': {0: 627700.0, 1: 294300.0, 2: 821200.0, 3: 80800.0, 4: 123000.0}, '2001-03': {0: 634500.0, 1: 291400.0, 2: 829200.0, 3: 80600.0, 4: 120500.0}, '2001-04': {0: 641000.0, 1: 290800.0, 2: 837000.0, 3: 80300.0, 4: 118300.0}, '2001-05': {0: 647000.0, 1: 291700.0, 2: 844400.0, 3: 80000.0, 4: 116600.0}, '2001-06': {0: 652700.0, 1: 293000.0, 2: 851600.0, 3: 79800.0, 4: 115200.0}, '2001-07': {0: 658100.0, 1: 293600.0, 2: 858600.0, 3: 79500.0, 4: 114200.0}, '2001-08': {0: 663300.0, 1: 292900.0, 2: 865300.0, 3: 79200.0, 4: 113500.0}, '2001-09': {0: 668400.0, 1: 290500.0, 2: 871800.0, 3: 78900.0, 4: 113200.0}, '2001-10': {0: 673400.0, 1: 286700.0, 2: 878200.0, 3: 78600.0, 4: 113100.0}, '2001-11': {0: 678300.0, 1: 282200.0, 2: 884700.0, 3: 78400.0, 4: 113200.0}, '2001-12': {0: 683200.0, 1: 276900.0, 2: 891300.0, 3: 78200.0, 4: 113400.0}, '2002-01': {0: 688300.0, 1: 271000.0, 2: 898000.0, 3: 78200.0, 4: 113700.0}, '2002-02': {0: 693300.0, 1: 264200.0, 2: 904700.0, 3: 78200.0, 4: 114000.0}, '2002-03': {0: 698000.0, 1: 257000.0, 2: 911200.0, 3: 78300.0, 4: 114300.0}, '2002-04': {0: 702400.0, 1: 249700.0, 2: 917600.0, 3: 78400.0, 4: 114700.0}, '2002-05': {0: 706400.0, 1: 243100.0, 2: 923800.0, 3: 78600.0, 4: 115100.0}, '2002-06': {0: 710200.0, 1: 237000.0, 2: 929800.0, 3: 78900.0, 4: 115500.0}, '2002-07': {0: 714000.0, 1: 231700.0, 2: 935700.0, 3: 79200.0, 4: 116100.0}, '2002-08': {0: 717800.0, 1: 227100.0, 2: 941400.0, 3: 79500.0, 4: 116700.0}, '2002-09': {0: 721700.0, 1: 223300.0, 2: 947100.0, 3: 79900.0, 4: 117200.0}, '2002-10': {0: 725700.0, 1: 220300.0, 2: 952800.0, 3: 80300.0, 4: 117800.0}, '2002-11': {0: 729900.0, 1: 217300.0, 2: 958900.0, 3: 80700.0, 4: 118200.0}, '2002-12': {0: 733400.0, 1: 214700.0, 2: 965100.0, 3: 81000.0, 4: 118500.0}, '2003-01': {0: 735600.0, 1: 213800.0, 2: 971000.0, 3: 81200.0, 4: 118800.0}, '2003-02': {0: 737200.0, 1: 215100.0, 2: 976400.0, 3: 81400.0, 4: 119100.0}, '2003-03': {0: 739000.0, 1: 217300.0, 2: 981400.0, 3: 81500.0, 4: 119300.0}, '2003-04': {0: 740900.0, 1: 219600.0, 2: 985700.0, 3: 81500.0, 4: 119500.0}, '2003-05': {0: 742600.0, 1: 221400.0, 2: 989400.0, 3: 81600.0, 4: 119600.0}, '2003-06': {0: 744400.0, 1: 222300.0, 2: 992900.0, 3: 81700.0, 4: 119700.0}, '2003-07': {0: 746000.0, 1: 222700.0, 2: 996800.0, 3: 81900.0, 4: 119900.0}, '2003-08': {0: 747200.0, 1: 223000.0, 2: 1000800.0, 3: 82000.0, 4: 120200.0}, '2003-09': {0: 748000.0, 1: 223700.0, 2: 1004600.0, 3: 82200.0, 4: 120500.0}, '2003-10': {0: 749000.0, 1: 225100.0, 2: 1008000.0, 3: 82500.0, 4: 120900.0}, '2003-11': {0: 750200.0, 1: 227200.0, 2: 1010600.0, 3: 82900.0, 4: 121500.0}, '2003-12': {0: 752300.0, 1: 229600.0, 2: 1012600.0, 3: 83400.0, 4: 122500.0}, '2004-01': {0: 755300.0, 1: 231800.0, 2: 1014500.0, 3: 84000.0, 4: 123900.0}, '2004-02': {0: 759200.0, 1: 233100.0, 2: 1017000.0, 3: 84700.0, 4: 125300.0}, '2004-03': {0: 764000.0, 1: 233500.0, 2: 1020500.0, 3: 85500.0, 4: 126600.0}, '2004-04': {0: 769600.0, 1: 233000.0, 2: 1024900.0, 3: 86400.0, 4: 127500.0}, '2004-05': {0: 775600.0, 1: 232100.0, 2: 1029800.0, 3: 87200.0, 4: 128100.0}, '2004-06': {0: 781900.0, 1: 231300.0, 2: 1035100.0, 3: 88000.0, 4: 128500.0}, '2004-07': {0: 787900.0, 1: 230700.0, 2: 1040500.0, 3: 88900.0, 4: 128800.0}, '2004-08': {0: 793200.0, 1: 230800.0, 2: 1046000.0, 3: 89700.0, 4: 128900.0}, '2004-09': {0: 798200.0, 1: 231500.0, 2: 1052100.0, 3: 90400.0, 4: 129000.0}, '2004-10': {0: 803100.0, 1: 232700.0, 2: 1058600.0, 3: 91100.0, 4: 129200.0}, '2004-11': {0: 807900.0, 1: 234000.0, 2: 1065000.0, 3: 91900.0, 4: 129400.0}, '2004-12': {0: 812900.0, 1: 235500.0, 2: 1071900.0, 3: 92700.0, 4: 129800.0}, '2005-01': {0: 818100.0, 1: 237000.0, 2: 1079000.0, 3: 93600.0, 4: 130100.0}, '2005-02': {0: 823200.0, 1: 238700.0, 2: 1086000.0, 3: 94400.0, 4: 130200.0}, '2005-03': {0: 828300.0, 1: 240600.0, 2: 1093100.0, 3: 95200.0, 4: 130300.0}, '2005-04': {0: 834000.0, 1: 241800.0, 2: 1100500.0, 3: 95800.0, 4: 130400.0}, '2005-05': {0: 839800.0, 1: 241700.0, 2: 1107400.0, 3: 96300.0, 4: 130400.0}, '2005-06': {0: 845600.0, 1: 240700.0, 2: 1113500.0, 3: 96700.0, 4: 130300.0}, '2005-07': {0: 851700.0, 1: 239300.0, 2: 1118800.0, 3: 97200.0, 4: 130100.0}, '2005-08': {0: 858000.0, 1: 238000.0, 2: 1123700.0, 3: 97700.0, 4: 129800.0}, '2005-09': {0: 864300.0, 1: 236900.0, 2: 1129200.0, 3: 98400.0, 4: 129400.0}, '2005-10': {0: 870600.0, 1: 235700.0, 2: 1135400.0, 3: 99000.0, 4: 129000.0}, '2005-11': {0: 876200.0, 1: 234700.0, 2: 1141900.0, 3: 99600.0, 4: 128800.0}, '2005-12': {0: 880600.0, 1: 233400.0, 2: 1148000.0, 3: 100200.0, 4: 128800.0}, '2006-01': {0: 884500.0, 1: 231700.0, 2: 1152800.0, 3: 101000.0, 4: 129000.0}, '2006-02': {0: 887800.0, 1: 230100.0, 2: 1155900.0, 3: 102000.0, 4: 129200.0}, '2006-03': {0: 890600.0, 1: 229000.0, 2: 1157900.0, 3: 103000.0, 4: 129400.0}, '2006-04': {0: 893200.0, 1: 228500.0, 2: 1159500.0, 3: 104300.0, 4: 129500.0}, '2006-05': {0: 895500.0, 1: 228700.0, 2: 1161000.0, 3: 105800.0, 4: 129700.0}, '2006-06': {0: 897300.0, 1: 229400.0, 2: 1162800.0, 3: 107400.0, 4: 130000.0}, '2006-07': {0: 898900.0, 1: 230400.0, 2: 1165300.0, 3: 109100.0, 4: 130300.0}, '2006-08': {0: 900300.0, 1: 231600.0, 2: 1168100.0, 3: 111000.0, 4: 130700.0}, '2006-09': {0: 902000.0, 1: 233000.0, 2: 1171300.0, 3: 113000.0, 4: 131200.0}, '2006-10': {0: 904300.0, 1: 234700.0, 2: 1174400.0, 3: 115000.0, 4: 131800.0}, '2006-11': {0: 907000.0, 1: 237100.0, 2: 1176700.0, 3: 117000.0, 4: 132300.0}, '2006-12': {0: 909500.0, 1: 240200.0, 2: 1178400.0, 3: 118800.0, 4: 132700.0}, '2007-01': {0: 912000.0, 1: 242900.0, 2: 1179900.0, 3: 120600.0, 4: 133000.0}, '2007-02': {0: 913400.0, 1: 244600.0, 2: 1181100.0, 3: 122200.0, 4: 133200.0}, '2007-03': {0: 913200.0, 1: 245200.0, 2: 1182800.0, 3: 124000.0, 4: 133600.0}, '2007-04': {0: 911800.0, 1: 245200.0, 2: 1184800.0, 3: 126000.0, 4: 134100.0}, '2007-05': {0: 909200.0, 1: 245000.0, 2: 1185300.0, 3: 128000.0, 4: 134700.0}, '2007-06': {0: 905200.0, 1: 245600.0, 2: 1183700.0, 3: 129600.0, 4: 135400.0}, '2007-07': {0: 901300.0, 1: 246900.0, 2: 1181000.0, 3: 130700.0, 4: 136000.0}, '2007-08': {0: 897900.0, 1: 248700.0, 2: 1177900.0, 3: 131400.0, 4: 136600.0}, '2007-09': {0: 895300.0, 1: 250700.0, 2: 1175400.0, 3: 132000.0, 4: 137000.0}, '2007-10': {0: 893500.0, 1: 252500.0, 2: 1173800.0, 3: 132300.0, 4: 137300.0}, '2007-11': {0: 891100.0, 1: 254000.0, 2: 1171700.0, 3: 132300.0, 4: 137400.0}, '2007-12': {0: 886700.0, 1: 254800.0, 2: 1167900.0, 3: 132000.0, 4: 137200.0}, '2008-01': {0: 881900.0, 1: 254000.0, 2: 1162900.0, 3: 131300.0, 4: 136500.0}, '2008-02': {0: 876500.0, 1: 252400.0, 2: 1157000.0, 3: 130300.0, 4: 135600.0}, '2008-03': {0: 870600.0, 1: 250900.0, 2: 1150700.0, 3: 129300.0, 4: 134700.0}, '2008-04': {0: 864900.0, 1: 249600.0, 2: 1144200.0, 3: 128300.0, 4: 133800.0}, '2008-05': {0: 859000.0, 1: 248400.0, 2: 1135900.0, 3: 127300.0, 4: 133000.0}, '2008-06': {0: 851600.0, 1: 247900.0, 2: 1125700.0, 3: 126300.0, 4: 132000.0}, '2008-07': {0: 843800.0, 1: 247700.0, 2: 1114200.0, 3: 125400.0, 4: 131200.0}, '2008-08': {0: 836400.0, 1: 247800.0, 2: 1102200.0, 3: 124600.0, 4: 130500.0}, '2008-09': {0: 830700.0, 1: 247900.0, 2: 1092100.0, 3: 123900.0, 4: 130000.0}, '2008-10': {0: 827300.0, 1: 247800.0, 2: 1085300.0, 3: 123300.0, 4: 129400.0}, '2008-11': {0: 824800.0, 1: 247600.0, 2: 1079400.0, 3: 122600.0, 4: 128700.0}, '2008-12': {0: 821400.0, 1: 247500.0, 2: 1072500.0, 3: 122100.0, 4: 128200.0}, '2009-01': {0: 818500.0, 1: 246600.0, 2: 1065400.0, 3: 121600.0, 4: 127600.0}, '2009-02': {0: 815200.0, 1: 245700.0, 2: 1057900.0, 3: 121200.0, 4: 127100.0}, '2009-03': {0: 810200.0, 1: 245600.0, 2: 1048900.0, 3: 120800.0, 4: 126400.0}, '2009-04': {0: 803500.0, 1: 246000.0, 2: 1037900.0, 3: 120300.0, 4: 125900.0}, '2009-05': {0: 795400.0, 1: 246300.0, 2: 1024300.0, 3: 119700.0, 4: 125300.0}, '2009-06': {0: 786800.0, 1: 246800.0, 2: 1010100.0, 3: 119100.0, 4: 124700.0}, '2009-07': {0: 780500.0, 1: 247200.0, 2: 999000.0, 3: 118700.0, 4: 124300.0}, '2009-08': {0: 776800.0, 1: 247600.0, 2: 990800.0, 3: 118400.0, 4: 124100.0}, '2009-09': {0: 774600.0, 1: 247900.0, 2: 985400.0, 3: 118200.0, 4: 124100.0}, '2009-10': {0: 774200.0, 1: 248100.0, 2: 983300.0, 3: 117900.0, 4: 124200.0}, '2009-11': {0: 774500.0, 1: 248200.0, 2: 982800.0, 3: 117600.0, 4: 124400.0}, '2009-12': {0: 775800.0, 1: 248000.0, 2: 983000.0, 3: 117500.0, 4: 124500.0}, '2010-01': {0: 774600.0, 1: 249800.0, 2: 985000.0, 3: 117300.0, 4: 124700.0}, '2010-02': {0: 774500.0, 1: 250500.0, 2: 988000.0, 3: 117300.0, 4: 125000.0}, '2010-03': {0: 773800.0, 1: 250100.0, 2: 986200.0, 3: 116900.0, 4: 125100.0}, '2010-04': {0: 769500.0, 1: 250400.0, 2: 978800.0, 3: 116100.0, 4: 124600.0}, '2010-05': {0: 765800.0, 1: 251800.0, 2: 974700.0, 3: 115700.0, 4: 124200.0}, '2010-06': {0: 767300.0, 1: 251300.0, 2: 975300.0, 3: 116100.0, 4: 124100.0}, '2010-07': {0: 765500.0, 1: 251200.0, 2: 973600.0, 3: 116400.0, 4: 124100.0}, '2010-08': {0: 761300.0, 1: 250600.0, 2: 967500.0, 3: 116700.0, 4: 123700.0}, '2010-09': {0: 756700.0, 1: 250000.0, 2: 957800.0, 3: 117400.0, 4: 123400.0}, '2010-10': {0: 747800.0, 1: 250000.0, 2: 945800.0, 3: 118200.0, 4: 123000.0}, '2010-11': {0: 738600.0, 1: 249700.0, 2: 935500.0, 3: 118700.0, 4: 122400.0}, '2010-12': {0: 732000.0, 1: 248100.0, 2: 927000.0, 3: 118800.0, 4: 121400.0}, '2011-01': {0: 730800.0, 1: 247400.0, 2: 924800.0, 3: 119000.0, 4: 120800.0}, '2011-02': {0: 732200.0, 1: 248500.0, 2: 926800.0, 3: 118800.0, 4: 120200.0}, '2011-03': {0: 732500.0, 1: 249400.0, 2: 925200.0, 3: 118300.0, 4: 119900.0}, '2011-04': {0: 731300.0, 1: 249200.0, 2: 918500.0, 3: 118100.0, 4: 120100.0}, '2011-05': {0: 731500.0, 1: 249300.0, 2: 914200.0, 3: 117600.0, 4: 120000.0}, '2011-06': {0: 731400.0, 1: 249500.0, 2: 912100.0, 3: 116800.0, 4: 119600.0}, '2011-07': {0: 732400.0, 1: 249500.0, 2: 913700.0, 3: 116500.0, 4: 119000.0}, '2011-08': {0: 735100.0, 1: 249400.0, 2: 919800.0, 3: 116100.0, 4: 118100.0}, '2011-09': {0: 736500.0, 1: 248900.0, 2: 924800.0, 3: 114800.0, 4: 117100.0}, '2011-10': {0: 736600.0, 1: 248000.0, 2: 925000.0, 3: 113500.0, 4: 116800.0}, '2011-11': {0: 735900.0, 1: 247100.0, 2: 924800.0, 3: 112800.0, 4: 116700.0}, '2011-12': {0: 739000.0, 1: 247000.0, 2: 930400.0, 3: 112700.0, 4: 116400.0}, '2012-01': {0: 739300.0, 1: 248600.0, 2: 930800.0, 3: 112400.0, 4: 116000.0}, '2012-02': {0: 735600.0, 1: 251200.0, 2: 925800.0, 3: 112200.0, 4: 115900.0}, '2012-03': {0: 735700.0, 1: 252600.0, 2: 927300.0, 3: 112400.0, 4: 115800.0}, '2012-04': {0: 741600.0, 1: 252600.0, 2: 940100.0, 3: 112800.0, 4: 115200.0}, '2012-05': {0: 746200.0, 1: 252700.0, 2: 954200.0, 3: 113200.0, 4: 114700.0}, '2012-06': {0: 752200.0, 1: 252700.0, 2: 967900.0, 3: 113400.0, 4: 114700.0}, '2012-07': {0: 762000.0, 1: 252400.0, 2: 978100.0, 3: 113100.0, 4: 115000.0}, '2012-08': {0: 772800.0, 1: 252500.0, 2: 986000.0, 3: 112800.0, 4: 115500.0}, '2012-09': {0: 781400.0, 1: 253300.0, 2: 995100.0, 3: 112900.0, 4: 115800.0}, '2012-10': {0: 788800.0, 1: 254200.0, 2: 1002400.0, 3: 112900.0, 4: 115900.0}, '2012-11': {0: 795800.0, 1: 255200.0, 2: 1005000.0, 3: 112900.0, 4: 116200.0}, '2012-12': {0: 800900.0, 1: 256600.0, 2: 1005100.0, 3: 112800.0, 4: 116700.0}, '2013-01': {0: 804200.0, 1: 257000.0, 2: 1008500.0, 3: 113000.0, 4: 117300.0}, '2013-02': {0: 808100.0, 1: 256500.0, 2: 1015700.0, 3: 113400.0, 4: 117900.0}, '2013-03': {0: 813200.0, 1: 256600.0, 2: 1027500.0, 3: 113600.0, 4: 118500.0}, '2013-04': {0: 819200.0, 1: 257300.0, 2: 1040800.0, 3: 113500.0, 4: 119300.0}, '2013-05': {0: 827900.0, 1: 258400.0, 2: 1055300.0, 3: 113300.0, 4: 120500.0}, '2013-06': {0: 838200.0, 1: 260700.0, 2: 1071300.0, 3: 113000.0, 4: 121800.0}, '2013-07': {0: 848300.0, 1: 263900.0, 2: 1090600.0, 3: 112900.0, 4: 123000.0}, '2013-08': {0: 853800.0, 1: 266900.0, 2: 1108500.0, 3: 112900.0, 4: 124300.0}, '2013-09': {0: 856500.0, 1: 269100.0, 2: 1123600.0, 3: 112700.0, 4: 125400.0}, '2013-10': {0: 856800.0, 1: 270900.0, 2: 1135600.0, 3: 112500.0, 4: 126100.0}, '2013-11': {0: 855400.0, 1: 273100.0, 2: 1142400.0, 3: 112300.0, 4: 126800.0}, '2013-12': {0: 854500.0, 1: 275800.0, 2: 1145800.0, 3: 112000.0, 4: 127600.0}, '2014-01': {0: 858500.0, 1: 277700.0, 2: 1148400.0, 3: 111500.0, 4: 128400.0}, '2014-02': {0: 862700.0, 1: 279600.0, 2: 1150700.0, 3: 111500.0, 4: 129100.0}, '2014-03': {0: 866500.0, 1: 282100.0, 2: 1152700.0, 3: 112100.0, 4: 130100.0}, '2014-04': {0: 874900.0, 1: 284500.0, 2: 1157700.0, 3: 112600.0, 4: 131300.0}, '2014-05': {0: 885100.0, 1: 286200.0, 2: 1162400.0, 3: 112700.0, 4: 132600.0}, '2014-06': {0: 890800.0, 1: 288300.0, 2: 1165200.0, 3: 113100.0, 4: 133700.0}, '2014-07': {0: 893800.0, 1: 290700.0, 2: 1169400.0, 3: 113900.0, 4: 134500.0}, '2014-08': {0: 894100.0, 1: 293100.0, 2: 1174900.0, 3: 114300.0, 4: 135300.0}, '2014-09': {0: 891300.0, 1: 295600.0, 2: 1175700.0, 3: 114400.0, 4: 136400.0}, '2014-10': {0: 889700.0, 1: 298200.0, 2: 1174000.0, 3: 114300.0, 4: 137600.0}, '2014-11': {0: 891900.0, 1: 300200.0, 2: 1176300.0, 3: 114200.0, 4: 138800.0}, '2014-12': {0: 894300.0, 1: 301500.0, 2: 1180100.0, 3: 114300.0, 4: 140000.0}, '2015-01': {0: 895000, 1: 301800, 2: 1178600, 3: 114700, 4: 141000}, '2015-02': {0: 897300, 1: 302200, 2: 1176700, 3: 115000, 4: 142000}, '2015-03': {0: 903700, 1: 303700, 2: 1180800, 3: 115100, 4: 143300}, '2015-04': {0: 911300, 1: 306600, 2: 1187600, 3: 115300, 4: 144800}, '2015-05': {0: 915600, 1: 309300, 2: 1193500, 3: 115700, 4: 146100}, '2015-06': {0: 916200, 1: 311900, 2: 1198300, 3: 115900, 4: 147200}, '2015-07': {0: 916700, 1: 314100, 2: 1199600, 3: 115600, 4: 148500}, '2015-08': {0: 918600, 1: 316000, 2: 1198000, 3: 115300, 4: 149700}, '2015-09': {0: 924400, 1: 318600, 2: 1199200, 3: 115300, 4: 151100}, '2015-10': {0: 935600, 1: 321800, 2: 1206600, 3: 115400, 4: 152200}, '2015-11': {0: 947200, 1: 324400, 2: 1218000, 3: 115700, 4: 153000}, '2015-12': {0: 950900, 1: 326400, 2: 1226400, 3: 116200, 4: 154100}, '2016-01': {0: 952700, 1: 327400, 2: 1230300, 3: 116200, 4: 156000}, '2016-02': {0: 959000, 1: 326900, 2: 1234700, 3: 115700, 4: 157800}, '2016-03': {0: 966400, 1: 327300, 2: 1240300, 3: 115100, 4: 159600}, '2016-04': {0: 970300, 1: 328900, 2: 1244700, 3: 114700, 4: 161700}, '2016-05': {0: 973200, 1: 330000, 2: 1245800, 3: 114300, 4: 164200}, '2016-06': {0: 973300, 1: 330000, 2: 1245300, 3: 114000, 4: 166100}, '2016-07': {0: 970600, 1: 328900, 2: 1243700, 3: 114000, 4: 167400}, '2016-08': {0: 971800, 1: 327500, 2: 1243400, 3: 113800, 4: 168100}, '2016-09': {0: 977800, 1: 326300, 2: 1245000, 3: 114000, 4: 168400}, '2016-10': {0: 985200, 1: 325300, 2: 1250800, 3: 114800, 4: 168400}, '2016-11': {0: 992900, 1: 324700, 2: 1259300, 3: 115600, 4: 168400}, '2016-12': {0: 997600, 1: 324700, 2: 1266600, 3: 116200, 4: 168400}, '2017-01': {0: 996000, 1: 323700, 2: 1270800, 3: 116800, 4: 168200}, '2017-02': {0: 993100, 1: 322100, 2: 1274500, 3: 117400, 4: 167900}, '2017-03': {0: 991500, 1: 320800, 2: 1278900, 3: 117800, 4: 167400}, '2017-04': {0: 990000, 1: 320400, 2: 1282600, 3: 118200, 4: 167000}, '2017-05': {0: 991400, 1: 320300, 2: 1285800, 3: 118700, 4: 166900}, '2017-06': {0: 998200, 1: 320900, 2: 1288100, 3: 119000, 4: 166800}, '2017-07': {0: 1004000, 1: 320900, 2: 1288500, 3: 119100, 4: 166800}, '2017-08': {0: 1006800, 1: 320300, 2: 1287500, 3: 119400, 4: 167300}, '2017-09': {0: 1008400, 1: 319800, 2: 1289200, 3: 119900, 4: 168300}, '2017-10': {0: 1011300, 1: 320200, 2: 1295000, 3: 120200, 4: 169500}, '2017-11': {0: 1015500, 1: 320800, 2: 1301100, 3: 120200, 4: 170700}, '2017-12': {0: 1022000, 1: 321100, 2: 1304300, 3: 120100, 4: 172100}, '2018-01': {0: 1028900, 1: 322700, 2: 1310100, 3: 120300, 4: 173500}, '2018-02': {0: 1034500, 1: 326500, 2: 1315300, 3: 120500, 4: 174600}, '2018-03': {0: 1037400, 1: 330400, 2: 1317900, 3: 120800, 4: 175500}, '2018-04': {0: 1038700, 1: 332700, 2: 1321100, 3: 121300, 4: 176400}, '2018-05': {0: 1041500, 1: 334500, 2: 1325300, 3: 122200, 4: 176900}, '2018-06': {0: 1042800, 1: 335900, 2: 1323800, 3: 123000, 4: 176900}, '2018-07': {0: 1042900, 1: 337000, 2: 1321200, 3: 123600, 4: 177300}, '2018-08': {0: 1044400, 1: 338300, 2: 1320700, 3: 124500, 4: 178000}, '2018-09': {0: 1047800, 1: 338400, 2: 1319500, 3: 125600, 4: 178500}, '2018-10': {0: 1049700, 1: 336900, 2: 1318800, 3: 126300, 4: 179300}, '2018-11': {0: 1048300, 1: 336000, 2: 1319700, 3: 126800, 4: 180200}, '2018-12': {0: 1047900, 1: 336500, 2: 1323300, 3: 127400, 4: 180700}}
I am new to Python, so please provide explanation with your code.
You can perform a groupby and sum on the columns:
df.iloc[:,7:].groupby(by=lambda x: x.split('-')[0], axis=1).sum().add_suffix('_sum')
We extract the monthly data and aggregate by the year. For this, I specify a callback to split the column name and return the year. So, for example x.split('-')[0] returns 1997 whenever x is 1997-XX.