I have this dataframe
Region 2021 2022 2023
0 Europe 0.00 0.00 0.00
1 N.Amerca 0.50 0.50 0.50
2 N.Amerca 4.40 4.40 4.40
3 N.Amerca 0.00 8.00 8.00
4 Asia 0.00 0.00 1.75
5 Asia 0.00 0.00 0.00
6 Asia 0.00 0.00 2.00
7 N.Amerca 0.00 0.00 0.50
8 Eurpoe 6.00 6.00 6.00
9 Asia 7.50 7.50 7.50
10 Asia 3.75 3.75 3.75
11 Asia 3.50 3.50 3.50
12 Asia 3.80 3.80 3.80
13 Asia 0.00 0.00 0.00
14 Europe 6.52 6.52 6.52
Once a value in 2021 is found it should carry a 0 to the rest (2022 and 2023)
and if a value in 2022 is found -it should carry 0 to the rest. In other words, once value in found in columns 2021 and forth it should zero the rest on the right.
expected result would be:
Region 2021 2022 2023
0 Europe 0.00 0.00 0.00
1 N.Amerca 0.50 0.00 0.00
2 N.Amerca 4.40 0.00 0.00
3 N.Amerca 0.00 8.00 0.00
4 Asia 0.00 0.00 1.75
5 Asia 0.00 0.00 0.00
6 Asia 0.00 0.00 2.00
7 N.Amerca 0.00 0.00 0.50
8 Eurpoe 6.00 0.00 0.00
9 Asia 7.50 0.00 0.00
10 Asia 3.75 0.00 0.00
11 Asia 3.50 0.00 0.00
12 Asia 3.80 0.00 0.00
13 Asia 0.00 0.00 0.00
14 Europe 6.52 0.00 0.00
I have tried to apply a lambda:
def foo(r):
#if r['2021')>0: then 2020 and forth should be zero)
df = df.apply(lambda x: foo(x), axis=1)
but the challange is that there are 2021 - to 2030 and the foo becomes a mess)
Let us try duplicated
df = df.mask(df.T.apply(pd.Series.duplicated).T,0)
Out[57]:
Region 2021 2022 2023
0 Europe 0.00 0.0 0.00
1 N.Amerca 0.50 0.0 0.00
2 N.Amerca 4.40 0.0 0.00
3 N.Amerca 0.00 8.0 0.00
4 Asia 0.00 0.0 1.75
5 Asia 0.00 0.0 0.00
6 Asia 0.00 0.0 2.00
7 N.Amerca 0.00 0.0 0.50
8 Eurpoe 6.00 0.0 0.00
9 Asia 7.50 0.0 0.00
10 Asia 3.75 0.0 0.00
11 Asia 3.50 0.0 0.00
12 Asia 3.80 0.0 0.00
13 Asia 0.00 0.0 0.00
14 Europe 6.52 0.0 0.00
This is another way:
df2 = df.set_index('Region').diff(axis=1).reset_index()
df2['2021'] = df['2021']
or:
df.iloc[:,1:].where(df.iloc[:,1:].ne(0).cumsum(axis=1).eq(1),0)
Output:
2021 2022 2023
0 0.00 0.0 0.00
1 0.50 0.0 0.00
2 4.40 0.0 0.00
3 0.00 8.0 0.00
4 0.00 0.0 1.75
5 0.00 0.0 0.00
6 0.00 0.0 2.00
7 0.00 0.0 0.50
8 6.00 0.0 0.00
9 7.50 0.0 0.00
10 3.75 0.0 0.00
11 3.50 0.0 0.00
12 3.80 0.0 0.00
13 0.00 0.0 0.00
14 6.52 0.0 0.00
Related
I have a dataframe like this,
ds 0 1 2 4 5 6
0 1991Q3 nan nan nan nan 1.0 nan
1 2014Q2 1.0 3.0 nan nan 1.0 nan
2 2014Q3 1.0 nan nan 1.0 4.0 nan
3 2014Q4 nan nan nan 2.0 3.0 nan
4 2015Q1 nan 1.0 2.0 4.0 4.0 nan
I would like the proportions for each column 0-6 like this,
ds 0 1 2 4 5 6
0 1991Q3 0.00 0.00 0.00 0.00 1.00 0.00
1 2014Q2 0.20 0.60 0.00 0.00 0.20 0.00
2 2014Q3 0.16 0.00 0.00 0.16 0.67 0.00
3 2014Q4 0.00 0.00 0.00 0.40 0.60 0.00
4 2015Q1 0.00 0.09 0.18 0.36 0.36 0.00
Is there a pandas way to this? Any suggestion would be great.
You can do this:
df = df.replace(np.nan, 0)
df = df.set_index('ds')
In [3194]: df.div(df.sum(1),0).reset_index()
Out[3194]:
ds 0 1 2 4 5 6
0 1991Q3 0.00 0.00 0.00 0.00 1.00 0.00
1 2014Q2 0.20 0.60 0.00 0.00 0.20 0.00
2 2014Q3 0.17 0.00 0.00 0.17 0.67 0.00
3 2014Q4 0.00 0.00 0.00 0.40 0.60 0.00
4 2015Q1 0.00 0.09 0.18 0.36 0.36 0.00
OR you can use df.apply:
In [3196]: df = df.replace(np.nan, 0)
In [3197]: df.iloc[:,1:] = df.iloc[:,1:].apply(lambda x: x/x.sum(), axis=1)
In [3198]: df
Out[3197]:
ds 0 1 2 4 5 6
0 1991Q3 0.00 0.00 0.00 0.00 1.00 0.00
1 2014Q2 0.20 0.60 0.00 0.00 0.20 0.00
2 2014Q3 0.17 0.00 0.00 0.17 0.67 0.00
3 2014Q4 0.00 0.00 0.00 0.40 0.60 0.00
4 2015Q1 0.00 0.09 0.18 0.36 0.36 0.00
Set the first column as the index, get the sum of each row, and divide the main dataframe by the sums, and filling the null entries with 0
res = df.set_index("ds")
res.fillna(0).div(res.sum(1),axis=0)
I am looking a way to aggregate (in pandas) a subset of values based on a particular partition, an equivalent of
select table.*,
sum(income) over (order by id, num_yyyymm rows between 3 preceding and 1 preceding) as prev_income_3,
sum(income) over (order by id, num_yyyymm rows between 1 following and 3 following) as next_income_3
from table order by a.id_customer, num_yyyymm;
I tried with the following solution but it has some problems:
1) Takes ages to complete
2) I have to merge all the results at the end of
for x, y in df.groupby(['id_customer']):
print(y[['num_yyyymm', 'income']])
y['next3'] = y['income'].iloc[::-1].rolling(3).sum()
print(y[['num_yyyymm', 'income', 'next3']])
break
Results:
num_yyyymm income next3
0 201501 0.00 0.00
1 201502 0.00 0.00
2 201503 0.00 0.00
3 201504 0.00 0.00
4 201505 0.00 0.00
5 201506 0.00 0.00
6 201507 0.00 0.00
7 201508 0.00 0.00
8 201509 0.00 0.00
9 201510 0.00 0.00
10 201511 0.00 0.00
11 201512 0.00 0.00
12 201601 0.00 0.00
13 201602 0.00 0.00
14 201603 0.00 0.00
15 201604 0.00 0.00
16 201605 0.00 0.00
17 201606 0.00 0.00
18 201607 0.00 0.00
19 201608 0.00 0.00
20 201609 0.00 1522.07
21 201610 0.00 1522.07
22 201611 0.00 1522.07
23 201612 1522.07 0.00
24 201701 0.00 -0.00
25 201702 0.00 1.52
26 201703 0.00 1522.07
27 201704 0.00 1522.07
28 201705 1.52 1520.55
29 201706 1520.55 0.00
30 201707 0.00 NaN
31 201708 0.00 NaN
32 201709 0.00 NaN
Does anybody have an alternative solution?
I have created a script which is used to extract a certain section of data from the file, which works properly. But in the specific data which I want to extract Form Feed and in next line [] is present which I don't want.
The Form Feed which appear in the file is:-
['STAAD', 'SPACE', '--', 'PAGE', 'NO.', '']
import csv
found_type = False
filename = (r'C:\Users\GURBHEJ SINGH\Desktop\3d_frame.txt')
with open(filename, 'r') as f:
for line in f:
if line == '\n':
continue
if '0.00 MAX' in line:
found_type = True
if found_type:
if '|---------------------------------------------------------------------------' in line:
found_type = False
else:
t_line = str(line)
t_line = list(t_line.split())
print(t_line)
At some positions in the result From feed and in next line [] is present which I don't want like as shown below
['STAAD', 'SPACE', '--', 'PAGE', 'NO.', '8']
[]
To elaborate my problem I am giving a short part of the data I want which is from the file I used.
FILE
2 0.00 MAX 64.08 2 6.17 2 0.00 2 0.00 2
MIN 42.72 1 4.11 1 0.00 2 0.00 2
STAAD SPACE -- PAGE NO. 6
0.31 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
0.62 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
0.92 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
1.23 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
1.54 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
1.85 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.15 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.46 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
2.77 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
3.08 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
3.38 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
3.69 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
4.00 MAX 42.72 1 4.11 1 0.00 2 0.00 2
MIN 64.08 2 6.17 2 0.00 2 0.00 2
3 0.00 MAX 64.08 2 6.17 2 0.00 2 0.00 2
MIN 42.72 1 4.11 1 0.00 2 0.00 2
0.31 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
0.62 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
0.92 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
1.23 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
1.54 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
1.85 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.15 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
STAAD SPACE -- PAGE NO. 7
2.46 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
2.77 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
3.08 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
3.38 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
3.69 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
4.00 MAX 42.72 1 4.11 1 0.00 2 0.00 2
MIN 64.08 2 6.17 2 0.00 2 0.00 2
But the output which I want is like as shown below:-
2 0.00 MAX 64.08 2 6.17 2 0.00 2 0.00 2
MIN 42.72 1 4.11 1 0.00 2 0.00 2
0.31 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
0.62 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
0.92 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
1.23 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
1.54 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
1.85 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.15 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.46 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
2.77 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
3.08 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
3.38 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
3.69 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
4.00 MAX 42.72 1 4.11 1 0.00 2 0.00 2
MIN 64.08 2 6.17 2 0.00 2 0.00 2
3 0.00 MAX 64.08 2 6.17 2 0.00 2 0.00 2
MIN 42.72 1 4.11 1 0.00 2 0.00 2
0.31 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
0.62 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
0.92 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
1.23 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
1.54 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
1.85 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.15 MAX 4.93 2 57.54 2 0.00 2 0.00 2
MIN 3.29 1 38.36 1 0.00 2 0.00 2
2.46 MAX 14.79 2 54.51 2 0.00 2 0.00 2
MIN 9.86 1 36.34 1 0.00 2 0.00 2
2.77 MAX 24.65 2 48.44 2 0.00 2 0.00 2
MIN 16.43 1 32.29 1 0.00 2 0.00 2
3.08 MAX 34.51 2 39.34 2 0.00 2 0.00 2
MIN 23.00 1 26.23 1 0.00 2 0.00 2
3.38 MAX 44.37 2 27.20 2 0.00 2 0.00 2
MIN 29.58 1 18.14 1 0.00 2 0.00 2
3.69 MAX 54.23 2 12.04 2 0.00 2 0.00 2
MIN 36.15 1 8.02 1 0.00 2 0.00 2
4.00 MAX 42.72 1 4.11 1 0.00 2 0.00 2
MIN 64.08 2 6.17 2 0.00 2 0.00 2
I have a dataframe like this:
Ind TIME PREC ET PET YIELD
0 1 1.21 0.02 0.02 0.00
1 2 0.00 0.03 0.04 0.00
2 3 0.00 0.03 0.05 0.00
3 4 0.00 0.04 0.05 0.00
4 5 0.00 0.05 0.07 0.00
5 6 0.00 0.03 0.05 0.00
6 7 0.00 0.02 0.04 0.00
7 8 1.14 0.03 0.04 0.00
8 9 0.10 0.02 0.03 0.00
9 10 0.00 0.03 0.04 0.00
10 11 0.10 0.05 0.11 0.00
11 12 0.00 0.06 0.15 0.00
12 13 2.30 0.14 0.44 0.00
13 14 0.17 0.09 0.29 0.00
14 15 0.00 0.13 0.35 0.00
15 16 0.00 0.14 0.39 0.00
16 17 0.00 0.10 0.31 0.00
17 18 0.00 0.15 0.51 0.00
18 19 0.00 0.22 0.58 0.00
19 20 0.10 0.04 0.09 0.00
20 21 0.00 0.04 0.06 0.00
21 22 0.27 0.13 0.43 0.00
22 23 0.00 0.10 0.25 0.00
23 24 0.00 0.03 0.04 0.00
24 25 0.00 0.04 0.05 0.00
25 26 0.43 0.04 0.15 0.00
26 27 0.17 0.06 0.23 0.00
27 28 0.50 0.02 0.04 0.00
28 29 0.00 0.03 0.04 0.00
29 30 0.00 0.04 0.08 0.00
30 31 0.00 0.04 0.08 0.00
31 1 6.48 1.97 5.10 0.03
32 32 0.00 0.22 0.70 0.00
33 33 0.00 0.49 0.88 0.00
In this dataframe column 'TIME' shows ordinal day number in a year, and after the end of every month - an ordinal number of month in a year, which messes up all dataframe calculations, so, for this reason, I would like to drop all rows that contain month value. First, I tried to use .shift():
df = df.loc[df.TIME == df.TIME.shift() +1],
however, in this case, I delete twice as many rows as it supposed to be. I also tried to delete every value after the end of every month:
for i in indexes:
df = df.loc[df.index != i],
where indexes is a list, containing row indexes after day value is equal to 31, 59, ... 365 or end of every month. However, in a leap year, these values would be different, and I could create another list for a leap year, but this method would be very non-pythonist. So, I wonder, is there any better way to delete non-consecutive values from a dataframe (excluding when one year ends and another one starts: 364, 365, 1, 2)?
EDIT: I should, probably, add that there are twenty years in this dataframe, so this is how the dataframe looks like at the end of each year:
TIME PREC ET PET YIELD
370 360 0.00 0.14 0.26 0.04
371 361 0.00 0.15 0.27 0.04
372 362 0.00 0.14 0.25 0.04
373 363 0.11 0.18 0.32 0.04
374 364 0.00 0.15 0.25 0.04
375 365 0.00 0.17 0.29 0.04
376 12 16.29 4.44 7.74 1.89
377 1 0.00 0.16 0.28 0.03
378 2 0.00 0.18 0.32 0.03
379 3 0.00 0.22 0.40 0.03
df
TIME PREC ET PET YIELD
0 360 0.00 0.14 0.26 0.04
1 361 0.00 0.15 0.27 0.04
2 362 0.00 0.14 0.25 0.04
3 363 0.11 0.18 0.32 0.04
4 364 0.00 0.15 0.25 0.04
5 365 0.00 0.17 0.29 0.04
6 12 16.29 4.44 7.74 1.89
7 1 1.21 0.02 0.02 0.00
8 2 0.00 0.03 0.04 0.00
9 3 0.00 0.03 0.05 0.00
10 4 0.00 0.04 0.05 0.00
11 5 0.00 0.05 0.07 0.00
12 6 0.00 0.03 0.05 0.00
13 7 0.00 0.02 0.04 0.00
14 8 1.14 0.03 0.04 0.00
15 9 0.10 0.02 0.03 0.00
16 10 0.00 0.03 0.04 0.00
17 11 0.10 0.05 0.11 0.00
18 12 0.00 0.06 0.15 0.00
19 13 2.30 0.14 0.44 0.00
20 14 0.17 0.09 0.29 0.00
21 15 0.00 0.13 0.35 0.00
22 16 0.00 0.14 0.39 0.00
23 17 0.00 0.10 0.31 0.00
24 18 0.00 0.15 0.51 0.00
25 19 0.00 0.22 0.58 0.00
26 20 0.10 0.04 0.09 0.00
27 21 0.00 0.04 0.06 0.00
28 22 0.27 0.13 0.43 0.00
29 23 0.00 0.10 0.25 0.00
30 24 0.00 0.03 0.04 0.00
31 25 0.00 0.04 0.05 0.00
32 26 0.43 0.04 0.15 0.00
33 27 0.17 0.06 0.23 0.00
34 28 0.50 0.02 0.04 0.00
35 29 0.00 0.03 0.04 0.00
36 30 0.00 0.04 0.08 0.00
37 31 0.00 0.04 0.08 0.00
38 1 6.48 1.97 5.10 0.03
39 32 0.00 0.22 0.70 0.00
40 33 0.00 0.49 0.88 0.00
Look at the diffs in TIME. Drop the rows where diff is between -360 and -1
df[~df.TIME.diff().le(-12)]
TIME PREC ET PET YIELD
0 360 0.00 0.14 0.26 0.04
1 361 0.00 0.15 0.27 0.04
2 362 0.00 0.14 0.25 0.04
3 363 0.11 0.18 0.32 0.04
4 364 0.00 0.15 0.25 0.04
5 365 0.00 0.17 0.29 0.04
7 1 1.21 0.02 0.02 0.00
8 2 0.00 0.03 0.04 0.00
9 3 0.00 0.03 0.05 0.00
10 4 0.00 0.04 0.05 0.00
11 5 0.00 0.05 0.07 0.00
12 6 0.00 0.03 0.05 0.00
13 7 0.00 0.02 0.04 0.00
14 8 1.14 0.03 0.04 0.00
15 9 0.10 0.02 0.03 0.00
16 10 0.00 0.03 0.04 0.00
17 11 0.10 0.05 0.11 0.00
18 12 0.00 0.06 0.15 0.00
19 13 2.30 0.14 0.44 0.00
20 14 0.17 0.09 0.29 0.00
21 15 0.00 0.13 0.35 0.00
22 16 0.00 0.14 0.39 0.00
23 17 0.00 0.10 0.31 0.00
24 18 0.00 0.15 0.51 0.00
25 19 0.00 0.22 0.58 0.00
26 20 0.10 0.04 0.09 0.00
27 21 0.00 0.04 0.06 0.00
28 22 0.27 0.13 0.43 0.00
29 23 0.00 0.10 0.25 0.00
30 24 0.00 0.03 0.04 0.00
31 25 0.00 0.04 0.05 0.00
32 26 0.43 0.04 0.15 0.00
33 27 0.17 0.06 0.23 0.00
34 28 0.50 0.02 0.04 0.00
35 29 0.00 0.03 0.04 0.00
36 30 0.00 0.04 0.08 0.00
37 31 0.00 0.04 0.08 0.00
39 32 0.00 0.22 0.70 0.00
40 33 0.00 0.49 0.88 0.00
df[df['TIME'].shift().fillna(0) <= df['TIME']]
Gives what you're looking for. You were almost there with
df.loc[df.TIME == df.TIME.shift() +1]
But you don't need to get rid of cases where .shift is smaller, because that's just the first of the month.
The addition of .fillna(0) takes care of the NaN in the first row of df['TIME'].shift().
Edit:
For the end of year case, just be sure to also take those with a difference of 11, to catch where the 12th month ends.
That would give
df[(df['TIME'].shift().fillna(0) <= df['TIME']+11)]
Edit2:
By the by, I checked solution runtimes, and the current version(df[~df.TIME.diff().le(-12)]) of #piRSquared's seems to run fastest.
For completeness, of the one presented in this post and the original version posted by #piRSquared,
the former was a bit faster on datasets on the order of 10000 rows or fewer, the latter somewhat faster on those larger.
I have calculated a series of totals tips by day of a week and appended it to the bottom of totalspt dataframe.
I have set the index.name for the totalspt dataframe to None.
However while the dataframe is displaying the default 0,1,2,3 index it doesn't display the default empty cell in the top left directly above the index.
How could I make this cell empty in the dataframe?
total_bill tip sex smoker day time size tip_pct
0 16.54 1.01 F N Sun D 2 0.061884
1 12.54 1.40 F N Mon D 2 0.111643
2 10.34 3.50 M Y Tue L 4 0.338491
3 20.25 2.50 M Y Wed D 2 0.123457
4 16.54 1.01 M Y Thu D 1 0.061064
5 12.54 1.40 F N Fri L 2 0.111643
6 10.34 3.50 F Y Sat D 3 0.338491
7 23.25 2.10 M Y Sun B 3 0.090323
pivot = tips.pivot_table('total_bill', index=['sex', 'size'],columns=['day'],aggfunc='sum').fillna(0)
print pivot
day Fri Mon Sat Sun Thu Tue Wed
sex size
F 2 12.54 12.54 0.00 16.54 0.00 0.00 0.00
3 0.00 0.00 10.34 0.00 0.00 0.00 0.00
M 1 0.00 0.00 0.00 0.00 16.54 0.00 0.00
2 0.00 0.00 0.00 0.00 0.00 0.00 20.25
3 0.00 0.00 0.00 23.25 0.00 0.00 0.00
4 0.00 0.00 0.00 0.00 0.00 10.34 0.00
totals_row = tips.pivot_table('total_bill',columns=['day'],aggfunc='sum').fillna(0).astype('float')
totalpt = pivot.reset_index('sex').reset_index('size')
totalpt.index.name = None
totalpt = totalpt[['Fri', 'Mon','Sat', 'Sun', 'Thu', 'Tue', 'Wed']]
totalpt = totalpt.append(totals_row)
print totalpt
**day** Fri Mon Sat Sun Thu Tue Wed #problem text day
0 12.54 12.54 0.00 16.54 0.00 0.00 0.00
1 0.00 0.00 10.34 0.00 0.00 0.00 0.00
2 0.00 0.00 0.00 0.00 16.54 0.00 0.00
3 0.00 0.00 0.00 0.00 0.00 0.00 20.25
4 0.00 0.00 0.00 23.25 0.00 0.00 0.00
5 0.00 0.00 0.00 0.00 0.00 10.34 0.00
total_bill 12.54 12.54 10.34 39.79 16.54 10.34 20.25
That's the columns' name.
In [11]: df = pd.DataFrame([[1, 2]], columns=['A', 'B'])
In [12]: df
Out[12]:
A B
0 1 2
In [13]: df.columns.name = 'XX'
In [14]: df
Out[14]:
XX A B
0 1 2
You can set it to None to clear it.
In [15]: df.columns.name = None
In [16]: df
Out[16]:
A B
0 1 2
An alternative, if you wanted to keep it, is to give the index a name:
In [21]: df.columns.name = "XX"
In [22]: df.index.name = "index"
In [23]: df
Out[23]:
XX A B
index
0 1 2
You can use rename_axis. Since 0.17.0
In [3939]: df
Out[3939]:
XX A B
0 1 2
In [3940]: df.rename_axis(None, axis=1)
Out[3940]:
A B
0 1 2
In [3942]: df = df.rename_axis(None, axis=1)
In [3943]: df
Out[3943]:
A B
0 1 2