I want to display mean and standard deviation values above each of the boxplots in the grouped boxplot (see picture).
My code is
import pandas as pd
import seaborn as sns
from os.path import expanduser as ospath
df = pd.read_excel(ospath('~/Documents/Python/Kandidatspeciale/TestData.xlsx'),'Ark1')
bp = sns.boxplot(y='throw angle', x='incident angle',
data=df,
palette="colorblind",
hue='Bat type')
bp.set_title('Rubber Comparison',fontsize=15,fontweight='bold', y=1.06)
bp.set_ylabel('Throw Angle [degrees]',fontsize=11.5)
bp.set_xlabel('Incident Angle [degrees]',fontsize=11.5)
Where my dataframe, df, is
Bat type incident angle throw angle
0 euro 15 28.2
1 euro 15 27.5
2 euro 15 26.2
3 euro 15 27.7
4 euro 15 26.4
5 euro 15 29.0
6 euro 30 12.5
7 euro 30 14.7
8 euro 30 10.2
9 china 15 29.9
10 china 15 31.1
11 china 15 24.9
12 china 15 27.5
13 china 15 31.2
14 china 15 24.4
15 china 30 9.7
16 china 30 9.1
17 china 30 9.5
I tried with the following code. It needs to be independent of number of x (incident angles), for instance it should do the job for more angles of 45, 60 etc.
m=df.mean(axis=0) #Mean values
st=df.std(axis=0) #Standard deviation values
for i, line in enumerate(bp['medians']):
x, y = line.get_xydata()[1]
text = ' μ={:.2f}\n σ={:.2f}'.format(m[i], st[i])
bp.annotate(text, xy=(x, y))
Can somebody help?
This question brought me here since I was also looking for a similar solution with seaborn.
After some trial and error, you just have to change the for loop to:
for i in range(len(m)):
bp.annotate(
' μ={:.2f}\n σ={:.2f}'.format(m[i], st[i]),
xy=(i, m[i]),
horizontalalignment='center'
)
This change worked for me (although I just wanted to print the actual median values). You can also add changes like the fontsize, color or style (i.e., weight) just by adding them as arguments in annotate.
Related
i want to plot the data which is shown below and compere it to a function which gives me the theoretical plot. I am able to plot the data with its uncertainty, but i am struguling to plot the mathematical function function which gives me the theoretical plot.
amplitude uncertainty position
5.2 0.429343685 0
12.2 1.836833144 1
21.4 0.672431409 2
30.2 0.927812481 3
38.2 1.163321108 4
44.2 1.340998136 5
48.4 1.506088975 6
51 1.543016526 7
51.2 1.587229032 8
49.8 1.507327436 9
46.2 1.400355669 10
40.6 1.254401849 11
32.5 0.995301462 12
24.2 0.753044487 13
14 0.58 14
7 0.29 15
here is my code so far:
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
data = pd.read_excel("Verdier_6.xlsx")
verdier = data.values
frekvens = verdier [:,3]
effektresonans = verdier [:,0]
usikkerhet = verdier [:,1]
x = np.arange(0,15,0.1)
p= 28.2
r=0.8156
v= 343.8
f= 1117
y=p*np.sqrt(1+r**2+2*r*np.cos(((2*np.pi)/(v/f))*x))
plt.plot(x,y)
plt.plot(frekvens, effektresonans)
plt.errorbar(frekvens, effektresonans, usikkerhet, fmt = "o")
plt.title("")
plt.xlabel("Posisjon, X [cm]")
plt.ylabel("Amplitude, U [mV] ")
plt.grid()
plt.show()
And here is here is a image of the plot with only experimental data shown above:
and here is an image of how my experimental and theoretical plot look:
and here is an image of how the experimental and theoretical plot should look:
I have a mass pandas DataFrame df:
year count
1983 5
1983 4
1983 7
...
2009 8
2009 11
2009 30
and I aim to sample 10 data points per year 100 times and get the mean and standard deviation of count per year. The signs of the count values are determined randomly.
I want to randomly sample 10 data per year, which can be done by:
new_df = pd.DataFrame(columns=['year', 'count'])
ref = df.year.unique()
for i in range(len(ref)):
appended_df = df[df['year'] == ref[i]].sample(n=10)
new_df = pd.concat([new_df,appended_df])
Then, I assign a sign to count randomly (so that by random chance the count could be positive or negative) and rename it to value, which can be done by:
vlist = []
for i in range(len(new_df)):
if randint(0,1) == 0:
vlist.append(new_df.count.iloc[i])
else:
vlist.append(new_df.count.iloc[i] * -1)
new_data['value'] = vlist
Getting a mean and standard deviation per each year is quite simple:
xdf = new_data.groupby("year").agg([np.mean, np.std]).reset_index()
But I can't seem to find an optimal way to try this sampling 100 times per year, store the mean values, and get the mean and standard deviation of those 100 means per year. I could think of using for loop, but it would take too much of a runtime.
Essentially, the output should be in the form of the following (the values are arbitrary here):
year mean_of_100_means total_sd
1983 4.22 0.43
1984 -6.39 1.25
1985 2.01 0.04
...
2007 11.92 3.38
2008 -5.27 1.67
2009 1.85 0.99
Any insights would be appreciated.
Try:
def fn(x):
_100_means = [x.sample(10).mean() for i in range(100)]
return {
"mean_of_100_means": np.mean(_100_means),
"total_sd": np.std(_100_means),
}
print(df.groupby("year")["count"].apply(fn).unstack().reset_index())
EDIT: Changed the computation of means.
Prints:
year mean_of_100_means total_sd
0 1983 48.986 8.330787
1 1984 48.479 10.384896
2 1985 48.957 7.854900
3 1986 50.821 10.303847
4 1987 50.198 9.835832
5 1988 47.497 8.678749
6 1989 46.763 9.197387
7 1990 49.696 8.837589
8 1991 46.979 8.141969
9 1992 48.555 8.603597
10 1993 50.220 8.263946
11 1994 48.735 9.954741
12 1995 49.759 8.532844
13 1996 49.832 8.998654
14 1997 50.306 9.038316
15 1998 49.513 9.024341
16 1999 50.532 9.883166
17 2000 49.195 9.177008
18 2001 50.731 8.309244
19 2002 48.792 9.680028
20 2003 50.251 9.384759
21 2004 50.522 9.269677
22 2005 48.090 8.964458
23 2006 49.529 8.250701
24 2007 47.192 8.682196
25 2008 50.124 9.337356
26 2009 47.988 8.053438
The dataframe was created:
data = []
for y in range(1983, 2010):
for i in np.random.randint(0, 100, size=1000):
data.append({"year": y, "count": i})
df = pd.DataFrame(data)
I think you can use pandas groupby and sample functions together to take 10 samples from each year of your DataFrame. If you put this in a loop, then you can sample it 100 times, and combine the results.
It sounds like you only need the standard deviation of the 100 means (and you don't need the standard deviation of the sample of 10 observations), so you can calculate only the mean in your groupby and sample, then calculate the standard deviation from each of those 100 means when you are creating the total_sd column of your final DataFrame.
import numpy as np
import pandas as pd
np.random.seed(42)
## create a random DataFrame with 100 entries for the years 1980-1999, length 2000
df = pd.DataFrame({
'year':[year for year in list(range(1980, 2000)) for _ in range(100)],
'count':np.random.randint(1,100,size=2000)
})
list_of_means = []
## sample 10 observations from each year, and repeat this process 100 times, storing the mean for each year in a list
for _ in range(100):
df_sample = df.groupby("year").sample(10).groupby("year").mean()
list_of_means.append(df_sample['count'].tolist())
array_of_means = [np.array(x) for x in list_of_means]
result = pd.DataFrame({
'year': df.year.unique(),
'mean_of_100_means': [np.mean(k) for k in zip(*array_of_means)],
'total_sd': [np.std(k) for k in zip(*array_of_means)]
})
This results in:
>>> result
year mean_of_100_means total_sd
0 1980 50.316 8.656948
1 1981 48.274 8.647643
2 1982 47.958 8.598455
3 1983 49.357 7.854620
4 1984 48.977 8.523484
5 1985 49.847 7.114485
6 1986 47.338 8.220143
7 1987 48.106 9.413085
8 1988 53.487 9.237561
9 1989 47.376 9.173845
10 1990 46.141 9.061634
11 1991 46.851 7.647189
12 1992 49.389 7.743318
13 1993 52.207 9.333309
14 1994 47.271 8.177815
15 1995 52.555 8.377355
16 1996 47.606 8.668769
17 1997 52.584 8.200558
18 1998 51.993 8.695232
19 1999 49.054 8.178929
i have been looking around and i can find examples for annotating a single line chart by using iterrows for the dataframe. what i am struggling with is
a) selecting the single line in the plot instead of ax.lines (using ax.lines[#]) is clearly not proper and
b) annotating the values for the line with values from a different column
the dataframe dfg is in a format such that (edited to provide a minimal, reproducible example):
week 2016 2017 2018 2019 2020 2021 min max avg WoW Change
1 8188.0 9052.0 7658.0 7846.0 6730.0 6239.0 6730 9052 7893.7
2 7779.0 8378.0 7950.0 7527.0 6552.0 6045.0 6552 8378 7588.0 -194.0
3 7609.0 7810.0 8041.0 8191.0 6432.0 5064.0 6432 8191 7529.4 -981.0
4 8256.0 8290.0 8430.0 7083.0 6660.0 6507.0 6660 8430 7687.0 1443.0
5 7124.0 9372.0 7892.0 7146.0 6615.0 5857.0 6615 9372 7733.7 -650.0
6 7919.0 8491.0 7888.0 6210.0 6978.0 5898.0 6210 8491 7455.3 41.0
7 7802.0 7286.0 7021.0 7522.0 6547.0 4599.0 6547 7802 7218.1 -1299.0
8 8292.0 7589.0 7282.0 5917.0 6217.0 6292.0 5917 8292 7072.3 1693.0
9 8048.0 8150.0 8003.0 7001.0 6238.0 5655.0 6238 8150 7404.0 -637.0
10 7693.0 7405.0 7585.0 6746.0 6412.0 5323.0 6412 7693 7135.1 -332.0
11 8384.0 8307.0 7077.0 6932.0 6539.0 6539 8384 7451.7
12 7748.0 8224.0 8148.0 6540.0 6117.0 6117 8224 7302.6
13 7254.0 7850.0 7898.0 6763.0 6047.0 6047 7898 7108.1
14 7940.0 7878.0 8650.0 6599.0 5874.0 5874 8650 7352.1
15 8187.0 7810.0 7930.0 5992.0 5680.0 5680 8187 7066.6
16 7550.0 8912.0 8469.0 7149.0 4937.0 4937 8912 7266.6
17 7660.0 8264.0 8549.0 7414.0 5302.0 5302 8549 7291.4
18 7655.0 7620.0 7323.0 6693.0 5712.0 5712 7655 6910.0
19 7677.0 8590.0 7601.0 7612.0 5391.0 5391 8590 7264.6
20 7315.0 8294.0 8159.0 6943.0 5197.0 5197 8294 7057.0
21 7839.0 7985.0 7631.0 6862.0 7200.0 6862 7985 7480.6
22 7705.0 8341.0 8346.0 7927.0 6179.0 6179 8346 7574.7
... ... ... ... ... ... ... ... ...
51 8167.0 7993.0 7656.0 6809.0 5564.0 5564 8167 7131.4
52 7183.0 7966.0 7392.0 6352.0 5326.0 5326 7966 6787.3
53 5369.0 5369 5369 5369.0
with the graph plotted by:
fig, ax = plt.subplots(1, figsize=[14,4])
ax.fill_between(dfg.index, dfg["min"], dfg["max"], label="5 Yr. Range", facecolor="oldlace")
ax.plot(dfg.index, dfg[2020], label="2020", c="grey")
ax.plot(dfg.index, dfg[2021], label="2021", c="coral")
ax.plot(dfg.index, dfg.avg, label="5 Yr. Avg.", c="goldenrod", ls=(0,(1,2)), lw=3).
I would like to label the dfg[2021] line with the values from dfg['WoW Change']. Additionally, if anyone knows how to get the calculate the first value in the WoW column based on the last value from 2020 and the first value from 2021, that would be wonderful! It's currently just dfg['WoW Change'] = dfg[2021].diff()
Thanks!
Figured it out. Zipped the index and two columns up as a tuple. I ended up deciding I only wanted the last value to be shown but using below code:
a = dfg.index.values
b = dfg[2021]
c = dfg['WoW Change']
#zip 3 columns separately
labels = list(zip(dfg.index.values,dfg[2021],dfg['WoW Change']))
#remove tuples with index + 2 nan values
labels_light = [i for i in labels if not any(isinstance(n,float) and math.isnan(n) for n in i)]
#label last point using list accessors
ax.annotate(str("w/w change: " + str("{:,}".format(int(labels_light[-1][2])))+link[1]),xy=(labels_light[-1][0],labels_light[-1][1]))
I'm sure this could have been done much better by someone who knows what they're doing, any feedback is appreciated.
Using Python 3, I work with a data frame which requires text preprocessing.
The data frame consists of historical sales for many different medical products with many different strengths. For simplification, the code below only shows a part of the strength column.
df = pd.DataFrame({'Strength': ['20 mg / 120 mg', ' 40/320 mg', '20mg/120mg', '150+750mg', '20/120MG', '62.5mg/375mg', '100 mg', 'Product1 20 mg, Product2 120 mg', '40mg/320mg', 'Product 20mg/120mg', 'Product1 20mg Product2 120mg', '100mg/1ml', '20 mg./ 120 mg..', '62.5 mg / 375 mg', '40/320mg 9s', '40/320', '50/125', '100mg..' '20/120']})
Strength
0 20 mg / 120 mg
1 40/320 mg
2 20mg/120mg
3 150+750mg
4 20/120MG
5 62.5mg/375mg
6 100 mg
7 Product1 20 mg, Product2 120 mg
8 40mg/320mg
9 Product 20mg/120mg
10 Product1 20mg Product2 120mg
11 100mg/1ml
12 20 mg./ 120 mg..
13 62.5 mg / 375 mg
14 40/320mg 9s
15 40/320
16 50/125
17 100mg..20/120
As you can see, there are different spellings for products which actually belong to the same Strength. For example, '20 mg / 120 mg' and 'Artemether 20 mg, Lumefantrine 120 mg' actually have the same strength.
Setting the text to lowercase, removing whitespaces and replacing + by / shown by the following code brings some standardization, but there are still lines with clearly the same strength.
df['Strength'] = df['Strength'].str.lower()
df['Strength'] = df['Strength'].str.replace(' ', '')
df['Strength'] = df['Strength'].str.replace('+', '/')
Adding commands like the following allows to further reduce the number of different notations, but this is way too manual.
df['Strength'].loc[df['Strength'].str.contains('Product1', case=False)
& df['Strength'].str.contains('Product2', case=False)] = '20mg/120mg'
Do you have any approaches for removing the number of unique notations in an efficient way?
Add a new column with fixed labels for each strength and train it based on a suitable ml classifier and predict the appropriate strength for the new item.
For each new notation, manually assign a new label and retrain again...
I am trying to make a model for predicting energy production, by using ARMA model.
The data I can use for training is as following;
（https://github.com/soma11soma11/EnergyDataSimulationChallenge/blob/master/challenge1/data/training_dataset_500.csv）
ID Label House Year Month Temperature Daylight EnergyProduction
0 0 1 2011 7 26.2 178.9 740
1 1 1 2011 8 25.8 169.7 731
2 2 1 2011 9 22.8 170.2 694
3 3 1 2011 10 16.4 169.1 688
4 4 1 2011 11 11.4 169.1 650
5 5 1 2011 12 4.2 199.5 763
...............
11995 19 500 2013 2 4.2 201.8 638
11996 20 500 2013 3 11.2 234 778
11997 21 500 2013 4 13.6 237.1 758
11998 22 500 2013 5 19.2 258.4 838
11999 23 500 2013 6 22.7 122.9 586
As shown above, I can use data from July 2011 to May 2013 for training.
Using the training, I want to predict energy production on June 2013 for each 500 house.
The problem is that the time series data is not stationary and has trend components and seasonal components (I checked it as following.).
import csv
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
data_train = pd.read_csv('../../data/training_dataset_500.csv')
rng=pd.date_range('7/1/2011', '6/1/2013', freq='M')
house1 = data_train[data_train.House==1][['EnergyProduction','Daylight','Temperature']].set_index(rng)
fig, axes = plt.subplots(nrows=1, ncols=3)
for i, column in enumerate(house1.columns):
house1[column].plot(ax=axes[i], figsize=(14,3), title=column)
plt.show()
With this data, I cannot implement ARMA model to get good prediction. So I want to get rid of the trend components and a seasonal components and make the time series data stationary. I tried this problem, but I could not remove these components and make it stationary..
I would recommend the Hodrick-Prescott (HP) filter, which is widely used in macroeconometrics to separate long-term trending component from short-term fluctuations. It is implemented statsmodels.api.tsa.filters.hpfilter.
import pandas as pd
import matplotlib.pyplot as plt
import statsmodels.api as sm
df = pd.read_csv('/home/Jian/Downloads/data.csv', index_col=[0])
# get part of the data
x = df.loc[df.House==1, 'Daylight']
# hp-filter, set parameter lamb=129600 following the suggestions for monthly data
x_smoothed, x_trend = sm.tsa.filters.hpfilter(x, lamb=129600)
fig, axes = plt.subplots(figsize=(12,4), ncols=3)
axes[0].plot(x)
axes[0].set_title('raw x')
axes[1].plot(x_trend)
axes[1].set_title('trend')
axes[2].plot(x_smoothed)
axes[2].set_title('smoothed x')