Excuse my bad english.
On a DataFrame like the following one :
-----------------
|index|var1|var2|
-----------------
there is lot of rows
var1 is between 0 and 4000
var2 is between -100 and 100
I'm looking to create an histogram that show how many rows there is according to var1.
On the Y axis, we can see how many rows there is, for example for 0 > var1 < 500, there is almost 500k rows.
Now I want to add var2, which show the quality of a row.
I want that for example the histgram become blue from 0 to 500 and another color from 500 to 1000 according to the value of var2 (like if the bar as values where the mean of var2 is 100, make it green, if the mean is 0, make it red).
I tried to hardcore this, but as soon as I change the bins or anything, my code break.
I also tried to do it using plot on the top of the hist, but it doesn't work.
My current code for the screenshot :
plt.hist(var1, bins=10, range=(0,4000), color='orange', alpha=0.7)
plt.title('Var 1',weight='bold', fontsize=18)
plt.yticks(weight='bold')
plt.xticks(weight='bold')
I feel like this is simple things to do, but I'm completely stuck in my learning because of this.
Many thanks for your help.
If you create a list containing the colors for each bar in your histogram you can use the following code snippet. It catches the return values of the plt.hist command, which include the individual patches. The color can be set individually while iterating through those patches.
n, bins, patches = plt.hist(var1, bins=8, range=(0,4000), color="orange", alpha=0.7)
for i, patch in enumerate(patches):
plt.setp(patch, "facecolor", colors[i])
Additionally, here is one possible way to create the mentioned color list based on the kind of data you have:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# create random values and store them in a DataFrame
y1 = np.random.randint(0,4000, 50)
y2 = np.random.randint(-100, 101, 50)
y = zip(y1,y2)
df = pd.DataFrame(y, columns=["Var1","Var2"])
var1 = df["Var1"].values
# pd.cut to bin the dataframe in the appropriate ranges of Var1
# then the mean of Var2 is calculated for each bin, results are stored in a list
mean = [df.groupby(pd.cut(df["Var1"], np.arange(0, 4000+500, 500)))["Var2"].mean()]
# how to color the bars based on Var2:
# -100 <= mean(Var2) < -33: blue
# -33 <= mean(Var2) < 33: red
# 33 <= mean(Var2) < 100: green
color_bins = np.array([-100,-33,33,100])
color_list = ["blue","red","green"]
# bin the means of Var2 according to the color_bins we just created
inds = np.digitize(mean, color_bins)
# list that assigns the appropriate color to each patch
colors = [color_list[value-1] for value in inds[0]]
n, bins, patches = plt.hist(var1, bins=8, range=(0,4000), color="orange", alpha=0.7)
for i, patch in enumerate(patches):
plt.setp(patch, "facecolor", colors[i])
plt.title('Var 1',weight='bold', fontsize=18)
plt.yticks(weight='bold')
plt.xticks(weight='bold')
plt.show()
Related
I have a DataFrame like below. It has Actual and Predicted columns. I want to compare Actual Vs Predicted in Bar plot in one on one. I have confidence value for Predicted column and default for Actual confidence is 1. So, I want to keep Each row in single bar group Actual and Predicted value will be X axis and corresponding Confidence score as y value.
I am unable to get the expected plot because X axis values are not aligned or grouped to same value in each row.
Actual Predicted Confidence
0 A A 0.90
1 B C 0.30
2 C C 0.60
3 D D 0.75
Expected Bar plot.
Any hint would be appreciable. Please let me know if further details required.
What I have tried so far.
df_actual = pd.DataFrame()
df_actual['Key']= df['Actual'].copy()
df_actual['Confidence'] = 1
df_actual['Identifier'] = 'Actual'
df_predicted=pd.DataFrame()
df_predicted = df[['Predicted', 'Confidence']]
df_predicted = df_predicted.rename(columns={'Predicted': 'Key'})
df_predicted['Identifier'] = 'Predicted'
df_combined = pd.concat([df_actual,df_predicted], ignore_index=True)
df_combined
fig = px.bar(df_combined, x="Key", y="Confidence", color='Identifier',
barmode='group', height=400)
fig.show()
I have found that adjusting the data first makes it easier to get the plot I want. I have used Seaborn, hope that is ok. Please see if this code works for you. I have considered that the df mentioned above is already available. I created df2 so that it aligns to what you had shown in the expected figure. Also, I used index as the X-axis column so that the order is maintained... Some adjustments to ensure xtick names align and the legend is outside as you wanted it.
Code
vals= []
conf = []
for x, y, z in zip(df.Actual, df.Predicted, df.Confidence):
vals += [x, y]
conf += [1, z]
df2 = pd.DataFrame({'Values': vals, 'Confidence':conf}).reset_index()
ax=sns.barplot(data = df2, x='index', y='Confidence', hue='Values',dodge=False)
ax.set_xticklabels(['Actual', 'Predicted']*4)
plt.legend(bbox_to_anchor=(1.0,1))
plt.show()
Plot
Update - grouping Actual and Predicted bars
Hi #Mohammed - As we have already used up hue, I don't think there is a way to do this easily with Seaborn. You would need to use matplotlib and adjust the bar position, xtick positions, etc. Below is the code that will do this. You can change SET1 to another color map to change colors. I have also added a black outline as the same colored bars were blending into one another. Further, I had to rotate the xlables, as they were on top of one another. You can change it as per your requirements. Hope this helps...
vals = df[['Actual','Predicted']].melt(value_name='texts')['texts']
conf = [1]*4 + list(df.Confidence)
ident = ['Actual', 'Predicted']*4
df2 = pd.DataFrame({'Values': vals, 'Confidence':conf, 'Identifier':ident}).reset_index()
uvals, uind = np.unique(df2["Values"], return_inverse=1)
cmap = plt.cm.get_cmap("Set1")
fig, ax=plt.subplots()
l = len(df2)
pos = np.arange(0,l) % (l//2) + (np.arange(0,l)//(l//2)-1)*0.4
ax.bar(pos, df2["Confidence"], width=0.4, align="edge", ec="k",color=cmap(uind) )
handles=[plt.Rectangle((0,0),1,1, color=cmap(i), ec="k") for i in range(len(uvals))]
ax.legend(handles=handles, labels=list(uvals), prop ={'size':10}, loc=9, ncol=8)
pos=pos+0.2
pos.sort()
ax.set_xticks(pos)
ax.set_xticklabels(df2["Identifier"][:l], rotation=45,ha='right', rotation_mode="anchor")
ax.set_ylim(0, 1.2)
plt.show()
Output plot
I updated #Redox answer to get the exact output.
df_ = pd.DataFrame({'Labels': df.reset_index()[['Actual', 'Predicted', 'index']].values.ravel(),
'Confidence': np.array(list(zip(np.repeat(1, len(df)), df['Confidence'].values, np.repeat(0, len(df))))).ravel()})
df_.loc[df_['Labels'].astype(str).str.isdigit(), 'Labels'] = ''
plt.figure(figsize=(15, 6))
ax=sns.barplot(data = df_, x=df_.index, y='Confidence', hue='Labels',dodge=False, ci=None)
ax.set_xticklabels(['Actual', 'Predicted', '']*len(df))
plt.setp(ax.get_xticklabels(), rotation=90)
ax.tick_params(labelsize=14)
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
Output:
Removed loop to improve performance
Added blank bar values to look alike group chart.
So, I have made a stripplot with seaborn the easiest way, with 5 different categories:
sns.set_style('whitegrid')
plt.figure(figsize=(35,20))
sns.set(font_scale = 3)
sns.stripplot(df.speed, df.routeID, hue=df.speed>50, jitter=0.2, alpha=0.5, size=10, edgecolor='black')
plt.xlabel("Speed", size=40)
plt.ylabel("route ID", size=40)
plt.title("Velocity stripplot", size=50)
Now, the thing is I want to have a different hue for each category, say speed greater than 50 kmh for first category, 30 kmh for second and so on. Is this possible? I tried to do it passing a list for hue:
hue=([("ROUTE 30">50),("ROUTE 104">0)])
but it marks: SyntaxError: invalid syntax
The thing is, I want to do it all at once (since the most obvious answer would be to plot separately) in the same plot, how can this be done?
EDIT: I followed the suggested answer. Used the same code:
plt.figure(figsize=(20,7))
my_palette = ['b' if x > 82 else 'g' for x in df.speed.values]
sns.stripplot(df.speed, df.routeID, jitter=0.2, alpha=0.5, size=8, edgecolor='black', palette = my_palette)
but didnt turned out like expected:
I dont understand what is wrong here. Any ideas?
I suggest to create separate column in df for dot color.
try this:
# INITIAL DATA
n = 1000
df = pd.DataFrame()
df['speed'] = np.random.randint(10,90,n)
df['routeID'] = np.random.choice(['ROUTE_5','ROUTE_66','ROUTE_95','ROUTE_101'], n)
# set hue indices to match your conditions
df['hue'] = 'normal' # new column with default value
df.loc[df.speed > 50, 'hue'] = 'fast'
df.loc[(df.routeID=="ROUTE_5") & (df.speed>40)|
(df.routeID=="ROUTE_66") & (df.speed>30)|
(df.routeID=="ROUTE_95") & (df.speed>60),
'hue'] = 'special'
palette = {'normal':'g','fast':'r','special':'magenta'}
sns.stripplot(x=df.speed, y=df.routeID, size=15,
hue=df.hue, palette=palette)
GOAL: I want to make a distribution function for registered dogs' ages in 2017 in Zurich from the 'Dogs of Zurich' dataset (Kaggle) (with Python). The variable I'm working with - 'GEBURTSJAHR_HUND' - gives the birth year for every registered dog as an int.
I have converted it to a 'dog_age' variable (= 2017 - birth_date) and want to plot the distribution function. See image below for sorted list of group size per age.
Size of dog age groups
PROBLEM: I'm running into is the fact that my distribution function's x axis has empty spaces/bars in it. Every age is shown on the graph, but in between some of these ages are empty bars.
Example: 1 and 2 are full bars, but between them is an empty space. Between 2 and 3, there is no empty space, but between 3 and 4 there is. Seemingly random which values have white spaces between them.
What my problematic distribution plot looks like at the moment
TRIED: I have previously tried three things to fix this.
plt.xticks(...)
Unfortunately this only changed the aesthetics of the x axis.
Tried ax = sns.distplot followed by ax.xaxis ticker lines, but this did not have the expected result.
ax.xaxis.set_major_locator(ticker.MultipleLocator())
ax.xaxis.set_major_formatter(ticker.ScalarFormatter(0))
Maybe problem is with 'dog_age' variable?
Used the original birth_date variable, but this had the same problem.
CODE:
dfnew = pd.read_csv(dog17_filepath,index_col='HALTER_ID')
dfnew.dropna(subset = ["ALTER"], inplace=True)
dfnew['dog_age'] = 2017 - dfnew['GEBURTSJAHR_HUND']
b = dfnew['dog_age']
sns.set_style("darkgrid")
plt.figure(figsize=(15,5))
sns.distplot(a=b,hist=True)
plt.xticks(np.arange(min(b), max(b)+1, 1))
plt.xlabel('Age Dog', fontsize=12)
plt.title('Distribution of age of dogs', fontsize=20)
plt.show()
Thanks in advance,
Arthur
The problem is that the age column is discrete: it only contains a short range of integers. Default the histogram divides the range of values (float) into a fixed number of bins, which usually don't align well with those integers. To get an appropriate histogram, the bins needs to be set explicitly, for example having a bin bound at every half.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
dfnew = pd.read_csv('hundehalter.csv')
dfnew.dropna(subset=["ALTER"], inplace=True)
dfnew['dog_age'] = 2017 - dfnew['GEBURTSJAHR_HUND']
b = dfnew['dog_age'][(dfnew['dog_age'] >= 0) & (dfnew['dog_age'] <= 25)]
sns.set_style("darkgrid")
plt.figure(figsize=(15, 5))
sns.distplot(a=b, hist=True, bins=np.arange(min(b)-0.5, max(b)+1, 1))
plt.xticks(np.arange(min(b), max(b) + 1, 1))
plt.xlabel('Age Dog', fontsize=12)
plt.title('Distribution of age of dogs', fontsize=20)
plt.xlim(min(b), max(b) + 1)
plt.show()
Currently, I'm working on an introductory paper on data manipulation and such; however... the CSV I'm working on has some things I wish to do a scatter graph on!
I want a scatter graph to show me the volume sold on certain items as well as their average price, differentiating all data according to their region (Through colours I assume).
So what I want is to know if I can add the region column as a quantitative value
or if there's a way to make this possible...
It's my first time using Python and I'm confused way too often
I'm not sure if this is what you mean, but here is some working code, assuming you have data in the format of [(country, volume, price), ...]. If not, you can change the inputs to the scatter method as needed.
import random
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
n_countries = 50
# get the data into "countries", for example
countries = ...
# in this example: countries is [('BS', 21, 25), ('WZ', 98, 25), ...]
df = pd.DataFrame(countries)
# arbitrary method to get a color
def get_color(i, max_i):
cmap = matplotlib.cm.get_cmap('Spectral')
return cmap(i/max_i)
# get the figure and axis - make a larger figure to fit more points
# add labels for metric names
def get_fig_ax():
fig = plt.figure(figsize=(14,14))
ax = fig.add_subplot(1, 1, 1)
ax.set_xlabel('volume')
ax.set_ylabel('price')
return fig, ax
# switch around the assignments depending on your data
def get_x_y_labels():
x = df[1]
y = df[2]
labels = df[0]
return x, y, labels
offset = 1 # offset just so annotations aren't on top of points
x, y, labels = get_x_y_labels()
fig, ax = get_fig_ax()
# add a point and annotation for each of the labels/regions
for i, region in enumerate(labels):
ax.annotate(region, (x[i] + offset, y[i] + offset))
# note that you must use "label" for "legend" to work
ax.scatter(x[i], y[i], color=get_color(i, len(x)), label=region)
# Add the legend just outside of the plot.
# The .1, 0 at the end will put it outside
ax.legend(loc='upper right', bbox_to_anchor=(1, 1, .1, 0))
plt.show()
I have the following dataframe and would like to differentiate the minor decimal differences in each "step" with a different color scheme in a heatmap.
Sample data:
Sample Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8
A 64.847 54.821 20.897 39.733 23.257 74.942 75.945
B 64.885 54.767 20.828 39.613 23.093 74.963 75.928
C 65.036 54.772 20.939 39.835 23.283 74.944 75.871
D 64.869 54.740 21.039 39.889 23.322 74.925 75.894
E 64.911 54.730 20.858 39.608 23.101 74.956 75.930
F 64.838 54.749 20.707 39.394 22.984 74.929 75.941
G 64.887 54.781 20.948 39.748 23.238 74.957 75.909
H 64.903 54.720 20.783 39.540 23.028 74.898 75.911
I 64.875 54.761 20.911 39.695 23.082 74.897 75.866
J 64.839 54.717 20.692 39.377 22.853 74.849 75.939
K 64.857 54.736 20.934 39.699 23.130 74.880 75.903
L 64.754 54.746 20.777 39.536 22.991 74.877 75.902
M 64.798 54.811 20.963 39.824 23.187 74.886 75.895
An example of what I am looking for:
My first approach would be based on a figure with multiple subplots. Number of plots would equal number of columns in your dataframe; the gap between the plots could be shrinked down to zero:
cm = ['Blues', 'Reds', 'Greens', 'Oranges', 'Purples', 'bone', 'winter']
f, axs = plt.subplots(1, df.columns.size, gridspec_kw={'wspace': 0})
for i, (s, a, c) in enumerate(zip(df.columns, axs, cm)):
sns.heatmap(np.array([df[s].values]).T, yticklabels=df.index, xticklabels=[s], annot=True, fmt='.2f', ax=a, cmap=c, cbar=False)
if i>0:
a.yaxis.set_ticks([])
Result:
Not sure if this will lead to a helpful or even self describing visualization of data, but that's your choice - perhaps this helps to start...
Supplemental:
Regarding adding the colorbars: of course you can. But - besides not knowing the background of your data and the purpose of the visualization - I'd like to add some thoughts on all that:
First: adding all those colorbars as a separate bunch of bars on one side or below the heatmap is probably possible, but I find it already quite hard to read the data, plus: you already have all those annotations - it would mess all up I think.
Additionally: in the meantime #ImportanceOfBeingErnest provided such a beutiful solution on that topic, that this would be not too meaningful imo here.
Second: if you really want to stick to the heatmap thing, perhaps splitting up and giving every column its colorbar would suit better:
cm = ['Blues', 'Reds', 'Greens', 'Oranges', 'Purples', 'bone', 'winter']
f, axs = plt.subplots(1, df.columns.size, figsize=(10, 3))
for i, (s, a, c) in enumerate(zip(df.columns, axs, cm)):
sns.heatmap(np.array([df[s].values]).T, yticklabels=df.index, xticklabels=[s], annot=True, fmt='.2f', ax=a, cmap=c)
if i>0:
a.yaxis.set_ticks([])
f.tight_layout()
However, all that said - I dare to doubt that this is the best visualization for your data. Of course, I don't know what you want to say, see or find with these plots, but that's the point: if the visualization type would fit to the needs, I guess I'd know (or at least could imagine).
Just for example:
A simple df.plot() results in
and I feel that this tells more about different characteristics of your columns within some tenths of a second than the heatmap.
Or are you explicitely after the differences to each columns' means?
(df - df.mean()).plot()
... or the distribution of each column around them?
(df - df.mean()).boxplot()
What I want to say: data visualization becomes powerful when a plot begins to tell sth about the underlying data before you begin/have to explain anything...
I suppose the problem can be divided into several parts.
Getting several heatmaps with different colormaps into the same picture. This can be done masking the complete array column-wise, plot each masked array seperately via imshow and apply a different colormap. To visualize the concept:
Obtaining variable number of distinct colormaps. Matplotlib provides a large number of colormaps, however, they are in general very different concerning luminosity and saturation. Here it seems desireable to have colormaps of differing hue, but otherwise same saturation and luminosity.
An option is to create the colormaps on the fly, choosing n different (and equally spaced) hues, and create a colormap using the same saturation and luminosity.
Obtaining a distinct colorbar for each column. Since the values within columns might be on totally different scales, a colorbar for each column would be needed to know the values shown, e.g. in the first column the brightest color may correspond to a value of 1, while in the second column it may correspond to a value of 100. Several colorbars can be created inside of the axes of a GridSpec which is placed next to the actual heatmap axes. The number of columns and rows of that gridspec would be dependent of the number of columns in the dataframe.
In total this may then look as follows.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
from matplotlib.gridspec import GridSpec
def get_hsvcmap(i, N, rot=0.):
nsc = 24
chsv = mcolors.rgb_to_hsv(plt.cm.hsv(((np.arange(N)/N)+rot) % 1.)[i,:3])
rhsv = mcolors.rgb_to_hsv(plt.cm.Reds(np.linspace(.2,1,nsc))[:,:3])
arhsv = np.tile(chsv,nsc).reshape(nsc,3)
arhsv[:,1:] = rhsv[:,1:]
rgb = mcolors.hsv_to_rgb(arhsv)
return mcolors.LinearSegmentedColormap.from_list("",rgb)
def columnwise_heatmap(array, ax=None, **kw):
ax = ax or plt.gca()
premask = np.tile(np.arange(array.shape[1]), array.shape[0]).reshape(array.shape)
images = []
for i in range(array.shape[1]):
col = np.ma.array(array, mask = premask != i)
im = ax.imshow(col, cmap=get_hsvcmap(i, array.shape[1], rot=0.5), **kw)
images.append(im)
return images
### Create some dataset
ind = list("ABCDEFGHIJKLM")
m = len(ind)
n = 8
df = pd.DataFrame(np.random.randn(m,n) + np.random.randint(20,70,n),
index=ind, columns=[f"Step {i}" for i in range(2,2+n)])
### Plot data
fig, ax = plt.subplots(figsize=(8,4.5))
ims = columnwise_heatmap(df.values, ax=ax, aspect="auto")
ax.set(xticks=np.arange(len(df.columns)), yticks=np.arange(len(df)),
xticklabels=df.columns, yticklabels=df.index)
ax.tick_params(bottom=False, top=False,
labelbottom=False, labeltop=True, left=False)
### Optionally add colorbars.
fig.subplots_adjust(left=0.06, right=0.65)
rows = 3
cols = len(df.columns) // rows + int(len(df.columns)%rows > 0)
gs = GridSpec(rows, cols)
gs.update(left=0.7, right=0.95, wspace=1, hspace=0.3)
for i, im in enumerate(ims):
cax = fig.add_subplot(gs[i//cols, i % cols])
fig.colorbar(im, cax = cax)
cax.set_title(df.columns[i], fontsize=10)
plt.show()