We Keep Coding

How to avoid overlapping error bars in matplotlib?

I want to create a plot for two different datasets similar to the one presented in this answer:
In the above image, the author managed to fix the overlapping problem of the error bars by adding some small random scatter in x to the new dataset.
In my problem, I must plot a similar graphic, but having some categorical data in the x axis:
Any ideas on how to slightly move one the error bars of the second dataset using categorical variables at the x axis? I want to avoid the overlapping between the bars for making the visualization easier.

You can translate each errorbar by adding the default data transform to a prior translation in data space. This is possible when knowing that categories are in general one data unit away from each other.
import numpy as np; np.random.seed(42)
import matplotlib.pyplot as plt
from matplotlib.transforms import Affine2D
x = list("ABCDEF")
y1, y2 = np.random.randn(2, len(x))
yerr1, yerr2 = np.random.rand(2, len(x))*4+0.3
fig, ax = plt.subplots()
trans1 = Affine2D().translate(-0.1, 0.0) + ax.transData
trans2 = Affine2D().translate(+0.1, 0.0) + ax.transData
er1 = ax.errorbar(x, y1, yerr=yerr1, marker="o", linestyle="none", transform=trans1)
er2 = ax.errorbar(x, y2, yerr=yerr2, marker="o", linestyle="none", transform=trans2)
plt.show()
Alternatively, you could translate the errorbars after applying the data transform and hence move them in units of points.
import numpy as np; np.random.seed(42)
import matplotlib.pyplot as plt
from matplotlib.transforms import ScaledTranslation
x = list("ABCDEF")
y1, y2 = np.random.randn(2, len(x))
yerr1, yerr2 = np.random.rand(2, len(x))*4+0.3
fig, ax = plt.subplots()
trans1 = ax.transData + ScaledTranslation(-5/72, 0, fig.dpi_scale_trans)
trans2 = ax.transData + ScaledTranslation(+5/72, 0, fig.dpi_scale_trans)
er1 = ax.errorbar(x, y1, yerr=yerr1, marker="o", linestyle="none", transform=trans1)
er2 = ax.errorbar(x, y2, yerr=yerr2, marker="o", linestyle="none", transform=trans2)
plt.show()
While results look similar in both cases, they are fundamentally different. You will observe this difference when interactively zooming the axes or changing the figure size.

Consider the following approach to highlight plots - combination of errorbar and fill_between with non-zero transparency:
import random
import matplotlib.pyplot as plt
# create sample data
N = 8
data_1 = {
'x': list(range(N)),
'y': [10. + random.random() for dummy in range(N)],
'yerr': [.25 + random.random() for dummy in range(N)]}
data_2 = {
'x': list(range(N)),
'y': [10.25 + .5 * random.random() for dummy in range(N)],
'yerr': [.5 * random.random() for dummy in range(N)]}
# plot
plt.figure()
# only errorbar
plt.subplot(211)
for data in [data_1, data_2]:
plt.errorbar(**data, fmt='o')
# errorbar + fill_between
plt.subplot(212)
for data in [data_1, data_2]:
plt.errorbar(**data, alpha=.75, fmt=':', capsize=3, capthick=1)
data = {
'x': data['x'],
'y1': [y - e for y, e in zip(data['y'], data['yerr'])],
'y2': [y + e for y, e in zip(data['y'], data['yerr'])]}
plt.fill_between(**data, alpha=.25)
Result:

Threre is example on lib site: https://matplotlib.org/stable/gallery/lines_bars_and_markers/errorbar_subsample.html
enter image description here
You need parameter errorevery=(m, n),
n - how often plot error lines, m - shift with range from 0 to n

How to draw a line with matplotlib?

I cannot find a way to draw an arbitrary line with matplotlib Python library. It allows to draw horizontal and vertical lines (with matplotlib.pyplot.axhline and matplotlib.pyplot.axvline, for example), but i do not see how to draw a line through two given points (x1, y1) and (x2, y2). Is there a way? Is there a simple way?

This will draw a line that passes through the points (-1, 1) and (12, 4), and another one that passes through the points (1, 3) et (10, 2)
x1 are the x coordinates of the points for the first line, y1 are the y coordinates for the same -- the elements in x1 and y1 must be in sequence.
x2 and y2 are the same for the other line.
import matplotlib.pyplot as plt
x1, y1 = [-1, 12], [1, 4]
x2, y2 = [1, 10], [3, 2]
plt.plot(x1, y1, x2, y2, marker = 'o')
plt.show()
I suggest you spend some time reading / studying the basic tutorials found on the very rich matplotlib website to familiarize yourself with the library.
What if I don't want line segments?
[edit]:
As shown by #thomaskeefe, starting with matplotlib 3.3, this is now builtin as a convenience: plt.axline((x1, y1), (x2, y2)), rendering the following obsolete.
There are no direct ways to have lines extend to infinity... matplotlib will either resize/rescale the plot so that the furthest point will be on the boundary and the other inside, drawing line segments in effect; or you must choose points outside of the boundary of the surface you want to set visible, and set limits for the x and y axis.
As follows:
import matplotlib.pyplot as plt
x1, y1 = [-1, 12], [1, 10]
x2, y2 = [-1, 10], [3, -1]
plt.xlim(0, 8), plt.ylim(-2, 8)
plt.plot(x1, y1, x2, y2, marker = 'o')
plt.show()

As of matplotlib 3.3, you can do this with plt.axline((x1, y1), (x2, y2)).

I was checking how ax.axvline does work, and I've written a small function that resembles part of its idea:
import matplotlib.pyplot as plt
import matplotlib.lines as mlines
def newline(p1, p2):
ax = plt.gca()
xmin, xmax = ax.get_xbound()
if(p2[0] == p1[0]):
xmin = xmax = p1[0]
ymin, ymax = ax.get_ybound()
else:
ymax = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmax-p1[0])
ymin = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmin-p1[0])
l = mlines.Line2D([xmin,xmax], [ymin,ymax])
ax.add_line(l)
return l
So, if you run the following code you will realize how does it work. The line will span the full range of your plot (independently on how big it is), and the creation of the line doesn't rely on any data point within the axis, but only in two fixed points that you need to specify.
import numpy as np
x = np.linspace(0,10)
y = x**2
p1 = [1,20]
p2 = [6,70]
plt.plot(x, y)
newline(p1,p2)
plt.show()

Just want to mention another option here.
You can compute the coefficients using numpy.polyfit(), and feed the coefficients to numpy.poly1d(). This function can construct polynomials using the coefficients, you can find more examples here
https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.poly1d.html
Let's say, given two data points (-0.3, -0.5) and (0.8, 0.8)
import numpy as np
import matplotlib.pyplot as plt
# compute coefficients
coefficients = np.polyfit([-0.3, 0.8], [-0.5, 0.8], 1)
# create a polynomial object with the coefficients
polynomial = np.poly1d(coefficients)
# for the line to extend beyond the two points,
# create the linespace using the min and max of the x_lim
# I'm using -1 and 1 here
x_axis = np.linspace(-1, 1)
# compute the y for each x using the polynomial
y_axis = polynomial(x_axis)
fig = plt.figure()
axes = fig.add_axes([0.1, 0.1, 1, 1])
axes.set_xlim(-1, 1)
axes.set_ylim(-1, 1)
axes.plot(x_axis, y_axis)
axes.plot(-0.3, -0.5, 0.8, 0.8, marker='o', color='red')
Hope it helps.

In case somebody lands here trying to plot many segments in one go, here is a way. Say the segments are defined by two 2-d arrays of same length, e.g. a and b. We want to plot segments between each a[i] and b[i]. In that case:
Solution 1
ab_pairs = np.c_[a, b]
plt_args = ab_pairs.reshape(-1, 2, 2).swapaxes(1, 2).reshape(-1, 2)
ax.plot(*plt_args, ...)
Example:
np.random.seed(0)
n = 32
a = np.random.uniform(0, 1, (n, 2))
b = np.random.uniform(0, 1, (n, 2))
fig, ax = plt.subplots(figsize=(3, 3))
ab_pairs = np.c_[a, b]
ab_args = ab_pairs.reshape(-1, 2, 2).swapaxes(1, 2).reshape(-1, 2)
# segments
ax.plot(*ab_args, c='k')
# identify points: a in blue, b in red
ax.plot(*a.T, 'bo')
ax.plot(*b.T, 'ro')
plt.show()
Solution 2
The above creates many matplotlib.lines.Line2D. If you'd like a single line, we can do it by interleaving NaN between pairs:
ax.plot(*np.c_[a, b, a*np.nan].reshape(-1, 2).T, ...)
Example:
# same init as example above, then
fig, ax = plt.subplots(figsize=(3, 3))
# segments (all at once)
ax.plot(*np.c_[a, b, a*np.nan].reshape(-1, 2).T, 'k')
# identify points: a in blue, b in red
ax.plot(*a.T, 'bo')
ax.plot(*b.T, 'ro')
plt.show()
(Same figure as above).

Based on #Alejandro's answer:
if you want to add a line to an existing Axes (e.g. a scatter plot), and
all you know is the slope and intercept of the desired line (e.g. a regression line), and
you want it to cover the entire visible X range (already computed), and
you want to use the object-oriented interface (not pyplot).
Then you can do this (existing Axes in ax):
# e.g. slope, intercept, r_value, p_value, std_err = scipy.stats.linregress(xs, ys)
xmin, xmax = ax.get_xbound()
ymin = (xmin * slope) + intercept
ymax = (xmax * slope) + intercept
l = matplotlib.lines.Line2D([xmin, xmax], [ymin, ymax])
ax.add_line(l)

How to make Matplotlib scatterplots transparent as a group?

I'm making some scatterplots using Matplotlib (python 3.4.0, matplotlib 1.4.3, running on Linux Mint 17). It's easy enough to set alpha transparency for each point individually; is there any way to set them as a group, so that two overlapping points from the same group don't change the color?
Example code:
import matplotlib.pyplot as plt
import numpy as np
def points(n=100):
x = np.random.uniform(size=n)
y = np.random.uniform(size=n)
return x, y
x1, y1 = points()
x2, y2 = points()
fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(111, title="Test scatter")
ax.scatter(x1, y1, s=100, color="blue", alpha=0.5)
ax.scatter(x2, y2, s=100, color="red", alpha=0.5)
fig.savefig("test_scatter.png")
Results in this output:
but I want something more like this one:
I can workaround by saving as SVG and manually grouping then in Inkscape, then setting transparency, but I'd really prefer something I can code. Any suggestions?

Yes, interesting question. You can get this scatterplot with Shapely. Here is the code :
import matplotlib.pyplot as plt
import matplotlib.patches as ptc
import numpy as np
from shapely.geometry import Point
from shapely.ops import cascaded_union
n = 100
size = 0.02
alpha = 0.5
def points():
x = np.random.uniform(size=n)
y = np.random.uniform(size=n)
return x, y
x1, y1 = points()
x2, y2 = points()
polygons1 = [Point(x1[i], y1[i]).buffer(size) for i in range(n)]
polygons2 = [Point(x2[i], y2[i]).buffer(size) for i in range(n)]
polygons1 = cascaded_union(polygons1)
polygons2 = cascaded_union(polygons2)
fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(111, title="Test scatter")
for polygon1 in polygons1:
polygon1 = ptc.Polygon(np.array(polygon1.exterior), facecolor="red", lw=0, alpha=alpha)
ax.add_patch(polygon1)
for polygon2 in polygons2:
polygon2 = ptc.Polygon(np.array(polygon2.exterior), facecolor="blue", lw=0, alpha=alpha)
ax.add_patch(polygon2)
ax.axis([-0.2, 1.2, -0.2, 1.2])
fig.savefig("test_scatter.png")
and the result is :

Interesting question, I think any use of transparency will result in the stacking effect you want to avoid. You could manually set a transparency type colour to get closer to the results you want,
import matplotlib.pyplot as plt
import numpy as np
def points(n=100):
x = np.random.uniform(size=n)
y = np.random.uniform(size=n)
return x, y
x1, y1 = points()
x2, y2 = points()
fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(111, title="Test scatter")
alpha = 0.5
ax.scatter(x1, y1, s=100, lw = 0, color=[1., alpha, alpha])
ax.scatter(x2, y2, s=100, lw = 0, color=[alpha, alpha, 1.])
plt.show()
The overlap between the different colours are not included in this way but you get,

This is a terrible, terrible hack, but it works.
You see while Matplotlib plots data points as separate objects that can overlap, it plots the line between them as a single object - even if that line is broken into several pieces by NaNs in the data.
With that in mind, you can do this:
import numpy as np
from matplotlib import pyplot as plt
plt.rcParams['lines.solid_capstyle'] = 'round'
def expand(x, y, gap=1e-4):
add = np.tile([0, gap, np.nan], len(x))
x1 = np.repeat(x, 3) + add
y1 = np.repeat(y, 3) + add
return x1, y1
x1, y1 = points()
x2, y2 = points()
fig = plt.figure(figsize=(4,4))
ax = fig.add_subplot(111, title="Test scatter")
ax.plot(*expand(x1, y1), lw=20, color="blue", alpha=0.5)
ax.plot(*expand(x2, y2), lw=20, color="red", alpha=0.5)
fig.savefig("test_scatter.png")
plt.show()
And each color will overlap with the other color but not with itself.
One caveat is that you have to be careful with the spacing between the two points you use to make each circle. If they're two far apart then the separation will be visible on your plot, but if they're too close together, matplotlib doesn't plot the line at all. That means that the separation needs to be chosen based on the range of your data, and if you plan to make an interactive plot then there's a risk of all the data points suddenly vanishing if you zoom out too much, and stretching if you zoom in too much.
As you can see, I found 1e-5 to be a good separation for data with a range of [0,1].

Just pass an argument saying edgecolors='none' to plt.scatter()

Here's a hack if you have more than just a few points to plot. I had to plot >500000 points, and the shapely solution does not scale well. I also wanted to plot a different shape other than a circle. I opted to instead plot each layer separately with alpha=1 and then read in the resulting image with np.frombuffer (as described here), then add the alpha to the whole image and plot overlays using plt.imshow. Note this solution forfeits access to the original fig object and attributes, so any other modifications to figure should be made before it's drawn.
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure
def arr_from_fig(fig):
canvas = FigureCanvas(fig)
canvas.draw()
img = np.frombuffer(fig.canvas.tostring_rgb(), dtype=np.uint8)
img = img.reshape(fig.canvas.get_width_height()[::-1] + (3,))
return img
def points(n=100):
x = np.random.uniform(size=n)
y = np.random.uniform(size=n)
return x, y
x1, y1 = points()
x2, y2 = points()
imgs = list()
figsize = (4, 4)
dpi = 200
for x, y, c in zip([x1, x2], [y1, y2], ['blue', 'red']):
fig = plt.figure(figsize=figsize, dpi=dpi, tight_layout={'pad':0})
ax = fig.add_subplot(111)
ax.scatter(x, y, s=100, color=c, alpha=1)
ax.axis([-0.2, 1.2, -0.2, 1.2])
ax.axis('off')
imgs.append(arr_from_fig(fig))
plt.close()
fig = plt.figure(figsize=figsize)
alpha = 0.5
alpha_scaled = 255*alpha
for img in imgs:
img_alpha = np.where((img == 255).all(-1), 0, alpha_scaled).reshape([*img.shape[:2], 1])
img_show = np.concatenate([img, img_alpha], axis=-1).astype(int)
plt.imshow(img_show, origin='lower')
ticklabels = ['{:03.1f}'.format(i) for i in np.linspace(-0.2, 1.2, 8, dtype=np.float16)]
plt.xticks(ticks=np.linspace(0, dpi*figsize[0], 8), labels=ticklabels)
plt.yticks(ticks=np.linspace(0, dpi*figsize[1], 8), labels=ticklabels);
plt.title('Test scatter');

I encountered the save issue recently, my case is there are too many points very close to each other, like 100 points of alpha 0.3 on top of each other, the alpha of the color in the generated image is almost 1. So instead of setting the alpha value in the cmap or scatter. I save it to a Pillow image and set the alpha channel there. My code:
import io
import os
import numpy as np
import numpy.ma as ma
import matplotlib.pyplot as plt
from matplotlib import colors
from PIL import Image
from dhi_base import DHIBase
class HeatMapPlot(DHIBase):
def __init__(self) -> None:
super().__init__()
# these 4 values are precalculated
top=75
left=95
width=1314
height=924
self.crop_box = (left, top, left+width, top+height)
# alpha 0.5, [0-255]
self.alpha = 128
def get_cmap(self):
v = [
...
]
return colors.LinearSegmentedColormap.from_list(
'water_level', v, 512)
def png3857(self):
"""Generate flooding images
"""
muids = np.load(os.path.join(self.npy_dir, 'myfilename.npy'))
cmap = self.get_cmap()
i = 0
for npyf in os.listdir(self.npy_dir):
if not npyf.startswith('flooding'):
continue
flooding_num = np.load(os.path.join(self.npy_dir, npyf))
image_file = os.path.join(self.img_dir, npyf.replace('npy', 'png'))
# if os.path.isfile(image_file):
# continue
# filter the water level value that is less than 0.001
masked_arr = ma.masked_where(flooding_num > 0.001, flooding_num)
flooding_masked = flooding_num[masked_arr.mask]
muids_masked = muids[masked_arr.mask, :]
plt.figure(figsize=(self.grid2D['numJ'] / 500, self.grid2D['numK'] / 500))
plt.axis('off')
plt.tight_layout()
plt.scatter(muids_masked[:, 0], muids_masked[:, 1], s=0.1, c=flooding_masked,
alpha=1, edgecolors='none', linewidths=0,
cmap=cmap,
vmin=0, vmax=1.5)
img_buf = io.BytesIO()
plt.savefig(img_buf, transparent=True, dpi=200, format='png')#, pad_inches=0)
plt.clf()
plt.close()
img_buf.seek(0)
img = Image.open(img_buf)
# Cropped image of above dimension
# (It will not change original image)
img = img.crop(self.crop_box)
alpha_channel = img.getchannel('A')
# Make all opaque pixels into semi-opaque
alpha_channel = alpha_channel.point(lambda i: self.alpha if i>0 else 0)
img.putalpha(alpha_channel)
img.save(image_file)
self.logger.info("PNG saved to {}".format(image_file))
i += 1
# if i > 15:
# break
if __name__ == "__main__":
hp = HeatMapPlot()
hp.png3857()

How to put circles on top of a polygon?

I use matplotlib to generate an image in the following way:
fig = plt.figure()
ax = fig.add_subplot(111)
ax.fill(border[0],border[1], color='g', linewidth=1, fill=True, alpha = 0.5)
patches = []
for x1,y1,r in zip(x, y, radii):
circle = Circle((x1,y1), r)
patches.append(circle)
p = PatchCollection(patches, cmap='cool', alpha=1.0)
p.set_array(c)
ax.add_collection(p)
plt.colorbar(p)
plt.savefig(fig_name)
What I want to have is a polygon (given by its border) and colored circles on the top of this polygon. However, I get the polygon on the top of the circles.
This is strange because I plot the polygon first and then I add circles to the plot.
Does anybody know why it happens and how this problem can be resolved?
ADDED
As requested, here is fully working example:
import pandas
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.collections import PatchCollection
from matplotlib.patches import Circle, Polygon
import numpy as np
def plot_xyc(df, x_col, y_col, c_col, radius, fig_name, title, zrange):
resolution = 50
x = df[x_col]
y = df[y_col]
c = df[c_col]
x0 = (max(x) + min(x))/2.0
y0 = (max(y) + min(y))/2.0
dx = (max(x) - min(x))
dy = (max(y) - min(y))
delta = max(dx, dy)
radii = [delta*radius for i in range(len(x))]
fig = plt.figure()
plt.title(title)
ax = fig.add_subplot(111)
border = ([-3, 3, 3, -3], [-3, -3, 3, 3])
ax.fill(border[0],border[1], color='g', linewidth=1, fill=True, alpha = 1.0)
patches = []
for x1,y1,r in zip(x, y, radii):
circle = Circle((x1,y1), r)
patches.append(circle)
patches.append(Circle((-100,-100), r))
patches.append(Circle((-100,-100), r))
p = PatchCollection(patches, cmap='cool', alpha=1.0)
p.set_array(c)
max_ind = max(c.index)
c.set_value(max_ind + 1, min(zrange))
c.set_value(max_ind + 2, max(zrange))
plt.xlim([x0 - delta/2.0 - 0.05*delta, x0 + delta/2.0 + 0.05*delta])
plt.ylim([y0 - delta/2.0 - 0.05*delta, y0 + delta/2.0 + 0.05*delta])
ax.add_collection(p)
plt.colorbar(p)
plt.savefig(fig_name)
if __name__ == '__main__':
df = pandas.DataFrame({'x':[1,2,3,4], 'y':[4,3,2,1], 'z':[1,1,2,2]})
plot_xyc(df, 'x', 'y', 'z', 0.1, 'test2.png', 'My Titlle', (0.0, 3.0))

You're looking for zorder.
In matplotlib, all additional arguments are just passed up the class heirarchy. zorder is a kwarg of the Artist class, so you just need to make sure that at some point it gets zorder.
You can do it two ways in your example;
either add it in here:
ax.fill(border[0],border[1], color='g', linewidth=1, fill=True, alpha = 1.0, zorder=1)
or here:
p = PatchCollection(patches, cmap='cool', alpha=1.0, zorder=2)
or if you want, both. Objects with a higher zorder sit on top of those with lower values.

axis limits for scatter plot not holding in matplotlib

I am trying to overlay a scatter plot onto a contour plot using matplotlib, which contains
plt.contourf(X, Y, XYprof.T, self.nLevels, extent=extentYPY, \
origin = 'lower')
if self.doScatter == True and len(xyScatter['y']) != 0:
plt.scatter(xyScatter['x'], xyScatter['y'], \
s=dSize, c=myColor, marker='.', edgecolor='none')
plt.xlim(-xLimHist, xLimHist)
plt.ylim(-yLimHist, yLimHist)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
What ends up happening is the resulting plots extend to include all of the scatter points, which can exceed the limits for the contour plot. Is there any way to get around this?

I used the following example to try and replicate your problem. If left to default, the range for x and y was -3 to 3. I input the xlim and ylim so the range for both was -2 to 2. It worked.
import numpy as np
import matplotlib.pyplot as plt
from pylab import *
# the random data
x = np.random.randn(1000)
y = np.random.randn(1000)
fig = plt.figure(1, figsize=(5.5,5.5))
X, Y = meshgrid(x, y)
Z1 = bivariate_normal(X, Y, 1.0, 1.0, 0.0, 0.0)
Z2 = bivariate_normal(X, Y, 1.5, 0.5, 1, 1)
Z = 10 * (Z1 - Z2)
origin = 'lower'
CS = contourf(x, y, Z, 10, # [-1, -0.1, 0, 0.1],
cmap=cm.bone,
origin=origin)
title('Nonsense')
xlabel('x-stuff')
ylabel('y-stuff')
# the scatter plot:
axScatter = plt.subplot(111)
axScatter.scatter(x, y)
# set axes range
plt.xlim(-2, 2)
plt.ylim(-2, 2)
show()