We Keep Coding

Pandas plot error : Missing category information for StrCategoryConverter; this might be caused by unintendedly mixing categorical and numeric data

I was trying to plot a line plot using pandas plot method, the same data with exactly same method runs fine if I use matplotlib methods, however if I use df.plot then annotate gives me error ValueError: Missing category information for StrCategoryConverter; this might be caused by unintendedly mixing categorical and numeric data
Say I have a dataframe,
data = {'Unit': {0: 'Admin ', 1: 'C-Level', 2: 'Engineering', 3: 'IT', 4: 'Manufacturing', 5: 'Sales'}, 'Mean': {0: 4.642857142857143, 1: 4.83, 2: 4.048, 3: 4.237317073170732, 4: 4.184319526627219, 5: 3.9904545454545453}}
result=pd.DataFrame(data)
When using matplotlib
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(18,9))
ax.plot(results['Unit'],results['Mean'])
for i, val in enumerate(zip(results['Unit'],results['Mean'])):
label = str(results.loc[i, 'Mean'])
ax.annotate(label, val, ha='center')
plt.show()
The above code works perfectly fine.
Using pandas plot function(which gives me error)
results.plot(x = 'Unit', y = 'Mean', marker = 'o', figsize=(8,5))
ax = plt.gca()
for i, val in enumerate(zip(results['Unit'],results['Mean'])):
label = str(results.loc[i, 'Mean'])
ax.annotate(label, val, ha='center')
plt.show()
which gives me error :
Traceback (most recent call last):
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\axis.py", line 1506, in convert_units
ret = self.converter.convert(x, self.units, self)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\category.py", line 49, in convert
raise ValueError(
ValueError: Missing category information for StrCategoryConverter; this might be caused by unintendedly mixing categorical and numeric data
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\backends\backend_qt.py", line 477, in _draw_idle
self.draw()
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\backends\backend_agg.py", line 436, in draw
self.figure.draw(self.renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\artist.py", line 73, in draw_wrapper
result = draw(artist, renderer, *args, **kwargs)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\artist.py", line 50, in draw_wrapper
return draw(artist, renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\figure.py", line 2837, in draw
mimage._draw_list_compositing_images(
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\image.py", line 132, in _draw_list_compositing_images
a.draw(renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\artist.py", line 50, in draw_wrapper
return draw(artist, renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\axes\_base.py", line 3091, in draw
mimage._draw_list_compositing_images(
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\image.py", line 132, in _draw_list_compositing_images
a.draw(renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\artist.py", line 50, in draw_wrapper
return draw(artist, renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\text.py", line 1969, in draw
if not self.get_visible() or not self._check_xy(renderer):
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\text.py", line 1559, in _check_xy
xy_pixel = self._get_position_xy(renderer)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\text.py", line 1552, in _get_position_xy
return self._get_xy(renderer, x, y, self.xycoords)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\text.py", line 1419, in _get_xy
x = float(self.convert_xunits(x))
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\artist.py", line 252, in convert_xunits
return ax.xaxis.convert_units(x)
File "C:\Users\hpoddar\AppData\Local\Programs\Python\Python310\lib\site-packages\matplotlib\axis.py", line 1508, in convert_units
raise munits.ConversionError('Failed to convert value(s) to axis '
matplotlib.units.ConversionError: Failed to convert value(s) to axis units: 'Admin '
Expected output :
Annotated graph
Why am I getting error in case of pandas plot, and how can I resolve the same

matplotlib.axes.Axes.annotate() takes parameter xy as below:
xy(float, float)
The point (x, y) to annotate. The coordinate system is determined by xycoords.
Therefore to fix the issue you could create a tuple that has (i, val[1]) rather than pass val which would contain string type.
ax = results.plot(x = 'Unit', y = 'Mean', marker = 'o', figsize=(8,5))
for i, val in enumerate(zip(results['Unit'],results['Mean'])):
label = str(results.loc[i, 'Mean'])
ax.annotate(text=label, xy=(i, val[1]), ha='center')
Another option is to use matplotlib.axes.Axes.text():
ax = results.plot(x = 'Unit', y = 'Mean', marker = 'o', figsize=(8,5))
for i, val in enumerate(zip(results['Unit'],results['Mean'])):
label = str(results.loc[i, 'Mean'])
ax.text(x=i, y=val[1], s=label, ha='center')
As you noted, when we call annotate() after matplotlib.axes.Axes.plot there is no ConversionError. But if we call annotate() after pandas.DataFrame.plot there is a ConversionError.
My best guess is that after using matplotlib to plot there will be no conversion on annotate. But with a pandas plot a conversion will be attempted. This could be due to the representation of the x-ticks being different for both cases.
To demonstrate, if we try the following code the same error will be thrown:
fig, ax = plt.subplots(figsize=(18,9))
# Call annotate without plotting! Throws ConversionError
# ax.plot(results['Unit'],results['Mean'])
for i, val in enumerate(zip(results['Unit'],results['Mean'])):
label = str(results.loc[i, 'Mean'])
ax.annotate(label, val, ha='center')
Whereas if we do the following, no error will be thrown:
fig, ax = plt.subplots(figsize=(18,9))
# Call annotate without plotting! No error
# ax.plot(results['Unit'],results['Mean'])
for i, val in enumerate(zip(results['Unit'],results['Mean'])):
label = str(results.loc[i, 'Mean'])
ax.annotate(text=label, xy=(i, val[1]), ha='center')

How to fix error running local Otsu threshold example?

I am very new to Python. I am trying to run this script:
https://scikit-image.org/docs/0.12.x/auto_examples/segmentation/plot_local_otsu.html
But, I'm getting this error:
Traceback (most recent call last):
File "/Users/janine/Downloads/plot_local_otsu.py", line 37, in <module>
fig, ax = plt.subplots(2, 2, figsize=(8, 5), sharex=True, sharey=True,
File "/usr/local/lib/python3.9/site-packages/matplotlib/_api/deprecation.py", line 471, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/matplotlib/pyplot.py", line 1440, in subplots
axs = fig.subplots(nrows=nrows, ncols=ncols, sharex=sharex, sharey=sharey,
File "/usr/local/lib/python3.9/site-packages/matplotlib/_api/deprecation.py", line 471, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/matplotlib/figure.py", line 908, in subplots
axs = gs.subplots(sharex=sharex, sharey=sharey, squeeze=squeeze,
File "/usr/local/lib/python3.9/site-packages/matplotlib/gridspec.py", line 307, in subplots
axarr[row, col] = figure.add_subplot(
File "/usr/local/lib/python3.9/site-packages/matplotlib/figure.py", line 781, in add_subplot
ax = subplot_class_factory(projection_class)(self, *args, **pkw)
File "/usr/local/lib/python3.9/site-packages/matplotlib/axes/_subplots.py", line 36, in __init__
self._axes_class.__init__(self, fig, [0, 0, 1, 1], **kwargs)
File "/usr/local/lib/python3.9/site-packages/matplotlib/_api/deprecation.py", line 471, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 648, in __init__
self.update(kwargs)
File "/usr/local/lib/python3.9/site-packages/matplotlib/artist.py", line 1064, in update
ret.append(func(v))
File "/usr/local/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 1531, in set_adjustable
_api.check_in_list(["box", "datalim"], adjustable=adjustable)
File "/usr/local/lib/python3.9/site-packages/matplotlib/_api/__init__.py", line 126, in check_in_list
raise ValueError(
ValueError: 'box-forced' is not a valid value for adjustable; supported values are 'box', 'datalim'
I have installed scikit-image exactly as recommended here:
https://scikit-image.org/docs/stable/install.html.
I am on macOS Mojave.

As you can see from the link, the example is from the outdated 0.12.x version(s) of skimage, whereas 0.18.0 is current stable version. And, as the error message indicates, the error comes from this line:
fig, ax = plt.subplots(2, 2, figsize=(8, 5), sharex=True, sharey=True,
subplot_kw={'adjustable': 'box-forced'})
Obviously, the handling of the adjustable member has changed in matplotlib.pyplot over the years. By simply removing the subplot_kw parameter at all, for example, the code runs perfectly fine:
fig, ax = plt.subplots(2, 2, figsize=(8, 5), sharex=True, sharey=True)
In fact, that's also, what the updated example from the skimage documentation looks like (second example there). Notice: You'll have to add two import statements on your own, since the given code there in incomplete.

Plot a dataframe of times

Hi I want to use a dataframe of times which are in the format hh:mm as the xticks of a figure.
Just to represent what I'm doing I have:
import matplotlib.pyplot as plt
import pandas as pd
#locate series to plot
df = pd.read_excel('Excel_file', header=None)
df1 = df.iloc[71:128, 3]
df2 = df.iloc[71:128, 0]
#Plot df2 on the x-axis and df1 on the y-axis
plt.plot(df2, df1)
plt.xticks(df2)
plt.show()
which gives me the (full) error:
Traceback (most recent call last):
File "C:/Users/Alessio/PycharmProjects/PeakAutomation/graphs.py", line 14, in <module>
plt.xticks(df2)
File "C:\Users\Alessio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\matplotlib\pyplot.py", line 1483, in xticks
locs = ax.set_xticks(ticks)
File "C:\Users\Alessio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\matplotlib\cbook\deprecation.py", line 400, in wrapper
return func(*args, **kwargs)
File "C:\Users\Alessio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\matplotlib\axes\_base.py", line 3306, in set_xticks
ret = self.xaxis.set_ticks(ticks, minor=minor)
File "C:\Users\Alessio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\matplotlib\cbook\deprecation.py", line 400, in wrapper
return func(*args, **kwargs)
File "C:\Users\Alessio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\matplotlib\axis.py", line 1765, in set_ticks
self.set_view_interval(min(ticks), max(ticks))
File "C:\Users\Alessio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\matplotlib\axis.py", line 1902, in setter
setter(self, min(vmin, vmax, oldmin), max(vmin, vmax, oldmax),
TypeError: '<' not supported between instances of 'float' and 'datetime.time'
How can I plot df2 as the x-axis values?
When I dont try to change the xticks, I get this:
Which is fine, but I want to have the x-values to be those from the dataFrame (df2)
Thanks

Basemap TypeError: input must be an array, list, tuple or scalar

I'm trying to create a map visualization using the basemap module in Python 3.0 but when I try to plot this figure I get the TypeError:
TypeError: input must be an array, list, tuple or scalar
My code looks like this:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
data = pd.ExcelFile('C:\\Users\\...xlsx')
data_input = pd.read_excel(data, 'Sheet2')
# Extract the data we're interested in
lat = data_input['value1'].values
lon = data_input['value2'].values
capacity = data_input['value3'].values
# 1. Draw the map background
fig = plt.figure(figsize=(8, 8))
m = Basemap(projection='lcc', resolution='h',
lat_0=31.1351682, lon_0=-99.3350553,
width=1.3E6, height=1.25E6)
m.shadedrelief()
m.drawcoastlines(color='gray')
m.drawcountries(color='gray')
m.drawstates(color='gray')
# 2. scatter city data, with color reflecting population
# and size reflecting area
m.scatter(lon, lat, latlon=True,
c=np.log10(capacity), s=capacity,
cmap='Reds', alpha=0.5)
I've tried changing all the inputs to data_input.values, data_input.to_list(), list(data_input) and just using the default pandas Series.
The error traceback occurs here:
File "<ipython-input-6-3a66206674c7>", line 3, in <module>
cmap='Reds', alpha=0.5)
File "C:\Users\...Continuum\anaconda3\lib\site-packages\mpl_toolkits\basemap\__init__.py", line 566, in with_transform
x, y = self(x,y)
File "C:\Users\...\Continuum\anaconda3\lib\site-packages\mpl_toolkits\basemap\__init__.py", line 1191, in __call__
xout,yout = self.projtran(x,y,inverse=inverse)
File "C:\Users\...\Continuum\anaconda3\lib\site-packages\mpl_toolkits\basemap\proj.py", line 288, in __call__
outx,outy = self._proj4(x, y, inverse=inverse)
File "C:\Users\...\Continuum\anaconda3\lib\site-packages\pyproj\__init__.py", line 397, in __call__
inx, xisfloat, xislist, xistuple = _copytobuffer(lon)
File "C:\Users\...\Continuum\anaconda3\lib\site-packages\pyproj\__init__.py", line 652, in _copytobuffer
raise TypeError('input must be an array, list, tuple or scalar')
No matter what form it gets it doesn't work. What am I missing here?

Matplotlib runs out of memory when plotting in a loop

I have a fairly simple plotting routine that looks like this:
from __future__ import division
import datetime
import matplotlib
matplotlib.use('Agg')
from matplotlib.pyplot import figure, plot, show, legend, close, savefig, rcParams
import numpy
from globalconstants import *
def plotColumns(columnNumbers, t, out, showFig=False, filenamePrefix=None, saveFig=True, saveThumb=True):
lineProps = ['b', 'r', 'g', 'c', 'm', 'y', 'k', 'b--', 'r--', 'g--', 'c--', 'm--', 'y--', 'k--', 'g--', 'b.-', 'r.-', 'g.-', 'c.-', 'm.-', 'y.-', 'k.-']
rcParams['figure.figsize'] = (13,11)
for i in columnNumbers:
plot(t, out[:,i], lineProps[i])
legendStrings = list(numpy.zeros(NUMCOMPONENTS))
legendStrings[GLUCOSE] = 'GLUCOSE'
legendStrings[CELLULOSE] = 'CELLULOSE'
legendStrings[STARCH] = 'STARCH'
legendStrings[ACETATE] = 'ACETATE'
legendStrings[BUTYRATE] = 'BUTYRATE'
legendStrings[SUCCINATE] = 'SUCCINATE'
legendStrings[HYDROGEN] = 'HYDROGEN'
legendStrings[PROPIONATE] = 'PROPIONATE'
legendStrings[METHANE] = "METHANE"
legendStrings[RUMINOCOCCUS] = 'RUMINOCOCCUS'
legendStrings[METHANOBACTERIUM] = "METHANOBACTERIUM"
legendStrings[BACTEROIDES] = 'BACTEROIDES'
legendStrings[SELENOMONAS] = 'SELENOMONAS'
legendStrings[CLOSTRIDIUM] = 'CLOSTRIDIUM'
legendStrings = [legendStrings[i] for i in columnNumbers]
legend(legendStrings, loc='best')
dt = datetime.datetime.now()
dtAsString = dt.strftime('%d-%m-%Y_%H-%M-%S')
if filenamePrefix is None:
filenamePrefix = ''
if filenamePrefix != '' and filenamePrefix[-1] != '_':
filenamePrefix += '_'
if saveFig:
savefig(filenamePrefix+dtAsString+'.eps')
if saveThumb:
savefig(filenamePrefix+dtAsString+'.png', dpi=300)
if showFig: f.show()
close('all')
When I plot this in single iterations, it works fine. However, the moment I put it in a loop, matplotlib throws a hissy fit...
Traceback (most recent call last):
File "c4hm_param_variation_h2_conc.py", line 148, in <module>
plotColumns(columnNumbers, timeVector, out, showFig=False, filenamePrefix='c
4hm_param_variation_h2_conc_'+str(hydrogen_conc), saveFig=False, saveThumb=True)
File "D:\phdproject\alexander paper\python\v3\plotcolumns.py", line 48, in plo
tColumns
savefig(filenamePrefix+dtAsString+'.png', dpi=300)
File "C:\Python25\lib\site-packages\matplotlib\pyplot.py", line 356, in savefi
g
return fig.savefig(*args, **kwargs)
File "C:\Python25\lib\site-packages\matplotlib\figure.py", line 1032, in savef
ig
self.canvas.print_figure(*args, **kwargs)
File "C:\Python25\lib\site-packages\matplotlib\backend_bases.py", line 1476, i
n print_figure
**kwargs)
File "C:\Python25\lib\site-packages\matplotlib\backends\backend_agg.py", line
358, in print_png
FigureCanvasAgg.draw(self)
File "C:\Python25\lib\site-packages\matplotlib\backends\backend_agg.py", line
314, in draw
self.figure.draw(self.renderer)
File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:\Python25\lib\site-packages\matplotlib\figure.py", line 773, in draw
for a in self.axes: a.draw(renderer)
File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:\Python25\lib\site-packages\matplotlib\axes.py", line 1735, in draw
a.draw(renderer)
File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
draw(artist, renderer, *kl)
File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 374, in draw
bbox = self._legend_box.get_window_extent(renderer)
File "C:\Python25\lib\site-packages\matplotlib\offsetbox.py", line 209, in get
_window_extent
px, py = self.get_offset(w, h, xd, yd)
File "C:\Python25\lib\site-packages\matplotlib\offsetbox.py", line 162, in get
_offset
return self._offset(width, height, xdescent, ydescent)
File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 360, in findof
fset
return _findoffset(width, height, xdescent, ydescent, renderer)
File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 325, in _findo
ffset_best
ox, oy = self._find_best_position(width, height, renderer)
File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 817, in _find_
best_position
verts, bboxes, lines = self._auto_legend_data()
File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 669, in _auto_
legend_data
tpath = trans.transform_path(path)
File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1911, in t
ransform_path
self._a.transform_path(path))
File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1122, in t
ransform_path
return Path(self.transform(path.vertices), path.codes,
File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1402, in t
ransform
return affine_transform(points, mtx)
MemoryError: Could not allocate memory for path
This happens on iteration 2 (counting from 1), if that makes a difference. The code is running on Windows XP 32-bit with python 2.5 and matplotlib 0.99.1, numpy 1.3.0 and scipy 0.7.1.
EDIT: The code has now been updated to reflect the fact that the crash actually occurs at the call to legend(). Commenting that call out solves the problem, though obviously, I would still like to be able to put a legend on my graphs...

Is each loop supposed to generate a new figure? I don't see you closing it or creating a new figure instance from loop to loop.
This call will clear the current figure after you save it at the end of the loop:
pyplot.clf()
I'd refactor, though, and make your code more OO and create a new figure instance on each loop:
from matplotlib import pyplot
while True:
fig = pyplot.figure()
ax = fig.add_subplot(111)
ax.plot(x,y)
ax.legend(legendStrings, loc = 'best')
fig.savefig('himom.png')
# etc....

I've also run into this error. what seems to have fixed it is
while True:
fig = pyplot.figure()
ax = fig.add_subplot(111)
ax.plot(x,y)
ax.legend(legendStrings, loc = 'best')
fig.savefig('himom.png')
#new bit here
pylab.close(fig) #where f is the figure
running my loop stably now with fluctuating memory but no consistant increase

Answer from ninjasmith worked for me too - pyplot.close() enabled my loops to work.
From the pyplot tutorial, Working with multiple figures and axes:
You can clear the current figure with clf() and the current
axes with cla(). If you find this statefulness, annoying, don’t
despair, this is just a thin stateful wrapper around an object
oriented API, which you can use instead (see Artist tutorial)
If you are making a long sequence of figures, you need to be aware of
one more thing: the memory required for a figure is not completely
released until the figure is explicitly closed with close(). Deleting
all references to the figure, and/or using the window manager to kill
the window in which the figure appears on the screen, is not enough,
because pyplot maintains internal references until close() is called.

In my case, matplotlib version 3.5.0, As Hui Liu san says,
Following method can keep memory usage low
import matplotlib
print(matplotlib.__version__) #'3.5.0'
import matplotlib.pyplot as plt
plt.savefig('your.png')
# Add both in this order for keeping memory usage low
plt.clf()
plt.close()

I had a similar issue when I was using it from jupyter, putting plt.clf() and plt.close() in the loop did not work.
But this helped:
import matplotlib
matplotlib.use('Agg')
This disables interactive backend for matplotlib.