We Keep Coding

Pcolormesh or column_stack function has memory error when I try creating images in a loop even after closing figures

I am quite new to python and am trying to plot data using pcolormesh, and have to create many images in a loop with quite large arrays. The code to generate the data works fine and creates the arrays, however the problem arises when I try to plot. It seems that the first iteration creates the image fine, however the second has a memory error unlike the usual ones I've seen before. This one comes from the column_stack function apparently multiplying my x and y arrays together. I'm not entirely sure why it would do this, I've tried closing the figure and deleting all variables before the next iteration, however the problems still there. Heres an example of my function within the loop for plotting the data:
import numpy as np
import matplotlib.pyplot as plt
import pathlib
import sys
import os
from scipy import stats
def dataplot(x, y, time, outputfolder, filenumber, gamma, colormap):
clrmap = plt.get_cmap(colormap)
xum = x * 1e6 # convert to microns
yum = y * 1e6
fig = plt.figure()
cax = plt.pcolormesh(xum, yum, gamma.T, cmap=clrmap)
cbar = plt.colorbar(cax, orientation='vertical')
cbar.set_label('$\gamma$')
plt.xlabel('x/$\mu$m')
plt.ylabel('y/$\mu$m')
plt.title('gamma ' + time + 'ps')
plt.tight_layout()
plt.savefig(outputfolder + filenumber + 'GammaGrid_' +'.png', dpi=300)
# delete everything after:
plt.clf()
plt.close(fig)
del x
del xum
del y
del yum
del gamma
Where:
>>> gamma.shape
(26000,3000)
>>> xum.shape
(26000,)
>>> yum.shape
(3000,)
And the following error arises on the second loop iteration:
Traceback (most recent call last):
File "gamma_grid.py", line 245, in <module>
dataplot(x,y,time,outputfolder,sdffilenumber, gamma_grid, colormap, stat)
File "gamma_grid.py", line 153, in dataplot
cax = plt.pcolormesh(xum, yum, gamma.T, cmap=clrmap)
File "/users/anaconda3/lib/python3.7/site-packages/matplotlib/pyplot.py", line 2758, in pcolormesh
**({"data": data} if data is not None else {}), **kwargs)
File "/users/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py", line 1599, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File "/users/anaconda3/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 6176, in pcolormesh
coords = np.column_stack((X, Y)).astype(float, copy=False)
MemoryError: Unable to allocate 1.16 GiB for an array with shape (78000000, 2) and data type float64
The shape of the array in the memory error is the same size as xum.shape * yum.shape, and I'm not quite sure why it is doing that. Any help would be greatly appreciated.

Calculating vorticity for multiple vertical levels in MetPy

I'm trying to calculate vorticity in MetPy for multiple (consecutive) vertical levels. When I try to calculate it for a single level, everything works fine.
Here's the code; I've used the example for cross sections from https://unidata.github.io/MetPy/latest/examples/cross_section.html#sphx-glr-examples-cross-section-py.
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.pyplot as plt
import numpy as np
import xarray as xr
import metpy.calc as mpcalc
from metpy.cbook import get_test_data
from metpy.interpolate import cross_section
from metpy.units import units
data = xr.open_dataset(get_test_data('narr_example.nc', False))
data = data.metpy.parse_cf().squeeze()
data_crs = data['Temperature'].metpy.cartopy_crs
lat = data['lat']
lon = data['lon']
f = mpcalc.coriolis_parameter(lat)
dx, dy = mpcalc.lat_lon_grid_deltas(lon, lat, initstring=data_crs.proj4_init)
Then the calculation of vorticity is performed.
vort = mpcalc.vorticity(data['u_wind'], data['v_wind'], dx, dy)
The traceback:
Traceback (most recent call last):
File "E:\Временные файлы\cross_section (1).py", line 63, in <module>
vort = mpcalc.vorticity(data['u_wind'], data['v_wind'], dx, dy)
File "C:\ProgramData\Miniconda3\lib\site-packages\metpy\xarray.py", line 436, in wrapper
return func(*args, **kwargs)
File "C:\ProgramData\Miniconda3\lib\site-packages\metpy\calc\kinematics.py", line 60, in wrapper
ret = func(*args, **kwargs)
File "C:\ProgramData\Miniconda3\lib\site-packages\metpy\calc\kinematics.py", line 121, in vorticity
dudy = first_derivative(u, delta=dy, axis=-2)
File "C:\ProgramData\Miniconda3\lib\site-packages\metpy\calc\tools.py", line 920, in wrapper
return preprocess_xarray(func)(f, **kwargs)
File "C:\ProgramData\Miniconda3\lib\site-packages\metpy\xarray.py", line 436, in wrapper
return func(*args, **kwargs)
File "C:\ProgramData\Miniconda3\lib\site-packages\metpy\calc\tools.py", line 1014, in first_derivative
combined_delta = delta[tuple(delta_slice0)] + delta[tuple(delta_slice1)]
File "C:\ProgramData\Miniconda3\lib\site-packages\pint\quantity.py", line 1400, in __getitem__
value = self._magnitude[key]
IndexError: too many indices for array
I'm absolutely stuck. Searching "metpy multiple levels calculations" (no actual quotes) gives no relevant results.
The doc says:
metpy.calc.vorticity(u, v, dx, dy)[source]
Calculate the vertical vorticity of the horizontal wind.
Parameters:
u ((M, N) ndarray) – x component of the wind
v ((M, N) ndarray) – y component of the wind
dx (float or ndarray) – The grid spacing(s) in the x-direction. If an array, there should be one item less than the size of u along the applicable axis.
dy (float or ndarray) – The grid spacing(s) in the y-direction. If an array, there should be one item less than the size of u along the applicable axis.
dim_order (str or None, optional) – The ordering of dimensions in passed in arrays. Can be one of None, 'xy', or 'yx'. 'xy' indicates that the dimension corresponding to x is the leading dimension, followed by y. 'yx' indicates that x is the last dimension, preceded by y. None indicates that the default ordering should be assumed, which is ‘yx’. Can only be passed as a keyword argument, i.e. func(…, dim_order=’xy’).
Returns:
(M, N) ndarray – vertical vorticity
I conclude that the input can have more than 2 dimensions, but 3-dimensional input (as it is in my case) gives errors. What can be done to fix them?
I'm absolutely new to Python, so I could've made a stupid mistake.

Unfortunately, the error message that comes up isn't that helpful in this case if you don't know what to look for!
The problem with the vorticity function call in your example is that the dimensionality of your input variables do not match. data['u_wind'] and data['v_wind'] are 3D arrays with shape (29, 118, 292), but dx and dy, since they were computed from lat_lon_grid_deltas, are 2D arrays with shapes (118, 291) and (117, 292) respectively. And so, we need to obtain arrays that broadcast appropriately...there are many different ways that you could do this, but here are two options that I would recommend:
Option 1: Manual Broadcasting
Since the "extra" dimension that dx and dy are missing is the first dimension (in the vertical), we can just make dx and dy into properly aligned 3D arrays by inserting a size-one leading dimension:
dx, dy = mpcalc.lat_lon_grid_deltas(lon, lat, initstring=data_crs.proj4_init)
dx = dx[None, :]
dy = dy[None, :]
vort = mpcalc.vorticity(data['u_wind'], data['v_wind'], dx, dy)
Option 2: Use the grid_deltas_from_dataarray() helper function
MetPy also has a helper function to make pulling the grid deltas from an xarray DataArray easy. It also ensures that the broadcasting occurs properly, so you don't have to do it yourself. Using it in your example, it would be:
dx, dy = mpcalc.grid_deltas_from_dataarray(data['u_wind'])
vort = mpcalc.vorticity(data['u_wind'], data['v_wind'], dx, dy)

Python equivalent of Matlab's hist3

for i=1:n
centersX(:,i)=linspace(min(xData)+dX/2,max(xData)-dX/2,nbins)';
centersY(:,i)=linspace(min(yData)+dY/2,max(phase)-dY/2,nbins)';
centers = {centersX(:,i),centersY(:,i)};
H(:,:,i) = hist3([xData yData],centers);
end
In each iteration, I construct centersX and centersY with linspace function. I then store them in a 2x1 cell array called centers. H is a nbins X nbins X n struct. In each iteration I fill a nbins X nbins slice of H with the data from hist3.
I'm looking for the Python equivalent. I'm having trouble with passing the arguments for numpy.histogram2d:
H[:,:,i] = numpy.histogram2d(xData,yData,centers)
I get the following error:
Traceback (most recent call last):
line 714, in histogramdd
N, D = sample.shape
AttributeError: 'list' object has no attribute 'shape'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
line 36, in <module>
H[:,:,i] = numpy.histogram2d(xData, yData, centers)
line 714, in histogram2d
hist, edges = histogramdd([x, y], bins, range, normed, weights)
line 718, in histogramdd
N, D = sample.shape
ValueError: too many values to unpack (expected 2)
Since Python doesn't have cell arrays, I changed centers to be an array of arrays where centers[0] = centersX and centers[1] = centersY. What do I need to change such that that assuming the data are the same between matlab and python that the outputs will match?
EDIT:
I have also tried H[:,:,i] = numpy.histogram2d(xData,yData, bins=(centersX,centersY)) to cutout the combining step into centers but no luck.

Have you tried combing them with square brackets?
Maybe you can also use matplotlib.pyplot.hist2d.
H[:,:,i], *_ = numpy.histogram2d(xData,yData,bins=[centers[0], centers[1]])
H[:,:,i], *_ = matplotlib.pyplot.hist2d(xData,yData,bins=[centers[0], centers[1]])
In both, the values in centers are the bin edges, not the centers. You have to adjust the calculation. I think it is enough to remove the dX/2:
centersX(:,i)=linspace(min(xData),max(xData),nbins)';
centersY(:,i)=linspace(min(yData),max(phase),nbins)';

Fitting unorganized data to a sphere using scipy.interpolate.SmoothSphereBivariateSpline

I have 974 data points located on the surface of the unit sphere. The points are not ordered in any particular way. I wish to read in these data points, along with their corresponding polar coordinates (phi, theta). I then wish to interpolate onto a regular latitude-longitude grid using scipy.interpolate.SmoothSphereBivariateSpline. So far I have the following code:
import numpy as np
from scipy.interpolate import SmoothSphereBivariateSpline
#Read in the unorganized grid points
# and also shift so that phi in [0,pi] and theta in [0,2*pi)
leb = np.genfromtxt('grid.txt')
u, v = np.hsplit(leb, 2)
phi, theta = u[:,0], v[:,0]
theta += np.pi
#Read in the unorganized data values
data1 = np.genfromtxt('0p0_97p03.txt')
#Create the interpolator object
lut = SmoothSphereBivariateSpline(phi, theta, data1, s=350)
#Generate a regular lat-long grid to interpolate onto
N = 100
lat = np.linspace(0.0, np.pi, N)
lon = np.linspace(0.0, 2.0*np.pi, N)
lat, lon = np.meshgrid(lat, lon)
#Now interpolate onto the regular grid
data_inerp = lut(lat, lon)
In the above code, the arrays phi, theta and data1 are the unorganized data points, with each of these three arrays having a shape (974,). When i run this code I get the error message:
Traceback (most recent call last):
File "spherebiv.py", line 25, in <module>
data_inerp = lut(lat, lon)
File "/usr/lib/python2.7/dist-packages/scipy/interpolate/fitpack2.py", line 958, in __call__
raise ValueError("Error code returned by bispev: %s" % ier)
ValueError: Error code returned by bispev: 10
Now I don't understand this error code - it seems to be something to do with the underlying Fortran routine. Is there anything obvious I'm doing wrong here?

Length-1 Arrays and Python Scalars Via plt.text

I'm trying to use plt.text to plot temperature values at their associated lat/lon points on a plot.
After reviewing the plt.text documentation, it appears that the plotted value (third arg) has to be a number and that the number has to be a whole number, NOT a number with decimals.
Below is the code that I'm trying to work with and the associated traceback error that I'm receiving:
Script Code:
data = np.loadtxt('/.../.../.../tmax_day0', delimiter=',', skiprows=1)
grid_x, grid_y = np.mgrid[-85:64:dx, 34:49:dx]
temp = data[:,2]
#print temp
grid_z = griddata((data[:,1],data[:,0]), data[:,2], (grid_x,grid_y), method='linear')
x,y = m(data[:,1], data[:,0]) # flip lat/lon
grid_x,grid_y = m(grid_x,grid_y)
#m.plot(x,y, 'ko', markersize=2)
def str_to_float(str):
try:
number = float(str)
except ValueError:
number = 0.0
return number
fmt = str_to_float(temp)
#annotate point temperature on plot
plt.text(grid_x, grid_y, fmt, fontdict=None)
Traceback Error:
Traceback (most recent call last):
File "plotpoints.py", line 56, in <module>
fmt = str_to_float(temp)
File "plotpoints.py", line 51, in str_to_float
number = float(str)
TypeError: only length-1 arrays can be converted to Python scalars
Data sample from text file tmax_day0:
latitude,longitude,value
36.65408,-83.21783,90
41.00928,-74.73628,92.02
43.77714,-71.75598,90
44.41944,-72.01944,88.8
39.5803,-79.3394,79
38.3154,-76.5501,86
38.91444,-82.09833,94
40.64985,-75.44771,92.6
41.25389,-70.05972,81.2
39.45202,-74.56699,90.88

I was able to achieve plotting data values only by using the following code:
for i in range(len(temp)):
plt.text(x[i], y[i], temp[i], va="top", family="monospace")
Result:

You aren't using a "proper" array, and are instead using a numpy array. Numpy arrays don't play well with non-numpy functions.
Going from your comment, this has been edited.
You would first need to fix the string so it's a proper array.
fmt = fmt[0].split()
I think should work to create a new (normal) array of strings. And then this to map that to an array of floats:
list_of_floats = np.array(map(float, fmt))