We Keep Coding

MetPy suface_based_cape_cin returning error with units

At my site, we're having an issue with MetPy returning a units error when trying to call surface_based_cape_cin
I am seeing the following error:
Traceback (most recent call last):
File "Advanced_Sounding_3Dnetcdf2.py", line 202, in <module>
sbcape, sbcin = mpcalc.surface_based_cape_cin(p1, T1, Td1)
File "/gpfs/group/kal6112/default/sw/anaconda3/lib/python3.6/site-packages/metpy/xarray.py", line 677, in wrapper
return func(*args, **kwargs)
File "/gpfs/group/kal6112/default/sw/anaconda3/lib/python3.6/site-packages/metpy/units.py", line 320, in wrapper
return func(*args, **kwargs)
File "/gpfs/group/kal6112/default/sw/anaconda3/lib/python3.6/site-packages/metpy/calc/thermo.py", line 1851, in surface_based_cape_cin
return cape_cin(p, t, td, profile)
File "/gpfs/group/kal6112/default/sw/anaconda3/lib/python3.6/site-packages/metpy/xarray.py", line 677, in wrapper
return func(*args, **kwargs)
File "/gpfs/group/kal6112/default/sw/anaconda3/lib/python3.6/site-packages/metpy/units.py", line 319, in wrapper
raise ValueError(msg)
ValueError: `cape_cin` given arguments with incorrect units: `temperature` requires "[temperature]" but given "none", `dewpt` requires "[temperature]" but given "none".
When I check the incoming values p1, T1, and Td1 they all have the correct units (hectopascal, degree_Celcius).
Just to be sure I added the following and checked the results prior to the call to surface_based_cape_cin:
p1 = units.hPa * phPa
T1 = units.degC * TdegC
Td1 = units.degC * TddegC
I'm running the following version of MetPy
# Name Version Build Channel
metpy 0.12.2 py_0 conda-forge
I don't recall having this prior to updating to this version but I can't be certain the problem I'm seeing arose after the update or not.
Thanks for any help you can provide.

This is definitely a bug in MetPy, likely due to more challenges with masked arrays and preserving units. I've opened a new issue. In the meanwhile as a work-around, it's probably best to just eliminate masked arrays with something like:
p1 = p1.compressed() * p1.units
T1 = T1.compressed() * T1.units
Td1 = Td1.compressed() * Td1.units
This will work so long as the data have no actual masked values or if all 3 arrays are masked in the same spot. If not, you'll need to do some more work to remove any of the levels where one of the values is masked.

Numba Indexing Error: TypeError: Can't index at [0] in i8*

I'm learning how to use Numba to speed up functions with jit and vectorize. I didn't have any issues with the jit version of this code, but I am getting an index error with vectorize. I suspect this question's answer is getting at the right idea that there is a type error, but I'm not confident on which direction to take on changing the indexing. Included below is the function I've been playing around with, which outputs the Fibonacci numbers up to a chosen index of the sequence. What is going wrong with the indexing, and how I can correct my code to account for it?
from numba import vectorize
import numpy as np
from timeit import timeit
#vectorize
def fib(n):
'''
Adjusted from:
https://lectures.quantecon.org/py/numba.html
https://en.wikipedia.org/wiki/Fibonacci_number
https://www.geeksforgeeks.org/program-for-nth-fibonacci-number/
'''
if n == 1:
return np.ones(1)
elif n > 1:
x = np.empty(n)
x[0] = 1
x[1] = 1
for i in range(2,n):
x[i] = x[i-1] + x[i-2]
return x
else:
print('WARNING: Check validity of input.')
print(timeit('fib(10)', globals={'fib':fib}))
Which results in the following error output.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/llvmlite/ir/instructions.py", line 619, in __init__
typ = typ.elements[i]
AttributeError: 'PointerType' object has no attribute 'elements'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/galen/Projects/myjekyllblog/test_code/quantecon_2.py", line 27, in <module>
print(timeit('fib(10)', globals={'fib':fib}))
File "/usr/lib/python3.6/timeit.py", line 233, in timeit
return Timer(stmt, setup, timer, globals).timeit(number)
File "/usr/lib/python3.6/timeit.py", line 178, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 6, in inner
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/dufunc.py", line 166, in _compile_for_args
return self._compile_for_argtys(tuple(argtys))
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/dufunc.py", line 188, in _compile_for_argtys
cres, actual_sig)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/ufuncbuilder.py", line 157, in _build_element_wise_ufunc_wrapper
cres.objectmode, cres)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 220, in build_ufunc_wrapper
env=envptr)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 130, in build_fast_loop_body
env=env)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 23, in _build_ufunc_loop_body
store(retval)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 126, in store
out.store_aligned(retval, ind)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 276, in store_aligned
self.context.pack_value(self.builder, self.fe_type, value, ptr)
File "/usr/local/lib/python3.6/dist-packages/numba/targets/base.py", line 482, in pack_value
dataval = self.data_model_manager[ty].as_data(builder, value)
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 558, in as_data
elems = self._as("as_data", builder, value)
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 530, in _as
self.get(builder, value, i)))
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 558, in as_data
elems = self._as("as_data", builder, value)
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 530, in _as
self.get(builder, value, i)))
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 624, in get
name="extracted." + self._fields[pos])
File "/usr/local/lib/python3.6/dist-packages/llvmlite/ir/builder.py", line 911, in extract_value
instr = instructions.ExtractValue(self.block, agg, idx, name=name)
File "/usr/local/lib/python3.6/dist-packages/llvmlite/ir/instructions.py", line 622, in __init__
% (list(indices), agg.type))
TypeError: Can't index at [0] in i8*

The error is because you are trying to vectorize a function which you can say is essentially not vectorizable. I think you are confusing the functionality of how #jit and #vectorize work. In order to speed up your functions, you use #jit, while #vectorize is used to create numpy universal functions. See the official documentation here :
Using vectorize(), you write your function as operating over input
scalars, rather than arrays. Numba will generate the surrounding loop
(or kernel) allowing efficient iteration over the actual inputs.
So it is essentially not possible to create a numpy universal function which has the same functionality as your fibonacci function. Here is the link for official documentation on universal functions if you are interested.
So in order to use #vectorize, you need to create a function which can be essentially used as a numpy universal function. For your purpose of speeding up your code, you simply need to use #jit.

My created function won't accept an array as one of the arguments

I've written a function that takes two arguments, one for no. dimensions and another for no. simulations. The function does exactly what is needed (calculating the volume of a unit hypersphere), however when I wish to plot the function over a range of dimensions it returns an error: ''list' object cannot be interpreted as an integer'.
My function is the following,
def hvolume(ndim, nsim):
ob = [np.random.uniform(0.0,1.0,(nsim, ndim))]
ob = np.concatenate(ob)
i = 0
res = []
while i <= nsim-1:
arr = np.sqrt(np.sum(np.square(ob[i])))
i += 1
res.append(arr)
N = nsim
n = ndim
M = len([i for i in res if i <= 1])
return ((2**n)*M/N)
The error traceback is:
Traceback (most recent call last):
File "<ipython-input-192-4c4a2c778637>", line 1, in <module>
runfile('H:/Documents/Python Scripts/Q4ATTEMPT.py', wdir='H:/Documents/Python Scripts')
File "C:\Users\u1708511\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 668, in runfile
execfile(filename, namespace)
File "C:\Users\u1708511\AppData\Local\Continuum\anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "H:/Documents/Python Scripts/Q4ATTEMPT.py", line 20, in <module>
print(hvolume(d, 2))
File "H:/Documents/Python Scripts/Q4ATTEMPT.py", line 4, in hvolume
ob = [np.random.uniform(0.0,1.0,(nsim, ndim))]
File "mtrand.pyx", line 1307, in mtrand.RandomState.uniform
File "mtrand.pyx", line 242, in mtrand.cont2_array_sc
TypeError: 'list' object cannot be interpreted as an integer
I really have no idea where to go from here, and have searched thoroughly online for how to resolve this. Unfortunately I'm a beginner with this!
Any help is appreciated.

If you simply try your first line in the function;
ob = [np.random.uniform(0.0,1.0,(nsim, ndim))]
with a list as one of the variables like so;
[np.random.uniform(0.0,1.0,([1,2], 2))]
you will get the error:
TypeError: 'list' object cannot be interpreted as an integer
This is because the uniform command it looking for an integer, not a list. You will need to make a for loop if you would like to handle lists.

One pattern I use for situations like this would be to begin the function with a block to handle the case of if they're iterators. Something like this for example.
from collections import Iterator
def hvolume(ndim, nsim):
outputs = []
if isinstance(ndim, Iterator):
for ndim_arg in ndim:
outputs.append(hvolume(ndim_arg, nsim))
if isinstance(nsim, Iterator):
for nsim_arg in nsim:
outputs.append(hvolume(ndim, nsim_arg))
if len(outputs) == 0: # neither above is an Iterator
# ... the rest of the function but it appends to outputs
return outputs

Check the input parameters of your method "hvolume", it seems that you give a list either nsim or ndim, which should be both integer values. That makes the uniform throw a TypeError Exception.

multi-threading in Python with pulp

I have a function which accepts a list R. In this function, I have defined an optimization problem using "pulp", This is my function:
import pulp
from multiprocessing.dummy import Pool as ThreadPool
def optimize(R):
variables = ["x1","x2","x3","x4"]
costs = {"x1":R[0], "x2":R[1], "x3":R[2], "x4":R[3]}
constraint = {"x1":5, "x2":7, "x3":4, "x4":3}
prob_variables = pulp.LpVariable.dicts("Intg",variables,
lowBound=0,
upBound=1,
cat=pulp.LpInteger)
prob = pulp.LpProblem("test1", pulp.LpMaximize)
# defines the constraints
prob += pulp.lpSum([constraint[i]*prob_variables[i] for i in variables]) <= 14
# defines the objective function to maximize
prob += pulp.lpSum([costs[i]*prob_variables[i] for i in variables])
pulp.GLPK().solve(prob)
# Solution
return pulp.value(prob.objective)
To get the output, I used a list as my input and the output is correct:
my_input = [[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]]
results =[]
for i in range(0,len(my_input)):
results.append(optimize(my_input[i]))
print("*"*20)
print(results)
But, I want to use multi-threading instead of the for loop. So, I used:
my_input = [[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]]
pool = ThreadPool(4)
results = pool.map(optimize, my_input)
But it gives me some errors:
Traceback (most recent call last):
File "/Users/Mohammad/PycharmProjects/untitled10/multi_thread.py", line 35, in <module>
results = pool.map(optimize, my_input)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/Users/Mohammad/PycharmProjects/untitled10/multi_thread.py", line 27, in optimize
pulp.GLPK().solve(prob)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/PuLP-1.6.1-py3.5.egg/pulp/solvers.py", line 179, in solve
return lp.solve(self)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/PuLP-1.6.1-py3.5.egg/pulp/pulp.py", line 1643, in solve
status = solver.actualSolve(self, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/PuLP-1.6.1-py3.5.egg/pulp/solvers.py", line 377, in actualSolve
raise PulpSolverError("PuLP: Error while executing "+self.path)
pulp.solvers.PulpSolverError: PuLP: Error while executing glpsol
Can anybody help me?
In my actual code, my_input list has the length of 27 (instead of 4 in the above code) and for each one, in my function I have to perform 80k optimizations (instead of one in the above code). So, multi-threading is a big help for me.

I have seen that class pulp.solvers.COIN_CMD has a threads argument, although the documentation is quite laconic. Taking a look at the code source, it seems to be indeed a way to provide threads to the solver.
If naming is indeed the issue, consider adding the desired name index for a given problem as an input argument to the function. Something like:
def optimize(tup): # here, tup contains (idx, R), so as to be callable using pool.map
...
prob = pulp.LpProblem('test'+str(idx), pulp.LpMaximize)
...
and then something like:
my_input = [[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]]
pool = ThreadPool(4)
results = pool.map(optimize, enumerate(my_input))

python error using PaCal statistical package

I recently started exploring Python and have encountred a problem with a package named PaCal
Everything looks to be working fine except that I keep having this error anytime I want to print out some data (like in print A.mean() )
the error line is :
Traceback (most recent call last):
File "C:\Users\rmobenta\Desktop\tt.py", line 12, in <module>
print A.interval(0.95)
File "C:\Python27\lib\site-packages\pacal\distr.py", line 229, in interval
return self.quantile(p_lim), self.quantile(1.0 - p_lim)
File "C:\Python27\lib\site-packages\pacal\distr.py", line 215, in quantile
return self.get_piecewise_cdf().inverse(y)
File "C:\Python27\lib\site-packages\pacal\segments.py", line 1721, in inverse
x = findinv(segi.f, a = segi.a, b = segi.b, c = y, rtol = params.segments.cumint.reltol, maxiter = params.segments.cumint.maxiter) # TODO PInd, MInf
File "C:\Python27\lib\site-packages\pacal\utils.py", line 384, in findinv
return brentq(lambda x : fun(x) - c, a, b, **kwargs)
File "C:\Python27\lib\site-packages\scipy\optimize\zeros.py", line 414, in brentq
raise ValueError("rtol too small (%g < %g)" % (rtol, _rtol))
ValueError: rtol too small (1e-16 < 4.44089e-16)
I am using a two-line script that I got for a demo (given by the author of this package) and have no idea how to tackle this issue.
Here is the script:
from pacal import *
Y = UniformDistr(1, 2)
X = UniformDistr(3, 4)
A = atan(Y / X)
A.plot()
print A.mean()
print A.interval(0.95)

The problem comes from PaCal that defines in l.141 of params.py: segments.vumint.reltol = 1e-16.
This is the value passed as rtol in segments.py to the SciPy function brentq().
Finally it is compared to numpy.finfo(float).eps * 2 (l.413 and l.10 of scipy/optimize/zeros.py) and is unfortunately lesser.
So it could be a problem of PaCal implementation, not related to your code.
Note that the value you provided to interval() corresponds to the default value (l.222 of distr.py).
I think you should contact the PaCal developers to get more informations and probably open an issue.