We Keep Coding

mpi4py not letting me send and receive array with dtype = object

I have a numpy array of dtype = object that I am trying to send and receive using comm.Send() and comm.Recv, but I'm running into errors and can't seem to debug it. The array I'm trying to send consists of 2 columns: 1 column of strings and 1 column of integers.
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
data_array = np.empty(100,2), dtype=object)
data_array[:,0] = var_1
data_array[:,1] = var_2
(data_array_0, data_array_1) = np.array_split(data_array, 2)
data_array_0 = np.ascontiguousarray(data_array_0, dtype = object)
data_array_1 = np.ascontiguousarray(data_array_1, dtype = object)
if rank == 0:
comm.Send(data_array_1, dest=1)
elif rank == 1:
data_array_1 = np.empty([data_array_row_1, data_array_col], dtype = object)
comm.Recv(data_array_1, source=0) # <--- the line that's causing the error
I get the following error message:
Traceback (most recent call last):
File "data_clean_parallel_1.py", line 156, in <module>
comm.Recv(data_array_1, source=0)
File "mpi4py/MPI/Comm.pyx", line 283, in mpi4py.MPI.Comm.Recv
File "mpi4py/MPI/msgbuffer.pxi", line 402, in mpi4py.MPI.message_p2p_recv
File "mpi4py/MPI/msgbuffer.pxi", line 388, in mpi4py.MPI._p_msg_p2p.for_recv
File "mpi4py/MPI/msgbuffer.pxi", line 155, in mpi4py.MPI.message_simple
File "mpi4py/MPI/msgbuffer.pxi", line 101, in mpi4py.MPI.message_basic
KeyError: 'O'
I don't really understand what's causing this issue, and if there's any possible alternative that exists where I could send/receive string data using mpi4py.

You will have to use send and recv (lowercase) for numpy arrays:
As seen on https://mpi4py.readthedocs.io/en/stable/overview.html:
The variants MPI.Comm.send(), MPI.Comm.recv() and MPI.Comm.sendrecv()
can communicate general Python objects.

Numba Indexing Error: TypeError: Can't index at [0] in i8*

I'm learning how to use Numba to speed up functions with jit and vectorize. I didn't have any issues with the jit version of this code, but I am getting an index error with vectorize. I suspect this question's answer is getting at the right idea that there is a type error, but I'm not confident on which direction to take on changing the indexing. Included below is the function I've been playing around with, which outputs the Fibonacci numbers up to a chosen index of the sequence. What is going wrong with the indexing, and how I can correct my code to account for it?
from numba import vectorize
import numpy as np
from timeit import timeit
#vectorize
def fib(n):
'''
Adjusted from:
https://lectures.quantecon.org/py/numba.html
https://en.wikipedia.org/wiki/Fibonacci_number
https://www.geeksforgeeks.org/program-for-nth-fibonacci-number/
'''
if n == 1:
return np.ones(1)
elif n > 1:
x = np.empty(n)
x[0] = 1
x[1] = 1
for i in range(2,n):
x[i] = x[i-1] + x[i-2]
return x
else:
print('WARNING: Check validity of input.')
print(timeit('fib(10)', globals={'fib':fib}))
Which results in the following error output.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/llvmlite/ir/instructions.py", line 619, in __init__
typ = typ.elements[i]
AttributeError: 'PointerType' object has no attribute 'elements'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/galen/Projects/myjekyllblog/test_code/quantecon_2.py", line 27, in <module>
print(timeit('fib(10)', globals={'fib':fib}))
File "/usr/lib/python3.6/timeit.py", line 233, in timeit
return Timer(stmt, setup, timer, globals).timeit(number)
File "/usr/lib/python3.6/timeit.py", line 178, in timeit
timing = self.inner(it, self.timer)
File "<timeit-src>", line 6, in inner
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/dufunc.py", line 166, in _compile_for_args
return self._compile_for_argtys(tuple(argtys))
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/dufunc.py", line 188, in _compile_for_argtys
cres, actual_sig)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/ufuncbuilder.py", line 157, in _build_element_wise_ufunc_wrapper
cres.objectmode, cres)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 220, in build_ufunc_wrapper
env=envptr)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 130, in build_fast_loop_body
env=env)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 23, in _build_ufunc_loop_body
store(retval)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 126, in store
out.store_aligned(retval, ind)
File "/usr/local/lib/python3.6/dist-packages/numba/npyufunc/wrappers.py", line 276, in store_aligned
self.context.pack_value(self.builder, self.fe_type, value, ptr)
File "/usr/local/lib/python3.6/dist-packages/numba/targets/base.py", line 482, in pack_value
dataval = self.data_model_manager[ty].as_data(builder, value)
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 558, in as_data
elems = self._as("as_data", builder, value)
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 530, in _as
self.get(builder, value, i)))
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 558, in as_data
elems = self._as("as_data", builder, value)
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 530, in _as
self.get(builder, value, i)))
File "/usr/local/lib/python3.6/dist-packages/numba/datamodel/models.py", line 624, in get
name="extracted." + self._fields[pos])
File "/usr/local/lib/python3.6/dist-packages/llvmlite/ir/builder.py", line 911, in extract_value
instr = instructions.ExtractValue(self.block, agg, idx, name=name)
File "/usr/local/lib/python3.6/dist-packages/llvmlite/ir/instructions.py", line 622, in __init__
% (list(indices), agg.type))
TypeError: Can't index at [0] in i8*

The error is because you are trying to vectorize a function which you can say is essentially not vectorizable. I think you are confusing the functionality of how #jit and #vectorize work. In order to speed up your functions, you use #jit, while #vectorize is used to create numpy universal functions. See the official documentation here :
Using vectorize(), you write your function as operating over input
scalars, rather than arrays. Numba will generate the surrounding loop
(or kernel) allowing efficient iteration over the actual inputs.
So it is essentially not possible to create a numpy universal function which has the same functionality as your fibonacci function. Here is the link for official documentation on universal functions if you are interested.
So in order to use #vectorize, you need to create a function which can be essentially used as a numpy universal function. For your purpose of speeding up your code, you simply need to use #jit.

How to use numba.SmartArrays for vector addition?

I have written this code for vector addition using numba.SmartArrays. I am using this numba.SmartArrays for the first time. I am not sure how to use that.
This code is not working and it is throwing errors.
import numpy as np
from numba import SmartArray,cuda, jit, uint32
li1=np.uint32([1,2,3,4])
li=np.uint32([1,2,3,4])
b=SmartArray(li,where="host",copy=True)
a=SmartArray(li1,where="host",copy=True)
c=np.uint32([1,1,1,1])
print type(li)
print type(a)
#cuda.jit('void(uint32[:],uint32[:],uint32[:])',type="gpu")
def additionG(c,a,b):
idx=cuda.threadIdx.x+cuda.blockDim.x*cuda.blockIdx.x
if idx< len(a):
a[idx]=c[idx]+b[idx]
dA=cuda.to_device(a)
dB=cuda.to_device(b)
dC=cuda.to_device(c)
additionG[1, 128](c,a,b)
print a.__array__()
Errors:
<type 'numpy.ndarray'>
<class 'numba.smartarray.SmartArray'>
Traceback (most recent call last):
File "C:\Users\hp-pc\My Documents\LiClipse Workspace\cuda\blowfishgpu_smart_arrays.py", line 20, in <module>
dA=cuda.to_device(a)
File "C:\Anaconda\lib\site-packages\numba\cuda\cudadrv\devices.py", line 257, in _require_cuda_context
return fn(*args, **kws)
File "C:\Anaconda\lib\site-packages\numba\cuda\api.py", line 55, in to_device
to, new = devicearray.auto_device(obj, stream=stream, copy=copy)
File "C:\Anaconda\lib\site-packages\numba\cuda\cudadrv\devicearray.py", line 403, in auto_device
devobj.copy_to_device(obj, stream=stream)
File "C:\Anaconda\lib\site-packages\numba\cuda\cudadrv\devicearray.py", line 148, in copy_to_device
sz = min(_driver.host_memory_size(ary), self.alloc_size)
File "C:\Anaconda\lib\site-packages\numba\cuda\cudadrv\driver.py", line 1348, in host_memory_size
s, e = host_memory_extents(obj)
File "C:\Anaconda\lib\site-packages\numba\cuda\cudadrv\driver.py", line 1333, in host_memory_extents
return mviewbuf.memoryview_get_extents(obj)
TypeError: expected a readable buffer object

Its been a while since I posted this question. Still posting the answer so that someone may find it helpful in future.
import numpy as np
from numba import SmartArray,cuda, jit, uint32,autojit
li1=np.uint32([6,7,8,9])
li=np.uint32([1,2,3,4])
a=SmartArray(li1,where='host',copy=True)
b=SmartArray(li,where="host",copy=True)
c=np.uint32([1,1,1,1])
def additionG(a,c):
idx=cuda.threadIdx.x+cuda.blockDim.x*cuda.blockIdx.x
if idx < len(c):
a[idx]=a[idx]+c[idx]
cuda.syncthreads()
bpg=1
tpb=128
dC=cuda.to_device(c)
cfunc = cuda.jit()(additionG)
cfunc[bpg, tpb](a,dC)
print a.__array__()

It looks to me like cuda.to_device doesn't handle smart arrays, which would sort of make sense, because smart arrays are supposed to do away with explicit copy management.
If my reading of the documentation is correct (I have never tried SmartArray before), you should just be able to change this
dA=cuda.to_device(a)
dB=cuda.to_device(b)
dC=cuda.to_device(c)
additionG[1, 128](c,a,b)
to just
dC=cuda.to_device(c)
additionG[1, 128](dC,a.gpu(),b.gpu())
The .gpu() method should return a GPU resident object that the kernel can understand and access.

Remove the numba.lowering.LoweringError: Internal error

I'm using numba to speed up my code which is working fine without numba. But after using #jit, it crashes with this error:
Traceback (most recent call last):
File "C:\work_asaaki\code\gbc_classifier_train_7.py", line 54, in <module>
gentlebooster.train(X_train, y_train, boosting_rounds)
File "C:\work_asaaki\code\gentleboost_c_class_jit_v7_nolimit.py", line 298, in train
self.g_per_round, self.g = train_function(X, y, H)
File "C:\Anaconda\lib\site-packages\numba\dispatcher.py", line 152, in _compile_for_args
return self.jit(sig)
File "C:\Anaconda\lib\site-packages\numba\dispatcher.py", line 143, in jit
return self.compile(sig, **kws)
File "C:\Anaconda\lib\site-packages\numba\dispatcher.py", line 250, in compile
locals=self.locals)
File "C:\Anaconda\lib\site-packages\numba\compiler.py", line 183, in compile_bytecode
flags.no_compile)
File "C:\Anaconda\lib\site-packages\numba\compiler.py", line 323, in native_lowering_stage
lower.lower()
File "C:\Anaconda\lib\site-packages\numba\lowering.py", line 219, in lower
self.lower_block(block)
File "C:\Anaconda\lib\site-packages\numba\lowering.py", line 254, in lower_block
raise LoweringError(msg, inst.loc)
numba.lowering.LoweringError: Internal error:
NotImplementedError: ('cast', <llvm.core.Instruction object at 0x000000001801D320>, slice3_type, int64)
File "gentleboost_c_class_jit_v7_nolimit.py", line 103
Line 103 is below, in a loop:
weights = np.empty([n,m])
for curr_n in range(n):
weights[curr_n,:] = 1.0/(n) # this is line 103
where n is a constant already defined somewhere above in my code.
How can I remove the error? What "lowering" is going on? I'm using Anaconda 2.0.1 with Numba 0.13.x and Numpy 1.8.x on a 64-bit machine.

Based on this: https://gist.github.com/cc7768/bc5b8b7b9052708f0c0a,
I figured out what to do to avoid the issue. Instead of using the colon : to refer to any row/column, I just opened up the loop into two loops to explicitly refer to the indices in each dimension of the array:
weights = np.empty([n,m])
for curr_n in range(n):
for curr_m in range (m):
weights[curr_n,curr_m] = 1.0/(n)
There were other instances in my code after this where I used the colon, but they didn't cause errors further down, not sure why.

MemoryError when using np.median()

I have two files that I am reading data from, doing calculations, and plotting a graph with. One file is quite small ~50 KB and raises no problem with the script. The other file is ~ 702, 900 KB (this is the file that causes the problem). I am able to read in the data perfectly fine, though when I calculate the row-by-row medians for this particular file, the script fails and gives me a MemoryError. It looks like the following:
RMSDataS1 = [y01S1, y02S1, y03S1, y04S1, y05S1, y06S1, y07S1, y08S1, y09S1,
y010S1, y011S1, y012S1, y013S1, y014S1, y015S1, y016S1, y017S1,
y018S1, y019S1, y020S1, y021S1, y022S1, y023S1, y024S1, y025S1,
y026S1, y027S1, y028S1, y029S1, y030S1, y031S1, y032S1, y033S1,
y034S1, y035S1, y036S1, y037S1, y038S1, y039S1, y040S1, y041S1,
y042S1, y043S1, y044S1, y045S1, y046S1, y047S1, y048S1, y049S1,
y050S1, y051S1, y052S1, y053S1, y054S1, y055S1, y056S1, y057S1,
y058S1, y059S1, y060S1, y061S1, y062S1, y063S1, y064S1, y065S1,
y066S1, y067S1, y068S1, y069S1, y070S1, y071S1, y072S1, y073S1,
y074S1, y075S1, y076S1, y077S1, y078S1, y079S1, y080S1, y081S1,
y082S1, y083S1, y084S1, y085S1, y086S1, y087S1, y088S1, y089S1,
y090S1, y091S1, y092S1, y093S1, y094S1, y095S1, y096S1, y097S1,
y098S1, y099S1, y0100S1, y0101S1, y0102S1, y0103S1, y0104S1, y0105S1,
y0106S1, y0107S1, y0108S1]
MediansS1 = []
MediansS1 = np.median(RMSDataS1, axis = 0)
Is there any convenient way to get around this? I believe the script is failing when trying to sort the data when calculating the medians.
The error:
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\xy\RMSTrialOriginal-Aera.py", line 511, in <module>
MediansS1 = np.average(RMSDataS1, axis = 0)
File "C:\Python27\lib\site-packages\numpy\lib\function_base.py", line 486, in average
a = np.asarray(a)
File "C:\Python27\lib\site-packages\numpy\core\numeric.py", line 235, in asarray
return array(a, dtype, copy=False, order=order)
MemoryError
Any help would be greatly appreciated!