MPI4Py Scatter sendbuf Argument Type? - python

I'm having trouble with the Scatter function in the MPI4Py Python module.
My assumption is that I should be able to pass it a single list for the sendbuffer. However, I'm getting a consistent error message when I do that, or indeed add the other two arguments, recvbuf and root:
File "code/step3.py", line 682, in subbox_grid
i = mpi_communicator.Scatter(station_range, station_data)
File "Comm.pyx", line 427, in mpi4py.MPI.Comm.Scatter (src/
mpi4py_MPI.c:44993)
File "message.pxi", line 321, in mpi4py.MPI._p_msg_cco.for_scatter
(src/mpi4py_MPI.c:14497)
File "message.pxi", line 232, in mpi4py.MPI._p_msg_cco.for_cco_send
(src/mpi4py_MPI.c:13630)
File "message.pxi", line 36, in mpi4py.MPI.message_simple (src/
mpi4py_MPI.c:11904)
ValueError: message: expecting 2 or 3 items
Here is the relevant code snipped, starting a few lines above 682
mentioned above.
for station in stations
#snip--do some stuff with station
station_data = []
station_range = range(1,len(station))
mpi_communicator = MPI.COMM_WORLD
i = mpi_communicator.Scatter(station_range, nsm)
#snip--do some stuff with station[i]
nsm = combine(avg, wt, dnew, nf1, nl1, wti[i], wtm, station[i].id)
station_data = mpi_communicator.Gather(station_range, nsm)
I've tried a number of combinations initializing station_range, but I
must not be understanding the Scatter argument types properly.
Does a Python/MPI guru have a clarification this?

If you want to move raw buffers (as with Gather), you provide a triplet [buffer, size, type]. Look at the demos for examples of this. If you want to send Python objects, you should use the higher level interface and call gather (note the lowercase) which uses pickle internally.

Related

code runs in VScode, but not in Dynamo embedded python

i’m trying to connect a module that provides a straight skeleton algorithm to the embedded python in Autodesk Revit's Dynamo environment.
i’m trying to debug things in VS code using a serialized data from Revit, so i think i have VScode using the embedded python that runs in dynamo.
my test code runs in VScode. Dynamo is able to find the module.
i’m trying to bring it over incrementally. when i add in the line that actually calls the function, i get this error:
Warning: RuntimeError : generator raised StopIteration [' File "<string>", line 40, in <module>\n', ' File "C:\\Users\\mclough\\src\\polyskel2\\polyskel\\polyskel.py", line 437, in skeletonize\n slav = _SLAV(polygon, holes)\n', ' File "C:\\Users\\mclough\\src\\polyskel2\\polyskel\\polyskel.py", line 207, in __init__\n self._original_edges = [_OriginalEdge(LineSegment2(vertex.prev.point, vertex.point), vertex.prev.bisector, vertex.bisector) for vertex in chain.from_iterable(self._lavs)]\n', ' File "C:\\Users\\mclough\\src\\polyskel2\\polyskel\\polyskel.py", line 207, in <listcomp>\n self._original_edges = [_OriginalEdge(LineSegment2(vertex.prev.point, vertex.point), vertex.prev.bisector, vertex.bisector) for vertex in chain.from_iterable(self._lavs)]\n']
i did debug this error (or a similar one?) in VScode, which seems to be related to changes in python 3.8. Is it maybe possible that Dynamo’s embedded python is somehow looking at an old version of the file? Just seems weird that it runs in VScode, but not in dynamo when both versions of python are the same.
I don't understand how i'm supposed to interpret the last line in the Error. I don't see anything that generates a StopIteration exception. The only thing i'm thinking about is the call of 'chain.from_iterable', which is part of itertools. Does it make any sense that would be the issue?
Code from the python node:
`
# Load the Python Standard and DesignScript Libraries
import sys
import clr
#this is temporary. final path should be the KS 'common lib'
sys.path.append(r'C:\Users\mclough\src\polyskel2\polyskel')
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
#sys.path.append(r'C:\ProgramData\Anaconda3\envs\Dynamo212\Lib\site-packages')
import polyskel
#import skgeom as sg
# The inputs to this node will be stored as a list in the IN variables.
# input data is serialized geometry of surface polycurves
# first element is coordinates of the boundary, listed counterclockwise
# remainder elements describe any holes, listed clockwise.
data = IN[0]
# Place your code below this line
skeletons=[]
for shape in data:
bdyPoints = []
boundary = shape.pop(0)
for b in boundary:
x,y=b[0]
bdyPoints.append([float(x),float(y)])
#remaining entries are the holes
holes = []
for curve in shape:
hlePoints=[]
for h in curve:
x,y = h[0]
hlePoints.append([float(x),float(y)])
holes.append(hlePoints)
sk = polyskel.skeletonize(bdyPoints,holes)
#shapes.append([bdyPoints,holes])
skeletons.append(sk)
# Assign your output to the OUT variable.
OUT = []
`
I've checked that VScode and Dynamo are using the same version of python.
I'm using the same input (geometry) data in both instances.
I verified that the modules are discoverable in both environments
I've searched for information on the Runtime Error

Passing Argument to MATLAB Function in Python Results in Error: Size inputs must be numeric

I'm currently working on a python script that passes an argument to a function getWeight.m written in MATLAB.
getWeight.m returns a weight matrix with the same dimensions given the size of an RGB image. It's source code can be found here.
I'm running the following python script:
import matlab.engine
eng = matlab.engine.start_matlab()
import cv2
image = cv2.imread('warped_image.png')
weight = eng.getWeight(image.shape) # (575, 503, 3)
print(weight)
Passing image.shape of class tuple as an argument to the function yields the following error:
Size inputs must be numeric
Are there any workarounds to this error?
Here's the error after following #Chris method:
Traceback (most recent call last):
File "test.py", line 10, in <module>
weight = eng.getWeight(matlab.int8(list(image.shape))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matlab/
mlarray.py", line 106, in __init__
raise ex
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matlab/
mlarray.py", line 104, in __init__
super(int8, self).__init__('b', initializer, size, is_complex)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matlab/
_internal/mlarray_sequence.py", line 57, in __init__
self._data = flat(self, initializer, init_dims, typecode)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/matlab/
_internal/mlarray_sequence.py", line 338, in flat
return array.array(typecode, nested_list)
OverflowError: signed char is greater than maximum
Check out how matlab converts python types: https://www.mathworks.com/help/matlab/matlab_external/pass-data-to-matlab-from-python.html
A python tuple will be converted to a matlab cell array, so in your getWeight function when you access the indeces with size(3) you're selecting a cell rather than the value of the cell.
You can change your getWeight function to pull out the values of the cells (e.g., size{3}) and it should work. If this function is used in many other places you may want to add some checks within getWeight to see if size is a normal array (when it should use size(3)) vs a cell array.
You could also change the input to accept integers instead, e.g., getWeight(s1,s2,s3) and then call it with eng.getWeight(image.shape[0],image.shape[1],image.shape[2]). But again, if getWeight is used elsewhere you'll have to refactor every call.
Finally, you could try converting the tuple to a "matlab numeric array object" and then passing that array to getWeight. See the documentation here for how to create a matlab array in python: https://www.mathworks.com/help/matlab/matlab_external/matlab-arrays-as-python-variables.html I don't have matlab on my present computer, but I believe the following code would work:
import matlab.engine
eng = matlab.engine.start_matlab()
import cv2
image = cv2.imread('warped_image.png')
weight = eng.getWeight(matlab.int8(image.shape))
print(weight)
The examples of creating the matlab arrays in the matlab documentation all use lists (for example A = matlab.int8([1,2,3,4,5])), so you may need to convert that image.shape tuple to a list, but I suspect the tuple will work...

How to iteratively load read_pixel and write to envi file; python3

I want to load hyperspectral data per pixel into an array and write this pixel out again using Python 3.5. I want to calculate something with the spectral information of this Pixel.
I have tried two different ways and both don't work the way I want.
First of all I have updated spectral package since the last version was stated not to work with iteratively envi.save_image but still my approach does not work.
Second my approaches both are not very good with my double for loop - I know -
If anyone could please help me on my problem.
1st:
myfile=open_image('input.hdr')
for i in range(0,myfile.shape[0]):
for j in range(0,myfile.shape[1]):
mypixel = (myfile.read_pixel(i,j))
envi.save_image('output.hdr', mypixel, dtype=np.int16)
1st example does not save the image rather gives me the error code
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 699, in runfile
execfile(filename, namespace)
File "/usr/local/lib/python3.5/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
File "/dtc/Python/Masking.py", line 132, in <module>
envi.save_image('test.hdr', mypixel, dtype=np.int16)#, metadata=myfile.metadata)
File "/usr/local/lib/python3.5/site-packages/spectral/io/envi.py", line 415, in save_image
data, metadata = _prepared_data_and_metadata(hdr_file, image, **kwargs)
File "/usr/local/lib/python3.5/site-packages/spectral/io/envi.py", line 568, in _prepared_data_and_metadata
add_image_info_to_metadata(image, metadata)
File "/usr/local/lib/python3.5/site-packages/spectral/io/envi.py", line 613, in add_image_info_to_metadata
metadata['samples'] = image.shape[1]
IndexError: tuple index out of range
2nd:
myfile=open_image('input.hdr')
envi.create_image('test.hdr',ext='.bip', interleave='bip',dtype='h',force=True,metadata=myfile.metadata)
open('test.bip', 'w').close() # empties the created file
file = open('test.bip', 'ab')#ab #opens the created file for appending the new bands
for i in range(0,myfile.shape[0]):
for j in range(0,myfile.shape[1]):
mypixel = (myfile.read_pixel(i,j))
file.write(mypixel)
file.close()
myfile.close()
The second example saves the image but stores the Pixel in a different order and messes up my image.
So this is the very short, fast and easy solution thanks to a colleague.
myfile=envi.open('input.hdr') #opens image for calculating with it
imageArray = 10000*myfile[:,:,:] #do some math with it;
#10000* is needed because input data are thresholded between {0;10000}
#and during processing get thresholded between {0;1}.
#For preventing 0 in the output with datatype int the thresholding to {0;10000} is necessary again
envi.save_image('test.hdr',imageArray,dtype=np.int16,metadata=myfile.metadata,force=True)
I have to say in advance that I am not familiar with the spectral package and envi and therefore unfortunately cannot offer a ready-to-use solution. Besides, I am not sure if I correctly understood what you are trying to do with your image.
But just some thoughts: Could the write/save function inside the for loop cause your problem, because every pixel is treated in the exact same way and it gets overwritten? I can not relate to the IndexError though.
Maybe you need a function where you can rather write a certain pixel to an empty image passing also i and j. A second option could be to save each pixel in an array and save it to an image at once after the for loop.

ValueError: too many boolean indices

I am trying to visualize some data using matpolid, but i got this error
File "C:\Python27\lib\site-packages\matplotlib\mlab.py", line 2775, in griddata
tri = delaunay.Triangulation(x,y)
File "C:\Python27\lib\site-packages\matplotlib\delaunay\triangulate.py", line 98, in __init__
duplicates = self._get_duplicate_point_indices()
File "C:\Python27\lib\site-packages\matplotlib\delaunay\triangulate.py", line 137, in _get_duplicate_point_indices
return j_sorted[mask_duplicates]
ValueError: too many boolean indices
It happens when i call function
data=griddata(self.dataX,self.dataY,self.dataFreq,xi,yi)
Does anyone know why I got that error? I suppoes it it something with parameters, but I can figure out what
Might be worth updating your matplotlib. There has been a lot of work on the triangulation code that has made it into v1.3.0.
The what's new page for matplotlib v1.3.0 can be found at http://matplotlib.org/users/whats_new.html#triangular-grid-interpolation

Factors Analysis using MDP in Python

Excuse my ignorance, I'm very new to Python. I'm trying to perform factor analysis in Python using MDP (though I can use another library if there's a better solution).
I have an m by n matrix (called matrix) and I tried to do:
import mdp
mdp.nodes.FANode()(matrix)
but I get back an error. I'm guessing maybe my matrix isn't formed properly? My goal is find out how many components are in the data and find out which rows load onto which components.
Here is the traceback:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "mdp/signal_node.py", line 630, in __call__
return self.execute(x, *args, **kwargs)
File "mdp/signal_node.py", line 611, in execute
self._pre_execution_checks(x)
File "mdp/signal_node.py", line 480, in _pre_execution_checks
self.train(x)
File "mdp/signal_node.py", line 571, in train
self._check_input(x)
File "mdp/signal_node.py", line 429, in _check_input
if not x.ndim == 2:
AttributeError: 'list' object has no attribute 'ndim'
Does anyone have any idea what's going on, and feel like explaining it to a Python newbie?
I have absolutely no experience with mdp, but it looks like it expects your matrices to be passed as a Numpy array instead of a list. Numpy is a package for high performance scientific computing. You can go to the Numpy home page and install it. After doing so, try altering your code to this:
import mdp, numpy
mdp.nodes.FANode()(numpy.array(matrix))
As Stephen said, the data must be a numpy array. More precisely it must be a 2D array, with the first index representing the different sampes and the second index representing the data dimensions (using the wrong order here can lead to the "singular matrix" error).
You should also take a look at the MDP documentation, which should answer all your questions. If that doesn't help there is the MDP user mailing list.

Categories