I created a module in c++ and need to use the results in python.
Already wrote a wrapper and it is working with this code
a = np.empty([r, hn])
for i in xrange(r):
for j in xrange(hn):
a[i,j]=self.thisptr.H[i*hn+j]
return a
The code is working, but I think there should be an easier and faster way to handle the pointer data.
Sadly I am not used to python and cython and can't figure it out myself.
Any help would be appreciated. :)
Typed memoryviews (http://docs.cython.org/src/userguide/memoryviews.html) are your friend here.
a = np.empty([r,hn])
# interpret the array as a typed memoryview of shape (r, hn)
# and copy into a
# I've assumed the array is of type double* for the sake of answering the question
a[...] = <double[:r,:hn]>self.thisptr.H
It well may not be a huge amount faster (internally it's a loop pretty similar to what your wrote), but it is easier.
Alternatively, even simpler, just using the example from the documentation (http://docs.cython.org/src/userguide/memoryviews.html#coercion-to-numpy)
a = np.asarray(<double[:r,:hn]>self.thisptr.H)
A possible approach is to manually write the wrapper in C. The struct of your Python object can contain a pointer to the C++ object. Looking at my code (I did this is 2005), I see that I tested for NULL in C functions that need the C++ object and created it on the fly.
Nice side effect is that you don't have to expose all C++ methods 1:1 to Python and you can adjust the interface to make it more Pythonic. In my wrapper, I stored some additional information in the struct to be able to emulate Python list behaviour and to make loading data into the C++ object more efficient.
Related
I'm trying to create something and I don't know if it's possible or "clean" :
From python, call function A of my C++ code to compute something complicated
The C++ code returns just the pointer to the python
Do other things in python...
From python, call function B of my C++ code, it takes the pointer and other things as arguments.
I really don't need to use my complicated C++ class in my Python algorithm, that's why I just want to save the pointer in python.
Anyone has any advice on how to do that ?
Edit : In the end I wrapped the c++ class in python, thank you everyone.
A pointer is just data that can be marshaled and sent to anything. It is however a very bad idea because when doing that, you have to assure that that pointer remains valid as long as the python part has the pointer. There is no possibility to check whether the pointer is still valid, so dereferencing a pointer that you receive from an external party could crash your program.
A better idea in a lot of situations is to send a key to a table. When that key is sent back, it can be used to get the needed information from that table and it can be handled when the table doesn't have the key anymore. It is easiest to use std::map for the table. Of course, you could store the pointer in a container and check for that, but a string or number is easier to debug.
It would be better to create a class in C++ and store that pointer in the class itself as private. Then create function calls to access those pointers.
Once the class is implemented generate the .so file of your lib and import it in python. This way you can simply use your C++ code in python and also will not have to save the pointer.
Note: More background and detail is provided before the question is repeated following the code block.
Big Picture
I’m building a library for use in both C++ and Python. The implementation is in C++, with a thin wrapper for users to be able to build projects in Python using this library with minimal impact on performance. The option to use an API in both languages is a hard requirement.
Ultimately, I want to make things simple (in terms of how much users need to type on their keyboard to do something that will come up frequently) for users of this library. This question concerns the Python API.
A little deeper now...
I’m working on a project where I have a class written in C++, and I’m exposing it to Python. I’m using pybind11 to do this. I want to treat my class as a NumPy ndarray without creating a copy.
This will allow users of the Python API to work entirely within Python, which is a hard requirement of this project. The use of numpy without making copies of the underlying data is also a hard requirement.
Illustrative Example
Let’s say I’ve defined a class Foo in C++ and followed the instructions on pybind11 to allow for the following (actually, I have done this, except the part where it says, “I want this...”):
from mymodule import Foo
import NumPy as np
# assume arg1 and arg2 are defined
f = Foo(arg1)
g =Foo(arg2)
h = Foo() # Foo has multiple constructors
# I want this...
h[:] = f + g
# ...to be equivalent to this (this is the part I’ve done successfully):
H = np.array(h, copy=False)
H[:] = np.array(f, copy=False) + np.array(g, copy=False)
The Question
Is there a way to do what I described in the code block above?
This is not an XY Problem
The goal is to force the user to type less, given the requirement that data be accessible via numpy.ndarray objects without making copies of the underlying data. Ideally, that means treating the objects as if they are already numpy.ndarray objects without any additional function calls or operations.
Explored so far...
I’m aware that this asking a lot of Python because I’m operating on a different type from what f and g are — they don’t even inherit from numpy.ndarray.
I’ve tried looking into class decorators. I wonder if could import Foo, and then decorate it so that f=Foo() is equivalent to f=np.array(Foo(), copy=False), where Foo is implemented in C++. I’m not sure how decorators work, despite RTFM, if this is the right track, or if I’m close to proving this is impossible.
I'm extending python code with c++ functions acting on Numpy arrays (very large).
Due to legacy i currently have both PyBind and Python API functions, both for Python 3.6 and above.
As soon as I access memory via ptr, I would love to be sure that memory layout is exactly corresponding to c++ array under this ptr.
I found that transposed array has exactly the same content in ptr in both cases. I also found that subarrays being send via Python API give in c++ exactly the same ptr as if it was the full array. In the course of development and testing i also observed more weird examples I believe, but cannot reproduce them any longer.
I cannot find any recipes on the internet so far. My solution is to make copy of all input arrays in Python, like
f(a.copy(), b.copy())
It seem to work well.
Is this optimal/sufficient solution?
I do not have any limitations on how input arrays have been produced. Transpose, subarray, reshape, in any combinations.
With pybind11, you can use the py::array::c_style flag as described in Matt Eding’s link. Numpy’s C API provides much the same functionality via the NPY_ARRAY_C_CONTIGUOUS flag. In either case, the array will be copied implicitly if needed to satisfy the layout requirements; if you prefer to reject such arguments (to avoid silent inefficiency), you’ll have to check the array’s flags yourself.
http://www.swig.org/papers/PyTutorial98/PyTutorial98.pdf
It comes from above link:
I know that it is an old publication so it is possible that information is outdated.
I would like to ask:
"Seems to work fine with C++ if you aren't being too clever"
What does it mean, to be too clever?
Is there known situation/case that I shuold be very careful where I am programming C++ modules and extending Python using swig tool?
This PDF appears to be a copy of slides from a presentation given by David Beazley at the 7th International Python Conference. My guess is there was a joke or verbal explanation of what he meant by that phrase.
Seems to work fine with C++ if you aren't being too clever
Here is a link to his website if you want to get in touch with him and ask him directly. His twitter account is dabeaz, which may (or may not) be a better way of contacting him.
The slide is strange and misleading. SWIG does not transform pass-by-value into pass-by-reference at all. Let me try to clarify by an example:
Let's say that as in the example you have the C++ function
double dot_product(Vector a, Vector b);
Now in plain C++ (no SWIG, no wrapping) you may use this function as in the following examples:
1.
Vector a = Vector(1,0);
Vector b = Vector(0,1);
double zero = dot_product(a, b);
2.
Vector *a = new Vector(1,0);
Vector *b = new Vector(0,1);
double zero = dot_product(*a, *b);
In both cases, the function is in fact called in exactly the same way using call-by-value.
SWIG wraps all objects into a structure that contains a pointer to the object, so under the hood SWIG passes pointers around for everything, and therefore uses a syntax as in the second example. But there is no conversion / transformation of call semantics going on whatsoever.
To answer your questions:
"Seems to work fine with C++ if you aren't being too clever" What does it mean, to be too clever?
I have no idea. As stated in another answer, likely a joke.
Is there known situation/case that I shuold be very careful where I am programming C++ modules and extending Python using swig tool?
This is a very broad question, and there certainly are pitfalls, especially related to memory management. However, this particular "transformation" is not an issue.
For reference, here is the relevant entry in the SWIG manual. Note that it is worded differently: The function is transformed to accept pointers. Nothing is said about "call semantics" (since this is a non-issue).
I have been mulling over writing a peak-fitting library for a while. I know Python fairly well and plan on implementing everything in Python to begin with but envisage that I may have to re-implement some core routines in a compiled language eventually.
IIRC, one of Python's original remits was as a prototyping language, however Python is pretty liberal in allowing functions, functors, objects to be passed to functions and methods, whereas I suspect the same is not true of say C or Fortran.
What should I know about designing functions/classes which I envisage will have to interface into the compiled language? And how much of these potential problems are dealt with by libraries such as cTypes, bgen, SWIG, Boost.Python, Cython or Python SIP?
For this particular use case (a fitting library), I imagine allowing users to define mathematical functions (Guassian, Lorentzian etc.) as Python functions which can then to be passed an interpreted by the compiled code fitting library. Passing and returning arrays is also essential.
Finally a question that I can really put a value answer to :).
I have investigated f2py, boost.python, swig, cython and pyrex for my work (PhD in optical measurement techniques). I used swig extensively, boost.python some and pyrex and cython a lot. I also used ctypes. This is my breakdown:
Disclaimer: This is my personal experience. I am not involved with any of these projects.
swig:
does not play well with c++. It should, but name mangling problems in the linking step was a major headache for me on linux & Mac OS X. If you have C code and want it interfaced to python, it is a good solution. I wrapped the GTS for my needs and needed to write basically a C shared library which I could connect to. I would not recommend it.
Ctypes:
I wrote a libdc1394 (IEEE Camera library) wrapper using ctypes and it was a very straigtforward experience. You can find the code on https://launchpad.net/pydc1394. It is a lot of work to convert headers to python code, but then everything works reliably. This is a good way if you want to interface an external library. Ctypes is also in the stdlib of python, so everyone can use your code right away. This is also a good way to play around with a new lib in python quickly. I can recommend it to interface to external libs.
Boost.Python: Very enjoyable. If you already have C++ code of your own that you want to use in python, go for this. It is very easy to translate c++ class structures into python class structures this way. I recommend it if you have c++ code that you need in python.
Pyrex/Cython: Use Cython, not Pyrex. Period. Cython is more advanced and more enjoyable to use. Nowadays, I do everything with cython that i used to do with SWIG or Ctypes. It is also the best way if you have python code that runs too slow. The process is absolutely fantastic: you convert your python modules into cython modules, build them and keep profiling and optimizing like it still was python (no change of tools needed). You can then apply as much (or as little) C code mixed with your python code. This is by far faster then having to rewrite whole parts of your application in C; you only rewrite the inner loop.
Timings: ctypes has the highest call overhead (~700ns), followed by boost.python (322ns), then directly by swig (290ns). Cython has the lowest call overhead (124ns) and the best feedback where it spends time on (cProfile support!). The numbers are from my box calling a trivial function that returns an integer from an interactive shell; module import overhead is therefore not timed, only function call overhead is. It is therefore easiest and most productive to get python code fast by profiling and using cython.
Summary: For your problem, use Cython ;). I hope this rundown will be useful for some people. I'll gladly answer any remaining question.
Edit: I forget to mention: for numerical purposes (that is, connection to NumPy) use Cython; they have support for it (because they basically develop cython for this purpose). So this should be another +1 for your decision.
I haven't used SWIG or SIP, but I find writing Python wrappers with boost.python to be very powerful and relatively easy to use.
I'm not clear on what your requirements are for passing types between C/C++ and python, but you can do that easily by either exposing a C++ type to python, or by using a generic boost::python::object argument to your C++ API. You can also register converters to automatically convert python types to C++ types and vice versa.
If you plan use boost.python, the tutorial is a good place to start.
I have implemented something somewhat similar to what you need. I have a C++ function that
accepts a python function and an image as arguments, and applies the python function to each pixel in the image.
Image* unary(boost::python::object op, Image& im)
{
Image* out = new Image(im.width(), im.height(), im.channels());
for(unsigned int i=0; i<im.size(); i++)
{
(*out)[i] == extract<float>(op(im[i]));
}
return out;
}
In this case, Image is a C++ object exposed to python (an image with float pixels), and op is a python defined function (or really any python object with a __call__ attribute). You can then use this function as follows (assuming unary is located in the called image that also contains Image and a load function):
import image
im = image.load('somefile.tiff')
double_im = image.unary(lambda x: 2.0*x, im)
As for using arrays with boost, I personally haven't done this, but I know the functionality to expose arrays to python using boost is available - this might be helpful.
The best way to plan for an eventual transition to compiled code is to write the performance sensitive portions as a module of simple functions in a functional style (stateless and without side effects), which accept and return basic data types.
This will provide a one-to-one mapping from your Python prototype code to the eventual compiled code, and will let you use ctypes easily and avoid a whole bunch of headaches.
For peak fitting, you'll almost certainly need to use arrays, which will complicate things a little, but is still very doable with ctypes.
If you really want to use more complicated data structures, or modify the passed arguments, SWIG or Python's standard C-extension interface will let you do what you want, but with some amount of hassle.
For what you're doing, you may also want to check out NumPy, which might do some of the work you would want to push to C, as well as offering some additional help in moving data back and forth between Python and C.
f2py (part of numpy) is a simpler alternative to SWIG and boost.python for wrapping C/Fortran number-crunching code.
In my experience, there are two easy ways to call into C code from Python code. There are other approaches, all of which are more annoying and/or verbose.
The first and easiest is to compile a bunch of C code as a separate shared library and then call functions in that library using ctypes. Unfortunately, passing anything other than basic data types is non-trivial.
The second easiest way is to write a Python module in C and then call functions in that module. You can pass anything you want to these C functions without having to jump through any hoops. And it's easy to call Python functions or methods from these C functions, as described here: https://docs.python.org/extending/extending.html#calling-python-functions-from-c
I don't have enough experience with SWIG to offer intelligent commentary. And while it is possible to do things like pass custom Python objects to C functions through ctypes, or to define new Python classes in C, these things are annoying and verbose and I recommend taking one of the two approaches described above.
Python is pretty liberal in allowing functions, functors, objects to be passed to functions and methods, whereas I suspect the same is not true of say C or Fortran.
In C you cannot pass a function as an argument to a function but you can pass a function pointer which is just as good a function.
I don't know how much that would help when you are trying to integrate C and Python code but I just wanted to clear up one misconception.
In addition to the tools above, I can recommend using Pyrex
(for creating Python extension modules) or Psyco (as JIT compiler for Python).