I am trying to make a Cython wrapper so I can call Python code from C. I am having issues with import as I would like the wrapper to be separate from original code.
Code below ends in segfault when calling imported function. If the code is written as a python module and imported via import the program says that the name ... is not defined. The problem does not exhibit itself when everything is in one file and there's no import involved (indeed code generated by Cython fails when cimporting). The code works fine as well when libcimpy.pyx is imported from other python script (either compiled to .so or live)
I have prepared a minimal example. This is far from actual code but covers the principle.
cimpPy.pyx: Sample python code (converted to Cython)
cdef sum(a, b):
return a + b
cimpPy.pxd
cdef sum(a, b)
libcimpy.pyx (glue Cython code)
cimport cimpPy
cdef public int cSum(int a, int b):
return cimpPy.sum(a, b)
ci.c (c code from which we want to call cimpPy)
#include <stdio.h>
#include <stdlib.h>
#include <Python.h>
#include "libcimp.h"
int main(int argc, char **argv) {
Py_Initialize();
initlibcimp();
int a = 2;
int b = 3;
int c = cSum(a, b);
printf("sum of %d and %d is %d\n", a, b, c);
Py_Finalize();
return 0;
}
Makefile
EXECUTABLE = ci
OBJS = ci.o
CC = gcc
CFLAGS = -g -I/usr/include/python2.7 -I$(shell pwd)
LINKER = g++
LDFLAGS = -L$(shell pwd) $(shell python-config --ldflags) -lcimp
.PHONY: clean cython
all: cython $(EXECUTABLE)
cython:
python setup.py build_ext --inplace
.c.o:
$(CC) $(CFLAGS) -c $<
$(EXECUTABLE) : $(OBJS)
$(LINKER) -o $(EXECUTABLE) $(OBJS) $(LDFLAGS)
clean:
rm -rf *.o *.so libcimp.c libcimp.h core build $(EXECUTABLE)
setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
extensions = [
Extension("libcimp", ["libcimp.pyx"])
]
setup(
name = "CIMP",
cmdclass = {"build_ext": build_ext},
ext_modules = cythonize(extensions)
)
What I intend to achieve is being able to plug Python code into larger C system. The assumption is that users will be able to write Python themselves. The C code is a simulation engine which can operate on agents in a environment. The idea is that the behaviour of agents and environment can be specified in python and passed to the engine for evaluation when necessary. The best analogy would be a map reduce system where Python scripts are mappers. In this sense I want to call Python from C and not the other way round.
Converting everything to Cython, while compelling would be to large undertaking.
Is this the right approach? Why import works only under python interpreter and not when embedded externally? Any suggestions and reference articles or documentation are appreciated.
In this code, the initlibcimp() actually fails, but you don't see it right away because the error is reported by setting a python exception. I'm not 100% sure this is the correct way to do this, but I could see the error by adding the following code below that call:
if (PyErr_Occurred())
{
PyErr_Print();
exit(-1);
}
Then, the program will output:
Traceback (most recent call last):
File "libcimpy.pyx", line 1, in init libcimpy (libcimpy.c:814)
cimport cimpPy
ImportError: No module named cimpPy
The reason that the cimpPy module is not yet defined, is that you need to do a call to initcimpPy() before calling initlibcimp.
Related
i am trying to wrap the C++ library PyrochloreAFM.hpp which itself uses the lib boost/randoom.hpp so that i can import it as a python module. The C++ part itself works fine and i can succesfully import and use all of this from my main.
#include "PyrochloreAFM.hpp"
int main (int argc, const char *argv[]) {
PyrochloreAFM pyrochloreAFM (¶meters, &statistics, &rng);
}
Now following a tutorial i set up my c++ wrapper:
// pybind11_wrapper.cpp
#include <pybind11/pybind11.h>
#include <PyrochloreAFM.hpp>
PYBIND11_MODULE(pybind11_example, m) {
m.doc() = "pybind11 example plugin"; // Optional module docstring
m.def("cpp_function", &PyrochloreAFM, "A function that multiplies two numbers");
}
and my tasks.py file
# tasks.py
import invoke
invoke.run(
"g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC PyrochloreAFM.cpp "
"-o libpyro.so "
)
However now $invoke build-PyrochloreAFM or even $invoke --list seem to have lost the track of the standard C++ library.
In file included from PyrochloreAFM.cpp:1:
./Parameters.hpp:16:10: fatal error: 'boost/random.hpp' file not found
#include "boost/random.hpp" // tested with boost 1.53
^~~~~~~~~~~~~~~~~~
1 error generated.
This might be a simple PATH problem so i would be very glad for any tips!
Thank you, Andres!
"Of course" the tasks.py file which compiles the .cpp file through the python module invoke has to instruct the compiler g++ to use the libraries we desire. In my case it is the flag -I /opt/homebrew/Cellar/boost/1.80.0/include i also use in my makefile or directly in terminal.
I want to use SWIG to create Python bindings for C library, but I have some troubles with it. I used the following .c and .i files from tutorial.
example.c
/* Compute factorial of n */
int fact(int n) {
if (n <= 1)
return 1;
else
return n*fact(n-1);
}
/* Compute n mod m */
int my_mod(int n, int m) {
return(n % m);
}
double My_variable;
example.i
%module example
%{
extern double My_variable;
extern int fact(int);
extern int my_mod(int n, int m);
%}
extern double My_variable;
extern int fact(int);
extern int my_mod(int n, int m);
Commands that I used:
$ swig -python -py3 example.i
$ gcc -c -fpic example.c example_wrap.c -I/usr/include/python3.6
$ gcc -shared example.o example_wrap.o -o example.so
And when I try to import it in python3 I am getting this
>>> import example
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_example)
Any help appreciated
SWIG generates a module with underscore prepended by default, so you'll get a module called _example which needs to be in _example.so.
This is explained in the SWIG documentation 34.2.3 Hand compiling a dynamic module:
While the preferred approach to building an extension module is to use the distutils, some people like to integrate building extensions with a larger build system, and thus may wish to compile their modules without the distutils. To do this, you need to compile your program using commands like this (shown for Linux):
$ swig -python example.i
$ gcc -O2 -fPIC -c example.c
$ gcc -O2 -fPIC -c example_wrap.c -I/usr/local/include/python2.5
$ gcc -shared example.o example_wrap.o -o _example.so
The exact commands for doing this vary from platform to platform. However, SWIG tries to guess the right options when it is installed. Therefore, you may want to start with one of the examples in the SWIG/Examples/python directory. If that doesn't work, you will need to read the man-pages for your compiler and linker to get the right set of options. You might also check the SWIG Wiki for additional information.
When linking the module, the name of the output file has to match the name of the module prefixed by an underscore. If the name of your module is example, then the name of the corresponding object file should be _example.so or _examplemodule.so. The name of the module is specified using the %module directive or the -module command line option.
Compatibility Note: In SWIG-1.3.13 and earlier releases, module names did not include the leading underscore. This is because modules were normally created as C-only extensions without the extra Python support file (instead, creating Python code was supported as an optional feature). This has been changed in SWIG-1.3.14 and is consistent with other Python extension modules. For example, the socket module actually consists of two files; socket.py and _socket.so. Many other built-in Python modules follow a similar convention.
Hi everyone and thanks for trying to help me !
I encounter trouble when trying to import a python module generated by swig.
I have a basic library "example" containing few methods.
Next to it I have a main program dynamically linked to python.
This program imports the generated module and calls a function in it.
If my library example is a shared one, named _example.so, everything works perfectly, and I can import it in python.
But if my library is static, _example.a, and linked to the main program, then I will have the error "no module named _example was found" unless I add a call to SWIG_init() in the main function.
What exactly does SWIG_init() , and when should I use it ? It seems quite weird to me because it's never said in the documentation to do such a call.
I know that dealing with a .so shared library is better but I try to reproduce the behavior of what I have on a big project at work, so I really have to understand what happens when the module is static.
Here is my main file :
#include "Python.h"
#include <iostream>
#if PY_VERSION_HEX >= 0x03000000
# define SWIG_init PyInit__example
#else
# define SWIG_init init_example
#endif
#ifdef __cplusplus
extern "C"
#endif
#if PY_VERSION_HEX >= 0x03000000
PyObject*
#else
void
#endif
SWIG_init(void);
int main (int arc, char** argv)
{
Py_Initialize();
SWIG_init(); // needed only using the statically linked version of example ?
PyRun_SimpleString("print \"Hello world from Python !\"");
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\"/path/to/my/module\")");
PyRun_SimpleString("import example");
PyRun_SimpleString("a = example.Example()");
PyRun_SimpleString("print a.fact(5)");
}
Here is how things are generated :
swig -c++ -python example.i
g++ -fpic -c example.cpp example_wrap.cxx -I/include/python2.7 -lstdc++
ar rvs libexample.a example.o example_wrap.o
// to generate dynamic instead of static : g++ -shared example.o example_wrap.o -o _example.so
g++ main.cpp -I/include/python2.7 libexample.a -lstdc++ -L/lib/python -lpython2.7 -o main
What you are calling is the init function of the native python module _example that is loaded by the SWIG generated python wrapper. For python 2 this function is named init_example, and for python 3 it is named PyInit__example.
Every python extension with C or C++ needs such a function, it basically initializes everything and registers the name of the module and all the methods available for it. In your case SWIG has generated this function for you.
The reason you have to call this function yourself when you compiled the library statically is simply that the python wrapper example imports the native module _example which is by the python convention a shared object, which you did not compile, and which is thus not found.
By calling SWIG_init, you "preload" the module, so python does not try to reimport it, so it works even though there is no shared object anywhere on the python module path.
If you have the shared object for your module, python will call this function for you after loading the shared object and you don't have to worry about this.
I'm new to using SWIG and a bit out of my depth as a programmer. I would like to be able to call the functions of a C++ class in python 2 by importing the wrapped class as a module 'import C++_file' and then call it within my python class with something like 'C++_file.function(inputs)'.
Following http://intermediate-and-advanced-software-carpentry.readthedocs.io/en/latest/c++-wrapping.html, I am wrapping the header file multiplyChannel.h:
#include <vector>
#include <complex>
using namespace std;
class MultiplyChannel {
public:
MultiplyChannel(double in_alpha);
void applyToVector(vector<complex<double> > *in_signal);
private:
double alpha;
};
which corresponds to my example C++ file multiplyChannel.cpp:
#include "multiplyChannel.h"
#include <vector>
#include <complex>
using namespace std;
MultiplyChannel::MultiplyChannel(double in_alpha){
this->alpha = in_alpha;
}
void MultiplyChannel::applyToVector(vector<complex<double> > *in_signal){
unsigned int size = in_signal->size();
for (int i = 0; i < size; i++) {
in_signal->at(i).real() = in_signal->at(i).real() * this->alpha;
in_signal->at(i).imag() = in_signal->at(i).imag() * this->alpha;
}
}
With the makefile:
all:
swig -python -c++ -o mult.cpp swigdemo.i
python setup.py build_ext --inplace
the wrapper file swigdemo.i:
%module swigdemo
%{
#include <stdlib.h>
#include "multiplyChannel.h"
%}
%include "multiplyChannel.h"
and setup.py build file:
from distutils.core import setup, Extension
extension_mod = Extension("_swigdemo", ["mult.cpp"])
setup(name = "swigdemo", ext_modules=[extension_mod])
by typing in my Ubuntu command window:
$ make
swig -python -c++ -o multiplyChannel.cpp swigdemo.i
python setup.py build_ext --inplace
running build_ext
$ python setup.py build
running build
running build_ext
Testing the import using C++_tester.py, I try to multiply the vector [1, 2, 3] into [5, 10, 15] using the MultiplyChannel object 'demo' with instance variable 'in_alpha' of 5x, multiplying all inputs by 5:
#!/usr/bin/python
import swigdemo
if __name__ == '__main__':
demo = swigdemo.MultiplyChannel(in_alpha=5)
out = demo.applyToVector(in_signal=[1, 2, 3])
print(out)
I am not getting past even the import line, receiving the following error:
$ python C++_tester.py
ImportError: ./_swigdemo.so: undefined symbol: _ZN15MultiplyChannelC1Ed
And am unsure what to do as I cannot even gedit or vim into the .so file. I'm guessing my error lies in wrapping incorrectly in my wrapper, build, or makefile, as pretty much everything in the C++_tester.py file auto-completed in my Pycharm IDE.
Many thanks!
The problem was indeed related to extension build:
swigdemo.i - generates swig wrappers (mult.cpp)
MultiplyChannel class implementation is in multiplyChannel.cpp
When building the extension, since it's a shared object (.so), the linker (by default) doesn't complain about undefined symbols (like the 2 MultiplyChannel methods (as it doesn't know anything about them) - and others), but creates it, considering that the symbols could be available at runtime (when the .so will be loaded)
In short, modify setup.py by adding multiplyChannel.cpp to the extension source flies list:
extension_mod = Extension("_swigdemo", ["mult.cpp", "multiplyChannel.cpp"])
Check [SO]: SWIG: How to pass list of complex from c++ to python for the next problem you're going to run into.
I create a c file named test.c with two functions defined as follows:
#include<stdio.h>
void hello_1(void){
printf("hello 1\n");
}
void hello_2(void){
printf("hello 2\n");
}
After that, I create test.pyx as follows:
import cython
cdef extern void hello_1()
The setup file is as follows:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(cmdclass={'buld_ext':build_ext},
ext_modules=[Extension("test",["test.pyx", "test.c"],
include_dirs=[np.get_include()],
extra_compile_args=['-g', '-fopenmp'],
extra_link_args=['-g', '-fopenmp', '-pthread'])
])
When I run setup file, it always reports hello_1 and hello_2 have multiple definitions. Can anybody tell me the problem?
There are a number of things wrong with your files as posted, and I have no idea which one is causing the problem in your real code—especially since the code you showed us doesn't, and can't possibly, generate those errors.
But if I fix all of the obvious problems, everything works. So, let's go through all of them:
Your setup.py is missing the imports at the top, so it's going to fail with a NameError immediately.
Next, there are multiple typos—Extenson for Extension, buld_ext for build_ext, and I think one more that I fixed but don't remember.
I stripped out the numpy and openmp stuff because it's not relevant to your problem, and it was easier to just get it out of the way.
When you fix all of that and actually run the setup, the next problem becomes immediately obvious:
$ python setup.py build_ext -i
running build_ext
cythoning test.pyx to test.c
You're either overwriting your test.c file with the file that gets built from test.pyx, or, if you get lucky, ignoring the generated test.c file and using the existing test.c as if it were the cython-compiled output of test.pyx. Either way, you're compiling the same file twice and trying to link the results together, hence your multiple definitions.
You can either configure Cython to use a non-default name for that file or, more simply, follow the usual naming conventions and don't have a test.pyx that tries to use a test.c in the first place.
So:
ctest.c:
#include <stdio.h>
void hello_1(void){
printf("hello 1\n");
}
void hello_2(void){
printf("hello 2\n");
}
test.pyx:
import cython
cdef extern void hello_1()
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(cmdclass={'build_ext':build_ext},
ext_modules=[Extension("test",["test.pyx", "ctest.c"],
extra_compile_args=['-g'],
extra_link_args=['-g', '-pthread'])
])
And running it:
$ python setup.py build_ext -i
running build_ext
cythoning test.pyx to test.c
# ...
clang: warning: argument unused during compilation: '-pthread'
$ python
>>> import test
>>>
Tada.