I am currently working on creating a Python extension using C, which includes the definition of a custom object type. In order to maintain a clear and organized codebase, I have chosen to define and implement this object in a separate file. However, as per the Python C API, functions and objects must be defined as static.
Despite this, I have used the __attribute__((visibility("hidden"))) attribute for this object. Unfortunately, I am encountering a recurring error from the linker, specifically, "multiple definition of symbol."
The code provided below is a minimal and easily replicable demonstration of the issue encountered in my larger project. Due to the large number of type objects present in the main project, I am seeking to organize them into separate files for improved maintainability.
My project structure:
.
├── example.c
├── include
│ ├── lcg.h
│ └── macros.h
├── lcg.c
└── setup.py
example.c
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "lcg.h"
static PyMethodDef example_methods[] = {
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef example_module = {
PyModuleDef_HEAD_INIT,
"example",
"An example module",
-1,
example_methods
};
PyMODINIT_FUNC PyInit_example(void) {
PyObject *module;
if (PyType_Ready(&LCG) < 0)
return NULL;
module = PyModule_Create(&example_module);
if (module == NULL)
return NULL;
Py_INCREF(&LCG);
PyModule_AddObject(module, "LCGType", (PyObject*)&LCG);
return module;
}
lcg.c
SYM_PRIVATE is defined in macros.h and expands to __attribute__((visibility("hidden"))).
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "structmember.h"
#include "macros.h" /* for SYM_PRIVATE */
#include "lcg.h"
static PyObject *LCG__dummy(LCG_Type *self)
{
int value = 10;
return PyFloat_FromDouble(value);
}
static PyMemberDef LCG_members[] = {
{NULL} /* sentinel */
};
static PyMethodDef LCG_methods[] = {
{"dummy", (PyCFunction)LCG__dummy, METH_NOARGS, "A simple dummy function on LCG module."},
{NULL} /* sentinel */
};
SYM_PRIVATE PyTypeObject LCG = {
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "example.LCG",
.tp_doc = PyDoc_STR("Linear Congruential Generator"),
.tp_basicsize = sizeof(LCG_Type),
.tp_itemsize = 0,
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_members = LCG_members,
.tp_methods = LCG_methods,
};
lcg.h
#ifndef LCG_H
#define LCG_H
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "structmember.h"
#include "macros.h" /* for SYM_PRIVATE */
typedef struct {
PyObject_HEAD
int foo;
} LCG_Type;
SYM_PRIVATE PyTypeObject LCG;
#endif /* LCG_H */
setup.py
from setuptools import Extension
from setuptools import setup
example_module = Extension('example', sources=['example.c', 'lcg.c'], include_dirs=['include'])
setup(name = 'example',
version = '1.0',
description = 'An example module',
ext_modules = [example_module]
)
When executing python3.10 setup.py build I got:
running build
running build_ext
building 'example' extension
creating build
creating build/temp.linux-x86_64-cpython-310
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Iinclude -I/usr/include/python3.10 -c example.c -o build/temp.linux-x86_64-cpython-310/example.o
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Iinclude -I/usr/include/python3.10 -c lcg.c -o build/temp.linux-x86_64-cpython-310/lcg.o
creating build/lib.linux-x86_64-cpython-310
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/example.o build/temp.linux-x86_64-cpython-310/lcg.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/example.cpython-310-x86_64-linux-gnu.so
/usr/bin/ld: build/temp.linux-x86_64-cpython-310/lcg.o:/home/user/workspaces/python-extension-example/src/include/lcg.h:14: multiple definition of `LCG'; build/temp.linux-x86_64-cpython-310/example.o:/home/user/workspaces/python-extension-example/src/include/lcg.h:14: first defined here
collect2: error: ld returned 1 exit status
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
I am running gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38 and Python 3.10.6.
In lcg.h, comment out:
// SYM_PRIVATE PyTypeObject LCG;
In example.c, add:
extern LCG;
Then you will build the Python module successfully.
Related
Is there any place I can find a working example to wrap a C++ function (using VTK) to be used by
Python?
Searched and found this page
mix VTK and SWIG Python
but there is no compiling details, and this simple example on this page
https://public.kitware.com/pipermail/vtkusers/2002-October/013980.html
But for my system Ubuntu 16.04, vtk 8.2.0, Python 3.6.8, I could not make the example work.
I have to modify them to let the compiling done without error.
Details:
#============simple.cpp===================
// simple.c
//based on https://public.kitware.com/pipermail/vtkusers/2002-October/013980.html
#include "Python.h"
#include "vtkPythonUtil.h"
#include "vtkObject.h"
#include "vtkObjectBase.h"
#include <stdio.h>
#include "vtkRenderer.h"
extern "C"
{
static PyObject* simple_vtk_function(PyObject *self, PyObject *args);
}
static PyObject* simple_vtk_function(PyObject *self, PyObject *args)
{
PyObject *vtkpy;
if (!PyArg_ParseTuple(args, "O", &vtkpy))
return NULL;
vtkObjectBase *obj = vtkPythonUtil::GetPointerFromObject(vtkpy,"PyObject");
/* Do what you want to do here. */
//printf("%s\n", obj->GetClassName());
printf("inside module simple_vtk, function simple_vtk_function\n");
Py_RETURN_NONE;
}
static char simple_docs[] =
"simple( ): Any message you want to put here!!\n";
static PyMethodDef module_methods[] =
{
{
"simple_vtk_function", (PyCFunction) simple_vtk_function,
METH_VARARGS, simple_docs
},
{NULL}
};
static struct PyModuleDef simple_vtk =
{
PyModuleDef_HEAD_INIT,
"simple_vtk",/* name of module */
"",/* module documentation, may be NULL */
-1,
module_methods
};
PyMODINIT_FUNC PyInit_simple_vtk(void)
{
return PyModule_Create(&simple_vtk);
}
#============setup.py===================
#!/usr/bin/env python3
import distutils.core as dc
module = dc.Extension('simple_vtk',
sources = ['simple.cpp'],
include_dirs=['/usr/local/include/vtk-8.2'],
libraries_dirs = ['/usr/local/include/vtk-8.2'],
#libraries=['vtkPythonUtil']
)
dc.setup(name = 'simple_vtk',
version = '1.0',
ext_modules = [module])
compile command
CFLAGS="-std=c++11" python setup.py build
and get this printing out
/home/jun/anaconda3/lib/python3.6/distutils/extension.py:131: UserWarning: Unknown Extension options: 'libraries_dirs'
warnings.warn(msg)
running build
running build_ext
building 'simple_vtk' extension
gcc -pthread -B /home/jun/anaconda3/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++11 -fPIC -I/usr/local/include/vtk-8.2 -I/home/jun/anaconda3/include/python3.6m -c simple.cpp -o build/temp.linux-x86_64-3.6/simple.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
simple.cpp: In function ‘PyObject* simple_vtk_function(PyObject*, PyObject*)’:
simple.cpp:22:19: warning: unused variable ‘obj’ [-Wunused-variable]
vtkObjectBase *obj = vtkPythonUtil::GetPointerFromObject(vtkpy,"PyObject");
^
g++ -pthread -shared -B /home/jun/anaconda3/compiler_compat -L/home/jun/anaconda3/lib -Wl,-rpath=/home/jun/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ -std=c++11 build/temp.linux-x86_64-3.6/simple.o -o build/lib.linux-x86_64-3.6/simple_vtk.cpython-36m-x86_64-linux-gnu.so
to test in python
>>> import simple_vtk
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/jun/project/play/swig/for_vtk_v2/simple_vtk.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN13vtkPythonUtil20GetPointerFromObjectEP7_objectPKc
The C++ function is so simple, it just to convert a PyObject and convert to vtkObjectBase. I didn't even remove the simple print
printf("%s\n", obj->GetClassName());
But I could never make it work. I think it is a common need for people to wrap a custom C++ vtk function for Python, a little bit surpring there is almost no woring example on internet.
Could you please help me on this? Thanks!
Intro
As a follow-up to my earlier question here, i still try to:
add new functionality to one of scipy's modules (e.g. scipy.optimize)
based on C++-code
which defines an interface-function to be called with ctypes
automatically setup through setup.py in scipy/optimize
Code incorporation
To keep it simple, let's assume we got the following (dumb and) simple C++-code:
extern "C" void hallo()
{
int a = 0;
}
where we use the extern keyword to handle name-mangling.
We introduce new files and folders in scipy.optimize:
scipy/optimize/__init__py
scipy/optimize/lbfgsb/...
scipy/optimize/lbfgsb.py
...
scipy/optimize/_mylib/README.md # new
scipy/optimize/_mylib/LICENSE # new
scipy/optimize/_mylib/src/debug.cpp # new
scipy/optimize/mylib.py # new
Setup preparation
We prepare the module-setup in scipy/optimize/setup.py:
from __future__ import division, print_function, absolute_import
from os.path import join
from scipy._build_utils import numpy_nodepr_api
def configuration(parent_package='',top_path=None):
from numpy.distutils.misc_util import Configuration
from numpy.distutils.system_info import get_info
config = Configuration('optimize',parent_package, top_path)
# MODIFICATION START
# OTHER EXTENSIONS OMITTED
# INSPIRED BY scipy.spatial: ckdtree/src & setup.py
mylib_src = ['debug.cpp']
mylib_src = [join('_mylib', 'src', x) for x in mylib_src]
mylib_headers = []
mylib_headers = [join('_mylib', 'src', x) for x in mylib_headers]
mylib_dep = mylib_headers + mylib_src
config.add_extension('_mylib',
sources=mylib_src,
depends=mylib_dep,
include_dirs=[join('_mylib', 'src')])
# MODIFICATION END
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration(top_path='').todict())
Run scipy install
Installing scipy now (from base-dir) with:
python3 setup.py build_ext --inplace
works and we will see those shared-libraries in scipy/optimize/:
_lbfgsb.cpython-35m-x86_64-linux-gnu.so
...
_mylib.cpython-35m-x86_64-linux-gnu.so
Let's try to use it / a look into scipy/optimize/mylib.py
As seen in the other question, we can get the lib with (we are in scipy/optimize/mylib.py) and get the function we want to use:
import scipy as scp
import numpy.ctypeslib as ctl
lib = ctl.load_library('_mylib', scp.optimize.__file__)
myfunc = lib.hallo
Now here is the problem: this fails with:
AttributeError: /.../_mylib.cpython-35m-x86_64-linux-gnu.so: undefined symbol: hallo
Trying it per hand:
import ctypes
lib = ctypes.CDLL('full path to above so')
myfun = lib.hallo
fails too.
But, checking (on my linux OS): nm --defined-only _mylib.cpython-35m-x86_64-linux-gnu.so outputs:
...
...
0000000000000530 t hallo
...
which should be ok. (Disclaimer: i absolutely miss any knowledge about C++ linking). Edit: maybe it's not okay. See later observation in regards to t vs. T!
Doing it manually: this works
Going to scipy/optimize/_mylib/src:
g++ -shared -fPIC debug.cpp -o mylib.so
followed by nm --defined-only mylib.so shows the same functions, but some t became T.
...
...
0000000000000600 T hallo
...
Probably this is the reason, and there is some related general question here.
As mentioned, this works:
# in src
import numpy.ctypeslib as ctl
lib = ctl.load_library('mylib.so', '.')
lib.hallo
# < FuncPtr object at 0x....
But what needs to be done in setup.py or my sources to make it work?
During running scipy-install, output looks like:
building 'scipy.optimize._mylib' extension
compiling C++ sources
C compiler: x86_64-linux-gnu-g++ -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src
compile options: '-Iscipy/optimize/_mylib/src -I/usr/local/lib/python3.5/dist-packages/numpy-1.15.0.dev0+e4d678a-py3.5-linux-x86_64.egg/numpy/core/include -I/usr/include/python3.5m -c'
x86_64-linux-gnu-g++: scipy/optimize/_mylib/src/debug.cpp
scipy/optimize/_mylib/src/debug.cpp: In function ‘void hallo()’:
scipy/optimize/_mylib/src/debug.cpp:3:9: warning: unused variable ‘a’ [-Wunused-variable]
int a = 0;
^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src/debug.o -Lbuild/temp.linux-x86_64-3.5 -o scipy/optimize/_mylib.cpython-35m-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.5/link-version-scipy.optimize._mylib.map
I have the following files for generating a Python binding to a C++ project with SWIG and CMake:
test.h
int add(int a, int b);
test.cpp
int add(int a, int b)
{
return a+b;
}
test.i
%module test
%{
#include "test.h"
%}
CMakeLists.txt
cmake_minimum_required(VERSION 2.6)
project(swig-test)
# This is a CMake example for Python
add_library(testcpp SHARED test.cpp)
FIND_PACKAGE(SWIG REQUIRED)
INCLUDE(${SWIG_USE_FILE})
FIND_PACKAGE(PythonLibs)
INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_PATH})
INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR})
SET(CMAKE_SWIG_FLAGS "")
SET_SOURCE_FILES_PROPERTIES(test.i PROPERTIES CPLUSPLUS ON)
SET_SOURCE_FILES_PROPERTIES(test.i PROPERTIES SWIG_FLAGS "-includeall")
set(${CMAKE_CXX_FLAGS} "${CMAKE_CXX_FLAGS} -fPIC")
SWIG_ADD_MODULE(test python test.i)
SWIG_LINK_LIBRARIES(test testcpp)
It compiles successfully and libtestcpp.so, _test.so, and test.py are created. strings libtestcpp.so and strings _test.so both have an entry _Z3addii and import test works in Python, but there is nothing under the test namespace in Python.
I've also tried manually compiling with
swig -c++ -python test.i
g++ -c -fpic test.cpp test_wrap.cxx -I/usr/include/python2.7 -I.
g++ -shared test.o test_wrap.o -o _test.so
with the same result.
It may be useful to note that import test isn't a completely empty module; import test; dir(test) yields
['__builtin__',
'__builtins__',
'__doc__',
'__file__',
'__name__',
'__package__',
'_newclass',
'_object',
'_swig_getattr',
'_swig_property',
'_swig_repr',
'_swig_setattr',
'_swig_setattr_nondynamic',
'_test']
and import test; help(test) has a description that it is created by SWIG.
You need to to add a line %include "test.h" at the end in test.i
%module test
%{
#include "test.h"
%}
%include "test.h"
This question already has an answer here:
Build Python (2.7) module on GCC 4.8 fails
(1 answer)
Closed 7 years ago.
I am currently working on building a python module in C. I have followed a tutorial series to get my self to a building point, and I have a pretty good grasp of what is going on.
However, when I run the script to set everything up I have the following printout. Looks like the error is coming from one of the header files in python-dev; however, I wanted to reach out here to make sure it isn't something I am doing. I am going to include the error code, and I can provide the other files on request.
sudo python setup.py build
running build
running build_ext
building 'lidar' extension
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/include -I/usr/include/python2.7 -c lidar_module.c -o build/temp.linux-armv6l-2.7/lidar_module.o
lidar_module.c: In function ‘lidar_test’:
lidar_module.c:8:6: warning: unused variable ‘sts’ [-Wunused-variable]
int sts = 0;
^
In file included from /usr/include/python2.7/Python.h:58:0,
from lidar_module.c:1:
lidar_module.c: At top level:
/usr/include/python2.7/pyport.h:802:39: error: expected ‘,’ or ‘;’ before ‘void’
# define PyMODINIT_FUNC void
^
lidar_module.c:24:1: note: in expansion of macro ‘PyMODINIT_FUNC’
PyMODINIT_FUNC initlidar(){
^
lidar_module.c:3:18: warning: ‘lidarError’ defined but not used [-Wunused-variable]
static PyObject *lidarError;
^
lidar_module.c:19:20: warning: ‘lidar_methods’ defined but not used [-Wunused-variable]
static PyMethodDef lidar_methods[] = {
^
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1
I peeked into the header file and found this around the area of the error.
#ifndef PyMODINIT_FUNC
# if defined(__cplusplus)
# fndef PyMODINIT_FUNC
# if defined(__cplusplus)
# define PyMODINIT_FUNC extern "C" void
# else /* __cplusplus */
# define PyMODINIT_FUNC void
# endif /* __cplusplus */
#endif
I literally had to search the line number output from the console, but I did find the solution here:
Build Python (2.7) module on GCC 4.8 fails
I'm trying to embed Python into a MATLAB mex function on OS X. I've seen references that this can be done (eg here) but I can't find any OS X specific information. So far I can successfully build an embedded Python (so my linker flags must be OK) and I can also build example mex files without any trouble and with the default options:
jm-g26b101:mex robince$ cat pytestnomex.c
#include <Python/Python.h>
int main() {
Py_Initialize();
PyRun_SimpleString("print 'hello'");
Py_Finalize();
return 0;
}
jm-g26b101:mex robince$ gcc -arch i386 pytestnomex.c -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -L/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config -ldl -lpython2.5
jm-g26b101:mex robince$ ./a.out
hello
But when I try to build a mex file that embeds Python I run into a problem with undefined symbol main. Here is my mex function:
#include <Python.h>
#include <mex.h>
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray*prhs[])
{
mexPrintf("hello1\n");
Py_Initialize();
PyRun_SimpleString("print 'hello from python'");
Py_Finalize();
}
Here are the mex compilation steps:
jm-g26b101:mex robince$ gcc -c -I/Applications/MATLAB_R2009a.app/extern/include -I/Applications/MATLAB_R2009a.app/simulink/include -DMATLAB_MEX_FILE -arch i386 -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -DMX_COMPAT_32 -O2 -DNDEBUG "pytest.c"
jm-g26b101:mex robince$ gcc -O -arch i386 -L/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/config -ldl -lpython2.5 -o "pytest.mexmaci" pytest.o -L/Applications/MATLAB_R2009a.app/bin/maci -lmx -lmex -lmat -lstdc++
Undefined symbols:
"_main", referenced from:
start in crt1.10.6.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
I've tried playing around with arch settings (I added -arch i386 it to try and keep everything 32bit - I am using the python.org 32 bit 2.5 build), and the order of the linker flags, but haven't been able to get anywhere. Can't find much online either. Does anyone have any ideas of how I can get this to build?
[EDIT: should probably add I'm on OS X 10.6.1 with MATLAB 7.8 (r2009a), Python 2.5.4 (python.org) - I've tried both gcc-4.0 and gcc-4.2 (apple)]
I think I found the answer - by including the mysterious apple linker flags:
-undefined dynamic_lookup -bundle
I was able to get it built and it seems to work OK. I'd be very interested if anyone has any references about these flags or library handling on OS X in general. Now I see them I remember being bitten by the same thing in the past - yet I'm unable to find any documentation on what they actually do and why/when they should be needed.