I was trying to build this package I wrote (which I know to be working), first in the usual way through distutils:
# python2.7 setup.py build
running build
running build_py
running build_ext
building 'uptime._posix' extension
gcc -fno-strict-aliasing -Wno-error -march=i586 -DHAVE_DB_185_H -I/usr/pkg/include -I/usr/include -DNDEBUG -Wno-error -march=i586 -DHAVE_DB_185_H -I/usr/pkg/include -I/usr/include -I/usr/pkg/include/python2.7 -c src/_posix.c -o build/temp.minix-3-i686-2.7/src/_posix.o
ld -L/usr/tmp/work/lang/python27/work/Python-2.7.2 -lcompat_minix -minlib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/lib -Wl,-R/usr/lib build/temp.minix-3- i686-2.7/src/_posix.o -o build/lib.minix-3-i686-2.7/uptime/_posix.so
ld: unrecognized option '-Wl,-R/usr/pkg/lib'
ld: use the --help option for usage information
build failed: uptime._posix (no big deal)
Well, fine; distutils is one of those modules that often break on less popular platforms. So I tried compiling the extension part manually:
# gcc -fno-strict-aliasing -march=i586 -DNDEBUG -I/usr/pkg/include/python2.7 -fPIC -shared -o _posix.so src/_posix.c
This worked—that is, it produced _posix.so without complaining—but, trying to import the resulting module in Python didn't:
>>> import _posix
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Service unavailable
Google suggests this is related to the ability of the OS to load shared libraries (which Python extensions are). I went looking, and it turns out MINIX didn't support shared libraries at all until last year, but now it should. In fact, /usr/pkg/include/python2.7/pyconfig.h defines both HAVE_DLOPEN and HAVE_DYNAMIC_LOADING, so it clearly does.
What's going on?
MINIX does support shared libraries now, but the binary Python packages provided by pkgin are statically linked, which prevents them from being able to load shared libraries anyway. The only way around this is to compile Python yourself, ideally through pkgsrc. All pkgsrc packages are built dynamically by default.
(You'll still have to build the extensions manually, too.)
Related
This question is related to a couple of questions in SO like this one and this other one. Unfortunately the solution there is not working for me so far. I have a module.pxc file which I am compiling by a setup.py file, such as the following one:
# setup.py
module_extension = Extension(
name="iolif",
sources=["/home/maurizio/Ongoing.Projects/c_libraries/dcomplex_libc.c",
"/home/maurizio/Ongoing.Projects/c_libraries/special_functions_libc.c",
"/home/maurizio/Ongoing.Projects/c_libraries/models/freq_cv_libc.c",
"module.pyx"],
libraries=['gsl', 'gslcblas', 'm'],
# library_dirs=["lib"],
include_dirs=["/home/maurizio/Ongoing.Projects/pycustommodules",
"/home/maurizio/Ongoing.Projects/c_libraries",
"/home/maurizio/Ongoing.Projects/c_libraries/models"]
)
setup(
name="iolif",
ext_modules=cythonize([module_extension])
)
From command line, in the same directory of module.pxc, when writing python setup.py build_ext --inplace compilation works fine, and the iolif.so library is produced. The issue is that I can only import this library if I use Python2.7, whereas if I attempt to import it in Python3.x I get the known ImportError: dynamic module does not define module export function (PyInit_iolif).
Googling around, and as pointed out in the two questions linked above, it seems that this is due to the fact that cython is looking at Python2.7 rather than Python3.x (which is the one I work with instead). Accordingly, I attempted asking cythonize in my setup.py to use Python3.x by:
...
setup(
name="iolif",
ext_modules=cythonize([module_extension],
compiler_directives={'language_level': "3"})
)
but it still does not work. The last compilation message indeed produces:
gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -
specs=/usr/lib/rpm/redhat/redhat-hardened-ld build/temp.linux-x86_64-2.7/pylif_io.o
build/temp.linux-x86_64-2.7/home/maurizio/Ongoing.Projects/c_libraries/dcomplex_libc.o
build/temp.linux-x86_64-2.7/home/maurizio/Ongoing.Projects/c_libraries/special_functions_libc.o build/temp.linux-x86_64-2.7/home/maurizio/Ongoing.Projects/c_libraries/models/freq_cv_libc.o -L/usr/lib64
-lgsl -lgslcblas -lm -lpython2.7 -o /home/maurizio/Ongoing.Projects/DePitta.PNAS/Software/LIF.Analysis/iolif.so
where you can see that it is still linking with the -lpython2.7 library (whereas it should use for example -lpython3.7m). How do I solve it? What am I missing?
Easy solution. My python command was still associated with python2.7 (I recently moved to Python3.x). Sorry about it. Hence:
python3 setup.py build_ext --inplace
will make the trick. Indeed compilation now reads:
gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g build/temp.linux-x86_64-3.7/pylif_io.o build/temp.linux-x86_64-3.7/home/maurizio/Ongoing.Projects/c_libraries/dcomplex_libc.o build/temp.linux-x86_64-3.7/home/maurizio/Ongoing.Projects/c_libraries/special_functions_libc.o build/temp.linux-x86_64-3.7/home/maurizio/Ongoing.Projects/c_libraries/models/freq_cv_libc.o -L/usr/lib64 -lgsl -lgslcblas -lm -lpython3.7m -o /home/maurizio/Ongoing.Projects/DePitta.PNAS/Software/LIF.Analysis/iolif.cpython-37m-x86_64-linux-gnu.so
as required.
When I try to install python-pcl(PCL is the point cloud library for presentation like laser radar data. I followed the instruction on https://github.com/strawlab/python-pcl ,and I have already copied travis/pcl-2d-1.8.pc file to /usr/local/lib/pkgconfig folder) in my computer. I typed AppledeMacBook-Pro-3:python-pcl-0.3.0rc1 apple$ python setup.py install in my terminal.Then I encountered a problem below:
running install
running bdist_egg
running egg_info
writing requirements to python_pcl.egg-info/requires.txt
writing python_pcl.egg-info/PKG-INFO
writing top-level names to python_pcl.egg-info/top_level.txt
writing dependency_links to python_pcl.egg-info/dependency_links.txt
reading manifest file 'python_pcl.egg-info/SOURCES.txt'
writing manifest file 'python_pcl.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.6-x86_64/egg
running install_lib
running build_py
running build_ext
skipping 'pcl/_pcl_180.cpp' Cython extension (up-to-date)
building 'pcl._pcl' extension
/usr/bin/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/apple/miniconda3/include -I/Users/apple/miniconda3/include -DEIGEN_YES_I_KNOW_SPARSE_MODULE_IS_NOT_STABLE_YET=1 -I/Users/apple/miniconda3/pkgs/python-3.5.4-h8f450c2_22/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include/pcl-1.8 -I/usr/local/Cellar/openni/1.5.7.10/include/ni -I/usr/local/include/pcl-1.8 -I/usr/local/Cellar/flann/1.9.1_6/include -I/usr/local/include/pcl-1.8 -I/opt/local/include/eigen3 -I/usr/include/ni -I/usr/include/vtk-5.8 -I/usr/local/include/vtk-8.0 -I/usr/local/Cellar/vtk/8.0.1/include -I/Users/apple/miniconda3/pkgs/python-3.5.4-h8f450c2_22/include/python3.5m -c pcl/_pcl_180.cpp -o build/temp.macosx-10.6-x86_64-3.5/pcl/_pcl_180.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the
command line to use the libc++ standard library instead
[-Wstdlibcxx-not-found]
pcl/_pcl_180.cpp:447:10: fatal error: 'vector' file not found
#include <vector>
^~~~~~~~
Somebody says it has something to do with my Cython version. But I have already switched Cython to version 0.25.2, still not working. Does anyone have the same problem? Many thanks. By the way, my python version is 3.5.4 and my Mac Version is macOS Mojave 10.14.1
This is a special issue with current MacOS-installations. You could tweak setup.py and add, as the warning suggest, -std=libc++ to the compile-options, i.e.
from distutils.core import setup
from Cython.Build import cythonize
... some stuff
#passing `-stdlib=libc++` to compiler and linker:
ext_modules = [Extension(...,
language='c++',
extra_compile_args=["-stdlib=libc++"], # + anything else you need
extra_link_args= ["-stdlib=libc++"] # + anything else you need]
... some more stuff
I have also added -stdlib=libc++ to the linker options, because it will be probably the next problem you will run into.
More background: In the MacOS world, for long time, there where two different implementations of c++'s standard library: -libstdc++ associated with gcc and libc++ associated with clang. At the beginning-libstdc++ was also used per default with clang-compiler. However, this is no longer the case - it is not even installed now and that is the reason why the headers cannot be found. I'm not sure why your clang-version doesn't take libc++ per default - so you have to pass it manually to compiler/linker.
I have exactly copied the example code given in the Cython documentation for wrapping C++ classes. I can successfully build and import the rect.so extension using distutils and the cythonize() method, i.e. by:
Putting the following directives at the top of rect.pyx:
# distutils: language = c++
# distutils: sources = Rectangle.cpp
Writing a setup.py file that contains this:
from distutils.core import setup
from Cython.Build import cythonize
setup(
name = "rectangleapp",
ext_modules = cythonize('*.pyx'),
)
Calling
$ python setup.py build_ext --inplace
However, when I'm wrapping C code in Cython I often find it more convenient to compile individual extensions manually from the command line, i.e.:
Generate the .c code using the command line Cython compiler
$ cython foo.pyx
Manually compile it using gcc:
$ gcc -shared -fPIC -O3 -I /usr/lib/python2.7 -L /usr/lib/python2.7 \
foo.c -lpython2.7 -o foo.so
I've tried applying the same process to build the rect.so example above:
$ cython --cplus rect.pyx
$ g++ -shared -fPIC -O3 -I /usr/lib/python2.7 -L /usr/lib/python2.7 \
rect.cpp -lpython2.7 -o rect.so
Both the Cython and g++ compilation steps seem to succeed - I don't get any command line output, and at the end I have a rect.so built. However, when I then try to import the module I get an undefined symbol error:
In [1]: import rect
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-ba16f97c2145> in <module>()
----> 1 import rect
ImportError: ./rect.so: undefined symbol: _ZN6shapes9Rectangle9getLengthEv
What's the correct procedure for manually compiling Cython code that wraps C++ classes?
The problem here is that you said that somewhere you will provide the definition of a class called Rectangle -- where the example code states
cdef extern from "Rectangle.h" namespace "shapes":
cdef cppclass Rectangle:
...
However, when you compiled the library you didn't provide the code for Rectangle, or a library that contained it, so rect.so has no idea where to find this Rectangle class.
To run your code you must first create the Rectangle object file.
gcc -c Rectangle.cpp # creates a file called Rectangle.o
Now you can either create a library to dynamically link against, or statically link the object file into rect.so. I'll cover statically linking first as it's simplest.
gcc -shared -fPIC -I /usr/include/python2.7 rect.cpp Rectangle.o -o rect.so
Note that I haven't included the library for python. This is because you expect your library to be loaded by the python interpreter, thus the python libraries will already be loaded by the process when your library is loaded. In addition to providing rect.cpp as a source I also provide Rectangle.o. So lets try running a program using your module.
run.py
import rect
print(rect.PyRectangle(0, 0, 1, 2).getLength())
Unfortunately, this produces another error:
ImportError: /home/user/rectangle/rect.so undefined symbol: _ZTINSt8ios_base7failureE
This is because cython needs the c++ standard library, but python hasn't loaded it. You can fix this by adding the c++ standard library to the required libraries for rect.so
gcc -shared -fPIC -I/usr/include/python2.7 rect.cpp Rectangle.o -lstdc++ \
-o rect.so
Run run.py again and all should work. However, the code for rect.so is larger than it needs to be, especially if you produce multiple libraries that depend on the same code. You can dynamically link the Rectangle code, by making it a library as well.
gcc -shared -fPIC Rectangle.o -o libRectangle.so
gcc -shared -fPIC -I/usr/include/python2.7 -L. rect.cpp -lRectangle -lstdc++ \
-o rect.so
We compile the Rectangle code into a shared library in the current directory and provide -L. so gcc knows to look for libraries in the current directory and -lRectangle so gcc knows to look for the Rectangle library. Finally, to be able to run your code you must tell python where the Rectangle library lives. Before running python enter
export LD_LIBRARY_PATH="/home/user/rectangle" # where libRectangle.so lives
You can use a shell script to make sure this is done every time before you run your program, but it makes things messier. Best to just stick with statically linking Rectangle.
I am trying to get cython to realize I have a c compiler in MinGW 32-bit and I've tried everything I can find on the web but it's still not working. I am running Windows 7 Professional 64-bit. Here is what I have tried:
(1) I have Python 2.7 and I just installed MinGW with options gcc and g++ and some other options
(2) I edited the PATH environmental variable so it includes
C:\MinGW\bin;C:\MinGW\MSYS\1.0\local\bin;C:\MinGW\MSYS\1.0\bin
(3) I told Python to use MinGW as the default compiler by creating a file named
C:\Python27\Lib\distutils\distutils.cfg, containing
[build]
compiler = mingw32
(I do have MinGW32 by the way)
(4) I removed all instances of -mno-cygwin from the file C:\Python27\Lib\distutils\cygwincompiler.py
(5) I have a file called setup.py and a module called tryingcython.pyx that is written in python. My setup.py says
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext':build_ext},
ext_modules=[Extension("tryingcython",["tryingcython.pyx"])]
)
So then I open Command Prompt and get into the directory that contains setup.py and tryingcython.pyx, and I type
python setup.py build_ext --inplace --compiler=mingw32
Then it tells me:
running build_ext
skipping 'tryingcython.c' Cython extension (up-to-date)
building 'tryingcython.c' extension
gcc -mdll -O -Wall -IC:\Python27\include -IC:\Python27\PC -c tryingcython.c -o build\
temp.win32-2.7\Release\tryingcython.o
error: command 'gcc' failed: No such file or directory
So I guess Cython can't tell that I have gcc and it can't find it or what, even though I've tried about every single piece of advice I can find online for making it realize that I have MinGW which has gcc included.
Any help/additional ideas on how I can get cython to actually work would be much appreciated.
You are using exactly the same operational system and versions than me.
Try to cal gcc using:
SET input=intput.c
SET output=output.pyd
gcc -shared -IC:\Python27\include -LC:\Python27\libs -O2 -o %output% %input% -lpython27
Usually I put this call in a cythongcc.bat file, in a directory recognized by the PATH environment variable as:
gcc -shared -IC:\Python27\include -LC:\Python27\libs -O3 -mtune=native -o %1.pyd %2.c -lpython27
So that I can , from where my cython .pyx files are, just do:
cython input.pyx
cythongcc input input
To get the compiled .pyd working!
I've been struggling all morning to get PIL installed on Cygwin. The errors I get are not consistent with common errors I find using Google. Perhaps a linux guru can see an obvious problem in this output:
$ python setup.py install
running install
running build
running build_py
running build_ext
building '_imaging' extension
gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ -I/usr/include/freetype2 -IlibImaging -I/usr/include -I/usr/include/python2.5 -c _imaging.c -o build/temp.cygwin-1.7.2-i686-2.5/_imaging.o
In file included from /usr/lib/gcc/i686-pc-cygwin/3.4.4/include/syslimits.h:7,
from /usr/lib/gcc/i686-pc-cygwin/3.4.4/include/limits.h:11,
from /usr/include/python2.5/Python.h:18,
from _imaging.c:75:
/usr/lib/gcc/i686-pc-cygwin/3.4.4/include/limits.h:122:61: limits.h: No such file or directory
In file included from _imaging.c:75:
/usr/include/python2.5/Python.h:32:19: stdio.h: No such file or directory
/usr/include/python2.5/Python.h:34:5: #error "Python.h requires that stdio.h define NULL."
/usr/include/python2.5/Python.h:37:20: string.h: No such file or directory
/usr/include/python2.5/Python.h:39:19: errno.h: No such file or directory
/usr/include/python2.5/Python.h:41:20: stdlib.h: No such file or directory
/usr/include/python2.5/Python.h:43:20: unistd.h: No such file or directory
/usr/include/python2.5/Python.h:55:20: assert.h: No such file or directory
In file included from /usr/include/python2.5/Python.h:57,
from _imaging.c:75:
/usr/include/python2.5/pyport.h:7:20: stdint.h: No such file or directory
In file included from /usr/include/python2.5/Python.h:57,
from _imaging.c:75:
/usr/include/python2.5/pyport.h:89: error: parse error before "Py_uintptr_t"
/usr/include/python2.5/pyport.h:89: warning: type defaults to `int' in declaration of `Py_uintptr_t'
/usr/include/python2.5/pyport.h:89: warning: data definition has no type or storage class
/usr/include/python2.5/pyport.h:90: error: parse error before "Py_intptr_t"
/usr/include/python2.5/pyport.h:90: warning: type defaults to `int' in declaration of `Py_intptr_t'
... more lines like this
Looks like you need to install GNU libC, because
syslimits.h; limits.h; errno.h; stdlib.h; assert.h etc. all belong to it.
For windows you get binary installers; incase you don't want to build PIL under CYGWIN, you can choose from one of these, http://www.pythonware.com/products/pil/#Downloads
You may download a fresh cygwin binary and install imaging libraries in Python section. That will make PIL available for using with cygwin python 2.7.