I wrote a simple program in python:
// main.py
import re
links = re.findall('(https?://\S+)', 'http://google.pl http://youtube.com')
print(links)
Then I execute this:
cython main.py
It was generated a file: main.c
And then I tried this:
gcc main.c
And I have an error:
main.c:8:10: fatal error: 'pyconfig.h' file not found
#include "pyconfig.h"
^
1 error generated.
How to compile python to c ? How to get started with cython with xcode on mac ?
You have to tell the gcc compiler where is the pyconfig.h file on your system using the -I flag. You can find it using the find program.
A simpler way to compile the module is using a setup.py module. Cython provides a cythonize function that starts this process for a .pyx module.
Another point you are missing is that Cython files usually define helper functions to be used from a main Python module.
Suppose you have the following setup as for dirs and files:
cython-start/
├── main.py
├── setup.py
└── split_urls.pyx
The contents of the setup.py are
from distutils.core import setup
from Cython.Build import cythonize
setup(name="My first Cython app",
ext_modules=cythonize('split_urls.pyx'), # accepts a glob pattern
)
The contents of the split_urls.pyx file are
import re
def do_split(links):
return re.findall('(https?://\S+)', links)
And it is the main.py module which uses the defined Cython function:
import split_urls
URLS = 'http://google.pl http://youtube.com'
print split_urls.do_split(URLS)
Compile the Cython module by issuing:
$ python setup.py build_ext --inplace
Cythonizing split_urls.pyx
running build_ext
building 'split_urls' extension
creating build
creating build/temp.macosx-10.9-x86_64-2.7
... compiler output ...
And check that your main module is doing what it is supposed to do:
$ python main.py
['http://google.pl', 'http://youtube.com']
Related
I have a large C/C++ project which I usually just build as a standalone executable. It also requires some libraries (system libraries like pthread, but also MKL from intel for example) which I specify when compiling the project.
Now, I want to use some functions from this project in Python. So my first plan was to simply build the project as a static library and then use that to write a cython wrapper. I.e.
Build the c project: icc -c src/main.cpp .. options and stuff.. linking to all libraries needed .. -o main.o
Create a static library (including the libraries I needed in step 1): ar -rc libmain.a main.o ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ... /usr/lib/x86_64-linux-gnu/libpthread.a ...
I use the generated static library to build my cython wrapper
when I try to execute a test python script calling a random function from my C program I get this error ImportError: /home/.../main.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_ok_to_fork
It seems like I need to tell cython to link to the corresponding libraries again or something, but I'm not really sure how to resolve this.. Also I don't want the python project to be dependent on some special libraries that I have installed on my system (that's what I try to achieve by adding all the libraries in step 2), is this possible at all?
My setup.py looks like this
from setuptools import setup, Extension
from Cython.Build import cythonize
import os
os.environ["CC"] = "icc"
os.environ["CXX"] = "icc"
setup(
ext_modules = cythonize([Extension("test", ["test.pyx"], language="c++",
library_dirs=[r'.'], libraries=['main'])])
)
Create a shared library (.so on linux). Python can only support dynamic or shared libraries. The linker is looking for a .so not a .a.
So your setup.py would be remain the same,
from setuptools import setup, Extension
from Cython.Build import cythonize
import os
os.environ["CC"] = "icc"
os.environ["CXX"] = "icc"
setup(
ext_modules = cythonize([Extension("test", ["test.pyx"], language="c++",
library_dirs=[r'.'], libraries=['main'])])
)
Related - Using Cython To Link Python To A Shared Library.
I am trying to invoke a Cython file in a Python script. I have read this answer and followed the instruction. However, even though I have successfully compiled the C code, PyCharm does not recognize the Cython file 'hello.pyx' while executing the import command, as shown in the screenshot below. What is the remedy?
The Cython file hello.c is generated by setup.py the content of which is shown below.
from distutils.core import setup
from Cython.Build import cythonize
setup(name='Hello world app',
ext_modules=cythonize("hello.pyx")
)
When you compile your module with cythonize, the resulting module will be put into a subfolder. By default, this subfolder is not in the PATHS list. To fix this problem you can just move the resulting file .dll file into the folder where your hello.py and hello.pyx are located.
An alternative approach is to add Extension like so:
from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
exts = [Extension(name='hello',sources=['hello.pyx'])]
setup(name='Hello world app',
ext_modules=cythonize(exts)
)
And then compile with:
python setup.py build_ext --inplace
This code will put resulting .dll into the same folder as your source files.
I'm working on wrapping a set of C functions using Cython into a module. I would like to be able to cimport this first module into subsequent Cython-based projects, but I am running into an 'undefined symbol' issue when importing the derivative modules in a python script.
Consider the following minimal working example of two modules developed in separate directories:
# exModA wraps the functions provided in exModAFxn.c
exModA/
__init__.pxd
__init__.py
exModAFxn.c
exModA.pyx
setup.py
# exModB attempts to use the cdef functions from exModA
exModB/
__init__.py
exModB.pyx
setup.py
# The test script attempts to make use of exModB
test.py
exModA/__init__.pxd:
cdef extern void myCMessage()
exModA/__init__.py:
from exModA import *
exModA/exModAFxn.c:
#include <stdio.h>
void myCMessage() { printf( "This is a test!\n" ); }
exModA/exModA.pyx:
cdef extern from "exModAFxn.c" :
void myCMessage()
# Use myCMessage in a python function
def foo() :
myCMessage()
exModA/setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(
name = 'exModA',
ext_modules = cythonize( ['exModA/exModA.pyx'] ),
)
exModB/__init__.py:
from exModB import *
exModB/exModB.pyx:
cimport exModA
# Use a cdef function from exModA in a python function
def bar() :
exModA.myCMessage()
exModB/setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(
name = 'exModB',
ext_modules = cythonize( ['exModB/exModB.pyx'] ),
)
The test.py script is called after compiling the two cython modules.
python extModA/setup.py build_ext --inplace
python extModB/setup.py build_ext --inplace
test.py:
import exModA
exModA.foo() # successfully prints message
import exModB # raises ImportError: undefined symbol: myCMessage
exModB.bar() # we never get to this line :(
The successful printing of the message from the exModA.foo function suggests to me that myCMessage is in fact available, but it isn't found when exModB is imported. I know this problem goes away if I merged exModA and exModB into a single project, but I'd like to avoid that approach if possible -- I'm trying to expose a common set of wrappers for a C library for use in several different applications.
I was able to get this to work by doing the following:
First, specify the exModA library for linking in your exModB/setup.py file as follows:
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension('exModB/exModB',
sources=['exModB/exModB.pyx'],
libraries=['exModA'],
library_dirs=['/home/MyFolder/exModA'],
runtime_library_dirs=['/home/MyFolder/exModA']
)
setup(name='exModB', ext_modules = cythonize(ext))
(Instead of specifying a runtime library directory, I recommend moving exModA.so to a folder on your library search path.)
I also had to rename exModA.so to libexModA.so so that gcc could find it. Do this after building exModA.so, but before building exModB.so:
python exModA/setup.py build_ext --inplace
cp exModA/exModA.so exModA/libexModA.so
python exModB/setup.py build_ext --inplace
You will probably want to tweak this, but it should get you started.
I am trying to build a Python multi-file code with PyInstaller. For that I have compiled the code with Cython, and am using .so files generated in place of .py files.
Assuming the 1st file is main.py and the imported ones are file_a.py and file_b.py, I get file_a.so and file_b.so after Cython compilation.
When I put main.py, file_a.so and file_b.so in a folder and run it by "python main.py", it works.
But when I build it with PyInstaller and try to run the executable generated, it throws errors for imports done in file_a and file_b.
How can this be fixed? One solution is to import all standard modules in main.py and this works. But if I do not wish to change my code, what can be the solution?
So I got this to work for you.
Please have a look at Bundling Cython extensions with Pyinstaller
Quick Start:
git clone https://github.com/prologic/pyinstaller-cython-bundling.git
cd pyinstaller-cython-bundling
./dist/build.sh
This produces a static binary:
$ du -h dist/hello
4.2M dist/hello
$ ldd dist/hello
not a dynamic executable
And produces the output:
$ ./dist/hello
Hello World!
FooBar
Basically this came down to producing a simple setup.py that builds the extensions file_a.so and file_b.so and then uses pyinstaller to bundle the application the extensions into a single executeble.
Example setup.py:
from glob import glob
from setuptools import setup
from Cython.Build import cythonize
setup(
name="test",
scripts=glob("bin/*"),
ext_modules=cythonize("lib/*.pyx")
)
Building the extensions:
$ python setup.py develop
Bundling the application:
$ pyinstaller -r file_a.so,dll,file_a.so -r file_b.so,dll,file_b.so -F ./bin/hello
Just in case someone's looking for a quick fix.
I ran into the same situation and found a quick/dirty way to do the job. The issue is that pyinstaller is not adding the necessary libraries in the .exe file that are needed to run your program.
All you need to do is import all the libraries (and the .so files) needed into your main.py file (the file which calls file_a.py and file_b.py). For example, assume that file_a.py uses opencv library (cv2) and file_b.py uses matplotlib library. Now in your main.py file you need to import cv2 and matplotlib as well. Basically, whatever you import in file_a.py and file_b.py, you have to import that in main.py as well. This tells pyinstaller that the program needed these libraries and it includes those libraries in the exe file.
I am constantly working on a Python module which contains C++ extensions wrapped with Cython. The setup.py currently handles the building of the extension module, and is called as python3 setup.py --build_ext --inplace.
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
srcDir = "../src"
src = ["_MyProject.pyx"] # list of source files
print("source files: {0}".format(src))
modules = [Extension("_MyProject",
src,
language = "c++",
extra_compile_args=["-fopenmp", "-std=c++11", "-O3", "-DNOGTEST"],
extra_link_args=["-fopenmp", "-std=c++11"],
libraries=["MyProjectLib", "log4cxx"],
library_dirs=["../"])]
for e in modules:
e.cython_directives = {"embedsignature" : True}
setup(name="_MyProject",
cmdclass={"build_ext": build_ext},
ext_modules=modules)
On top of the Cython module _MyProject, there is a pure Python module MyProject which imports stuff from _MyProject.
Currently I use and test the module by cd-ing into its directory and importing it from there. How do I need to modify my setup.py so that I can install MyProject into my site packages and have the package always up to date?
Add the argument py_modules = ["MyProject.py",] to your setup() function.