I am working on a project which requires me to create some wrappers in Python for the C library that I need to call from Python. For context, the C library I am using is a bunch of header files (.h) and statically linked library files (.a)
I have decided to use either CFFI or Cython to get my work done. I followed examples similar to this for CFFI - Interfacing C code with CFFI, and this for Cython - Making your C library callable from Python by wrapping it with Cython. Now small sample programs I've tried in both these modules more or less have the following steps
Create the interfacing code to call C APIs
In CFFI, it's a python file declaring the C functions and headers needed
In Cython, it's a .pyx file and modifications to setup.py
Build the interfacing code to generated the .so files for the interfacing glue code.
Call the wrapped functions from a different python script, by importing the interfacing library from the .so file.
Now, this works perfectly for me. But, I'll have to go through two execution steps in the process (generating the .so file, and then actually running the python script with the C API being called).
What I need is to know if there is a way to do all the above in a single execution step. Like, I want to run my final python script, and it should build the interfacing code and import it on the fly in a single execution.
For more context, I have tried SWIG, but wasn't able to find a way to wrapped .a statically linked libraries with it. Same goes for ctypes.
Can you not do this?
import os
os.system('command to build your .so here')
...
import what_ever_you_need
...
For CFFI you just need to execute at runtime the code that is now in the builder script. Move it all in a function, and then you have a function you can call when needed.
I am trying to embed a piece of Cython code in a C++ project, such that I can compile a binary that has no dependencies on Python 2.7 (so users can run the executable without having Python installed). The Cython source is not pure Cython: There is also Python code in there.
I am compiling my Cython code using distutils in the following script (setup.py):
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize("test.pyx")
)
I then run the script using python setup.py build_ext --inplace. This generates a couple of files: test.c, test.h, test.pyd and some library files: test.exp, test.obj and test.lib.
What would be the proper procedure to import this into C++? I managed to get it working by including test.c and test.h during compilation and test.lib during linking.
I am then able to call the Cython functions after I issue
Py_Initialize();
inittest();
in my C++ code.
The issue is that there a numerous dependencies on Python, both during compilation (e.g., in test.h) as well in during linking. Bottom-line is that in order to run the executable, Python has to be installed (otherwise I get errors on missing python27.dll).
Am I going in the right direction with this approach? There are so many options that I am just very confused on how to proceed. Conceptually, it also does not make sense why I should call Py_Initialize() if I want the whole thing to be Python-independent. Furthermore, this is apparently the `Very High Level Embedding' method instead a low-level Cython embedding, but this is just how I got it to work.
If anybody has any insights on this, that would be really appreciated.
Cython cannot make Python code Python-independent; it calls into the Python library in order to handle Python types and function calls. If you want your program to be Python-independent then you should not write any Python code.
(This is primarily extra detail to
Ignacio Vazquez-Abrams's answer which says that you can't eliminate the Python dependency)
If you don't want to force your users to have Python installed themselves, you could always bundle python27.dll with your application (read the license agreement, but I'm almost certain it's fine!).
However, as soon as you do an import in your code, you either have to bundle the relevant module, or make sure it (and anything it imports!) is compiled with Cython. Unless you're doing something very trivial then you could end spending a lot of time chasing dependencies. This includes the majority of the standard library.
Well, I have a Python package. I need to compile it as dll before distribute it in a way easily importable. How? You may suggest that *.pyc. But I read somewhere any *.pyc can be easily decompiled!
Update:
Follow these:
1) I wrote a python package
2) want to distribute it
3) do NOT want distribute the source
4) *.pyc is decompilable >> source can be extracted!
5) dll is standard
Write everything you want to hide in Cython, and compile it to pyd. That's as close as you can get to making compiled python code.
Also, dll is not a standard, not in Python world. They're not portable, either.
Nowadays a simple solutino exists: use Nuitka compiler as described in Nuitka User Manual
Use Case 2 - Extension Module compilation
If you want to compile a single extension module, all you have to do is this:
python -m nuitka --module some_module.py
The resulting file some_module.so can then be used instead of some_module.py.
You need to compile for each platform you want to support and write some initialization code to import so/pyd file ~~appropriate for given platform/python version etc.~~
[EDIT 2021-12]: Actually in python 3 the proper so/dll is determined automatically based on the file name (if it includes python version and platform - can't find PEP for this feature at the moment but Nuitka creates proper names for compiled modules). So for python 2.7 the library name would be something.pyd or something.so whereas for python 3 this would change to something.cp36-win32.pyd or something.cpython-36m-x86_64-linux-gnu.so (for 32bit python 3.6 on x86).
The result is not DLL as requested but Python-native compiled binary format (it is not bytecode like in pyc files; the so/pyd format cannot be easily decompiled - Nuitka compiles to machine code through C++ translation)
EDIT [2020-01]: The compiled module is prone to evaluation methods using python standard mechanisms - e.g. it can be imported as any other module and get its methods listed etc. To secure implementation from being exposed that way there is more work to be done than just compiling to a binary module.
You can use py2exe.org to convert python scripts into windows executables. Granted this will only work on windows, but it's better then nothing.
You can embed python inside C. The real trick is converting between C values and Python values. Once you've done that, though, making a DLL is pretty straightforward.
However, why do you need to make a dll? Do you need to use this from a non-python program?
Python embedding is supported in CFFI version 1.5, you can create a .dll file which can be used by a Windows C application.
I would also using Cython to generate pyd files, like Dikei wrote.
But if you really want to secure your code, you should better write the important stuff in C++. The best would be to combine both C++ and Python. The idea: you would leave the python code open for adjustments, so that you don't have to compile everything over and over again. That means, you would write the "core" in C++ (which is the most secure solution these days) and use those dll files in your python code. It really depends what kind of tool or program you are building and how you want to execute it. I create mostly an execution file (exe,app) once I finish a tool or a program, but this is more for the end user. This could be done with py2exe and py2app (both 64 bit compatible). If you implement the interpreter, the end user's machine doesn't have to have python installed on the system.
A pyd file is the same like a dll and fully supported inside python. So you can normally import your module. You can find more information about it here.
Using and generating pyd files is the fastest and easiest way to create safe and portable python code.
You could also write real dll files in C++ and import them with ctypes to use them (here a good post and here the python description of how it works)
To expand on the answer by Nick ODell
You must be on Windows for DLLs to work, they are not portable.
However the code below is cross platform and all platforms support run-times so this can be re-compiled for each platform you need it to work on.
Python does not (yet) provide an easy tool to create a dll, however you can do it in C/C++
First you will need a compiler (Windows does not have one by default) notably Cygwin, MinGW or Visual Studio.
A basic knowledge of C is also necessary (since we will be coding mainly in C).
You will also need to include the necessary headers, I will skip this so it does not become horribly long, and will assume everything is set up correctly.
For this demonstration I will print a traditional hello world:
Python code we will be converting to a DLL:
def foo(): print("hello world")
C code:
#include "Python.h" // Includes everything to use the Python-C API
int foo(void); // Declare foo
int foo(void) { // Name of our function in our DLL
Py_Initialize(); // Initialise Python
PyRun_SimpleString("print('hello world')"); // Run the Python commands
return 0; // Finish execution
}
Here is the tutorial for embedding Python. There are a few extra things that should be added here, but for brevity I have left those out.
Compile it and you should have a DLL. :)
That is not all. You will need to distribute whatever dependencies are needed, that will mean the python36.dll run-time and some other components to run the Python script.
My C coding is not perfect, so if anyone can spot any improvements please comment and I will do my best to fix the it.
It might also be possible in C# from this answer How do I call a specific Method from a Python Script in C#?, since C# can create DLLs, and you can call Python functions from C#.
You can use pyinstaller for converting the .py files into executable with all required packages into .dll format.
Step 1. pip install pyinstaller,
step 2. new python file let's name it code.py .
step 3. Write some lines of code i.e print("Hello World")
step 4. Open Command Prompt in the same location and write pyinstaller code.py hit enter. Last Step see in the same location two folders name build, dist will be created. inside dist folder there is folder code and inside that folder there is an exe file code.exe along with required .dll files.
If your only goal is to hide your source code, it is much simpler to just compile your code to an executable(use PyInstaller, for example), and use an module with readable source for communication.
NOTE: You might need more converter functions as shown in this example.
Example:
Module:
import subprocess
import codecs
def _encode_str(str):
encoded=str.encode("utf-32","surrogatepass")
return codecs.encode(encoded,"base64").replace(b"\n",b"")
def _decode_str(b64):
return codecs.decode(b64,"base64").decode("utf-32","surrogatepass")
def strlen(s:str):#return length of str;int
proc=subprocess.Popen(["path_to_your_exe.exe","strlen",_encode_str(str).decode("ascii")],stdout=subprocess.PIPE)
return int(proc.stdout.read())
def random_char_from_string(str):
proc=subprocess.Popen(["path_to_your_exe.exe","randchr",_encode_str(str).decode("ascii")],stdout=subprocess.PIPE)
return _decode_str(proc.stdout.read())
Executable:
import sys
import codecs
import random
def _encode_str(str):
encoded=str.encode("utf-32","surrogatepass")
return codecs.encode(encoded,"base64").replace(b"\n",b"")
def _decode_str(b64):
return codecs.decode(b64,"base64").decode("utf-32","surrogatepass")
command=sys.argv[1]
if command=="strlen":
s=_decode_str(sys.argv[2].encode("ascii"))
print(len(str))
if command=="randchr":
s_decode_str(sys.argv[2].encode("ascii"))
print(_encode_str(random.choice(s)).decode("ascii"))
You might also want to think about compiling different executables for different platforms, if your package isn't a windows-only package anyways.
This is my idea, it might work. I don't know, if that work or not.
1.Create your *.py files.
2.Rename them into *.pyx
3.Convert them into *.c files using Cython
4.Compile *.c into *.dll files.
But I don't recommend you because it won't work on any other platforms, except Windows.
Grab Visual Studio Express and IronPython and do it that way? You'll be in Python 2.7.6 world though.
I currently have an executable compiled from C++ that embeds python. The embedded executable runs a python script which load several Cython modules. Both the Cython modules and the executable are linked against a shared library.
I want to move the shared library into the executable by statically linking the shared library against the executable.
Can I statically link the Cython modules into the executable which embeds python? What is the best way to handle this situation?
Yes it's possible, but if you have an hand on the python interpreter. What i'm going to describe have been done for python on IOS platform. You need to check more how to let python known about your module if you don't want to touch on the original python interpreter (Replace TEST everywhere with your own tag/libname)
One possible way to do it is:
Compile your own python with a dynload patch that prefer to not dlopen() your module, but use directly dlsym() to check if the module is already in memory.
Create an libTEST.a, including all the .o generated during the build process (not the .so). You can found it usually in the build/temp.*, and do something like this:
ar rc libTEST.a build/temp.*/*.o
ranlib libTEST.a
When compiling the main executable, you need to add a dependency to that new libTEST.a by appending in the compilation command line:
-lTEST -L.
The result will give you an executable with all the symbol from your cython modules, and python will be able to search them in memory.
(As an example, I'm using an enhanced wrapper that redirect ld during compilation to not produce .so, and create a .a at the end. On the kivy-ios project, you can grab liblink that is used to produce .o, and biglink that is used to grab all the .o in directories and produce .a. You can see how it's used in build_kivy.sh)
I reference this file "kbdext.c" and its headerfile listed on http://www.docdroppers.org/wiki/index.php?title=Writing_Keyloggers (the listings are at the bottom).
I've been trying to compile this into a dll for use in Python or Visual Basic, but have not succeeded. I'm not familiar with C or GCC to sort out the problems or do the dll compile correctly. (I also get an error about snprintf not being declared when doing a regular compile of all the files).
What are the steps I should do to make all functions available for other languages and external apps?
Or is it perhaps easier to use SWIG and make a python module, instead of compiling a DLL?
I've succeeded in compiling the dll with GCC, and am able to import its functions in C. I have yet to test the import in VB and Python but can't see why it would pose problems.