Reduce Python static library size - python

I'm trying to build a minimal Python static library for distribution in an app. This is the C API I need to link to, I believe it may be called CPython as well. I don't need to package actual Python scripts, just link to the library itself.
I've built 3.7.4 by doing:
./configure --disable-shared
make
This does build a library libpython3.7m.a which is 12.4MB. Is it possible to reduce this size at all? I need an absolute barebones distribution without any of the normal packages. Literally this is just for a scripting bridge and doesn't need any of the usual Python functionality.

If you're trying to link to Python's Extension API, you won't find much luck in shaving off size from the libpython shared object. This is because the Extension API uses the same types and headers as the core interpreter (which is why it's called extensions instead of something else like plugins), granting the extensions runtime access to the inner mechanics of the interpreter that normal python scripts wouldn't allow. Basically, the libpython object is just the interpreter compiled without an entry point and an additional frontend so that the expression of simple python operations (e.g. parsing function arguments) stays a clean development experience.
If you're using Python 3.2+, there is offered a "Limited API" for the version of python you are trying to target, but the point of this is to guarantee a stable ABI as the extensions API evolves, not to reduce the size of the compilation unit.
I would also like to mention that the term CPython is just a way to disambiguate the vanilla Python distribution from similar but incompatible interpreters like Pypy.

Related

Is it possible to access C++ headers/libraries and run a C++ script within a python console?

I need to access data via USB from a beam profiler. I've tried using the USB module in python to access it, but unfortunately the company who makes this device "does not support development in Python". The project I am working on is to eventually create a GUI (via Python) to automate a motor and pull data from the device. So it has to be done in Python, or I'm going to have to discard the first half of the code and redo it in C++.
I think the reason the device can only interface with C/C++ is because of the header and library files that come with the driver download.
I've looked at Cython but am still very unsure how it can help me. I'm just trying to access the header files for the driver in python and somehow execute the C commands in python.
BTW I am using Anaconda (if that matters).
Thank-you for any clarification and help!
Check out boost.python
Here is an intro:
The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).
It includes support for:
References and Pointers
Globally Registered Type Coercions
Automatic Cross-Module Type Conversions
Efficient Function Overloading
C++ to Python Exception Translation
Default Arguments
Keyword Arguments
Manipulating Python objects in C++
Exporting C++ Iterators as Python Iterators
Documentation Strings
and many more.

Wrapping C++ code with python (manually)

I have a main file(main.cpp) and a header file(nodes.hpp). The main file takes N(any positive integer) as input argument and by using the functions of header file it gives output say 'x & y' (both double).
Note:
Both main and header files are written in C++.
Both main and header files instead of using data structues as arrays,vectors, make use of Eigen Library.
I have to write a python wrapper for them, I have working knowledge of python but have never used any wrapper.
Can anybody please refer or give some notes about using python wrpper for such code?
Here are your options:
You can use ctypes, and I consider this the cleanest solution, because you convert your program to a shared library that can be called by any other software, not only Python. You, though, have to write a C-interface for your program yourself.
You can use Python C-Extension, and I consider this the worst solution, because it's very low level, and prone to memory leaks, and costs lots of time to implement one function, and is Python-version dependent. Basically this is good to start a Python interpreter inside your C++. You can create PyObjects (which is the main building block of any Python type) and deal with them insdie C/C++.
You can use SWIG, where it automatically creates the the interface that you have to create with ctypes through an interface file that you define. People say it's very good, but the documentation is not as good.
You can use Boost.Python, which is good, but it has a very ugly build system with bjam. If you can manage to bypass that, then it's even better than ctypes. I'm a big boost fan, but bjam is why I don't use this.
What I do typically is ctypes. I trust it because it emphasizes the single-reponsibility principle. The library has a job that's separate from the interface (the C-interface), which is also separate from your Python script that uses that interface and exposes "the easy functionality" to the user.
Use Boost.Python. Here is my tutorial, previously on SO Docs.
Using Boost.Python
Things are easy when you have to use a C++ library in a Python project. Just you can use Boost.
First of all here is a list of components you need:
A CMakeList.txt file, because you're going to use CMake.
The C++ files of the C++ project.
The python file - this is your python project.
Let's start with a small C++ file. Our C++ project has only one method which returns some string "This is the first try". Call it CppProject.cpp
char const *firstMethod() {
return "This is the first try.";
}
BOOST_PYTHON_MODULE(CppProject) {
boost::python::def("getTryString", firstMethod); // boost::python is the namespace
}
Have a CMakeLists.txt file a below:
cmake_minimum_required(VERSION 2.8.3)
FIND_PACKAGE(PythonInterp)
FIND_PACKAGE(PythonLibs)
FIND_PACKAGE(Boost COMPONENTS python)
INCLUDE_DIRECTORIES(${Boost_INCLUDE_DIRS} ${PYTHON_INCLUDE_DIRS})
PYTHON_ADD_MODULE(NativeLib CppProject)
FILE(COPY MyProject.py DESTINATION .) # See the whole tutorial to understand this line
By this part of the tutorial everything is so easy. you can import the library and call method in your python project. Call your python project MyProject.py.
import NativeLib
print (NativeLib.getTryString)
In order to run your project follow the instructions below:
Create a directory with the name build.
Enter into that directory.
Give the command cmake -DCMAKE_BUILD_TYPE=Release ..
make
python MyProject.py. Now, you have to see the string which the method in your C++ project returns.
Another tool for C++ wrapper generation is CLIF. Released in 2017, Google uses this for most everything these days. We no longer allow new SWIG wrappers to be written for Python internally.
It is built on top of Clang for the C++ parsing and requires relatively idiomatic modern C++ API use (unsurprisingly following Google's Style Guide) rather than any attempt to allow you to shoot yourself in the foot via SWIG's "support everything poorly" approach.
Try with official documentation:
https://docs.python.org/2/extending/extending.html
this link will provide you simple example of how to include a cpp module and use it from the python interpreter, or if this is possible try with Cython: http://cython.org/
Cython will allow you to write C-like, Python-like code which will be translated to CPP compiled and then will be easily accessible from the Python.
You can use Boost.Python
or go with the Python native interface
I would recommend Boost.Python if you already have Boost set up.

Python program into a standard assembly?

Is it possible to convert Python programs to a Microprocessor standard assembly language like IEEE-694? The assembly syntax is close to this one
or this other one: http://www.ethicalhacker.net/content/view/152/2/
Compile python to C, then use a C compiler of your choice to get it down to assembly.
Alternatively, use PyPy, specifying LLVM as the target, and use the LLVM Static Compiler to yield assembly language for your target architecture.
Not in the same way as C, FORTRAN, COBOL, etc. Languages that support lambda calculus or automatic memory management cannot be compiled directly to assembly. An interpreter can, however, be provided in microcode or in a bootstrap program to bridge the gap and allow "compiled" Python, LISP, etc. (Some operations, such as garbage collection, are still carried out within the embedded interpreter packaged into the compiled binary.)
Since Python is a dynamically typed language, this would only be possible if the assembly program would use the runtime environment / library of Python to dynamically get objects.
So it would only be possible with some overhead.
But there is RPython from the PyPy project. It is a restricted subset of the Python language (it is not longer dynamically typed and lacks most modules from Python's standard library). RPython programs can be translated to machine code (AFAIK it generates C code as a intermediate code).
Python itself generates a intermediate code for it's virtual machine. If you want to have a look at this code, use the dis module from the Python standard library. This generates a assembly-like representation of your Python function. Keep in mind that a "real" microprocessor would not be able to use this and that the result might change with the Python version you are using.

Can i take package of cpython?

I used cpython api to load py from C/C++.
But, if i want not setup cpython in client, can I take package dll of cpython in my program?
How to do that?
Installer-builders like PyInstaller (cross-platform) and py2exe (Windows only) basically do that job for you in a general way, except that the executable at the heart of the produced package is their own instead of yours.
But basically, you can imitate their behavior in terms of setting up a .zip file with all the Python library modules you need (or just zip up everything in the standard python library if you want to allow python code running form your app to import anything from there), and follow the simple advice in the Embedding Python in Another Application section of the Python docs.
Note that embedding Python equals extending Python plus a little bit of code to initialize and finalize the interpreter itself and a little bit of packaging as I just mentioned; if you've never writted Python extensions I would suggest practicing that first since it's the most substantial part of the task (not all that hard with helpers such as boost python, but more work if you choose to do it as the "bare C" level instead).
You don't need to setup Python to embed it in applications. The core of the Python interpreter is available as a shared library which you can dynamically load in your application and distribute with it.
Read on embedding Python in the official docs. Also, this article seems nice and comprehensive for Linux. For Windows, read the notes here.
Here's another SO question that discusses this issue.
The Python license is probably hard to understand for a non-lawyer, non-native English speaker. So yes, you can redistribute the unmodified DLL as it contains the copyright notice within it.
It would be polite to give credit like "This program contains the Python Language Interpreter version X.XX http://python.org for more information" or similar somewhere in the program or documentation.

Combined Python & Ruby extension module

I have a C extension module for Python and I want to make it available to Rubyists.
The source has a number of C modules, with only one being Python-dependent. The rest depend only on each other and the standard library. I can build it with python setup.py build in the usual way.
I've been experimenting with adding Ruby support using newgem and I can build a version of the extension with rake gem. However, the combined source has an ugly directory layout (mixing Gem-style and Setuptools-style structures) and the build process is a kludge.
I can't just keep all the sources in the same directory because mkmf automatically picks up the Python-dependent module and tries to build that, and users shouldn't have to install Python to compile a module that won't be used. My current hack is for extconf.rb to copy the Python-independent source-files into the same directory as the Ruby-dependent extension module.
Is there a saner way to make the code available to both languages? Should I just duplicate the Python-independent code in a separate Gem? Should I release the independent code as a separate lib built with autotools? Is there a version of mkmf that can skip the unwanted module?
One way to solve it is to create three different projects:
The library itself, independent on python & ruby
Python bindings
Ruby bindings
That's probably the cleanest solution, albeit it requires a bit more work when doing releases, but it has the advantage that you can release a new version of the Ruby bindings without having to ship a new library/python bindings version.
Complementing on what Johan said, I've used a couple c/c++ support libraries in Python thanks to swig. You write your code in c/c++ then make an intermediary template for each language that you want to support. Its rather painless for Python, but some considerations must be made for Ruby... namely I don't think pthread support is to happy with ruby or vice versa.
http://www.swig.org/
It's got a somewhat steep learning curve so it might be best to find an example project out there that demonstrates how to use the wrapper for your target languages.
This is definitely a useful tool as it makes your code a lot cleaner while still providing robust bindings to multiple languages (PHP, Python, Ruby, and I believe c#)

Categories