How does python venv manage C++ dependencies - python

I'm using a library which offers a python wrapper to a c++ executable.
I installed it (https://github.com/bulletphysics/bullet3) using venv (https://docs.python.org/3/library/venv.html) - and all is working great.
I'm considering trying to build https://github.com/bulletphysics/bullet3
From the root of the venv folder I found gym/lib/python3.7/site-packages/pybullet.cpython-37m-x86_64-linux-gnu.so. I'm guessing this is the executable that is invoked eventually from python.
What steps are involved in calling from Python to the correct external binary executable? How does import pybullet as p resolve to gym/lib/python3.7/site-packages/pybullet.cpython-37m-x86_64-linux-gnu.so?
This seems to be close to the end of the c++ world; but I can't find the right key word searches to see exactly how that allows python usages.
Thanks

In short words: C-python just looks for correctly named dynamic libraries in PYTHONPATH, loads such library and uses predefined interface to understand what exactly from this library shall be visible as contents of the module inside Python.
In long words details of how to prepare such shared object and what are required contents of it are described in https://docs.python.org/3/extending/index.html
So venv just puts dynamic library in directory which is part of PYTHONPATH of virtual environment.

python just looks for correctly named dynamic libraries in PYTHONPATH, loads such library and uses predefined interface
for further info :
https://docs.python.org/3/extending/index.html

Related

Run Python Library as script

Some important background upfront, I am using a computer that does not give me access to pip. In fact I do not have access to the command prompt. This make is it impossible for me to install additional libraries unfortunately (at least the standard way).
My question is whether I can run a python library without formally installing it. Could I download the library, and then store it the same directory as my main script, and then import it like I would with a multi .py script project with functions being defined in other files, almost as if I had written the script natively on my computer?
Specifically, I would like to use pdfminer.six. Apparently it is written completely in python, however, I realize that may not mean what I think it does. It may be similar to numpy which I understand has C++ code associated with it.
You can import any script or lib from your current folder (example). You can find any lib you want by googling 'lib_name github'. Download the zip and unpack it in your folder, it should work.
You can also go to your python Lib folder on another computer and copy libs from there (By default: C:\Users\User\AppData\Local\Programs\Python\Python310\Lib)
Maybe you can use a web-based-interpreter solution like Google Colab and work in your browser.
https://colab.research.google.com

Specify a path for ctypes.util.find_library() under macOS

I'd like to specify a path for ctypes.util.find_library() to search. How can I do this?
I'd like to do it from within Python.
I'm using macOS.
If I wanted to do it from outside Python, I can specify the location using LD_LIBRARY_PATH. However I have read that I cannot modify this environment variable from within Python as it is cached on Python's startup. Modifying the value and then restarting Python seems like a very unusable idea; for example, what would happen if the library was imported part way through execution?
Why would I like to do this? Because I would like to add a MacOS wheel to a Python library that works under Windows. Currently they're packaging the DLLs into the same directory as the Python source files and adding that path to Windows' PATH environment, which ctypes.util.find_library() searches--a technique that I can't seem to replicate under Mac.
I have tried to understand delocate. It seems to state that the Python library doesn't depend on any shared objects. I suspect this is because the dylibs are loaded dynamically using ctypes.util.find_library() rather than being compiled code within Python.
Any suggestions would be gratefully received!
Although there are environment variables (LD_LIBRARY_PATH and DYLD_LIBRARY_PATH) that impact the search path for shared libraries, they are read and fixed by Python when Python starts up. Thus that option was eliminated.
Instead, we took the approach hinted at in the Python documentation:
If wrapping a shared library with ctypes, it may be better to determine the shared library name at development time, and hardcode that into the wrapper module instead of using find_library() to locate the library at runtime.
We hard-coded the names of the libraries that were included for each operating system, plus we included a generic name for operating systems where the libraries were not included:
names = {
"Windows": "libogg.dll",
"Darwin": "libogg.0.dylib",
"external": "ogg"
}
The library loading code checks if it can find the specified library for the current operating system. If it isn't successful, a search is made for the 'external' library.
The library loading code can be found inside the PyOgg project; see Library.load().
Further to this, inspired by the delocate project, we were required to edit the paths found inside certain dylibs. For example, we edited the opusfile binary to point to the ogg binary found in our package:
install_name_tool -change /usr/local/opt/libogg/lib/libogg.0.dylib #loader_path/libogg.0.dylib libopusfile.0.dylib
For the details on this process please see the README file associated with the macOS binaries inside the PyOgg project.
Further discussion of the approach can be found in the issue raised on PyOgg's GitHub page (#32).

How to distribute C++ application which calls Python?

I know there is some way to call Python from C++, like Python/C API or Boost.Python. My question is, how can I distribute the application? For example, does user still need to install Python and Python packages on their machine?
My user case is: I want to use some Python code from my C++ code. The main application is written in C++. Then I am going to deploy my app. The goal is to make the app self contained, and user don't need to install Python and Python packages at all.
The possible steps may be :
1, calling Python from C++ via Python/C API or boost.Python from source code.
2, bring Python/C libraries together with application.
I hope after these 2 steps, my app will be a self-contained and standalone software. User can just copy the app folder to any other machines which has no Python installed.
Note that due to license issue, I can not use PyInstaller. I also meet some problems when trying to use "Nuitka" to make the Python part self contained. So I am now trying directly calling Python from C++. I know it will run on my developer machine. But needs to confirm that this solution can also make app self-contained and won't ask user to install Python.
Update: Now I feel I need to do something to make my app self-contained if I use Python/C to call python from C++ :
1, I need to bring all needed runtime with my app. (C++ runtime of course, and the python_version.dll)
2, I need to deploy a Python interpreter inside my app. Simply copy the Python folder from Python installation and remove some not needed files (like header files, lib files)
3, use Py_SetPythonHome function to points to the copied Python interpreter inside the app.
I'd say you're on the right track. Basically, you should obtain a Python (shared or static) library, compile your program with it, and of course bundle the Python dependencies you have with your program. The best documentation I've read is available here: https://docs.python.org/3.8/extending/embedding.html#embedding-python-in-another-application. Roughly, the process is:
Get a Python library from python.org and compile with ./configure --enable-shared (I believe omitting --enable-shared does only produce the python binary).
Compile your program. Have it reference the headers under Include and link the library. Note that you can obtain the compiler and linker flags you need as described here.
Call Python code from within your application using e.g. PyRun_SimpleString() or other functions from the C API. Note that you may also depend on the Python standard library (under Lib in the distribution) if there's any functionality you use from it.
If you linked against Python statically, at this point you're done, aside from bundling any Python code you depend on, which I'm not sure is relevant in your case.
I am suffering from the same problem, I had a project which is made up of C++ and python(embedded) and there is a problem of deployment/distribution.
After research, I got a solution which is not perfect (means it will be helpful to run your app in other system)
change visual studio in release mode and compile(you got a folder in your working directory)
install pyinstaller (pip install pyinstaller)
then navigate to pyinstaller folder and command:-pyinstaller.exe "your script_file_path.py"
-it will create a dist folder
copy that folder in working folder where exe exists.
remember.
dist folder and c/python code compiled by same version of python.
now good to go.
it will work.

how to add precompiled binaries to a python project (cloud-functions python)?

I have to deploy a python function to GCP. The libraries I want to use a library (USD by Pixar specifically) which needs I need to build myself. To make the library accessible I need to make changes to $PATH an $PYTHONPATH.
So the problem is that I have to include everything in one project to deploy the function and I don't know where to start.
I have tried appending to PYTHONPATH on run-time but it gives no module error. Also I have no idea how can I change PATH variable to be able to use executables inside usd/bin folder
import sys
sys.path.append('lib/python') # relative path.

Using external modules in a Pycharm project

I am new to Pycharm and need some help. I am working on a project that makes use of a large library of modules (specifically, Schrodinger; which allows for a lot of cool chemistry programs). Schrodinger requires the use of Python 2.7, if that makes any difference.
There are too many modules to install to the project directory. When I move the project directory to the location of the modules, my script becomes stuck on 'initializing'. I have attempted to import it as a package to no avail.
I have also tried to use the sys.path command, however a lot of the modules make use of other modules as well. So I that has become a pain very quickly.
How can I use these modules within Pycharm? And if there is no easy way, do you have a recommendation for an IDE that does have this feature?
Thanks
Pycharm doesn't identifies user defined modules which are not imported to Pycharm.
I usually mask the module as a Sources Root see the picture for more details. if the modules are in same project.
Alternative way: In your case import the external modules using File -> Open modules with open -> open in current window -> add to currently opened project this looks like two different projects. Now you can mark Sources Root for the complete module (i.e. learning) which you have imported.
import stackoverflow
Now pycharm can identifies the user defined modules.

Categories