We Keep Coding

Python/C++ wrapper Using external dll with Pybind11

Python version: 3.8.1
Spyder version: 3.3.6
Qt version: 5.12.9
Wrapper: develop using PyBind11
I am wrapping a dll develop in C++ which use Qt dlls to be used with Python. I wrote the wrapper with Visual Studio 2019 using the compiler MSVC (as my dll is compiled with MSVC). After generating the solution in VS2019 I obtain a .pyd file which can be import with python.
It works good when I use python on line command:
Start cmd.exe
$python
import MyLibName
I can use the functions/classes ...
But if I try with Spyder, I get the following error:
ImportError: DLL load failed while importing PyStack: The specified module could not be found..
So here are my questions :
Is there a way to get more information about ImportError like the name of the missing dll or something?
I don't understand why the issue only happen with spyder. I tried with IPython Qt Console and it work. Does spyder use a embeded python version or something ?
I don't fully understand how dll shall be managed, I mean shall I provide dll like libGLESV2.dll with the .pyd or just give a path where to find it ?
Thank you in advance.

My guess
I think I find out which part of Qt/python is producing this issue, but I still don't know how to solve it.
My dll use signals/slots which need an event loop to be performed. If an event loop is already running the dll will try to use it, if the loop version (ex : PyQt5==5.14.1) isn’t the same as mine (ex Qt==5.15.1) import will be impossible.
Note that the reverse is true, if I run my dll an then try to start a loop with %gui qt the command will throw an error.
How to reproduce the issue :
Compile a Qt project available here.
Copy the output dll in the folder PyMyStack/dependencies of the VS Project (available here)
Compile the VS project.
Open an IPython console (without using qt has event loop)
Import the module created with VS (Import PyMyStack)
Run the magic command %gui qt
Last command shall print : ERROR:root:DLL load failed while importing QtSvg: The specified procedure could not be found.
How to hide/solve the problem:
Disclaimer : The solutions presented here are surely not the best, if you know a better one please share it ☺
If you just want to import your lib in Spyder, you can use another event loop. Here are the steps to change this:
In Spyder menus go to Tools→Preferences
Select “IPython console”
Go to “Graphics” tab and change the backend combo box to any other values than Qt or Automatic
If you want to use Qt event loop you will have to update it. You can do this with pip command, but remember than Spyder is not compatible with some version. Here is the pip command:
Pip install PyQt5==X.Y.Z
Where X and Y are the same version use to compile your Qt project. The last digit version seems to not be important.

C++ - Python Embedding with numpy - error on deployment

I'm trying to embed python with numpy in a C++ application. I'm using Windows 10 and Visual Studio 2015.
Currently I have Anaconda and WinPython installed (because I'm using Python scripts that work only with one or the other). I don't have any environment variable related to python.
For my C++ application, I'm using the WinPython python, that have numpy and a handful of other packages installed. I managed to embed python and numpy in my application when using Visual studio, both for Debug and Release. Everything is working, python is initialized and I can use numpy array and functions. WinPython is correctly used. As a simple test in my code I have:
_putenv_s("PYTHONPATH", ".");
Py_InitializeEx(0);
PyRun_SimpleString("import sys");
PyRun_SimpleString("print(sys.path)");
PyRun_SimpleString("print(sys.prefix)");
PyRun_SimpleString("print(sys.executable)");
PyRun_SimpleString("import importlib.machinery");
PyRun_SimpleString("print(importlib.machinery.all_suffixes())");
init_numpy2();
That prints:
['C:\\DevC++\\Tesseler-Cmake\\build', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\python37.zip', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\DLLs', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\lib', 'C:\\DevC++\\Tesseler-Cmake\\build\\Release', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\lib\\site-packages', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\lib\\site-packages\\win32', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\lib\\site-packages\\win32\\lib', 'C:\\Git\\WPy64-3741\\python-3.7.4.amd64\\lib\\site-packages\\Pythonwin']
C:\Git\WPy64-3741\python-3.7.4.amd64
C:\DevC++\Tesseler-Cmake\build\Release\Tesseler.exe
['.py', '.pyw', '.pyc', '.cp37-win_amd64.pyd', '.pyd']
I then set-up an installer using Wix in Release, and checked that the Winpython python37.dll is shipped with my application. But when I run my program, I'm getting this error when calling import_numpy2():
['C:\\Tesseler', 'C:\\Tesseler\\python37.zip', 'C:\\Users\\Florian\\anaconda3\\Lib', 'C:\\Users\\Florian\\anaconda3\\DLLs', 'C:\\Users\\Florian\\anaconda3', 'C:\\Users\\Florian\\anaconda3\\lib\\site-packages', 'C:\\Users\\Florian\\anaconda3\\lib\\site-packages\\win32', 'C:\\Users\\Florian\\anaconda3\\lib\\site-packages\\win32\\lib', 'C:\\Users\\Florian\\anaconda3\\lib\\site-packages\\Pythonwin']
C:\Users\Florian\anaconda3
C:\Tesseler\Tesseler.exe
['.py', '.pyw', '.pyc', '.cp37-win_amd64.pyd', '.pyd']
ModuleNotFoundError: No module named 'numpy'
I don't understand why anaconda is added to the sys.path and sys.prefix since I never have any reference to it in my visual studio project, nor any environment variable referencing it.
I see why using a dll from anaconda could lead to some problem but I checked with Process Explorer that my application is using the python37.dll shipped with it, which is the case.
Any idea what could cause this error?
Update:
Following ideas described in this thread, I managed to make it work by creating a python subfolder and copying the whole numpy, scipy, pandas and statsmodels folders in it (these 4 modules are needed by my script). These folders were copied from C:\Git\WPy64-3741\python-3.7.4.amd64\Lib\site-packages. I also added this python subfolder to PYTHONPATH: _putenv_s("PYTHONPATH", ".;./python");
Anyway, if someone has a better solution I'm eager to hear it as I find it annoying to have to bundle these modules (more than 600 Mo) when my application is roughly 20 Mo...

Embedding Python in a Qt Creator project

I'm working on a project that requires C++ to call a program written in Python that relies on Python exclusive modules.
The project is handled using Qt Creator, and Python 3.7.5 and its packages are installed via Miniconda. I've gotten a basic embedding working using Pybind11 where basic interfacing works, however, most external modules cannot be imported.
For example, when importing Numpy through Pybind11, the following error is thrown (reduced for brevity):
Importing the numpy c-extensions failed.
Original error was: /home/brentnallt/miniconda3/envs/car_class_nogpu/lib/python3.7/site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyMemoryView_FromObject
A similar error occurs when importing tensorflow through Pybind11:
ImportError: /home/brentnallt/miniconda3/envs/car_class_nogpu/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyUnicode_FromFormat
It appears to be a problem with Python's C API being found when reading C extension shared libraries. However, modules like lxml which use C source files import just fine. Additionally, I can import problem modules in projects separate from the project I'm working on, implying it's a setup problem. Note that this test project setup doesn't actually use any QT functionality, whereas the main one does.
My PYTHONHOME environment variable looks like:
['/home/brentnallt/miniconda3/envs/car_class_nogpu/lib/python3.7', '/home/brentnallt/miniconda3/envs/car_class_nogpu/lib/python3.7/site-packages', '/home/brentnallt/miniconda3/envs/car_class_nogpu/lib/python37.zip', '/home/brentnallt/miniconda3/envs/car_class_nogpu/lib/python3.7/lib-dynload', '.']
Are there any special considerations I have to make when embedding with Qt Creator? Or is this likely a different problem from a setup error?

Maybe you can consider using PythonQt as an alternative module for calling and importing python libraries from Qt application.
I've used it a lot in my projects and it never failed, but never used it with any kinda data scientific modules maybe you could give it a chance
https://mevislab.github.io/pythonqt/

Python 3.3.5 time module missing

When I try to import time I get : No module named time
I have tried other time modules(datetime and timeit) and they work fine. I decided to check my installation and I can't find time.py anywhere. I checked the Lib, Scripts, libs and include folders, but can't find it anywhere.
Anyone know what I can do to fix this? Maybe download the .py and put it in Lib myself?
I am using Python 3.3.5 with PyCharm IDE. Only extra scripts I've installed is EasyInstall and PRAW.

The import does work. When PyCharm said No module named time, I assumed I would get a compiler error and started trying to fix it.
However when I eventually just ran the code it worked fine. I expect PyCharm doesn't detect the time module as it's a dll and not a py as noted by Martijn in the comments. This is on PyCharm Community Edition 4.0.4.

I tried playing with virtualenv and a host of other things, but I eventually went to Preferences -> Build, Execution, Deployment -> Console -> Python Console, and in the "starting script" box, I added two lines:
sys.builtin_module_names.append('sys')
sys.builtin_module_names.append('time')
This got rid of errors I had with both sys and time. Once I did that, I even get autocomplete for both of those modules... weird.

error on importing pyIpopt

I'm trying to install and configure pyIpopt. Ipopt is already installed and the examples run fine.
From the shell, when I do import pyIpopt, I get the error:
ImportError: /***PATH***/libipopt.so.1: undefined symbol: MPI_Init
The FAQ section of the pyIpopt git project has this to offer for these kinds of errors:
Do a Google search to find the library file, and add
-lWhateverLibrary in the makefile of pyipopt.
I've googled and found this: http://www.mcs.anl.gov/research/projects/mpi/www/www3/MPI_Init.html.
I don't know how to get the library or add it to the makefile... Any assistance would be much appreciated!

Just had a similar problem on ubuntu.
Using libmumps-seq worked for me:
installed libmumps-seq-4.9.2 (just with apt-get, along side the ordinary libmumps)
in setup.py changed in the libraries list argument 'coinmumps' to 'dmumps_seq-4.9.2'
rebuilt and installed.
If I understand it correctly, the default mumps is distributed (using MPI lib which can be a world of pain), and all i needed is the sequential one, which mumps-seq provides.