We Keep Coding

Python Initialization fails with dynamically loaded DLL [duplicate]

I have a C++ program and it has sort of plugin structure: when program starts up, it's looking for dll in the plugin folder with certain exported function signatures, such as:
void InitPlugin(FuncTable* funcTable);
Then the program will call the function in the dll to initialize and pass function pointers to the dll. From that time on, the dll can talk to the program.
I know Cython let you call C function in Python, but I'm not sure can I write a Cython code and compile it to a dll so my C++ program can initialize with it. An example code would be great.

Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.
The first step would be to mark cdef-function which should be used from external C-code with public, for example:
#cyfun.pyx:
#doesn't need python interpreter
cdef public int double_me(int me):
return 2*me;
#needs initialized python interpreter
cdef public void print_me(int me):
print("I'm", me);
cyfun.c and cyfun.h can be generated with
cython -3 cyfun.pyx
These files will be used for building of the dll.
The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail). Note: the initialization/clean-up could be put also in DllMain - see such a version further bellow.
The header-file for the dll would look like following:
//cyfun_dll.h
#ifdef BUILDING_DLL
#define DLL_PUBLIC __declspec(dllexport)
#else
#define DLL_PUBLIC __declspec(dllimport)
#endif
//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();
DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);
So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.
The implementation follows in cyfun_dll.c-file:
//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"
DLL_PUBLIC int cyfun_init(){
int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
if(status==-1){
return -1;//error
}
Py_Initialize();
PyObject *module = PyImport_ImportModule("cyfun");
if(module==NULL){
Py_Finalize();
return -1;//error
}
return 0;
}
DLL_PUBLIC void cyfun_finalize(){
Py_Finalize();
}
DLL_PUBLIC int cyfun_double_me(int me){
return double_me(me);
}
DLL_PUBLIC void cyfun_print_me(int me){
print_me(me);
}
Noteworthy details:
we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
we use cyfun.h generated by cython from cyfun.pyx.
cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29, PEP-489 is default. More information can be found in this SO-post. If the Python-interpreter isn't initialized or if the module cyfun is not imported, the chances are high, that calling the functionality from cyfun.h will end in a segmentation fault.
cyfun_double_me just wraps double_me so it becomes visible outside of the dll.
Now we can build the dll!
:: set up tool chain
call "<path_to_vcvarsall>\vcvarsall.bat" x64
:: build cyfun.c generated by cython
cl /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include>
:: build dll-wrapper
cl /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>
:: link both obj-files into a dll
link cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>
The dll is now built, but the following details are noteworthy:
<other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to run cythonize some_file.pyx` and to inspect the log.
we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
we have the dependency on the python-dll, so later on it must be somewhere where it can be found.
Were you go from here depends on your task, we test our dll with a simple main:
//test.c
#include "cyfun_dll.h"
int main(){
if(0!=cyfun_init()){
return -1;
}
cyfun_print_me(cyfun_double_me(2));
cyfun_finalize();
return 0;
}
which can be build via
...
:: build main-program
cl /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>
:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>
And now calling test_prog.exe leads to the expected output "I'm 4".
Depending on your installation, following things must be considered:
test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
The embeded python interpreter needs an installation, see this and/or this SO-posts.
IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.
Thus, it may make sense to put initialization/clean-up code into DllMain (and make cyfun_init() and cyfun_finalize() private), e.g.
BOOL WINAPI DllMain(
HINSTANCE hinstDLL, // handle to DLL module
DWORD fdwReason, // reason for calling function
LPVOID lpReserved ) // reserved
{
// Perform actions based on the reason for calling.
switch( fdwReason )
{
case DLL_PROCESS_ATTACH:
return cyfun_init()==0;
case DLL_PROCESS_DETACH:
cyfun_finalize();
break;
case DLL_THREAD_ATTACH:
// Do thread-specific initialization.
break;
case DLL_THREAD_DETACH:
// Do thread-specific cleanup.
break;
}
return TRUE;
}
If your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize the python-interpreter. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.

I'd imagine it'd be difficult to call it directly, with Cython depending so much on the Python runtime.
Your best bet is to embed a Python interpreter directly inside your app, e.g. as described in this answer, and call your Cython code from the interpreter. That's what I would do.

Embed / Include Python.h into C++ [Full Guide] (Python 3.9) (Windows) (Qt 5.15) [duplicate]

This question already has answers here:
how can i include python.h in QMake
(1 answer)
Embedding python 3.4 into C++ Qt Application?
(4 answers)
Closed 2 years ago.
When I was trying to embed a Python script into my Qt C++ program, I run into multiple problems when trying to include Python.h.
The following features, I would like to provide:
Include python.h
Execute Python Strings
Execute Python Scripts
Execute Python Scripts with Arguments
It should also work when Python is not installed on the deployed machine
Therefore I searched around the Internet to try to find a solution. And found a lot of Questions and Blogs, but non have them covered all my Problems and it still took me multiple hours and a lot of frustration.
That's why I have to write down a StackOverflow entry with my full solution so it might help and might accelerate all your work :)

(This answer and all its code examples work also in a non-Qt environment. Only 2. and 4. are Qt specific)
Download and install Python https://www.python.org/downloads/release
Alter the .pro file of your project and add the following lines (edit for your correct python path):
INCLUDEPATH = "C:\Users\Public\AppData\Local\Programs\Python\Python39\include"
LIBS += -L"C:\Users\Public\AppData\Local\Programs\Python\Python39\libs" -l"python39"
Example main.cpp code:
#include <QCoreApplication>
#pragma push_macro("slots")
#undef slots
#include <Python.h>
#pragma pop_macro("slots")
/*!
* \brief runPy can execut a Python string
* \param string (Python code)
*/
static void runPy(const char* string){
Py_Initialize();
PyRun_SimpleString(string);
Py_Finalize();
}
/*!
* \brief runPyScript executs a Python script
* \param file (the path of the script)
*/
static void runPyScript(const char* file){
FILE* fp;
Py_Initialize();
fp = _Py_fopen(file, "r");
PyRun_SimpleFile(fp, file);
Py_Finalize();
}
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
runPy("from time import time,ctime\n"
"print('Today is', ctime(time()))\n");
//uncomment the following line to run a script
//runPyScript("test/decode.py");
return a.exec();
}
Whenever you #include <Python.h> use the following code instead. (The Slots from Python will otherwise conflict with the Qt Slots
#pragma push_macro("slots")
#undef slots
#include <Python.h>
#pragma pop_macro("slots")
After compiling, add the python3.dll, python39.dll, as well as the DLLs and Lib Python folders to your compilation folder. You can find them in the root directory of your Python installation. This will allow you to run the embedded c++ code even when python is not installed.
With these steps, I was able to get python running in Qt with the 64 bit MinGW and MSVC compiler. Only the MSVC in debug mode got still a problem.
FURTHER:
If you want to pass arguments to the python script, you need the following function (It can be easy copy-pasted into your code):
/*!
* \brief runPyScriptArgs executs a Python script and passes arguments
* \param file (the path of the script)
* \param argc amount of arguments
* \param argv array of arguments with size of argc
*/
static void runPyScriptArgs(const char* file, int argc, char *argv[]){
FILE* fp;
wchar_t** wargv = new wchar_t*[argc];
for(int i = 0; i < argc; i++)
{
wargv[i] = Py_DecodeLocale(argv[i], nullptr);
if(wargv[i] == nullptr)
{
return;
}
}
Py_SetProgramName(wargv[0]);
Py_Initialize();
PySys_SetArgv(argc, wargv);
fp = _Py_fopen(file, "r");
PyRun_SimpleFile(fp, file);
Py_Finalize();
for(int i = 0; i < argc; i++)
{
PyMem_RawFree(wargv[i]);
wargv[i] = nullptr;
}
delete[] wargv;
wargv = nullptr;
}
To use this function, call it like this (For example in your main):
int py_argc = 2;
char* py_argv[py_argc];
py_argv[0] = "Progamm";
py_argv[1] = "Hello";
runPyScriptArgs("test/test.py", py_argc, py_argv);
Together with the test.py script in the test folder:
import sys
if len(sys.argv) != 2:
sys.exit("Not enough args")
ca_one = str(sys.argv[0])
ca_two = str(sys.argv[1])
print ("My command line args are " + ca_one + " and " + ca_two)
you get the following output:
My command line args are Progamm and Hello

The correct CMakeLists.txt file to call a MAXON libarary in a Python script using pybind11

I'm very new to the whole CMake. Following this and this posts, now I want to call a MAXON function inside Python, using pybind11. What I have done so far:
The library can be downloaded from this page (direct download link).
wget https://www.maxongroup.com/medias/sys_master/root/8837358518302/EPOS-Linux-Library-En.zip
unzip:
unzip EPOS-Linux-Library-En.zip
make the install shell script executable and run it:
chmod +x ./install.sh
sudo ./install.sh
Then going to the example folder:
cd /opt/EposCmdLib_6.6.1.0/examples/HelloEposCmd/
Now combining the CMakeLists.txt files from here:
# CMakeLists.txt
cmake_minimum_required(VERSION 2.8.12)
project (HelloEposCmd)
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Wall")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -Wall")
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
find_package(pybind11 REQUIRED)
pybind11_add_module(${PROJECT_NAME} HelloEposCmd.cpp)
add_executable(${PROJECT_NAME} HelloEposCmd.cpp)
target_link_libraries(${PROJECT_NAME} -lEposCmd)
and the HelloEposCmd.cpp this line is added right after other header files:
#include <pybind11/pybind11.h>
the main function is renamed to:
int run(int argc, char** argv)
and the pybind11 syntax to add the module is written at the end:
PYBIND11_MODULE(HelloEposCmd, m) {
m.def("run", &run, "runs the HelloEposCmd");
}
However, When I run the cmake . I get the error:
CMake Error at CMakeLists.txt:13 (add_executable):
add_executable can not create target "HelloEposCmd" because another target with the same name already exists. The existing target is a module library created in source directory "/opt/EposCmdLib_6.6.1.0/examples/HelloEposCmd" See documentation for policy CMP0002 for more details.
...
I was wondering if you could be kind to help me get the right CMakeList.txt file. Ideally, I should be able to call the compiled module in python:
# HelloEposCmd.py
import HelloEposCmd
HelloEposCmd.run()
Thanks for your support in advance.

pybind11_add_module already creates a target for you. So you don't need add_executable anymore. Just remove that line and when you will build you will get a library with the name HelloEposCmd
add_executable is needed if you are building an executable (.exe), which I believe is not what you want.
Documenation of pybind11 says.
This function behaves very much like CMake’s builtin add_library (in fact, it’s a wrapper function around that command).

Thanks to abhilb post and his kind followup in the comments I was able to figure the problem out. well, at least find a temporary workaround:
According to this post, the last two lines of the CMakeLists.txt file should change to
# this line can be removed
# add_executable(${PROJECT_NAME} HelloEposCmd.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE -lEposCmd)
and then because according to this post pybind11 doesn't support double pointers we change the run function to:
int run() {
int argc = 1;
char* argv[] = {"./HelloEposCmd"};
...
}
which I suppose to be a horrible workaround (inspired by information from this page). Now running cmake ., make and python3 HelloEposCmd.py should work properly (except a small c++ warning!).
P.S.1. Maybe someone could use std::vector<std::string> as suggested here. This idea was proposed here and there are already some answers worth investigating.
P.S.2. Following this discussion, another workaround could be something like:
#include <stdio.h>
#include <stdlib.h>
void myFunc(int argc, char* argv[]) {
for (int i = 0; i < argc; ++i) {
printf("%s\n", argv[i]);
}
}
int run(int argc, long* argv_) {
char** argv = (char**)malloc(argc * sizeof(char*));
for (int i = 0; i < argc; ++i) {
argv[i] = (char*)(argv_[i]);
}
myFunc(argc, argv);
free(argv);
return 0;
}

compile python executable to work outside of main python directory

I am working on packaging and distributing some python applications in Windows by wrapping together a python runtime, the python packages for the applications, and some executables to run the python applications. The approach is just to modify the source for python.exe to launch the applications but accept command line arguments for things like data file names.
Below is an example C++ source for one of the executables:
// source for my_python_application1
#include "stdafx.h"
#include "Windows.h"
#include "Python.h"
wchar_t SWITCH[] = L"-m";
wchar_t APP[] = L"my_python_application1.main";
int wmain(int argc, wchar_t **argv) {
int newargc;
newargc = argc + 2;
// can use this to modify the PythonPath for specific distributions
// _putenv("PYTHONPATH=\"\"");
wchar_t **newargv = new wchar_t*[newargc];
newargv[0] = argv[0];
newargv[1] = SWITCH;
newargv[2] = APP;
for (int i = 1; i < argc; i++) {
newargv[i + 2] = argv[i];
}
return Py_Main(newargc, newargv);
// return Py_Main(argc, argv);
}
Functionally this achieves everything I need it to achieve, but I suffer from a certain OCD nature which leads me to want things organized in a certain way. I'd like to have a structure like the following
/application_suite
/python_runtime
python.exe
python36.dll
(and everything else in a python dir)
/python_applications
my_python_application1.exe
my_python_application2.exe
However, since mypythonapplication1/2.exe are basically modified python.exe files, in order for them to work properly (load the python dll, import modules, access all of the landmarking features necessary for modules to be interconnected) they need to be located in the /python_runtime directory.
I'm wondering is there a way to compile these executables so that they can be arranged in the directory structure that I presented, but know that they python_runtime directory and all of its structure are located in a relative path of './python_runtime' or whatever so that this all behaves well no matter where the distribution of applications is installed by the end user.

Pre-Answer Warning I am not a C/C++ programmer. It is possible there are bad C++ practices in here, so please use what you find in this answer with a grain of salt.
The requirements to achieve this behavior are the following:
We must get the directory of the custom executable
We must set the PYTHONHOME environment variable to %executable_dir%\runtime
We must set the PYTHONPATH environment variable to %executable_dir%\apps so that python knows where our python packages are living. This also clears out any system wide settings so that the distribution doesn't use other python environment settings
I don't know if it's necessary, but I am adding the runtime directory at the front of the path
We have to dynamically load the Py_Main function from the desired dll. Since we are not expecting the runtime to be on the path before execution, we must find the dll dynamically from %executable_dir%\runtime\python36.dll.
The following source code works when I compiled in Visual Studio 2017, with no Python Header files and no dll specified in the Linker
// source code for custom my_python_application1
// requires _CRT_SECURE_NO_WARNINGS flag to compile deprecated path operations
#include "stdafx.h"
#include <string>
#include <sstream>
#include <iostream>
#include "Windows.h"
#include "Shlwapi.h"
// #include "Python.h" // don't need this as we are dynamically loading the library of choice
#pragma comment(lib, "Shlwapi.lib")
__pragma(warning(disable:4996)) // # _CRT_SECURE_NO_DEPRECIATE
wchar_t SWITCH[] = L"-m";
wchar_t APP[] = L"my_python_application1.main";
typedef int(__stdcall *py_main_function)(int, wchar_t**);
int wmain(int argc, wchar_t **argv) {
int newargc;
newargc = argc + 2;
// determine the path of the executable so we know the absolute path
// of the python runtime and application directories
wchar_t executable_dir[MAX_PATH];
if (GetModuleFileName(NULL, executable_dir, MAX_PATH) == 0)
return -1;
PathRemoveFileSpec(executable_dir);
std::wstring executable_dir_string(executable_dir);
// now set the relevant environment variables so that the environment works as it is supposed to
std::wstring python_home(L"PYTHONHOME=" + executable_dir_string + L"\\runtime");
_wputenv(python_home.c_str());
std::wstring python_path(L"PYTHONPATH=" + executable_dir_string + L"\\apps");
_wputenv(python_path.c_str());
// put the python runtime at the front of the path
std::wstringstream ss;
ss << "PATH=" << executable_dir << "\\runtime;" << getenv("PATH");
std::wstring path_string (ss.str());
_wputenv(path_string.c_str());
wchar_t **newargv = new wchar_t*[newargc];
newargv[0] = argv[0];
newargv[1] = SWITCH;
newargv[2] = APP;
for (int i = 1; i < argc; i++) {
newargv[i + 2] = argv[i];
}
// dynamically load the python dll
std::wstring python_dll(executable_dir_string + L"\\runtime\\python36.dll");
HINSTANCE hGetProcIDDLL = LoadLibrary(python_dll.c_str());
py_main_function Py_Main = (py_main_function)GetProcAddress(hGetProcIDDLL, "Py_Main");
//now call Py_Main with our arguments
return Py_Main(newargc, newargv);
// return Py_Main(argc, argv);
}

fseek with SEEK_END returns a "Invalid argument" error to manage large data(7GB) with python x86 C extension lib on windows7 x64

I've been trying to manage large binary data(7GB) within python x86 original extension library.
But fseek with SEEK_END doesn't work well.
I put _FILE_OFFSET_BITS 64 macro. I also tried fseeko64, but it raises an error.
With Less than 2GB files or using SEEK_CUR, SEEK_SET it's works fine.
I've been stuck for a couple of days. Could anybody give me ideas?
#define _GNU_SOURCE
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#define _FILE_OFFSET_BITS 64
#include <Python.h>
#include "structmember.h"
#include <stdio.h>
static PyObject *
MyClass_load(MyClass* self, PyObject *args)
{
const char* file_path;
if (!PyArg_ParseTuple(args, "s", &file_path))
return NULL;
self->fp = fopen(file_path ,"rb");
if (self->fp == NULL) {
PyErr_SetString(PyExc_IOError, "File does not exist.");
return NULL;
}
off_t offset = 0;
if(fseek(self->fp, offset, SEEK_END) != 0){
printf("%s\n", strerror(errno)); // show "Invalid argument"
PyErr_SetString(PyExc_IOError, "Seek failed.");
return NULL;
}
Py_INCREF(Py_None);
return Py_None;
}
Environment:
Windows 7 x64
Python 2.7 x86
MinGW GCC

Using '_fseeki64' and '_ftelli64' like VC works perfectly.
I know I'm still using gcc to compile c file to python c library file, but I don't know why I can use VC code in gcc.
Anyway, problem was solved!