I'm currently in a class where I'm asked to establish integration between Python and C++ as we will be working with both later in the class. My instructor has provided some simple C++ and Python code so we can test the integration and has also provided instructions on how to set this up using Visual Studio. I currently only have access to Ubuntu and my (very old) laptop won't be able to handle a VM.
I'm currently using Clion as my IDE and Anaconda as a virtual environment. I've spent most of the day trying to figure this out and I believe the issue is related to my CMake file as I know pretty much nothing about CMake right now, but plan to learn it soon.
The CMakeLists.txt, main.cpp, and myPython.py are all in the main project directory.
CMakeLists.txt
cmake_minimum_required(VERSION 3.21)
project(myProject)
set(CMAKE_CXX_STANDARD 14)
add_executable(myProject main.cpp)
target_include_directories(myProject PUBLIC /usr/include/python3.9)
target_link_directories(myProject PUBLIC /usr/lib/python3.9)
target_link_libraries(myProject PUBLIC python3.9)
For CMakeLists.txt I also tried the following but got the same error/output regardless of which one I used
cmake_minimum_required(VERSION 3.21)
project(myProject)
set(CMAKE_CXX_STANDARD 14)
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
add_executable(myProject main.cpp)
target_link_libraries(myProject PUBLIC Python3::Python)
main.cpp
#include <python3.9/Python.h>
#include <iostream>
#include <string>
using namespace std;
// the main in the provided code is void but Clion gave me an error saying to change it to int
int main()
{
cout << "Start 1 \n";
Py_Initialize();
cout << "2\n";
PyObject* my_module = PyImport_ImportModule("myPython");
cerr << my_module << "\n";
PyErr_Print();
cout << "3\n";
PyObject* my_function = PyObject_GetAttrString(my_module,
"printsomething");
cout << "4\n";
PyObject* my_result = PyObject_CallObject(my_function, NULL);
Py_Finalize();
return 0;
}
myPython.py
import re
import string
def printsomething():
print("Hello from Python!")
The expected output is
Start 1
2
01592AE0 // this will vary since it's the .py files location
3
4
Hello from Python!
what I'm getting when I run it is
Start 1
2
0 // output is red
3
ModuleNotFoundError: No module named 'myPython' // output is red
Since I'm not familiar with CMake and would consider myself a "general user" when it comes to Ubuntu, I may need some additional details on any steps provided.
Add this in your main function after Py_Initialize();
PySys_SetPath(L".:/usr/lib/python3.8");
this is the search path for python module. Add . to search in the current path.
Related
This question already has answers here:
how can i include python.h in QMake
(1 answer)
Embedding python 3.4 into C++ Qt Application?
(4 answers)
Closed 2 years ago.
When I was trying to embed a Python script into my Qt C++ program, I run into multiple problems when trying to include Python.h.
The following features, I would like to provide:
Include python.h
Execute Python Strings
Execute Python Scripts
Execute Python Scripts with Arguments
It should also work when Python is not installed on the deployed machine
Therefore I searched around the Internet to try to find a solution. And found a lot of Questions and Blogs, but non have them covered all my Problems and it still took me multiple hours and a lot of frustration.
That's why I have to write down a StackOverflow entry with my full solution so it might help and might accelerate all your work :)
(This answer and all its code examples work also in a non-Qt environment. Only 2. and 4. are Qt specific)
Download and install Python https://www.python.org/downloads/release
Alter the .pro file of your project and add the following lines (edit for your correct python path):
INCLUDEPATH = "C:\Users\Public\AppData\Local\Programs\Python\Python39\include"
LIBS += -L"C:\Users\Public\AppData\Local\Programs\Python\Python39\libs" -l"python39"
Example main.cpp code:
#include <QCoreApplication>
#pragma push_macro("slots")
#undef slots
#include <Python.h>
#pragma pop_macro("slots")
/*!
* \brief runPy can execut a Python string
* \param string (Python code)
*/
static void runPy(const char* string){
Py_Initialize();
PyRun_SimpleString(string);
Py_Finalize();
}
/*!
* \brief runPyScript executs a Python script
* \param file (the path of the script)
*/
static void runPyScript(const char* file){
FILE* fp;
Py_Initialize();
fp = _Py_fopen(file, "r");
PyRun_SimpleFile(fp, file);
Py_Finalize();
}
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
runPy("from time import time,ctime\n"
"print('Today is', ctime(time()))\n");
//uncomment the following line to run a script
//runPyScript("test/decode.py");
return a.exec();
}
Whenever you #include <Python.h> use the following code instead. (The Slots from Python will otherwise conflict with the Qt Slots
#pragma push_macro("slots")
#undef slots
#include <Python.h>
#pragma pop_macro("slots")
After compiling, add the python3.dll, python39.dll, as well as the DLLs and Lib Python folders to your compilation folder. You can find them in the root directory of your Python installation. This will allow you to run the embedded c++ code even when python is not installed.
With these steps, I was able to get python running in Qt with the 64 bit MinGW and MSVC compiler. Only the MSVC in debug mode got still a problem.
FURTHER:
If you want to pass arguments to the python script, you need the following function (It can be easy copy-pasted into your code):
/*!
* \brief runPyScriptArgs executs a Python script and passes arguments
* \param file (the path of the script)
* \param argc amount of arguments
* \param argv array of arguments with size of argc
*/
static void runPyScriptArgs(const char* file, int argc, char *argv[]){
FILE* fp;
wchar_t** wargv = new wchar_t*[argc];
for(int i = 0; i < argc; i++)
{
wargv[i] = Py_DecodeLocale(argv[i], nullptr);
if(wargv[i] == nullptr)
{
return;
}
}
Py_SetProgramName(wargv[0]);
Py_Initialize();
PySys_SetArgv(argc, wargv);
fp = _Py_fopen(file, "r");
PyRun_SimpleFile(fp, file);
Py_Finalize();
for(int i = 0; i < argc; i++)
{
PyMem_RawFree(wargv[i]);
wargv[i] = nullptr;
}
delete[] wargv;
wargv = nullptr;
}
To use this function, call it like this (For example in your main):
int py_argc = 2;
char* py_argv[py_argc];
py_argv[0] = "Progamm";
py_argv[1] = "Hello";
runPyScriptArgs("test/test.py", py_argc, py_argv);
Together with the test.py script in the test folder:
import sys
if len(sys.argv) != 2:
sys.exit("Not enough args")
ca_one = str(sys.argv[0])
ca_two = str(sys.argv[1])
print ("My command line args are " + ca_one + " and " + ca_two)
you get the following output:
My command line args are Progamm and Hello
I'm very new to the whole CMake. Following this and this posts, now I want to call a MAXON function inside Python, using pybind11. What I have done so far:
The library can be downloaded from this page (direct download link).
wget https://www.maxongroup.com/medias/sys_master/root/8837358518302/EPOS-Linux-Library-En.zip
unzip:
unzip EPOS-Linux-Library-En.zip
make the install shell script executable and run it:
chmod +x ./install.sh
sudo ./install.sh
Then going to the example folder:
cd /opt/EposCmdLib_6.6.1.0/examples/HelloEposCmd/
Now combining the CMakeLists.txt files from here:
# CMakeLists.txt
cmake_minimum_required(VERSION 2.8.12)
project (HelloEposCmd)
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -Wall")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -Wall")
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
find_package(pybind11 REQUIRED)
pybind11_add_module(${PROJECT_NAME} HelloEposCmd.cpp)
add_executable(${PROJECT_NAME} HelloEposCmd.cpp)
target_link_libraries(${PROJECT_NAME} -lEposCmd)
and the HelloEposCmd.cpp this line is added right after other header files:
#include <pybind11/pybind11.h>
the main function is renamed to:
int run(int argc, char** argv)
and the pybind11 syntax to add the module is written at the end:
PYBIND11_MODULE(HelloEposCmd, m) {
m.def("run", &run, "runs the HelloEposCmd");
}
However, When I run the cmake . I get the error:
CMake Error at CMakeLists.txt:13 (add_executable):
add_executable can not create target "HelloEposCmd" because another target with the same name already exists. The existing target is a module library created in source directory "/opt/EposCmdLib_6.6.1.0/examples/HelloEposCmd" See documentation for policy CMP0002 for more details.
...
I was wondering if you could be kind to help me get the right CMakeList.txt file. Ideally, I should be able to call the compiled module in python:
# HelloEposCmd.py
import HelloEposCmd
HelloEposCmd.run()
Thanks for your support in advance.
pybind11_add_module already creates a target for you. So you don't need add_executable anymore. Just remove that line and when you will build you will get a library with the name HelloEposCmd
add_executable is needed if you are building an executable (.exe), which I believe is not what you want.
Documenation of pybind11 says.
This function behaves very much like CMake’s builtin add_library (in fact, it’s a wrapper function around that command).
Thanks to abhilb post and his kind followup in the comments I was able to figure the problem out. well, at least find a temporary workaround:
According to this post, the last two lines of the CMakeLists.txt file should change to
# this line can be removed
# add_executable(${PROJECT_NAME} HelloEposCmd.cpp)
target_link_libraries(${PROJECT_NAME} PRIVATE -lEposCmd)
and then because according to this post pybind11 doesn't support double pointers we change the run function to:
int run() {
int argc = 1;
char* argv[] = {"./HelloEposCmd"};
...
}
which I suppose to be a horrible workaround (inspired by information from this page). Now running cmake ., make and python3 HelloEposCmd.py should work properly (except a small c++ warning!).
P.S.1. Maybe someone could use std::vector<std::string> as suggested here. This idea was proposed here and there are already some answers worth investigating.
P.S.2. Following this discussion, another workaround could be something like:
#include <stdio.h>
#include <stdlib.h>
void myFunc(int argc, char* argv[]) {
for (int i = 0; i < argc; ++i) {
printf("%s\n", argv[i]);
}
}
int run(int argc, long* argv_) {
char** argv = (char**)malloc(argc * sizeof(char*));
for (int i = 0; i < argc; ++i) {
argv[i] = (char*)(argv_[i]);
}
myFunc(argc, argv);
free(argv);
return 0;
}
I have the following code in the Qt Quick Application together with Boost.
In this Cpp there is a personal module created using BOOST_PYTHON_MODULE(hello). The main goal is to be able to import hello in Python and call the methods of hello struct. My Python script only contains very simple structure as i just want to see no errors when importing hello.
import hello
print("Import was successful!")
Most of the codes below are copied from a different question in stackoverflow but not entirely so i had to repost the parts.
Main.cpp
#include <cstdlib> // setenv, atoi
#include <iostream> // cerr, cout, endl
#include <boost/python.hpp>
#include <QGuiApplication>
#include <QQmlApplicationEngine>
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
//---------------------------------------------------------------------------------------------------------------------
/// Staticly linking a Python extension for embedded Python.
BOOST_PYTHON_MODULE(hello)
{
namespace python = boost::python;
python::class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
;
}
//---------------------------------------------------------------------------------------------------------------------
int main(int argc, char *argv[])
{
QCoreApplication::setAttribute(Qt::AA_EnableHighDpiScaling);
QGuiApplication app(argc, argv);
namespace python = boost::python;
try
{
int uploaded = PyImport_AppendInittab("hello", &PyInit_hello);
//This executes else part
if(uploaded == -1)
std::cout<< "Module table was not extended: " << uploaded << std::endl;
else
std::cout<< "Module Table was extended" << std::endl;
Py_Initialize();
} catch (...)
{
PyErr_Print();
return 1;
}
return app.exec();
}
Finally, I run my QT application and the return app.exec(); keeps it running while i try and run my python script as mentioned above from the terminal. The python script is in the same directory as the currently running application, not sure if that makes any difference.
Then the error i get is:
Traceback (most recent call last):
File "test_hilton.py", line 1, in <module>
import hello
ModuleNotFoundError: No module named 'hello'
Not sure what i am missing here. According to the Python API:
PyImport_AppendInittab - Add a single module to the existing table of
built-in modules. This is a convenience wrapper around
PyImport_ExtendInittab(), returning -1 if the table could not be
extended. The new module can be imported by the name name, and uses
the function initfunc as the initialization function called on the
first attempted import.
And the If-else part inside the try-catch block inside the main proves that the hello module is being added to the table. Out of ideas on what to do, looked in different places. But still stuck with this part of the problem.
Since the hello module is defined in that Qt program it is available only in that program. Executing the program doesn't make it available to python interpreter that expects to find hello.py or hello.so (the file extension may vary depending on the operating system) when importing hello by import hello.
You need to build a python module, answer might help.
I am working on packaging and distributing some python applications in Windows by wrapping together a python runtime, the python packages for the applications, and some executables to run the python applications. The approach is just to modify the source for python.exe to launch the applications but accept command line arguments for things like data file names.
Below is an example C++ source for one of the executables:
// source for my_python_application1
#include "stdafx.h"
#include "Windows.h"
#include "Python.h"
wchar_t SWITCH[] = L"-m";
wchar_t APP[] = L"my_python_application1.main";
int wmain(int argc, wchar_t **argv) {
int newargc;
newargc = argc + 2;
// can use this to modify the PythonPath for specific distributions
// _putenv("PYTHONPATH=\"\"");
wchar_t **newargv = new wchar_t*[newargc];
newargv[0] = argv[0];
newargv[1] = SWITCH;
newargv[2] = APP;
for (int i = 1; i < argc; i++) {
newargv[i + 2] = argv[i];
}
return Py_Main(newargc, newargv);
// return Py_Main(argc, argv);
}
Functionally this achieves everything I need it to achieve, but I suffer from a certain OCD nature which leads me to want things organized in a certain way. I'd like to have a structure like the following
/application_suite
/python_runtime
python.exe
python36.dll
(and everything else in a python dir)
/python_applications
my_python_application1.exe
my_python_application2.exe
However, since mypythonapplication1/2.exe are basically modified python.exe files, in order for them to work properly (load the python dll, import modules, access all of the landmarking features necessary for modules to be interconnected) they need to be located in the /python_runtime directory.
I'm wondering is there a way to compile these executables so that they can be arranged in the directory structure that I presented, but know that they python_runtime directory and all of its structure are located in a relative path of './python_runtime' or whatever so that this all behaves well no matter where the distribution of applications is installed by the end user.
Pre-Answer Warning I am not a C/C++ programmer. It is possible there are bad C++ practices in here, so please use what you find in this answer with a grain of salt.
The requirements to achieve this behavior are the following:
We must get the directory of the custom executable
We must set the PYTHONHOME environment variable to %executable_dir%\runtime
We must set the PYTHONPATH environment variable to %executable_dir%\apps so that python knows where our python packages are living. This also clears out any system wide settings so that the distribution doesn't use other python environment settings
I don't know if it's necessary, but I am adding the runtime directory at the front of the path
We have to dynamically load the Py_Main function from the desired dll. Since we are not expecting the runtime to be on the path before execution, we must find the dll dynamically from %executable_dir%\runtime\python36.dll.
The following source code works when I compiled in Visual Studio 2017, with no Python Header files and no dll specified in the Linker
// source code for custom my_python_application1
// requires _CRT_SECURE_NO_WARNINGS flag to compile deprecated path operations
#include "stdafx.h"
#include <string>
#include <sstream>
#include <iostream>
#include "Windows.h"
#include "Shlwapi.h"
// #include "Python.h" // don't need this as we are dynamically loading the library of choice
#pragma comment(lib, "Shlwapi.lib")
__pragma(warning(disable:4996)) // # _CRT_SECURE_NO_DEPRECIATE
wchar_t SWITCH[] = L"-m";
wchar_t APP[] = L"my_python_application1.main";
typedef int(__stdcall *py_main_function)(int, wchar_t**);
int wmain(int argc, wchar_t **argv) {
int newargc;
newargc = argc + 2;
// determine the path of the executable so we know the absolute path
// of the python runtime and application directories
wchar_t executable_dir[MAX_PATH];
if (GetModuleFileName(NULL, executable_dir, MAX_PATH) == 0)
return -1;
PathRemoveFileSpec(executable_dir);
std::wstring executable_dir_string(executable_dir);
// now set the relevant environment variables so that the environment works as it is supposed to
std::wstring python_home(L"PYTHONHOME=" + executable_dir_string + L"\\runtime");
_wputenv(python_home.c_str());
std::wstring python_path(L"PYTHONPATH=" + executable_dir_string + L"\\apps");
_wputenv(python_path.c_str());
// put the python runtime at the front of the path
std::wstringstream ss;
ss << "PATH=" << executable_dir << "\\runtime;" << getenv("PATH");
std::wstring path_string (ss.str());
_wputenv(path_string.c_str());
wchar_t **newargv = new wchar_t*[newargc];
newargv[0] = argv[0];
newargv[1] = SWITCH;
newargv[2] = APP;
for (int i = 1; i < argc; i++) {
newargv[i + 2] = argv[i];
}
// dynamically load the python dll
std::wstring python_dll(executable_dir_string + L"\\runtime\\python36.dll");
HINSTANCE hGetProcIDDLL = LoadLibrary(python_dll.c_str());
py_main_function Py_Main = (py_main_function)GetProcAddress(hGetProcIDDLL, "Py_Main");
//now call Py_Main with our arguments
return Py_Main(newargc, newargv);
// return Py_Main(argc, argv);
}
I have a simple python script that uses a c/c++ library with ctypes. My c++ library also contains a main method, so I can compile it without the -shared flag and it can be excecuted and it runs without issues.
However, when I run the same code from a python script using ctypes, a part of the c++ program is excecuted (I can tell that from the cout calls). Then the entire application, including the python script, termiantes (I can tell that from the missing cout and print calls). There is no error message, no segfault, no python stacktrace.
My question is: How can I debug this? What are possible reasons for this to happen?
Here is part of the code, however, since there is no error message, I don't know which code is relevant.
import ctypes
interface = ctypes.CDLL("apprunner.so")
interface.start()
print "complete"
.
#include "../../app/ShaderApp.cpp"
#include <iostream>
#include "TestApp.cpp"
TestApp* app = 0;
extern "C" void start() {
app = new TestApp();
cout << "Running from library" << endl;
app->run();
}
int main( int argc, const char* argv[]) {
cout << "Running from excecutable" << endl;
start();
}
Typically you begin from a small mock-up library that just lets you test the function calls from python. When this is ready (all the debug prints are ok) you proceed further. In your example, comment out #include "testapp.cpp" and get the prints to cout working.