Firstly my problem is similar to this one:
Debugging embedded Python
However although the description in that case is very helpful, it doesn't alas contain all the information needed to debug a mixed C++/embedded Python app.
Background: I have a C++ app that is calling Python code with some argument:
bool
ui::runLUOvershoot4(PyObject*inData)
{
// Get the python module.
PyObject* const pName = PyBytes_FromString("lu_overshoot4");
PyObject* pModule = nullptr;
PyObject* const pMod = PyImport_AddModule("lu_overshoot4");
if (pMod != nullptr) {
pModule = PyImport_ReloadModule(pMod);
}
/* NB pMod is borrowed and must not be Py_DECREF-ed */
if (pModule == nullptr) {
PyErr_Print();
return false;
}
Py_DECREF(pName);
// Get the dictionary for the lu_overshoot4 module
PyObject* const pDict = PyModule_GetDict(pModule);
Py_DECREF(pModule);
// Get the function in the dictionary
PyObject* const pFunc = PyDict_GetItemString(pDict, "LU_Overshoot4");
if (pFunc == nullptr) {
PyErr_Print();
return false;
}
Py_INCREF(pFunc);
if (pFunc && PyCallable_Check(pFunc)) {
// Call the python func with inData as arg
PyObject* arglist = Py_BuildValue("(O)", inData);
PyObject* const pValue = PyObject_Call(pFunc, arglist, nullptr);
if (pValue == nullptr) {
PyErr_Print();
return false;
}
Py_DECREF(pFunc);
return true;
}
return false;
}
All works as expected. But I want to be able to step in the debugger from the C++ PyObject_Call into the python code. So far I have been able to do this only if I start the C++ code without the debugger, then attach the debugger to the running process. If I try and start in the debugger, the breakpoints in the python code are disabled with the tooltip 'The breakpoint will not currently be hit. No symbols have been loaded for this document'.
A little detail on the setup: this is a Visual Studio 2022 solution with a C++ project (with the debugger to launch being the Python/Native Debugger) and a Python project containing the python files. It's not clear to me what settings to give the Python project, what I have is in the General tab,
Startup File -
Working Directory - .
Windows Application - ticked (makes no difference though)
Interpreter - env(Python 3.8 64-bit).
In the Debug tab,
Search Paths - .;\env
Script args -
Interpreter Path - \env
Interpreter Args -
Environment Variables -
Debug - 'Enable native code debugging' checked.
The python project's Python Environment was set up with a virtual env, set to the \env directory, which contains the include/Lib/Scripts/share dirs and python.exe, python38.dll, python.pdb etc. i.e. python installation and pdb files. This is how (I believe) 'Lambda' describes their setup.
One of the issues I faced was that at first VS2022 (v 17.2.6) did not give the option of setting the debugger to python/native; a fix is described in https://learn.microsoft.com/en-us/answers/questions/860222/the-option-34pythonnative-debugging34-is-missing-f.html. But with that set and starting the debugger with F5, it is not possible to step into Python code from C++ or break on Python code - only if debugging is done by attaching to the running process.
Any help gratefully received.
Related
On the second call of the following code, my app segfault, so I guess I am missing something :
Py_Initialize();
pName = PyString_FromString("comp_macbeth");
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if(pModule == NULL) {
PyErr_Print();
Py_Finalize();
return;
}
pFunc = PyObject_GetAttrString(pModule, "compute");
/* pFunc is a new reference */
if (!pFunc || !PyCallable_Check(pFunc) ) {
PyErr_Print();
Py_Finalize();
return;
}
Py_Finalize();
The comp_macbeth.py is importing numpy. If I remove the numpy import, everything is fine. Is it a numpy bug, or am I missing something about imports ?
From the Py_Finalize docs:
Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_Finalize() more than once.
Apparently Numpy is one of those. See also this message from Numpy-discussion.
Calling Py_Initialize() only once, and cleaning up at exit, is the way to go. (And it's should be faster, too!)
I have this in my module initialization part, but the URL does not exist anymore. In case it helps:
// http://numpy.scipy.org/numpydoc/numpy-13.html mentions this must be done in module init, otherwise we will crash
import_array();
This question already has answers here:
how can i include python.h in QMake
(1 answer)
Embedding python 3.4 into C++ Qt Application?
(4 answers)
Closed 2 years ago.
When I was trying to embed a Python script into my Qt C++ program, I run into multiple problems when trying to include Python.h.
The following features, I would like to provide:
Include python.h
Execute Python Strings
Execute Python Scripts
Execute Python Scripts with Arguments
It should also work when Python is not installed on the deployed machine
Therefore I searched around the Internet to try to find a solution. And found a lot of Questions and Blogs, but non have them covered all my Problems and it still took me multiple hours and a lot of frustration.
That's why I have to write down a StackOverflow entry with my full solution so it might help and might accelerate all your work :)
(This answer and all its code examples work also in a non-Qt environment. Only 2. and 4. are Qt specific)
Download and install Python https://www.python.org/downloads/release
Alter the .pro file of your project and add the following lines (edit for your correct python path):
INCLUDEPATH = "C:\Users\Public\AppData\Local\Programs\Python\Python39\include"
LIBS += -L"C:\Users\Public\AppData\Local\Programs\Python\Python39\libs" -l"python39"
Example main.cpp code:
#include <QCoreApplication>
#pragma push_macro("slots")
#undef slots
#include <Python.h>
#pragma pop_macro("slots")
/*!
* \brief runPy can execut a Python string
* \param string (Python code)
*/
static void runPy(const char* string){
Py_Initialize();
PyRun_SimpleString(string);
Py_Finalize();
}
/*!
* \brief runPyScript executs a Python script
* \param file (the path of the script)
*/
static void runPyScript(const char* file){
FILE* fp;
Py_Initialize();
fp = _Py_fopen(file, "r");
PyRun_SimpleFile(fp, file);
Py_Finalize();
}
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
runPy("from time import time,ctime\n"
"print('Today is', ctime(time()))\n");
//uncomment the following line to run a script
//runPyScript("test/decode.py");
return a.exec();
}
Whenever you #include <Python.h> use the following code instead. (The Slots from Python will otherwise conflict with the Qt Slots
#pragma push_macro("slots")
#undef slots
#include <Python.h>
#pragma pop_macro("slots")
After compiling, add the python3.dll, python39.dll, as well as the DLLs and Lib Python folders to your compilation folder. You can find them in the root directory of your Python installation. This will allow you to run the embedded c++ code even when python is not installed.
With these steps, I was able to get python running in Qt with the 64 bit MinGW and MSVC compiler. Only the MSVC in debug mode got still a problem.
FURTHER:
If you want to pass arguments to the python script, you need the following function (It can be easy copy-pasted into your code):
/*!
* \brief runPyScriptArgs executs a Python script and passes arguments
* \param file (the path of the script)
* \param argc amount of arguments
* \param argv array of arguments with size of argc
*/
static void runPyScriptArgs(const char* file, int argc, char *argv[]){
FILE* fp;
wchar_t** wargv = new wchar_t*[argc];
for(int i = 0; i < argc; i++)
{
wargv[i] = Py_DecodeLocale(argv[i], nullptr);
if(wargv[i] == nullptr)
{
return;
}
}
Py_SetProgramName(wargv[0]);
Py_Initialize();
PySys_SetArgv(argc, wargv);
fp = _Py_fopen(file, "r");
PyRun_SimpleFile(fp, file);
Py_Finalize();
for(int i = 0; i < argc; i++)
{
PyMem_RawFree(wargv[i]);
wargv[i] = nullptr;
}
delete[] wargv;
wargv = nullptr;
}
To use this function, call it like this (For example in your main):
int py_argc = 2;
char* py_argv[py_argc];
py_argv[0] = "Progamm";
py_argv[1] = "Hello";
runPyScriptArgs("test/test.py", py_argc, py_argv);
Together with the test.py script in the test folder:
import sys
if len(sys.argv) != 2:
sys.exit("Not enough args")
ca_one = str(sys.argv[0])
ca_two = str(sys.argv[1])
print ("My command line args are " + ca_one + " and " + ca_two)
you get the following output:
My command line args are Progamm and Hello
I want to call a Python function from C++ code in OMNeT++ simple module.
I debugged the code using gdb. It passes all lines fine, but at the end the
segmentation fault occurs after Py_Finalize();.
I found the following issue on GitHub that describes the same problem.
But it did not help me to resolve the problem.
double result=0;
// 1) Initialise python interpretator
if (!Py_IsInitialized()) {
Py_Initialize();
//Py_AtExit(Py_Finalize);
}
// 2) Initialise python thread mechanism
if (!PyEval_ThreadsInitialized()) {
PyEval_InitThreads();
assert(PyEval_ThreadsInitialized());
}
PyGILState_STATE s = PyGILState_Ensure();
PyRun_SimpleString("import sys; sys.path.append('/home/mypath/')");
PyObject *pName = PyUnicode_DecodeFSDefault((char*)"integrationTest");
PyObject* pModule = PyImport_Import(pName);
if (pModule != NULL)
{
PyObject* pFunction = PyObject_GetAttrString(pModule, (char*)"calculateExecutionTime");
/// changement will be held in this level Args and function result.
PyObject* pArgs = PyTuple_Pack(2,PyFloat_FromDouble(2.0),PyFloat_FromDouble(8.0));
PyObject* pResult = PyObject_CallObject(pFunction, pArgs);
result = (double)PyFloat_AsDouble(pResult);
///////
}
// Clean up
PyGILState_Release(s);
Py_DECREF(pName);
Py_DECREF(pModule);
Py_Finalize();
The problem occurs after the first initialization/uninitialization of python interpreter. What happens during the OmneT++ simulation is initializing/uninitializing/re-initializing/... the Python interpreter. However, Numpy doesn't support this;
So, I resolved this problem by initializing python interpreter just one time in the beginning of the simulation in initialize() method. Then, I called Py_Finalize(); in the destructor.
I'm trying to figure out why I can't simply get and set the python path through its C API. I am using Python3.6, on Ubuntu 17.10 with gcc version 7.2.0. Compiling with:
gcc pytest.c `python3-config --libs` `python3-config --includes`
#include <Python.h>
int main()
{
Py_Initialize(); // removes error if put after Py_SetPath
printf("setting path\n"); // prints
Py_SetPath(L"/usr/lib/python3.6"); // Error in `./a.out': free(): invalid size: 0x00007fd5a8365030 ***
printf("success\n"); // doesn't print
return 0;
}
Setting the path works fine, unless I also try to get the path prior to doing so. If I get the path at all, even just to print without modifying the returned value or anything, I get a "double free or corruption" error.
Very confused. Am I doing something wrong or is this a bug? Anyone know a workaround if so?
Edit: Also errors after calling Py_Initialize();. Updated code. Now errors even if I don't call Py_GetPath() first.
From alk it seems related to this bug: https://bugs.python.org/issue31532
Here is the workaround I am using. Since you can't call Py_GetPath() before Py_Initialize(), and also seemingly you can't call Py_SetPath() after Py_Initialize(), you can add to or get the path like this after calling Py_Initialize():
#include <Python.h>
int main()
{
Py_Initialize();
// get handle to python sys.path object
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
// make a list of paths to add to sys.path
PyObject *newPaths = PyUnicode_Split(PyUnicode_FromWideChar(L"a:b:c", -1), PyUnicode_FromWideChar(L":", 1), -1);
// iterate through list and add all paths
for(int i=0; i<PyList_Size(newPaths); i++) {
PyList_Append(path, PyList_GetItem(newPaths, i));
}
// print out sys.path after appends
PyObject *newlist = PyUnicode_Join(PyUnicode_FromWideChar(L":", -1), path);
printf("newlist = %ls\n", PyUnicode_AsWideCharString(newlist, NULL));
return 0;
}
[the below answer refers to this version of the question.]
From the docs:
void Py_Initialize()
Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the exception of Py_SetProgramName(), Py_SetPythonHome() and Py_SetPath().
But the code you show does call Py_GetPath() before it calls Py_Initialize();, which it per the above paragraph implicitly should not.
How to compile python 3.4.3 script with the module tkinter and ttk to an self-executable exe (standalone)? (py2exe, pyinstaller, freeze doesn't work.) any suggestions? Thank You
What I do is
download Portable Python
create an file in an other language that can be compiled to exe
make that executable call portable Python with my Python file.
Structure:
application_folder # the folder where everything is in
+--my_python_folder # the folder where your python files are in
| +--my_program.py # the python file that you want to start
+--Portable Python 3 # the Python version that you use
+--program.exe # the compiled program
The C++ source code:
// based on https://msdn.microsoft.com/en-us/library/ms682425%28VS.85%29.aspx
#include <windows.h>
#include <stdio.h>
#include <tchar.h>
int _tmain( int argc, TCHAR *argv[] )
{
STARTUPINFO si;
PROCESS_INFORMATION pi;
// choose between pythonw.exe and python.exe
TCHAR command[] = "\"Portable Python 3\\App\\pythonw.exe\" \"my_program.py\"";
// the directory where you start the Python program in
TCHAR directory[] = "my_python_folder";
ZeroMemory( &si, sizeof(si) );
si.cb = sizeof(si);
ZeroMemory( &pi, sizeof(pi) );
// Start the child process.
if( !CreateProcess( NULL, // No module name (use command line)
command, // Command line
NULL, // Process handle not inheritable
NULL, // Thread handle not inheritable
FALSE, // Set handle inheritance to FALSE
0, // No creation flags
NULL, // Use parent's environment block
directory, // Use parent's starting directory
&si, // Pointer to STARTUPINFO structure
&pi ) // Pointer to PROCESS_INFORMATION structure
)
{
printf( "CreateProcess failed (%d).\n", GetLastError() );
return 1;
}
/*
// Wait until child process exits.
WaitForSingleObject( pi.hProcess, INFINITE );
// Close process and thread handles.
CloseHandle( pi.hProcess );
CloseHandle( pi.hThread );
*/
return 0;
}
You can compile the file with devc++.
Evaluation
Pros:
An other way to do it.
Cons:
Needs whole Portable Python.
No command line arguments are passed.
You can do it with a .bat file and it works, too.
The current working directory is different than the caller's one.