I need to read local variables from Python in C/C++. When I try to PyEval_GetLocals, I get a NULL. This happens although Python is initialized. The following is a minimal example.
#include <iostream>
#include <Python.h>
Py_Initialize();
PyRun_SimpleString("a=5");
PyObject *locals = PyEval_GetLocals();
std::cout<<locals<<std::endl; //prints NULL (prints 0)
Py_Finalize();
In the manual, it says that it returns NULL if no frame is running, but... there's a frame running!
What am I doing wrong?
I'm running this in Debian Jessie.
Turns out the right way to access variables in the scope is:
Py_Initialize();
PyObject *main = PyImport_AddModule("__main__");
PyObject *globals = PyModule_GetDict(main);
PyObject *a = PyDict_GetItemString(globals, "a");
std::cout<<globals<<std::endl; //Not NULL
Py_Finalize();
Related
Background
I have some functions which are written in C++, which require high real-time performance. I want to quickly export these functions as dynamic link library to be exposed to Python so that I could do some high level programming.
In these functions, in order to simply usage, I use PyList_New in <Python.h> to collect some intermedia data. But I met some errors.
Code Example
I found the core problem is that I CAN'T event export a python object. After compiling the source to dll and use ctypes to load it, result shows
OSError: exception: access violation reading 0x0000000000000008
C++ code:
#include <Python.h>
#ifdef _MSC_VER
#define DLL_EXPORT __declspec( dllexport )
#else
#define DLL_EXPORT
#endif
#ifdef __cplusplus
extern "C"{
#endif
DLL_EXPORT PyObject *test3() {
PyObject* ptr = PyList_New(10);
return ptr;
}
#ifdef __cplusplus
}
#endif
Python test code:
if __name__ == "__main__":
import ctypes
lib = ctypes.cdll.LoadLibrary(LIB_DLL)
test3 = lib.test3
test3.argtypes = None
test3.restype = ctypes.py_object
print(test3())
Environment Config
Clion with Microsoft Visual Studio 2019 Community, and the arch is amd64.
I know that, the right way is to use the recommanded method to wrap C++ source using Python/C Api to a module, but it seems that I have to code a lot. Anyone can help?
ctypes is normally for calling "regular" C functions, not Python C API functions, but it can be done. You must use PyDLL to load a function that uses Python, as it won't release the GIL (global intepreter lock) required to be held when using Python functions. Your code as shown is invalid, however, because it doesn't populate the list it creates (using OP code as test.c):
>>> from ctypes import *
>>> lib = PyDLL('./test')
>>> lib.test3.restype=py_object
>>> lib.test3()
[<NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>, <NULL>]
Instead, write a C or C++ function normally:
test.cpp
#ifdef _MSC_VER
#define DLL_EXPORT __declspec( dllexport )
#else
#define DLL_EXPORT
#endif
#ifdef __cplusplus
extern "C"{
#endif
DLL_EXPORT int* create(int n) {
auto p = new int[n];
for(int i = 0; i < n; ++i)
p[i] = i;
return p;
}
DLL_EXPORT void destroy(int* p) {
delete [] p;
}
#ifdef __cplusplus
}
#endif
test.py
from ctypes import *
lib = CDLL('./test')
lib.create.argtypes = c_int,
lib.create.restype = POINTER(c_int)
lib.destroy.argtypes = POINTER(c_int),
lib.destroy.restype = None
p = lib.create(5)
print(p) # pointer to int
print(p[:5]) # convert to list...pointer doesn't have length so slice.
lib.destroy(p) # free memory
Output:
<__main__.LP_c_long object at 0x000001E094CD9DC0>
[0, 1, 2, 3, 4]
I solved it by myself. Just change to Release and all the problems are solved.
I want to call a Python function from C++ code in OMNeT++ simple module.
I debugged the code using gdb. It passes all lines fine, but at the end the
segmentation fault occurs after Py_Finalize();.
I found the following issue on GitHub that describes the same problem.
But it did not help me to resolve the problem.
double result=0;
// 1) Initialise python interpretator
if (!Py_IsInitialized()) {
Py_Initialize();
//Py_AtExit(Py_Finalize);
}
// 2) Initialise python thread mechanism
if (!PyEval_ThreadsInitialized()) {
PyEval_InitThreads();
assert(PyEval_ThreadsInitialized());
}
PyGILState_STATE s = PyGILState_Ensure();
PyRun_SimpleString("import sys; sys.path.append('/home/mypath/')");
PyObject *pName = PyUnicode_DecodeFSDefault((char*)"integrationTest");
PyObject* pModule = PyImport_Import(pName);
if (pModule != NULL)
{
PyObject* pFunction = PyObject_GetAttrString(pModule, (char*)"calculateExecutionTime");
/// changement will be held in this level Args and function result.
PyObject* pArgs = PyTuple_Pack(2,PyFloat_FromDouble(2.0),PyFloat_FromDouble(8.0));
PyObject* pResult = PyObject_CallObject(pFunction, pArgs);
result = (double)PyFloat_AsDouble(pResult);
///////
}
// Clean up
PyGILState_Release(s);
Py_DECREF(pName);
Py_DECREF(pModule);
Py_Finalize();
The problem occurs after the first initialization/uninitialization of python interpreter. What happens during the OmneT++ simulation is initializing/uninitializing/re-initializing/... the Python interpreter. However, Numpy doesn't support this;
So, I resolved this problem by initializing python interpreter just one time in the beginning of the simulation in initialize() method. Then, I called Py_Finalize(); in the destructor.
I have a C++ code which execute python script with boost_python package. Everything is fine, as longa as I extract int, string, or other not-array variables from python. However I have to extract a numpy::ndarray and convert it to cpp vector. I tried as follow:
main.cpp
#include <iostream>
#include <boost/python.hpp>
#include <boost/python/numpy.hpp>
using namespace boost::python;
int main()
double t_end=7
try
{
Py_Initialize();
object module = import("__main__");
object name_space = module.attr("__dict__");
exec_file("MyModule.py", name_space, name_space);
object MyFunc = name_space["MyFunc"];
object result = MyFunc(t_end);
auto result_array = extract<numpy::ndarray>(result);
const numpy::ndarray& ret = result_array();
int input_size = ret.shape(0);
double* input_ptr = reinterpret_cast<double*>(ret.get_data());
std::vector<double> v(input_size);
for (int i = 0; i < input_size; ++i)
v[i] = *(input_ptr + i);
}
catch (error_already_set)
{
PyErr_Print();
}
Py_Finalize();
And example py script:
MyModule.py
import numpy as np
def MyFunc(t_end):
result = np.array([2,3,1,t_end])
return results
However it ends with error:
read access violation BOOST_NUMPY_ARRAY_API was nullptr
I also was trying to declare numpy::ndarray directly like numpy::ndarray result_array = extract<numpy::ndarray>(result); But the error is exactly the same. I've checked if my ndarray is not empty by printing it directly from python, and it is not. At the python step all seems to be correct. So what is causing the violation and how to fix it?
That error occurs since you're using the numpy module without first initializing it.
Notice the beginning of the official tutorial:
Initialise the Python runtime, and the numpy module. Failure to call these results in segmentation errors:
namespace np = boost::python::numpy;
int main(int argc, char **argv)
{
Py_Initialize();
np::initialize();
Your code is lacking the call to np::initialize();.
I'm trying to figure out why I can't simply get and set the python path through its C API. I am using Python3.6, on Ubuntu 17.10 with gcc version 7.2.0. Compiling with:
gcc pytest.c `python3-config --libs` `python3-config --includes`
#include <Python.h>
int main()
{
Py_Initialize(); // removes error if put after Py_SetPath
printf("setting path\n"); // prints
Py_SetPath(L"/usr/lib/python3.6"); // Error in `./a.out': free(): invalid size: 0x00007fd5a8365030 ***
printf("success\n"); // doesn't print
return 0;
}
Setting the path works fine, unless I also try to get the path prior to doing so. If I get the path at all, even just to print without modifying the returned value or anything, I get a "double free or corruption" error.
Very confused. Am I doing something wrong or is this a bug? Anyone know a workaround if so?
Edit: Also errors after calling Py_Initialize();. Updated code. Now errors even if I don't call Py_GetPath() first.
From alk it seems related to this bug: https://bugs.python.org/issue31532
Here is the workaround I am using. Since you can't call Py_GetPath() before Py_Initialize(), and also seemingly you can't call Py_SetPath() after Py_Initialize(), you can add to or get the path like this after calling Py_Initialize():
#include <Python.h>
int main()
{
Py_Initialize();
// get handle to python sys.path object
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
// make a list of paths to add to sys.path
PyObject *newPaths = PyUnicode_Split(PyUnicode_FromWideChar(L"a:b:c", -1), PyUnicode_FromWideChar(L":", 1), -1);
// iterate through list and add all paths
for(int i=0; i<PyList_Size(newPaths); i++) {
PyList_Append(path, PyList_GetItem(newPaths, i));
}
// print out sys.path after appends
PyObject *newlist = PyUnicode_Join(PyUnicode_FromWideChar(L":", -1), path);
printf("newlist = %ls\n", PyUnicode_AsWideCharString(newlist, NULL));
return 0;
}
[the below answer refers to this version of the question.]
From the docs:
void Py_Initialize()
Initialize the Python interpreter. In an application embedding Python, this should be called before using any other Python/C API functions; with the exception of Py_SetProgramName(), Py_SetPythonHome() and Py_SetPath().
But the code you show does call Py_GetPath() before it calls Py_Initialize();, which it per the above paragraph implicitly should not.
I've been trying to write a DLL in C.
Install hook sets up the KeyboardProc. Calling the InstallHook() and UninstallHook() functions from Python always returns a 0, which I guess is because my callback function KeyboardProc isn't working.
The following is my C code for the DLL:
#include "stdafx.h"
#include <windows.h>
#include <string.h>
#include <ctype.h>
#include <stdio.h>
#include "ourdll.h"
//#pragma comment(linker, "/SECTION:.SHARED,RWS")
//#pragma data_seg(".SHARED")
HHOOK hKeyboardHook = 0;
int keypresses = 0;
HMODULE hInstance = 0;
//#pragma data_seg()
BOOL WINAPI DllMain (HANDLE hModule, DWORD dwFunction, LPVOID lpNot)
{
hInstance = hModule; //Edit
return TRUE;
}
LRESULT CALLBACK KeyboardProc(int hookCode, WPARAM vKeyCode, LPARAM flags)
{
if(hookCode < 0)
{
return CallNextHookEx(hKeyboardHook, hookCode, vKeyCode, flags);
}
keypresses++;;
return CallNextHookEx(hKeyboardHook, hookCode, vKeyCode, flags);
}
__declspec(dllexport) void InstallHook(void)
{
hKeyboardHook = SetWindowsHookEx(WH_KEYBOARD, KeyboardProc, hInstance, 0);
}
__declspec(dllexport) int UninstallHook(void)
{
UnhookWindowsHookEx(hKeyboardHook);
hKeyboardHook = 0;
return keypresses;
}
The Python code to use this is as follows:
>>> from ctypes import *
>>> dll = CDLL('C:\...\OurDLL.dll')
>>> dll.InstallHook()
[Type something at this point]
>>> result = dll.UninstallHook()
>>> result
0
EDIT: I should probably mention that I've also tried out a LowLevelKeyboardHook. I understand that the LowLevel hook is global and will catch all keystrokes, but that just caused my dll.InstallHook() Python code to freeze for a second or two before returning zero.
I am no expert in C, so any help would be greatly appreciated. Thanks.
hKeyboardHook = SetWindowsHookEx(WH_KEYBOARD, KeyboardProc, NULL, 0);
SetWindowsHookEx requires a hModule - save the hModule from DllMain and pass it here. (You can pass NULL only if the thread id is your own thread.)
One exception to this is for the _LL hook types; these don't need a hmodule param since these hook don't inject into the target process - that's why your code using KEYBOARD_LL is 'succeeding'.
As for why it might be blocking when you use KEYBOARD_LL - docs for LowLevelKeyboardHookProc mention that the thread that installs the hook (ie. calls SetWindowsHookEx) must have a message loop, which you might not have in your python code.
Debugging tips: it looks like SetWindowsHookEx should be returning NULL (with GetLastError() returning a suitable error code); while developing code, using some combination of assert/printf/OutputDebugString as appropriate to check these return values is a good way to ensure that your assumptions are correct and give you some clues as to where things are going wrong.
BTW, one other thing to watch for with KEYBOARD vs KEYBOARD_LL: the KEYBOARD hook gets loaded into the target process - but only if it's the same bitness - so a 32-bit hook only sees keys pressed by other 32-bit processes. OTOH, KEYBOARD_LL is called back in your own process, so you get to see all keys - and also don't need to deal with the shared segment (though as far as I know it's also less efficient as a KEYBOARD hook).