I have a C code which has embedded python in it using "Python.h" It works fine without any errors - But it doesn't completely do what I want it to.
What it does : After the C code starts running, it ignores all changes which I make to the python file until I restart the C code.
What I want : While the C code is running, if i make changes to the python file, it should start running the new code.
I tried using the function PyImport_ReloadModule in everytime before calling the function, but it does not work. Am I doing something wrong ?
My current code :
#include "Strategy.h"
#undef _DEBUG /* Link with python24.lib and not python24_d.lib */
#include <Python.h>
#include <stdlib.h>
#include <iostream>
using namespace std;
PyObject *pName, *pModule, *pDict, *pFunc;
PyObject *pArgs, *pValue;
void import_py() {
pName = PyString_FromString("main");
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule == NULL) {
cout << "ERR : Unable to load main.py\n";
return;
} else {
cout << "OK : Loaded main.py\n";
}
if ( PyObject_HasAttrString(pModule, "main") ) {
cout << "OK : main.py has function main\n";
} else {
cout << "ERR : main.py has no function main\n";
return;
}
pFunc = PyObject_GetAttrString(pModule, "main");
if ( pFunc == NULL ) {
cout << "OK : main.py's function main gave NULL when trying to take it\n";
return;
}
}
void remove_py() {
Py_XDECREF(pArgs);
Py_XDECREF(pModule);
Py_XDECREF(pFunc);
Py_Finalize();
}
void Construct() {
Py_Initialize();
import_py();
}
void Destruct() {
if ( pModule || pFunc ) {
remove_py();
}
}
void Loop () {
if ( ! ( pModule && pFunc ) ) {
cout << "Looped. But python values are null\n";
return;
}
cout << "Loop : ";
pArgs = PyTuple_New(2); // Create a tuple to send to python - sends 1,2
PyTuple_SetItem(pArgs, 0, PyInt_FromLong(1));
PyTuple_SetItem(pArgs, 1, PyInt_FromLong(2));
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
double t = 0; // Get the 2 return values
t = PyFloat_AsDouble(PyTuple_GetItem(pValue, 0));
cout << t << ", ";
t = PyFloat_AsDouble(PyTuple_GetItem(pValue, 1));
cout << t;
cout << "\n";
}
void main() {
Construct();
while(1) { // Using an infinite loop for now - to test
pModule = PyImport_ReloadModule(pModule);
Loop();
}
Destruct();
}
I found the issue.
Even after getting the new module with pModule = PyImport_ReloadModule(pModule) the p
variable pFunc doesn't get automatically updated. So, the variable pFunc is still referencing the old module !
hence, every variable needs to be got again. Like so :
void main() {
Construct();
while(1) { // Using an infinite loop for now - to test
pModule = PyImport_ReloadModule(pModule);
pFunc = PyObject_GetAttrString(pModule, "main");
Loop();
}
Destruct();
}
One thing I am not sure about here is whether DECREF should be made to the pFunc which is referencing the old pModule.
Related
I'm embedding Python 3.8.2 in C++ code (using Visual Studio 2019). Python has pandasn installed (through pip).
I manage to import pandas from a C++ program, however, when I try to import it a second time, it crashs.
#include <Python.h>
#include <iostream>
int main( int argc, char* argv[] )
{
{
Py_SetPythonHome( L"C:\\Python38" );
// Initialize the Python Interpreter
Py_Initialize();
std::cout << "Importing pandas..." << std::endl;
if ( PyRun_SimpleString( "import pandas" ) == 0 )
std::cout << "SUCCESS" << std::endl;
else
std::cout << "FAIL" << std::endl;
Py_Finalize();
}
{
Py_SetPythonHome( L"C:\\Python38" );
// Initialize the Python Interpreter
Py_Initialize();
std::cout << "Importing pandas..." << std::endl;
if ( PyRun_SimpleString( "import pandas" ) == 0 )
std::cout << "SUCCESS" << std::endl;
else
std::cout << "FAIL" << std::endl;
Py_Finalize();
}
return 0;
}
This crashs with an exception:
_multiarray_umath.cp38-win_amd64.pyd!00007ffbd5b8ca69() Inconnu
_multiarray_umath.cp38-win_amd64.pyd!00007ffbd5b8ffd6() Inconnu
_multiarray_umath.cp38-win_amd64.pyd!00007ffbd5b9d34d() Inconnu
python38.dll!00007ffbd22f6131() Inconnu
python38.dll!00007ffbd22f6092() Inconnu
Output is:
Importing pandas...
SUCCESS
Importing pandas...
Traceback (most recent call last):
File "<string>", line 1, in <module>
Is there any init/uninit step I missed that could make this fail while it shaould work?
Note that I cannot Debug as pandas cannot be loaded in Debug build.
Upon request from OP, I made a small demo for how we wrap user Python scripts in our application to prevent that global variables of user scripts become unintended persistent:
#include <iostream>
#include <string>
#include <Python.h>
const char *PyScript = R"(try:
print(a)
except:
print("a not (yet) defined")
a = 123
print(a)
)";
std::string wrapPyScript(const char* script)
{
std::string source = std::string("def __mainPythonFunction():\n") + script;
{ const char *const indent = "\n ";
for (size_t i = source.size(); i--;) {
size_t n = 1;
switch (source[i]) {
case '\n': if (i && source[i - 1] == '\r') n = 2, --i;
case '\r': source.replace(i, n, indent); break;
}
}
}
source += "\n"
"pass\n"
"\n"
"try:\n"
" __mainPythonFunction()\n"
"except:\n"
" rf.form.appContext.notifyAbort()\n"
" raise\n";
return source;
}
#define DEBUG(...) std::cout << #__VA_ARGS__ << ";\n"; __VA_ARGS__
int main()
{
DEBUG(Py_Initialize());
std::cout << "\nWithout wrapper:\n\n";
DEBUG(for (int i = 0; i < 2; ++i) {
DEBUG(PyRun_SimpleString(PyScript));
});
std::cout << "\nWith wrapper:\n\n";
DEBUG(for (int i = 0; i < 2; ++i) {
DEBUG(PyRun_SimpleString(wrapPyScript(PyScript).c_str()));
});
std::cout << '\n';
DEBUG(Py_Finalize());
}
Output:
Py_Initialize();
Without wrapper:
for (int i = 0; i < 2; ++i) { DEBUG(PyRun_SimpleString(PyScript)); };
PyRun_SimpleString(PyScript);
a not (yet) defined
123
PyRun_SimpleString(PyScript);
123
123
With wrapper:
for (int i = 0; i < 2; ++i) { DEBUG(PyRun_SimpleString(wrapPyScript(PyScript).c_str())); };
PyRun_SimpleString(wrapPyScript(PyScript).c_str());
a not (yet) defined
123
PyRun_SimpleString(wrapPyScript(PyScript).c_str());
a not (yet) defined
123
Py_Finalize();
However, I'm not quite sure whether this is enough to fix OPs issue with the imported Pandas library.
In our application (where we used the above trick), we import selected libraries once after the Py_Initialize().
(I remember roughly that this was our last desperate resort to fix similar issues like OP observed.)
I'm trying to embed simple python instructions in my c++ program.
I'm unable to extract c++ types from the python object types...
Would appreciate any help!
Sample Program:
#include <iostream>
#include <Python.h>
using namespace std;
int main()
{
Py_Initialize();
auto pModule = PyImport_ImportModule("math");
auto pFunc = PyObject_GetAttrString(pModule, "sin");
auto pIn = Py_BuildValue("(f)", 2.);
auto pRes = PyObject_CallObject(pFunc, pIn);
auto cRes = ???;
cout << cRes << endl;
Py_Finalize();
}
The program should simply print the result for sin(2).
You'll want to know what type(s) to expect from the function call, including errors... If the function raised an exception, the PyObject_CallObject should return NULL, so check for that first:
if (!pRes) {
PyErr_Print();
// don't do anything else with pRes
}
Otherwise, you can check for and interpret each type you might expect from the Python function call:
if (pRes == Py_None) {
cout << "result is None" << endl;
} else if (PyFloat_Check(pRes)) {
auto cRes = PyFloat_AsDouble(pRes);
cout << cRes << endl;
} else if (<other checks>) {
// Handle other types
} else {
cout << "Unexpected return type" << endl;
}
In the case of your math.sin() call, you can probably safely assume either an exception or a PyFloat return.
i am writing a Test to keep Data in a C++ map. I am using the Python C-Api.
Adding int values and getting the length of the map is no problem.
But when i get a value from the map and want to return it to Python than the Interpreter crashes.
This is my Code:
//some includes
int VerboseLog = 99;
typedef map<const char*, int> SessionKeeper;
static SessionKeeper openSessions;
static PyObject* getSession(PyObject* self, PyObject* args) {
cout << "get" << endl;
const char *key;
if (!PyArg_ParseTuple(args, "z*", &key)) {
PyErr_SetString(PyExc_Exception, "UUID konnte nicht ausgelesen werden");
PyErr_PrintEx(1);
return NULL;
}
int r = openSessions.at(key);
return Py_BuildValue("i", r);
}
static PyObject *addSession(PyObject* self, PyObject* args) {
cout << "Hello" << endl;
const char *key;
int toAppend;
if (!PyArg_ParseTuple(args, "si", &key, &toAppend)) {
PyErr_SetString(PyExc_Exception, "Ein Parameter konnte nicht ausgelesen werden");
PyErr_PrintEx(1);
return NULL;
}
openSessions[key] = toAppend;
cout << openSessions.size() << endl;
cout << openSessions[key] << endl;
Py_RETURN_NONE;
}
static PyObject* length(PyObject *self){
cout << "length: ";
int i = openSessions.size();
return Py_BuildValue("i",i);
}
static PyMethodDef SessionKeeper_methods[] = {
/*Note the third entry (METH_VARARGS). This is a flag telling the interpreter the calling convention
to be used for the C function. It should normally always be METH_VARARGS or METH_VARARGS | METH_KEYWORDS;
a value of 0 means that an obsolete variant of PyArg_ParseTuple() is used.*/
{ "length", (PyCFunction)length, METH_VARARGS, "return length of a Session" },
{ "addSession", (PyCFunction)addSession, METH_VARARGS, "add a Session." },
{ "getSession", (PyCFunction)getSession, METH_VARARGS, "get a Session" },
{ NULL }
};
static PyModuleDef sessionKeeperMod = {
PyModuleDef_HEAD_INIT,
u8"SessionKeeper",
NULL,
-1,
SessionKeeper_methods
};
static PyModuleDef CpluplusModules_ModDef = {
PyModuleDef_HEAD_INIT,
u8"CPlusPlusMouldes",
u8"Enthält alle Erweietrungen",
-1,
NULL, NULL, NULL, NULL, NULL
};
PyMODINIT_FUNC PyInit_CPlusPlusModules(void) {
PyObject* m, *sessionMod;
if (PyType_Ready(&TAModulType) < 0)
return NULL;
if (PyType_Ready(&DatabaseReader_Type) < 0)
return NULL;
if (!(sessionMod = PyModule_Create(&sessionKeeperMod))) {
cout << "fail" << endl;
return NULL;
}
m = PyModule_Create(&CpluplusModules_ModDef);
if (m == NULL)
return NULL;
Py_INCREF(&TAModulType);
Py_INCREF(&DatabaseReader_Type);
PyModule_AddObject(m, "TAModul", (PyObject *)&TAModulType);
PyModule_AddObject(m, "DBReader", (PyObject *)&DatabaseReader_Type);
PyModule_AddObject(m, "SessionKeeper", sessionMod);
return m;
}
The other Modules (DBReader and TAModule) workes fine. The goal will be to safe PythonObjects (containing DbReader Objects and TAModul Objects) in the map.
But bacck to my question why do the getSession crashes the Interpreter on return? And why do the length function works fine.
map<const char*,...> will match based on the address of the string rather than the contents. Therefore it's pretty likely that at will fail and throw an C++ out_of_range exception.
You should start by wrapping the line
int r = openSessions.at(key);
with a try {} catch block, to stop the out_of_range exception propagating through the Python interpreter (which isn't C++ so can't handle it).
You should then change the map to match the key on the string contents. The easiest way would be to use map<std::string,int>.
I'm back to programming in a project and i'm getting no return from the python script for some long hours now.
Funny thing is, a couple of months ago i managed to get this working, now i don't know what's wrong
Where's the C++ code:
int CPythonPlugIn::py_embed(int argc, char *argv[]){
ofstream textfile3;
textfile3.open("FP_python_embed.txt");
PyObject *pName, *pModule, *pDict, *pFunc, *pArgs, *pValue;
if(argc<3){
printf("Usage: exe_name python_source function_name\n");
return 1;
}
//To inform the interpreter about paths to Python run-time libraries
Py_SetProgramName(argv[0]);
// Initialize the Python Interpreter
Py_Initialize();
if( !Py_IsInitialized() ){
cout<<"Can't initialize"<<endl;
return -1;
}
// Build the name object
pName = PyString_FromString(argv[1]);
if( !pName ){
cout<<"Can't build the object "<<endl;
return -1;
}
// Load the module object
pModule = PyImport_Import(pName);
if( !pModule ){
cout<<"Can't import the module "<<endl;
return -1;
}
// pDict is a borrowed reference
pDict = PyModule_GetDict(pModule);
if( !pDict ){
cout<<"Can't get the dict"<<endl;
return -1;
}
// pFunc is also a borrowed reference
pFunc = PyDict_GetItemString(pDict, argv[2]);
if( !pFunc || !PyCallable_Check(pFunc) ){
cout<<"can't get the function"<<endl;
return -1;
}
if (PyCallable_Check(pFunc))
{
// Prepare the argument list for the call
if( argc > 3 )
{
pArgs = PyTuple_New(argc - 3);
for (int i = 0; i < argc - 3; i++)
{
pValue = PyInt_FromLong(atoi(argv[i + 3]));
if (!pValue)
{
PyErr_Print();
return 1;
}
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
textfile3<<PyInt_AsLong(pValue)<<endl<<" worked1";
if (pArgs != NULL)
{
Py_DECREF(pArgs);
}
}
else
{
pValue = PyObject_CallObject(pFunc, NULL);
}
if (pValue != NULL)
{
printf("Return of call : %d\n", PyInt_AsLong(pValue));
textfile3<<PyInt_AsLong(pValue)<<endl<<" worked2";
Py_DECREF(pValue);
}
else
{
PyErr_Print();
}
textfile3.close();
}
// Clean up
Py_DECREF(pModule);
Py_DECREF(pName);
// Finish the Python Interpreter
Py_Finalize();
return 0;
};
and this is how i call this function:
char *arg[4]={"PythonPlugIn2","bridge","test_callsign","MAH545"};
py_embed(4,arg);
the simple python script:
def test_callsign(b):
fp_txt=open('allomate.txt','w+')
fp_txt.write('WHAT')
fp_txt.write(b)
if b=='MAH545':
fp_txt.write('MAHHH')
fp_txt.close()
return 1
elif b=='MAH544':
fp_txt.close()
return 2
elif b=='AFR545':
fp_txt.close()
return 3
else:
fp_txt.write('MAHHH22')
print 'No such airplane'
fp_txt.close()
return 10
The allomate.txt is created and it has only "WHAT" written on it. The FP_python_embed.txt also is created and has "-1, worked1" so the problem needs to be on pValue which is giving NULL for some reason
Thank you in advance for the help
I finally found the solution. I was parsing the pValue not as a string but as an int.
So, where i had:
pValue = PyInt_FromLong(atoi(argv[i + 3]));
it really should be:
pValue = PyString_FromString(argv[i+3]);
I have a solution in VisualStudio that contains two projects. The first project is a C++ console application with Pure Embedding as follows:
#include <Python.h>
int main(int argc, char *argv[]){
PyObject *pName, *pModule, *pDict, *pFunc;
PyObject *pArgs, *pValue;
int i;
if (argc < 3) {
fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
return 1;
}
Py_Initialize();
pName = PyUnicode_FromString(argv[1]);
/* Error checking of pName left out */
pModule = PyImport_Import(pName);
Py_DECREF(pName);
if (pModule != NULL) {
pFunc = PyObject_GetAttrString(pModule, argv[2]);
/* pFunc is a new reference */
if (pFunc && PyCallable_Check(pFunc)) {
pArgs = PyTuple_New(argc - 3);
for (i = 0; i < argc - 3; ++i) {
pValue = PyLong_FromLong(atoi(argv[i + 3]));
if (!pValue) {
Py_DECREF(pArgs);
Py_DECREF(pModule);
fprintf(stderr, "Cannot convert argument\n");
return 1;
}
/* pValue reference stolen here: */
PyTuple_SetItem(pArgs, i, pValue);
}
pValue = PyObject_CallObject(pFunc, pArgs);
Py_DECREF(pArgs);
if (pValue != NULL) {
printf("Result of call: %ld\n", PyLong_AsLong(pValue));
Py_DECREF(pValue);
}
else {
Py_DECREF(pFunc);
Py_DECREF(pModule);
PyErr_Print();
fprintf(stderr,"Call failed\n");
return 1;
}
}
else {
if (PyErr_Occurred())
PyErr_Print();
fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
}
Py_XDECREF(pFunc);
Py_DECREF(pModule);
}
else {
PyErr_Print();
fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
return 1;
}
Py_Finalize();
return 0;}
When running a python file any works perfectly.
The second project contains some C++ classes compiled with SWIG. This project works perfectly using the Python script as below.
import example
x = 42
y = 105
example.gcd = g (x, y)
My problem is when the console application needs to run a python script that imports classes from the SWIG. If I use the python lib python32.lib works commenting on some parts of the code as "Py_DECREF (pArgs)" but I can only run once the python script, the second attempt, an error occurs the read access of memory. If I use python32_d.lib the construction of the modules do not work if the file import python SWIG:
pName = PyUnicode_FromString(argv[1]);
pModule = PyImport_Import(pName);
NOTE: I need to compile in DEBUG mode and not in RELEASE mode.