Is it possible to pass a python dict into a function expecting a nlohmann::json (nlohmann/json) object via cppyy? This question has to have come up by now, but I wasn't able to find anything on it.
Minimal example to reproduce (without regard to performance/safety, pls forgive):
test-json.h
#include <iostream>
#include <nlohmann/json.hpp>
using nlohmann::json;
void print_name_and_age(json j) {
std::cout << j["name"] << "\n"
<< j["age"] << "\n";
}
test-cppyy.py
import cppyy
cppyy.include('test-json.h')
from cppyy.gbl import print_name_and_age
some_dict = {
"name": "alfred",
"age": 25
}
print_name_and_age(some_dict)
runs into
print_name_and_age(some_dict)
NotImplementedError: void ::print_name_and_age(nlohmann::json j) =>
NotImplementedError: could not convert argument 1 (this method cannot (yet) be called)
I would like to be able to pass a python dict into the C++ function, and receive it as a nlohmann::json object. I presume I would need to write some custom converter for this?
Design requirement/background (optional)
I have a reinforcement learning environment class (written in C++) that needs to accept some configuration to initialize it (in its constructor). Everything's all fine passing a nlohmann::json object into the constructor while in the C++ domain, but I have a Python wrapper around the class too, written with cppyy that provides similar functionality to the C++ interface.
Uptill now, because of the aforementioned issue, I've been forced to receive a const std::map<std::string, float>& in the constructor instead of a nlohmann::json, which is what a python dict containing only str -> float mappings easily gets converted to by cppyy. But this obviously limits my input json files to only contain floats as values (my usecase requires having strings as keys but strings, ints, floats and bools as values in the JSON file). I can ofcourse write some pre-processing code to encode my heterogenous python dict into a homogenous str->float mapping on the python front (and do the same for C++) but I'd like a cleaner solution, if possible.
Could anyone please help me achieve passing a python dict into the cppyy-imported C++ function and have it converted into a nlohmann::json object in the C++ function? If this requires forking cppyy to add extra converter code/too much trouble I presume I would need to use a std::map<std::string, std::any / variant> alternative? I haven't worked alot with std::any/variant, would like to ask if this would even be possible - python dict to map<string, any> - if this is the best alternative to a custom converter - in terms of performance / clean elegant code.
Environment:
Python 3.9.13
C++17 ( I believe cppyy-2.4.0 doesn't support C++20 yet, I don't have any constraint on the C++ standard)
cppyy==2.3.1
cppyy-backend==1.14.8
cppyy-cling==6.25.3
cppyythonizations==1.2.1
MacOS Monterey, Apple M1
This has been answered at GitHub: Conversion from python dict into nlohmann::json
Related
Please read this post before answering: Pass a closure from Cython to C++
In the accepted answers, it is neatly shown how a python function is converted into a std::function using Boost Python.
Following this example I'm able to wrap functions taking an std::function as an argument and call them using a python function as input. However, this only works when the std::function parameters are primitives like int, double, string etc.
Any guidance on how to make this work for custom types as well will be highly appreciated.
This won't be a complete answer - it assumes you can fill in the gaps from my previous answer that the question was based on. Unfortunately it is a little bit more complicated than that case.
Just to define the problem - assume you have a parameter of a custom C++ class, like:
class cpp_class {
// some non-trivial contents
};
and thus your C++ interface looks like this:
void call_some_std_func(std::function<void(cpp_class&)> callback) {
callback(5,std::string("hello"));
}
The first thing to do is to write a Cython wrapper for your C++ class (in principle you could make a Boost Python wrapper instead). Here you need to make a choice of about "ownership" of the C++ object. The first choice is to make a copy:
cdef extern from "cpp_file.hpp":
cppclass cpp_class:
pass # details
cdef class CyWrapper:
cdef cpp_class* ptr
def __dealloc__(self):
del self.ptr
# other details following standard wrapper pattern
cdef public make_CyWrapper(cpp_class& x):
obj = CyWrapper()
obj.ptr = new cpp_class(x)
return obj
I've created a wrapper class with a destructor that handles the memory and a publicly accessible constructor function that can be called from external code. This version is safe because your wrapper owns the object it holds and so there can be no writes to invalid memory. However, because it makes a copy, you can't make changes to the original C++ object.
A second option is to hold a pointer to an object you don't own. The code is basically identical except you remove the __dealloc__ and avoid making a copy in make_CyWrapper:
obj.ptr = &x // instead of new cpp_class(x)
This is unsafe - you need to ensure your C++ object outlives the Cython wrapper - but allows you to modify the object.
You could also imagine a few other options: you could take ownership of an existing object with your Cython wrapper (Such a scheme would have to pass by pointer rather than reference, or it could use move constructors); you could deconstruct your C++ class into a representation expressed in basic types and pass those to Python; you could use shared pointers to split the ownership; or you have a more elaborate way of marking your Cython wrapper as "invalid" once your held C++ instance is destructed.
What you do next depends on whether you're using Boost Python (for it's convenient, callable wrapping of Python objects) or if you're making your own version. (I showed both possibilities in the previous answer).
Assuming Boost Python, you need to do two things - tell it about the conversion and make sure that it imports the module that your wrapper is defined in (if you don't do this you get exciting segmentation faults)
struct convert_to_PyWrapper {
static PyObject* convert(const cpp_class& rhs) {
// the const_cast here is a bit dodgy, but was needed to make it work
return make_CyWrapper(const_cast<cpp_class&>(rhs));
}
};
inline void setup_boost_python() {
PyInit_your_module_name(); // named inityour_module_name in Python 2
boost::python::to_python_converter<
cpp_class,
convert_to_PyWrapper>();
}
You need to make sure that your Python/Cython code calls "setup_boost_python" before attempting to use the callback (if you put it at module level it's done on import, which is ideal).
If you're following my "manual" scheme (avoiding the dependency on Boost Python) then you need to modify the call_obj Cython function that does the C++ to Cython type conversion.
cdef public void call_obj(obj, cpp_class& c):
obj(make_CyWrapper(c))
You also need to ensure the wrapper Cython module is imported before use (otherwise you get segmentation faults). I did this in "py_object_wrapper.hpp" but providing it's done once somewhere you can place it where you like.
void operator()(cpp_class& a) {
PyInit_your_module_name();
if (held) {
call_obj(held,a);
}
}
I am using SWIG to generate Python bindings for a library (lets call it Spam) that is written in C++. The library internally defines its own Vector datatype, defined in the Spam::Vector class.
Consider the following functions to be wrapped:
void ham(Spam::Vector &vec_in, Spam::Vector &vec_out);
void eggs(Spam::Vector &vec_in, double arg2, double result);
I would like to be able to call these functions using Python lists AND NumPy arrays as inputs (instead of having to create a Spam::Vector object in Python and then populate it using the associated C++ methods - it is very unpythonic).
How would I go about writing the SWIG typemap to achieve this? Also, is there a way to incorporate/leverage numpy.i for this purpose?
The right way to do this is with a custom typemap. Precisely what this will look like depends a lot on the type Spam::Vector itself. In general though you can do this with something like:
%typemap(in) {
// Maybe you'd rather check for iterable here, with this check after numpy?
if (PyList_Check($input)) {
$1 = ... // Code to iterate over a list and prepare a Spam::Vector
}
else if (PyType_IsSubtype($input->ob_type, NumpyType)) {
$1 = ... // Code to convert from numpy input
}
else {
// code to raise an error
}
}
There are various hacks that might be possible in other more specific circumstances, but this is the general solution.
I have a C++ class that is able to output strings in normal ASCII or wide format. I want to get the output in Python as a string. I am using SWIG (version 3.0.4) and have read the SWIG documentation. I believe I need to use the typemap construct to achieve my goal. I have written the following:
%typemap(out) my_namespace::MyString *
{
$result = PyString_AsString($1);
}
with no success. When I try to access a C++ string from Python, I get the following output:
<Swig Object of type 'MyString *' at 0x02B6FC68>
Obviously, I'm doing something wrong. Can anyone point me in the right direction? Thanks in advance.
I use pyboost for C++/python interfaces and is amazing and easy to do that. If you can, I recommend it. A simply std::string is automatically mapped to python string. In your case, may be any solution is to define a
__str __
method for the object or directly pass a char* (I see that in swig docs but I never followed this way).
It turns out the correct solution was:
%typemap(out) my_namespace::MyString &
{
$result = PyString_FromFormat($1->c_str());
}
Not very intuitive, but it works.
I started on my first Python extension today and was only creating a very small wrapper around a C library as an exercise. As is typical with C libraries, you start of with an initialization function that yields a handler. You can pass that handler to functions and later you pass it to the cleanup function that frees memory.
When I started writing the wrapper I basically wanted to have a way to call each native C function from python. Quickly I hit the problem that I need to return an arbitrary pointer from C to Python only to give it from there to C again in another function. I doesn't matter how it looks as I don't use it in Python, I just store it and pass it around.
So how do you pass around a void pointer between Python and C?
Please note: I know it is not recommended to write such small wrappers using the extension system but rather ctypes and friends. This is just for practice right now.
PyLong_FromVoidPtr() and PyLong_AsVoidPtr() can be abused to inject malicious data into your program. I recommend against them.
Python has PyCapsule for exactly that job. Capsules provide a safe way to exchange void ptr between modules or Python space and C space. The capsules are type-safe, too. If you need some example, the socket / ssl modules and pyexpat / _elementtree modules use capsules to exchange CAPI structs.
http://docs.python.org/3/c-api/capsule.html
After some searching I found the functions PyLong_AsVoidPtr and PyLong_FromVoidPtr. This yields a nice way to convert between a void * and a PyObject:
# in init function
return PyLong_FromVoidPtr(handle);
# in function using handle
handle = PyLong_AsVoidPtr(python_handle);
The one problem now might be how to retrieve python_handle from the typical *args given to a function:
PyObject *python_handle;
PyArg_ParseTuple(args, "O", &python_handle);
Careful here: The argument given for the "O" object must be a pointer to a PyObject pointer: PyObject **. The "O" itself only denotes to pass this PyObject through without any handling and converting. And with this, you can pass around any pointers any way you like.
Note: I think this solution is not really pretty, because you now have to variables, one that is only needed for a short time.
I have dll, that builded in Delphi, and I try to call function from it. Declaration of function looks like this:
function GetUid(UID:Pointer):Integer; stdcall;
This is equivalent to this C function signature:
int GetUID(void *pointer);
Library handled using ctypes:
from ctypes import *
lib = cdll.LoadLibrary("mylib.dll")
But i stuck here:
res = lib.GetUid(?)
What I need to pass in this function?
Pointer is void *, but how make this rightly in python?
Python is a high level language. You do not typically import a DLL from a C or Pascal native library and invoke it and pass variables from Python into a C or Pascal function taking a void * type raw pointer and then manipulate raw memory this way.
In short if you knew what you were doing you would know better than to try to do what you're doing here.
Let's suppose that your implementation is like this:
function GetUid(UID:Pointer):Integer; stdcall;
var
P2:^Integer;
begin
P2 := UID;
P2^ := 0;
end;
Then, what you would want to do is pass in an address to a 32 bit integer. Of course my example above is absurd, because what would have made sense above is to just declare the parameter as an "int *pointer" (in C terms) rather than as a "void *pointer".
Whatever it is you're doing, the next thing that will likely happen is that you will corrupt your python interpreter's heap, and cause lots of fun crashes and errors.
A far more sensible approach is to read the Python documentation on writing C extensions that can manipulate native Python types (PyObject), and doing the same thing but in pascal, if you like.
p4d appears to be a workable way of writing extension DLLs in delphi:
https://code.google.com/p/python4delphi/source/list