For a better understanding of how to pass arguments from Python to C++ functions with the pybind library, I wanted to build a small dummy/demo code where I could receive a Python list on the C++ side, cast it to a float pointer object, and then print it.
Though I know I can use the py::list class I haven't figured out the methods available of this class. I looked in the documentation reference, and then in the code (list.h, stl.h) and couldn't figure out which methods where available.
What is the equivalent of __getitem__? Do I have every python method available for py::list?
The code you are looking for is here:
class list : public object {
public:
PYBIND11_OBJECT_CVT(list, object, PyList_Check, PySequence_List)
explicit list(size_t size = 0) : object(PyList_New((ssize_t) size), stolen_t{}) {
if (!m_ptr) pybind11_fail("Could not allocate list object!");
}
size_t size() const { return (size_t) PyList_Size(m_ptr); }
detail::list_accessor operator[](size_t index) const { return {*this, index}; }
detail::list_iterator begin() const { return {*this, 0}; }
detail::list_iterator end() const { return {*this, PyList_GET_SIZE(m_ptr)}; }
template <typename T> void append(T &&val) const {
PyList_Append(m_ptr, detail::object_or_cast(std::forward<T>(val)).ptr());
}
};
Also keep in mind that py::list inherits from py::object, which in turn inherits from py::handle (this also means that you are passing by reference). In my experience, there is very little documentation for this kind of usage, and reading the code is your best bet.
We can see from the class definition that we can use the member functions size, operator[], begin, end (C++ iterators!) and append (templated!). If this is not enough, you can use attr to access python attributes (including methods). Here is an example:
Python code (some_python.py):
import cppimport
cpp = cppimport.imp("some_cpp")
l = [1,2,3,4,5]
cpp.test(l)
print('after C++', l)
cpp.float_cast(l)
C++ code (some_cpp.cpp):
/* <%
setup_pybind11(cfg)
%> */
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
namespace py = pybind11;
void test(py::list l) {
l.attr("pop")();
std::cout << "List has length " << l.size() << std::endl;
for (py::handle obj : l) { // iterators!
std::cout << " - " << obj.attr("__str__")().cast<std::string>() << std::endl;
}
l.append(10); // automatic casting (through templating)!
}
void float_cast(py::list l) {
float f = l.cast<float>();
}
PYBIND11_MODULE(some_cpp, m) {
m.def("test", &test);
m.def("float_cast", &float_cast);
}
Output:
List has length 4
- 1
- 2
- 3
- 4
after C++ [1, 2, 3, 4, 10]
Traceback (most recent call last):
File "some_python.py", line 9, in <module>
cpp.float_cast(l)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
As you can see I've also included your specific question of casting to a float. Here I've used the cast method of py::handle, which gives a nice exception. You may try to "directly" cast the object (something like float* f = (float*) &l;) but this will give you garbage and I guess this is not what you're looking for.
One more remark: pybind/stl.h enables conversion between Python's standard types and C++ versions. For example, a list may be converted to a std::vector<int>, including typechecks. An important effect of this is that the data is passed as a copy instead of as a reference.
Related
I have the following sample code which obtains a py::list as the output of evaluating some python code.
I would like to convert it to a std::vector<std::string>, but am getting an error:
conversion from 'pybind11::list' to non-scalar type
'std::vector<std::__cxx11::basic_string<char> >' requested
Per the documentation:
When including the additional header file pybind11/stl.h, conversions
between
std::vector<>/std::deque<>/std::list<>/std::array<>/std::valarray<>,
std::set<>/std::unordered_set<>, and std::map<>/std::unordered_map<>
and the Python list, set and dict data structures are automatically
enabled.
As you can see from the below code example, I have included stl.h, but automatic conversion doesn't work.
#include <iostream>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/eval.h>
namespace py = pybind11;
py::list func()
{
py::object scope = py::module_::import("__main__").attr("__dict__");
return py::eval("[ 'foo', 'bar', 'baz' ]", scope);
}
int main()
{
Py_Initialize();
// call the function and iterate over the returned list of strings
py::list list = func();
for (auto it : list)
std::cout << py::str(it) << '\n';
// error
// conversion from 'pybind11::list' to non-scalar type 'std::vector<std::__cxx11::basic_string<char> >' requested
std::vector<std::string> vec = list;
for (auto str : vec)
std::cout << str << '\n';
return 0;
}
I can iterate over the py::list manually and call vector::push_back with each element
// populating the vector manually myself works
std::vector<std::string> vec;
vec.reserve(list.size());
for (auto it : list)
vec.push_back(py::str(it));
So I guess the linked documentation above only refers to c++ -> python conversions, and not the other way?
What is the recommended way to convert from py::list to std::vector?
You need to call .cast<>:
auto vec = list.cast<std::vector<std::string>>();
<pybind11/stl.h> simply brings specializations of the conversion templates that allow such cast, and that also allow implicit conversion when you bind function with vector arguments or returning vectors (or other standard containers).
How to convert a C++ function that return pointer? Just like:
int * getRandom( )
{
static int r[10];
srand( (unsigned)time( NULL ) );
for (int i = 0; i < 10; ++i)
{
r[i] = rand();
cout << r[i] << endl;
}
return r;
}
How should I bind to convert this C++ function to python?
m.def("getRandom", &getRandom, py::return_value_policy::reference);
The above method can only return the first number of the array. In python side, How should I do to get the array value with length of 10 that function getRandom generate in C++ code.
Like this
#include <pybind11/numpy.h>
// ...
pybind11::array_t<int> getRandomWrapper(void)
{
int* values = getRandom();
pybind11::capsule cleanup(values, [](void *f) {});
return pybind11::array_t<int>(
{10}, // shape
{sizeof(int)}, // stride
values, // pointer to data
cleanup // garbage collection callback
);
}
and then just
m.def("getRandom", &getRandomWrapper);
(return_value_policy::reference is not appropriate here as you are not directly referencing r.)
The fourth argument to the array_t<int> constructor is a wrapper or "capsule" around a function that describes how to free the array's underlying memory. In this case you are returning a pointer to something managed by "automatic storage" so there is nothing to cleanup, hence it does nothing.
If, on the other hand, you were dynamically allocating the returned array you would need something like
pybind11::capsule cleanup(values, [](void *f) {
delete[] static_cast<int*>(values);
});
I only recently started using cppyy and ctypes, so this may be a bit of a silly question. I have the following C++ function:
float method(const char* args[]) {
...
}
and from Python I want to pass args as a list of strings, i.e.:
args = *magic*
x = cppyy.gbl.method(args)
I have previously found this, so I used
def setParameters(strParamList):
numParams = len(strParamList)
strArrayType = ct.c_char_p * numParams
strArray = strArrayType()
for i, param in enumerate(strParamList):
strArray[i] = param
lib.SetParams(numParams, strArray)
and from Python:
args = setParameters([b'hello', b'world'])
c_types.c_char_p expects a bytes array. However, when calling x = cppyy.gbl.method(args) I get
TypeError: could not convert argument 1 (could not convert argument to buffer or nullptr)
I'm not entirely sure why this would be wrong since the args is a <__main__.c_char_p_Array_2> object, which I believe should be converted to a const char* args[].
For the sake of having a concrete example, I'll use this as the .cpp file:
#include <cstdlib>
extern "C"
float method(const char* args[]) {
float sum = 0.0f;
const char **p = args;
while(*p) {
sum += std::atof(*p++);
}
return sum;
}
And I'll assume it was compiled with g++ method.cpp -fPIC -shared -o method.so. Given those assumptions, here's an example of how you could use it from Python:
#!/usr/bin/env python3
from ctypes import *
lib = CDLL("./method.so")
lib.method.restype = c_float
lib.method.argtypes = (POINTER(c_char_p),)
def method(args):
return lib.method((c_char_p * (len(args) + 1))(*args))
print(method([b'1.23', b'45.6']))
We make a C array to hold the Python arguments. len(args) + 1 makes sure there's room for the null pointer sentinel.
ctypes does not have a public API that is usable from C/C++ for extension writers, so the handling of ctypes by cppyy is by necessity somewhat clunky. What's going wrong, is that the generated ctypes array of const char* is of type const char*[2] not const char*[] and since cppyy does a direct type match for ctypes types, that fails.
As-is, some code somewhere needs to do a conversion of the Python strings to low-level C ones, and hold on to that memory for the duration of the call. Me, personally, I'd use a little C++ wrapper, rather than having to think things through on the Python side. The point being that an std::vector<std::string> can deal with the necessary conversions (so no bytes type needed, for example, but of course allowed if you want to) and it can hold the temporary memory.
So, if you're given some 3rd party interface like this (putting it inline for cppyy only for the sake of the example):
import cppyy
cppyy.cppdef("""
float method(const char* args[], int len) {
for (int i = 0; i < len; ++i)
std::cerr << args[i] << " ";
std::cerr << std::endl;
return 42.f;
}
""")
Then I'd generate a wrapper:
# write a C++ wrapper to hide C code
cppyy.cppdef("""
namespace MyCppAPI {
float method(const std::vector<std::string>& args) {
std::vector<const char*> v;
v.reserve(args.size());
for (auto& s : args) v.push_back(s.c_str());
return ::method(v.data(), v.size());
}
}
""")
Then replace the original C API with the C++ version:
# replace C version with C++ one for all Python users
cppyy.gbl.method = cppyy.gbl.MyCppAPI.method
and things will be as expected for any other person downstream:
# now use it as expected
cppyy.gbl.method(["aap", "noot", "mies"])
All that said, obviously there is no reason why cppyy couldn't do this bit of wrapping automatically. I created this issue: https://bitbucket.org/wlav/cppyy/issues/235/automatically-convert-python-tuple-of
So this question is sort of one of translation. I am new to C++, and was looking through the class documentation. However, it looks like finding the answer to my question is a bit hard via the documentation.
I have code for generating a random number between 0 and 1 in C++: (obtained from here, since the rand() function solution for floats is integer based)
#include <random>
#include <iostream>
int main()
{
std::random_device rd;
std::mt19937 gen(rd());
std::uniform_real_distribution<> dis(0, 1); //corrected from 1,2
for (int n = 0; n < 10; ++n) {
std::cout << dis(gen) << ' ';
}
std::cout << '\n';
}
Next, I would like to create a class or struct or something (not really an OOP guy) that has an API like:
float x = my_RandomNumberGenerator.next();
In python, I might write something like:
class my_RNG():
def __init__(self):
self.rd = (the random device object I initialize in c code)
self.gen = (the mersenne_twister engine object)(rd)
self.distribution = (the uniform real distribution object)
def next():
return self.distribution(self.gen)
my_randomNumberGenerator = my_RNG()
print(my_randomNumberGenerator.next())
How would I implement this in C++?
update Here is what I have so far (it does not work... or compile...but there seems to be some strangeness in the way things are initialized in my template code that I got from the reference site that I don't understand):
#include <iostream>
#include <random>
class MyRNG
{
public:
float next(void);
private:
std::random_device randomDevice;
std::mt19937_64 randomGenerator;
std::uniform_real_distribution distribution;
MyRNG(float range_lower,float range_upper);
};
MyRNG::MyRNG(float range_lower, float range_upper)
{
randomGenerator = std::mersenne_twister_engine(randomDevice);
distribution = std::uniform_real_distribution<> distribution(range_lower,range_upper);
}
MyRNG::next(void)
{
return distribution(randomGenerator);
}
int main() {
MyRNG my_rng = MyRNG(0,1);
std::cout << my_rng.next() << std::endl;
return 0;
}
Seems like you just need some form of probability generation class, see below for a basic implementation which meets your question requirements:
template<class Ty = double,
class = std::enable_if_t<std::is_floating_point<Ty>::value>
> class random_probability_generator {
public:
// default constructor uses single random_device for seeding
random_probability_generator()
: mt_eng{std::random_device{}()}, prob_dist(0.0, 1.0) {}
// ... other constructors with custom seeds if necessary
Ty next() { return prob_dist(mt_eng); }
// ... other methods if necessary
private:
std::mt19937 mt_eng;
std::uniform_real_distribution<Ty> prob_dist;
};
Then you can use this simply via:
random_probability_generator<> pgen;
double p = pgen.next(); // double in range [0.0, 1.0]
Or if you want random floats instead (as part of your question seems to imply):
random_probability_generator<float> pgen;
float p = pgen.next(); // float in range [0.0f, 1.0f]
Also, to address why the class you posted isn't compiling, the error in your class is that you try to initialise a std::mt19937_64 type object (randomGenerator) with a std::mersenne_twister_engine instance but they are fundamentally different types. Instead you would need to do
randomGenerator = std::mt19937_64(randomDevice());
in MyRNG constructor, or construct via initialisation list as I have done in the example above.
As pointed out in the comments, a more suitable c++-esque implementation of this is to overload operator() instead of creating a next() method. See below for a better implementation of the above class,
template<class FloatType = double,
class Generator = std::mt19937,
class = std::enable_if_t<std::is_floating_point<FloatType>::value>
> class uniform_random_probability_generator {
public:
typedef FloatType result_type;
typedef Generator generator_type;
typedef std::uniform_real_distribution<FloatType> distribution_type;
// default constructor
explicit uniform_random_probability_generator(Generator&& _eng
= Generator{std::random_device{}()}) : eng(std::move(_eng)), dist() {}
// construct from existing pre-defined engine
explicit uniform_random_probability_generator(const Generator& _eng)
: eng(_eng), dist() {}
// generate next random value in distribution (equivalent to next() in above code)
result_type operator()() { return dist(eng); }
// will always yield 0.0 for this class type
constexpr result_type min() const { return dist.min(); }
// will always yield 1.0 for this class type
constexpr result_type max() const { return dist.max(); }
// resets internal state such that next call to operator()
// does not rely on previous call
void reset_distribution_state() { dist.reset(); }
private:
generator_type eng;
distribution_type dist;
};
Then you can use this similarly to the first class in this answer,
uniform_random_probability_generator<> urpg;
double next_prob = urpg();
Additionally, uniform_random_probability_generator can use a different Generator type as a template parameter so long as this type meets the requirements of UniformRandomBitGenerator. For example, if for any reason you needed to use std::knuth_b instead of std::mt19937 then you can do so as follows:
uniform_random_probability_generator<double, std::knuth_b> urpg_kb;
double next_prob = urpg_kb();
You can create a class that holds a random number generator as a private member variable (like std::mt19937) and seeds it in the constructor. Your next function could just invoke the stored generator to get the next value (applying whatever distribution you want (if any) of course).
This is not very complicated, so I'm afraid I'm missing the real point of your question..
Suppose there is a class MyArray in C++. It implements an array of SomeType In order to make a __getitem__ function for it in Python, I do something like this
const SomeType& getitem(const MyArray *arr, PyObject *slice) {
// ???
}
BOOST_PYTHON_MODULE(mymodule)
{
class_<MyArray>("MyArray")
.def("__getitem__", &getitem)
// probably some other methods...
;
}
It is possible to get indices in slice by using these functions. However, "Boost::Python is designed with the idea in mind that users never touch a PyObject*".
Is there a better 'boost way' to do this?
Boost.Python is designed to minimize the need to interact with PyObject, and it often accomplishes this by:
Providing higher-level type wrappers.
Allowing access to the Python object's interface through the associated boost::python::object.
For example, one can access the Python object's interface through C++ in a similar manner as one would do in Python. The following demonstrates accessing the start attribute of a boost::python::object that refers to a Python slice instance:
namespace python = boost::python;
python::object slice = get_slice_object();
python::object start = slice.attr("start");
std::size_t start_index = !start.is_none()
? python::extract<std::size_t>(start) // Extract index.
: 0; // Default.
While this approach works, it tends to result in much boilerplate code: creating defaults when None is provided, handling zero-length slices, and converting negative indexes to positive index. In this case, Boost.Python provides a higher-level type wrapper boost::python::slice that has a get_indices() member-function that will remove much of the boilerplate code. Here is a complete minimal example:
#include <vector>
#include <boost/range/algorithm.hpp>
#include <boost/range/irange.hpp>
#include <boost/python.hpp>
#include <boost/python/slice.hpp>
/// #brief Mockup class that creates a range from 0 to N.
struct counter
{
counter(std::size_t n)
{
data.reserve(n);
boost::copy(boost::irange(std::size_t(0), n), std::back_inserter(data));
}
std::vector<int> data;
};
/// #brief Handle slicing for counter object.
boost::python::list spam_getitem(
const counter& self,
boost::python::slice slice)
{
namespace python = boost::python;
python::list result;
// Boost.Python will throw std::invalid_argument if the range would be
// empty.
python::slice::range<std::vector<int>::const_iterator> range;
try
{
range = slice.get_indices(self.data.begin(), self.data.end());
}
catch (std::invalid_argument)
{
return result;
}
// Iterate over fully-closed range.
for (; range.start != range.stop; std::advance(range.start, range.step))
{
result.append(*range.start);
}
result.append(*range.start); // Handle last item.
return result;
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<counter>("Counter", python::init<int>())
.def("__getitem__", &spam_getitem)
;
}
Interactive usage:
>>> from example import Counter
>>> counter = Counter(5)
>>> assert(counter[:] == [0,1,2,3,4])
>>> assert(counter[:-2] == [0,1,2])
>>> assert(counter[-2:] == [3,4])
>>> assert(counter[::2] == [0,2,4])
>>> assert(counter[1::2] == [1,3])
>>> assert(counter[100:] == [])