I am fairly new to boost.python and trying to expose the return value of a function to python.
The function signature looks like this:
std::unique_ptr<Message> someFunc(const std::string &str) const;
When calling the function in python, I get the following error:
TypeError: No to_python (by-value) converter found for C++ type: std::unique_ptr<Message, std::default_delete<Message> >
My function call in python looks like this:
a = mymodule.MyClass()
a.someFunc("some string here") # error here
I tried to expose the std::unique_ptr but just cant get it to work..
Does someone know how to properly expose the pointer class?
Thanks!
Edit:
I tried the following:
class_<std::unique_ptr<Message, std::default_delete<Message>>, bost::noncopyable ("Message", init<>())
;
This example compiles, but I still get the error mentioned above.
Also, I tried to expose the class Message itself
class_<Message>("Message", init<unsigned>())
.def(init<unsigned, unsigned>())
.def("f", &Message::f)
;
In short, Boost.Python does not support move-semantics, and therefore does not support std::unique_ptr. Boost.Python's news/change log has no indication that it has been updated for C++11 move-semantics. Additionally, this feature request for unique_ptr support has not been touched for over a year.
Nevertheless, Boost.Python supports transferring exclusive ownership of an object to and from Python via std::auto_ptr. As unique_ptr is essentially a safer version of auto_ptr, it should be fairly straight forward to adapt an API using unique_ptr to an API that uses auto_ptr:
When C++ transfers ownership to Python, the C++ function must:
be exposed with CallPolicy of boost::python::return_value_policy with a boost::python::manage_new_object result converter.
have unique_ptr release control via release() and return a raw pointer
When Python transfers ownership to C++, the C++ function must:
accept the instance via auto_ptr. The FAQ mentions that pointers returned from C++ with a manage_new_object policy will be managed via std::auto_ptr.
have auto_ptr release control to a unique_ptr via release()
Given an API/library that cannot be changed:
/// #brief Mockup Spam class.
struct Spam;
/// #brief Mockup factory for Spam.
struct SpamFactory
{
/// #brief Create Spam instances.
std::unique_ptr<Spam> make(const std::string&);
/// #brief Delete Spam instances.
void consume(std::unique_ptr<Spam>);
};
The SpamFactory::make() and SpamFactory::consume() need to be wrapped via auxiliary functions.
Functions transferring ownership from C++ to Python can be generically wrapped by a function that will create Python function objects:
/// #brief Adapter a member function that returns a unique_ptr to
/// a python function object that returns a raw pointer but
/// explicitly passes ownership to Python.
template <typename T,
typename C,
typename ...Args>
boost::python::object adapt_unique(std::unique_ptr<T> (C::*fn)(Args...))
{
return boost::python::make_function(
[fn](C& self, Args... args) { return (self.*fn)(args...).release(); },
boost::python::return_value_policy<boost::python::manage_new_object>(),
boost::mpl::vector<T*, C&, Args...>()
);
}
The lambda delegates to the original function, and releases() ownership of the instance to Python, and the call policy indicates that Python will take ownership of the value returned from the lambda. The mpl::vector describes the call signature to Boost.Python, allowing it to properly manage function dispatching between the languages.
The result of adapt_unique is exposed as SpamFactory.make():
boost::python::class_<SpamFactory>(...)
.def("make", adapt_unique(&SpamFactory::make))
// ...
;
Generically adapting SpamFactory::consume() is a more difficult, but it is easy enough to write a simple auxiliary function:
/// #brief Wrapper function for SpamFactory::consume_spam(). This
/// is required because Boost.Python will pass a handle to the
/// Spam instance as an auto_ptr that needs to be converted to
/// convert to a unique_ptr.
void SpamFactory_consume(
SpamFactory& self,
std::auto_ptr<Spam> ptr) // Note auto_ptr provided by Boost.Python.
{
return self.consume(std::unique_ptr<Spam>{ptr.release()});
}
The auxiliary function delegates to the original function, and converts the auto_ptr provided by Boost.Python to the unique_ptr required by the API. The SpamFactory_consume auxiliary function is exposed as SpamFactory.consume():
boost::python::class_<SpamFactory>(...)
// ...
.def("consume", &SpamFactory_consume)
;
Here is a complete code example:
#include <iostream>
#include <memory>
#include <boost/python.hpp>
/// #brief Mockup Spam class.
struct Spam
{
Spam(std::size_t x) : x(x) { std::cout << "Spam()" << std::endl; }
~Spam() { std::cout << "~Spam()" << std::endl; }
Spam(const Spam&) = delete;
Spam& operator=(const Spam&) = delete;
std::size_t x;
};
/// #brief Mockup factor for Spam.
struct SpamFactory
{
/// #brief Create Spam instances.
std::unique_ptr<Spam> make(const std::string& str)
{
return std::unique_ptr<Spam>{new Spam{str.size()}};
}
/// #brief Delete Spam instances.
void consume(std::unique_ptr<Spam>) {}
};
/// #brief Adapter a non-member function that returns a unique_ptr to
/// a python function object that returns a raw pointer but
/// explicitly passes ownership to Python.
template <typename T,
typename ...Args>
boost::python::object adapt_unique(std::unique_ptr<T> (*fn)(Args...))
{
return boost::python::make_function(
[fn](Args... args) { return fn(args...).release(); },
boost::python::return_value_policy<boost::python::manage_new_object>(),
boost::mpl::vector<T*, Args...>()
);
}
/// #brief Adapter a member function that returns a unique_ptr to
/// a python function object that returns a raw pointer but
/// explicitly passes ownership to Python.
template <typename T,
typename C,
typename ...Args>
boost::python::object adapt_unique(std::unique_ptr<T> (C::*fn)(Args...))
{
return boost::python::make_function(
[fn](C& self, Args... args) { return (self.*fn)(args...).release(); },
boost::python::return_value_policy<boost::python::manage_new_object>(),
boost::mpl::vector<T*, C&, Args...>()
);
}
/// #brief Wrapper function for SpamFactory::consume(). This
/// is required because Boost.Python will pass a handle to the
/// Spam instance as an auto_ptr that needs to be converted to
/// convert to a unique_ptr.
void SpamFactory_consume(
SpamFactory& self,
std::auto_ptr<Spam> ptr) // Note auto_ptr provided by Boost.Python.
{
return self.consume(std::unique_ptr<Spam>{ptr.release()});
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<Spam, boost::noncopyable>(
"Spam", python::init<std::size_t>())
.def_readwrite("x", &Spam::x)
;
python::class_<SpamFactory>("SpamFactory", python::init<>())
.def("make", adapt_unique(&SpamFactory::make))
.def("consume", &SpamFactory_consume)
;
}
Interactive Python:
>>> import example
>>> factory = example.SpamFactory()
>>> spam = factory.make("a" * 21)
Spam()
>>> spam.x
21
>>> spam.x *= 2
>>> spam.x
42
>>> factory.consume(spam)
~Spam()
>>> spam.x = 100
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
None.None(Spam, int)
did not match C++ signature:
None(Spam {lvalue}, unsigned int)
My suggestion is to get the raw pointer from the std::unique_ptr container with get(). You will have to careful to keep the unique_ptr in scope for for whole time that you wish to use the raw pointer value, otherwise the object will be deleted and you'll have a pointer to an invalid area of memory.
Boost supports movable semantics and unique_ptr since v.1.55.
But in my project I used previous version and made such simple wrapper:
class_<unique_ptr<HierarchyT>, noncopyable>(typpedName<LinksT>("hierarchy", false)
, "hierarchy holder")
.def("__call__", &unique_ptr<HierarchyT>::get,
return_internal_reference<>(),
"get holding hierarchy")
.def("reset", &unique_ptr<HierarchyT>::reset,
"reset holding hierarhy")
;
to create unique_ptr<HierarchyT> as Python shierarchy and pass it to the function that accepts it by reference.
Python code:
hier = mc.shierarchy()
mc.clusterize(hier, nds)
where C++ function is float clusterize(unique_ptr<HierarchyT>& hier,...).
Then to access results in Python make a call hier() to get the wrapped object from the unique_ptr:
output(hier(), nds)
I think nowadays there is no way to do what you are looking for... The reason is because std::unique_ptr<Message> someFunc(const std::string &str) is returning by value, which means one of two things:
The return value is going to be copied (but unique_ptr is not copyable);
The return value is going to be moved (now the problem is that boost::python doesn't provide support to move semantics). (heyy, I'm using boost 1,53, not sure in the newest versions);
Is someFunc() creating the object? In case YES, I think the solution is to create a wrapper, in case NO, you can return by reference:
std::unique_ptr<Message>& someFunc(const std::string &str)
expose the class:
class_<std::unique_ptr<Message, std::default_delete<Message>>, boost::noncopyable>("unique_ptr_message")
.def("get", &std::unique_ptr<Message>::get, return_value_policy<reference_existing_object>())
;
and also the functions:
def("someFunc", someFunc, return_value_policy<reference_existing_object>());
Related
So, I know that pybind lets you set a return value policy for methods that you wrap up. However, that doesn't seem to be working for me when I try to use this policy on a constructor. I have a class to wrap my C++ type that looks like this:
class PyComponent{
public:
static Component* Create(ComponentType type) {
Component* c = new Component(type);
// Irrelevant stuff removed here
return c;
}
/// #brief Wrap a behavior for Python
static void PyInitialize(py::module_& m);
};
void PyComponent::PyInitialize(py::module_ & m)
{
py::class_<Component>(m, "Component")
.def(py::init<>(&PyComponent::Create), py::return_value_policy::reference)
;
}
However, this does NOT stop my Component type from getting deallocated from the Python side if I call Component() and the created object goes out of scope. Any suggestions?
I did figure out the solution to this. It's to pass py::nodelete to the wrapper for my class
void PyComponent::PyInitialize(py::module_ & m)
{
py::class_<Component, std::unique_ptr<Component, py::nodelete>>(m, "Component")
.def(py::init<>(&PyComponent::Create), py::return_value_policy::reference)
;
}
I am trying to wrap a C++ library for python, using SWIG. The library uses callback functions frequently, by passing callback functions of certain type to class methods.
Now, after wrapping the code, I would like to create the callback logic from python. Is this possible? Here is an experiment I was doing to find it out .. does not work at the moment.
The header and swig files are as follows:
paska.h :
typedef void (handleri)(int code, char* codename);
// handleri is now an alias to a function that eats int, string and returns void
void wannabe_handleri(int i, char* blah);
void handleri_eater(handleri* h);
paska.i :
%module paska
%{ // this section is copied in the front of the wrapper file
#define SWIG_FILE_WITH_INIT
#include "paska.h"
%}
// from now on, what are we going to wrap ..
%inline %{
// helper functions here
void wannabe_handleri(int i, char* blah) {
};
void handleri_eater(handleri* h) {
};
%}
%include "paska.h"
// in this case, we just put the actual .cpp code into the inline block ..
Finally, I test in python ..
import paska
def testfunc(i, st):
print i
print st
paska.handleri_eater(paska.wannabe_handleri(1,"eee")) # THIS WORKS!
paska.handleri_eater(testfunc) # THIS DOES NOT WORK!
The last line throws me "TypeError: in method 'handleri_eater', argument 1 of type 'handleri *'"
Is there any way to "cast" the python function to a type accepted by the SWIG wrapper?
Seems to me that a combination of ctypes and a SWIG typemap would be the easiest way to solve the problem. ctypes makes it easy to generate a C function that calls a Python callable. The Python code should be along the lines of:
import example
# python callback
def py_callback(i, s):
print( 'py_callback(%d, %s)'%(i, s) )
example.use_callback(py_callback)
On the SWIG side we have: (1) a Python function use_callback that wraps the Python callback with a ctypes wrapper, and passes the address the wrapper as an integer to _example.use_callback(), and (2) a SWIG typemap that extracts the address and casts itto the appropriate function pointer.
%module example
// a typemap for the callback, it expects the argument to be an integer
// whose value is the address of an appropriate callback function
%typemap(in) void (*f)(int, const char*) {
$1 = (void (*)(int i, const char*))PyLong_AsVoidPtr($input);;
}
%{
void use_callback(void (*f)(int i, const char* str));
%}
%inline
%{
// a C function that accepts a callback
void use_callback(void (*f)(int i, const char* str))
{
f(100, "callback arg");
}
%}
%pythoncode
%{
import ctypes
# a ctypes callback prototype
py_callback_type = ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_char_p)
def use_callback(py_callback):
# wrap the python callback with a ctypes function pointer
f = py_callback_type(py_callback)
# get the function pointer of the ctypes wrapper by casting it to void* and taking its value
f_ptr = ctypes.cast(f, ctypes.c_void_p).value
_example.use_callback(f_ptr)
%}
You can find this complete example with a CMakeLists.txt file here.
edit: incorporated #Flexo suggestion to move the Python part into the %pythoncode block of the SWIG file.
edit: incorporated #user87746 suggestion for Python 3.6+ compatibility.
You can implement the callback logic in Python by using "directors".
Basically, instead of passing callback functions, you pass callback objects instead. The base object can be defined in C++ and provide a virtual callback member function. This object can then be inherited from and the callback function overwritten in Python. The inherited object can then be passed to a C++ function instead of a callback function. For this to work, you need to enable the director feature for such a callback class.
This does require changing the underlying C++ library, though.
I am working on a C++ library with Python bindings (using boost::python) representing data stored in a file. Majority of my semi-technical users will be using Python to interact with it, so I need to make it as Pythonic as possible. However, I will also have C++ programmers using the API, so I do not want to compromise on the C++ side to accommodate Python bindings.
A large part of the library will be made out of containers. To make things intuitive for the python users, I would like them to behave like python lists, i.e.:
# an example compound class
class Foo:
def __init__( self, _val ):
self.val = _val
# add it to a list
foo = Foo(0.0)
vect = []
vect.append(foo)
# change the value of the *original* instance
foo.val = 666.0
# which also changes the instance inside the container
print vect[0].val # outputs 666.0
The test setup
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
#include <boost/python/register_ptr_to_python.hpp>
#include <boost/shared_ptr.hpp>
struct Foo {
double val;
Foo(double a) : val(a) {}
bool operator == (const Foo& f) const { return val == f.val; }
};
/* insert the test module wrapping code here */
int main() {
Py_Initialize();
inittest();
boost::python::object globals = boost::python::import("__main__").attr("__dict__");
boost::python::exec(
"import test\n"
"foo = test.Foo(0.0)\n" // make a new Foo instance
"vect = test.FooVector()\n" // make a new vector of Foos
"vect.append(foo)\n" // add the instance to the vector
"foo.val = 666.0\n" // assign a new value to the instance
// which should change the value in vector
"print 'Foo =', foo.val\n" // and print the results
"print 'vector[0] =', vect[0].val\n",
globals, globals
);
return 0;
}
The way of the shared_ptr
Using the shared_ptr, I can get the same behaviour as above, but it also means that I have to represent all data in C++ using shared pointers, which is not nice from many points of view.
BOOST_PYTHON_MODULE( test ) {
// wrap Foo
boost::python::class_< Foo, boost::shared_ptr<Foo> >("Foo", boost::python::init<double>())
.def_readwrite("val", &Foo::val);
// wrap vector of shared_ptr Foos
boost::python::class_< std::vector < boost::shared_ptr<Foo> > >("FooVector")
.def(boost::python::vector_indexing_suite<std::vector< boost::shared_ptr<Foo> >, true >());
}
In my test setup, this produces the same output as pure Python:
Foo = 666.0
vector[0] = 666.0
The way of the vector<Foo>
Using a vector directly gives a nice clean setup on the C++ side. However, the result does not behave in the same way as pure Python.
BOOST_PYTHON_MODULE( test ) {
// wrap Foo
boost::python::class_< Foo >("Foo", boost::python::init<double>())
.def_readwrite("val", &Foo::val);
// wrap vector of Foos
boost::python::class_< std::vector < Foo > >("FooVector")
.def(boost::python::vector_indexing_suite<std::vector< Foo > >());
}
This produces:
Foo = 666.0
vector[0] = 0.0
Which is "wrong" - changing the original instance did not change the value inside the container.
I hope I don't want too much
Interestingly enough, this code works no matter which of the two encapsulations I use:
footwo = vect[0]
footwo.val = 555.0
print vect[0].val
Which means that boost::python is able to deal with "fake shared ownership" (via its by_proxy return mechanism). Is there any way to achieve the same while inserting new elements?
However, if the answer is no, I'd love to hear other suggestions - is there an example in the Python toolkit where a similar collection encapsulation is implemented, but which does not behave as a python list?
Thanks a lot for reading this far :)
Due to the semantic differences between the languages, it is often very difficult to apply a single reusable solution to all scenarios when collections are involved. The largest issue is that the while Python collections directly support references, C++ collections require a level of indirection, such as by having shared_ptr element types. Without this indirection, C++ collections will not be able to support the same functionality as Python collections. For instance, consider two indexes that refer to the same object:
s = Spam()
spams = []
spams.append(s)
spams.append(s)
Without pointer-like element types, a C++ collection could not have two indexes referring to the same object. Nevertheless, depending on usage and needs, there may be options that allow for a Pythonic-ish interface for the Python users while still maintaining a single implementation for C++.
The most Pythonic solution would be to use a custom converter that would convert a Python iterable object to a C++ collection. See this answer for implementation details. Consider this option if:
The collection's elements are cheap to copy.
The C++ functions operate only on rvalue types (i.e., std::vector<> or const std::vector<>&). This limitation prevents C++ from making changes to the Python collection or its elements.
Enhance vector_indexing_suite capabilities, reusing as many capabilities as possible, such as its proxies for safely handling index deletion and reallocation of the underlying collection:
Expose the model with a custom HeldType that functions as a smart pointer and delegate to either the instance or the element proxy objects returned from vector_indexing_suite.
Monkey patch the collection's methods that insert elements into the collection so that the custom HeldType will be set to delegate to a element proxy.
When exposing a class to Boost.Python, the HeldType is the type of object that gets embedded within a Boost.Python object. When accessing the wrapped types object, Boost.Python invokes get_pointer() for the HeldType. The object_holder class below provides the ability to return a handle to either an instance it owns or to an element proxy:
/// #brief smart pointer type that will delegate to a python
/// object if one is set.
template <typename T>
class object_holder
{
public:
typedef T element_type;
object_holder(element_type* ptr)
: ptr_(ptr),
object_()
{}
element_type* get() const
{
if (!object_.is_none())
{
return boost::python::extract<element_type*>(object_)();
}
return ptr_ ? ptr_.get() : NULL;
}
void reset(boost::python::object object)
{
// Verify the object holds the expected element.
boost::python::extract<element_type*> extractor(object_);
if (!extractor.check()) return;
object_ = object;
ptr_.reset();
}
private:
boost::shared_ptr<element_type> ptr_;
boost::python::object object_;
};
/// #brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const object_holder<T>& holder)
{
return holder.get();
}
With the indirection supported, the only thing remaining is patching the collection to set the object_holder. One clean and reusable way to support this is to use def_visitor. This is a generic interface that allows for class_ objects to be extended non-intrusively. For instance, the vector_indexing_suite uses this capability.
The custom_vector_indexing_suite class below monkey patches the append() method to delegate to the original method, and then invokes object_holder.reset() with a proxy to the newly set element. This results in the object_holder referring to the element contained within the collection.
/// #brief Indexing suite that will resets the element's HeldType to
/// that of the proxy during element insertion.
template <typename Container,
typename HeldType>
class custom_vector_indexing_suite
: public boost::python::def_visitor<
custom_vector_indexing_suite<Container, HeldType>>
{
private:
friend class boost::python::def_visitor_access;
template <typename ClassT>
void visit(ClassT& cls) const
{
// Define vector indexing support.
cls.def(boost::python::vector_indexing_suite<Container>());
// Monkey patch element setters with custom functions that
// delegate to the original implementation then obtain a
// handle to the proxy.
cls
.def("append", make_append_wrapper(cls.attr("append")))
// repeat for __setitem__ (slice and non-slice) and extend
;
}
/// #brief Returned a patched 'append' function.
static boost::python::object make_append_wrapper(
boost::python::object original_fn)
{
namespace python = boost::python;
return python::make_function([original_fn](
python::object self,
HeldType& value)
{
// Copy into the collection.
original_fn(self, value.get());
// Reset handle to delegate to a proxy for the newly copied element.
value.reset(self[-1]);
},
// Call policies.
python::default_call_policies(),
// Describe the signature.
boost::mpl::vector<
void, // return
python::object, // self (collection)
HeldType>() // value
);
}
};
Wrapping needs to occur at runtime and custom functor objects cannot be directly defined on the class via def(), so the make_function() function must be used. For functors, it requires both CallPolicies and a MPL front-extensible sequence representing the signature.
Here is a complete example that demonstrates using the object_holder to delegate to proxies and custom_vector_indexing_suite to patch the collection.
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
/// #brief Mockup type.
struct spam
{
int val;
spam(int val) : val(val) {}
bool operator==(const spam& rhs) { return val == rhs.val; }
};
/// #brief Mockup function that operations on a collection of spam instances.
void modify_spams(std::vector<spam>& spams)
{
for (auto& spam : spams)
spam.val *= 2;
}
/// #brief smart pointer type that will delegate to a python
/// object if one is set.
template <typename T>
class object_holder
{
public:
typedef T element_type;
object_holder(element_type* ptr)
: ptr_(ptr),
object_()
{}
element_type* get() const
{
if (!object_.is_none())
{
return boost::python::extract<element_type*>(object_)();
}
return ptr_ ? ptr_.get() : NULL;
}
void reset(boost::python::object object)
{
// Verify the object holds the expected element.
boost::python::extract<element_type*> extractor(object_);
if (!extractor.check()) return;
object_ = object;
ptr_.reset();
}
private:
boost::shared_ptr<element_type> ptr_;
boost::python::object object_;
};
/// #brief Helper function used to extract the pointed to object from
/// an object_holder. Boost.Python will use this through ADL.
template <typename T>
T* get_pointer(const object_holder<T>& holder)
{
return holder.get();
}
/// #brief Indexing suite that will resets the element's HeldType to
/// that of the proxy during element insertion.
template <typename Container,
typename HeldType>
class custom_vector_indexing_suite
: public boost::python::def_visitor<
custom_vector_indexing_suite<Container, HeldType>>
{
private:
friend class boost::python::def_visitor_access;
template <typename ClassT>
void visit(ClassT& cls) const
{
// Define vector indexing support.
cls.def(boost::python::vector_indexing_suite<Container>());
// Monkey patch element setters with custom functions that
// delegate to the original implementation then obtain a
// handle to the proxy.
cls
.def("append", make_append_wrapper(cls.attr("append")))
// repeat for __setitem__ (slice and non-slice) and extend
;
}
/// #brief Returned a patched 'append' function.
static boost::python::object make_append_wrapper(
boost::python::object original_fn)
{
namespace python = boost::python;
return python::make_function([original_fn](
python::object self,
HeldType& value)
{
// Copy into the collection.
original_fn(self, value.get());
// Reset handle to delegate to a proxy for the newly copied element.
value.reset(self[-1]);
},
// Call policies.
python::default_call_policies(),
// Describe the signature.
boost::mpl::vector<
void, // return
python::object, // self (collection)
HeldType>() // value
);
}
// .. make_setitem_wrapper
// .. make_extend_wrapper
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
// Expose spam. Use a custom holder to allow for transparent delegation
// to different instances.
python::class_<spam, object_holder<spam>>("Spam", python::init<int>())
.def_readwrite("val", &spam::val)
;
// Expose a vector of spam.
python::class_<std::vector<spam>>("SpamVector")
.def(custom_vector_indexing_suite<
std::vector<spam>, object_holder<spam>>())
;
python::def("modify_spams", &modify_spams);
}
Interactive usage:
>>> import example
>>> spam = example.Spam(5)
>>> spams = example.SpamVector()
>>> spams.append(spam)
>>> assert(spams[0].val == 5)
>>> spam.val = 21
>>> assert(spams[0].val == 21)
>>> example.modify_spams(spams)
>>> assert(spam.val == 42)
>>> spams.append(spam)
>>> spam.val = 100
>>> assert(spams[1].val == 100)
>>> assert(spams[0].val == 42) # The container does not provide indirection.
As the vector_indexing_suite is still being used, the underlying C++ container should only be modified using the Python object's API. For instance, invoking push_back on the container may cause a reallocation of the underlying memory and cause problems with existing Boost.Python proxies. On the other hand, one can safely modify the elements themselves, such as was done via the modify_spams() function above.
Unfortunately, the answer is no, you can't do what you want. In python, everything is a pointer, and lists are a container of pointers. The C++ vector of shared pointers work because the underlying data structure is more or less equivalent to a python list. What you are requesting is to have the C++ vector of allocated memory act like a vector of pointers, which can't be done.
Let's see what's happening in python lists, with C++ equivalent pseudocode:
foo = Foo(0.0) # Foo* foo = new Foo(0.0)
vect = [] # std::vector<Foo*> vect
vect.append(foo) # vect.push_back(foo)
At this point, foo and vect[0] both point to the same allocated memory, so changing *foo changes *vect[0].
Now with the vector<Foo> version:
foo = Foo(0.0) # Foo* foo = new Foo(0.0)
vect = FooVector() # std::vector<Foo> vect
vect.append(foo) # vect.push_back(*foo)
Here, vect[0] has it's own allocated memory, and is a copy of *foo. Fundamentally, you can't make vect[0] be the same memory as *foo.
On a side note, be careful with lifetime management of footwo when using std::vector<Foo>:
footwo = vect[0] # Foo* footwo = &vect[0]
A subsequent append may require moving the allocated storage for the vector, and may invalidate footwo (&vect[0] may change).
I implemented a bunch of functions and they are dispatched from the same C function called by the Python interpreter:
PyObject *
CmdDispatch(PyObject *self, PyObject *args, PyObject *kwargs)
Unexpectedly, self is NULL, and I need to get the function name currently being called. Is there any way to get this information?
I have dozens of functions which are all going through this routine. This command processes all of the options into a C++ map and passes it along to the implementation of each command.
Update:
http://docs.python.org/extending/extending.html#a-simple-example specifically says "The self argument points to the module object for module-level functions; for a method it would point to the object instance.", but I am getting passed null when linking against python 2.6.
I've been trying to solve very similar problem.
The conclusion I've come to suggests there is no way to determine name of current function or caller(s) at the level of Python C API. The reason being, Python interpreter puts on call stack only pure Python functions (implemented in Python itself). Functions implemented in C, regardless if registered in module methods table, are not put on Python stack, thus it's not possible to find them inspecting the stack frames.
Here is a quick example in Python illustrating what I wanted to achieve (I assume Juan asks for similar behaviour):
import sys
def foo():
print('foo:', sys._getframe(0).f_code.co_name)
def bar():
print('bar:', sys._getframe(0).f_code.co_name)
foo()
bar()
Here is close equivalent of this example (based on the Python 3 docs) but implemented using Python C API:
// Based on example from Python 3.2.1 documentation, 5.4. Extending Embedded Python
// http://docs.python.org/release/3.2.1/extending/embedding.html#extending-embedded-python
//
#include <Python.h>
#include <frameobject.h>
static void foo()
{
PyThreadState * ts = PyThreadState_Get();
PyFrameObject* frame = ts->frame;
while (frame != 0)
{
char const* filename = _PyUnicode_AsString(frame->f_code->co_filename);
char const* name = _PyUnicode_AsString(frame->f_code->co_name);
printf("foo: filename=%s, name=%s\n", filename, name);
frame = frame->f_back;
}
}
static void bar()
{
PyRun_SimpleString(
"import sys\n"
"print(\"bar: filename=%s, name=%s\" % (sys._getframe(0).f_code.co_filename, sys._getframe(0).f_code.co_name))"
);
}
static PyObject* emb_numargs(PyObject *self, PyObject *args)
{
foo();
bar();
PyRun_SimpleString(
"import sys\n"
"print(\"emb_numargs: filename=%s, name=%s\" % (sys._getframe(0).f_code.co_filename, sys._getframe(0).f_code.co_name))"
);
return PyLong_FromLong(0);
}
static PyMethodDef EmbMethods[] = {
{"numargs", emb_numargs, METH_VARARGS, "Return number 0"},
{NULL, NULL, 0, NULL}
};
static PyModuleDef EmbModule = {
PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods,
NULL, NULL, NULL, NULL
};
static PyObject* PyInit_emb(void)
{
return PyModule_Create(&EmbModule);
}
int main(int argc, char* argv[])
{
PyImport_AppendInittab("emb", &PyInit_emb);
Py_Initialize();
PyRun_SimpleString(
"import emb\n"
"print('This is Zero: ', emb.numargs())\n"
);
Py_Finalize();
return 0;
}
I hope this completes Ned's answer too.
The Python api isn't built to tell you what function it is calling. You've created a function, and it is calling it, the API assumes you know what function you've written. You'll need to create a small wrapper function for each of your Python functions. The best way to do this is to register your one C function as one Python function that takes a string as its first argument. Then, in your Python layer, create as many Python functions as you need, each invoking your C function with the proper string argument identifying what function you really want to call.
Another alternative is to rethink the structure of your C code, and have as many functions as you need, each of which invokes your common helper code to process options, etc.
NOTICE error checking for API is not being provided for clarity;
This example insert a new function directly on python __builtin__ module, allowing to call the method without class.method schema. Just change mymodule to any other module as you wish.
PyObject* py_cb(PyObject *self, PyObject *args)
{
const char *name = (const char *) PyCObject_AsVoidPtr(self);
printf("%s\n", name);
Py_RETURN_NONE;
}
int main(int argc, char *argv)
{
PyObject *mod, *modname, *dict, *fnc, *usrptr;
const char *mymodule = "__builtin__";
PyMethodDef *m;
const char *method = "hello_python";
Py_Initialize();
mod = PyImport_ImportModule(mymodule);
modname = PyString_FromString(mymodule);
dict = PyModule_GetDict(mod);
m = (PyMethodDef *) calloc(1, sizeof(PyMethodDef));
m->ml_name = strdup(method);
m->ml_meth = py_cb;
usrptr = PyCObject_FromVoidPtr("I'm am the hello_python!", NULL);
fnc = PyCFunction_NewEx(m, usrptr, modname);
PyDict_SetItemString(dict, method, fnc);
...
When python script execute hello_python, the py_cb extension function will show:
I'm am the hello_python!
The self is used to send a real pointer such as the library context instead of this const char * of this example, this is now a matter of changing it to something interesting though.
I don't know if can be done directly from the C-API. At worst, you could call the traceback module from C, to get the name of the caller.
I'm working on embedding python into my C++ program using swig. At the moment I have a object written in C++ which I want to pass to a python function. I've created the swig interface to wrap the class.
What I'm trying to do is take this C++ object which I've created and pass it to a python function with the ability to use it like I would in C++. Is it possible for me to use code generate by swig to do this? If not how can I approach this?
You can use PyObject_CallMethod to pass a newly created object back to python. Assuming ModuleName.object is a python object with a method called methodName that you want to pass a newly created C++ object to you want to roughly (from memory, I can't test it right now) do this in C++:
int callPython() {
PyObject* module = PyImport_ImportModule("ModuleName");
if (!module)
return 0;
// Get an object to call method on from ModuleName
PyObject* python_object = PyObject_CallMethod(module, "object", "O", module);
if (!python_object) {
PyErr_Print();
Py_DecRef(module);
return 0;
}
// SWIGTYPE_p_Foo should be the SWIGTYPE for your wrapped class and
// SWIG_POINTER_NEW is a flag indicating ownership of the new object
PyObject *instance = SWIG_NewPointerObj(SWIG_as_voidptr(new Foo()), SWIGTYPE_p_Foo, SWIG_POINTER_NEW);
PyObject *result = PyObject_CallMethod(python_object, "methodName", "O", instance);
// Do something with result?
Py_DecRef(instance);
Py_DecRef(result);
Py_DecRef(module);
return 1;
}
I think I've got the reference counting right for this, but I'm not totally sure.