We Keep Coding

Create a copy class method for a Python object containing a Swig object

I have created a Python class with an attribute that is a Swig object (which happens to be a wrapper of a C structure). I want to be able to create copies of that class, e.g., by defining a __copy__ method, that contain independent copies of the Swig object (using the copy modules' copy class just creates a pointer to the original object, and deepcopy fails).
Does anyone know if you can just copy chunks of memory in Python, and use this to copy the attribute containing the Swig object? Or, could I create a __copy__ or __deepcopy__ method in the Swig interface file that created the Swig object, which is able to use Cs memcpy?

From looking at the __deepcopy__ implemented in the Swig interface for LAL, finding the Swig macros for allocating and deallocating memory, and looking at my own(!) example of extending the Swig interface to a C structure, I have figured out how to create a __deepcopy__ method for the Swig-wrapped structure.
Repeating my gist, and extending it to add a __deepcopy__ method is as follows:
Say you have some C code containing a structure like this:
/* testswig.h file */
#include <stdlib.h>
#include <stdio.h>
typedef struct tagteststruct{
double *data;
size_t len;
} teststruct;
teststruct *CreateStruct(size_t len);
where the structure will contain a data array of length len. The function CreateStruct() allocates
memory for an instantiation of the structure, and is defined as
/* testswig.c file */
#include "testswig.h"
/* function for allocating memory for test struct */
teststruct *CreateStruct(size_t len){
teststruct *ts = NULL;
ts = (teststruct *)malloc(sizeof(teststruct));
ts->data = (double *)malloc(sizeof(double)*len);
ts->len = len;
return ts;
}
If you wrap this with SWIG for use in python, then it might be useful to have some python list-like methods available to,
e.g., add or get items from the data array. To do this you can create the following SWIG interface file:
/* testswig.i */
%module testswig
%include "exception.i"
%{
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include "testswig.h"
static int teststructErr = 0; // flag to save test struct error state
%}
%include "testswig.h"
// set exception handling for __getitem__
%exception tagteststruct::__getitem__ {
assert(!teststructErr);
$action
if ( teststructErr ){
teststructErr = 0; // clear flag for next time
SWIG_exception(SWIG_IndexError, "Index out of bounds");
}
}
// set exception handling for __setitem__
%exception tagteststruct::__setitem__ {
assert(!teststructErr);
$action
if ( teststructErr ){
teststructErr = 0; // clear flag for next time
SWIG_exception(SWIG_IndexError, "Index out of bounds");
}
}
// set exception handling for insert()
%exception tagteststruct::insert {
assert(!teststructErr);
$action
if ( teststructErr ){
teststructErr = 0; // clear flag for next time
SWIG_exception(SWIG_IndexError, "Index out of bounds");
}
}
// "extend" the structure with various methods
%extend tagteststruct{
// add a __getitem__ method to the structure to get values from the data array
double __getitem__(size_t i) {
if (i >= $self->len) {
teststructErr = 1;
return 0;
}
return $self->data[i];
}
// add a __setitem__ method to the structure to set values in the data array
void __setitem__(size_t i, double value) {
if ( i >= $self->len ){
teststructErr = 1;
return;
}
$self->data[i] = value;
}
size_t __len__(){
return $self->len;
}
void insert(size_t i, double value) {
if ( i >= $self->len ){
teststructErr = 1;
return;
}
$self->data[i] = value;
}
%typemap(in, noblock=1) const void *memo "";
struct tagteststruct * __deepcopy__(const void *memo) {
// copy structure
struct tagteststruct * scopy = %new_copy(*$self, struct tagteststruct);
// copy array within the structure
scopy->data = %new_copy_array($self->data, $self->len, double);
return scopy;
}
%clear const void *memo;
}
In the above example, it adds the following methods to the structure:
__getitem__: this allows the structure's data array to be accessed like a list item in python, e.g., using x[0] returns the value in teststruct->data[0]
__setitem__: this allows the structure's data array values to be set like a list item in python, e.g., using x[0] = 1.2 sets the value in teststruct->data[0]
__len__: this returns the length of the data array when using len(x)
insert(): this inserts a value into a particular index in the array like with __getitem__
__deepcopy__: this allows the use of deepcopy to create a copy of the structure.
The example also shows how to perform some exception checking for these methods, in particular, making sure the requested index does not exceed the size of the array.
To compile and use this example, you could do the following (see, e.g., SWIG's tutorial):
$ swig -python testswig.i
$ gcc -c testswig.c testswig_wrap.c -fPIC -I/usr/include/python2.7
$ ld -shared testswig.o testswig_wrap.o -o _testswig.so
where, in this case, the -I/usr/include/python2.7 flag points to the path containing the Python.h file. The
testswig_wrap.c file is generated by the swig command.
The structure can then be used in python as in the following example:
>>> from testswig import CreateStruct
>>> # create an instance of the structure with 10 elements
>>> x = CreateStruct(10)
>>> # set the 5th element of the data array to 1.3
>>> x[4] = 1.3
>>> # output the 5th element of the array
>>> print(x[4])
1.3
>>> # output the length of the array
>>> print(len(x))
10
>>> # create a copy
>>> import copy
>>> y = copy.deepcopy(x)
>>> print(len(y))
10
>>> print(y[4])
1.3
>>> y[4] = 3.4
>>> print(y[4])
3.4
>>> print(x[4]) # check x hasn't been altered
1.3
The Swig-wrapped structure could itself be in a class, e.g.,:
from testswig import CreateStruct
class mystruct():
def __init__(self, size):
self.array = CreateStruct(size)
self.name = 'array'
def __len__(self):
return len(self.array)
def __getitem__(self, idx):
return self.array[idx]
def __setitem__(self, idx, val):
self.array[idx] = val
which we can test:
>>> x = mystruct(10)
>>> x[4] = 1.2
>>> print(x[4])
1.2
>>> import copy
>>> y = copy.deepcopy(x)
>>> print(y[4])
1.2
>>> y[4] = 3.4
>>> print(y[4])
3.4
>>> print(x[4]) # check it hasn't changed
1.2

pybind how can I operate over a py::list object

For a better understanding of how to pass arguments from Python to C++ functions with the pybind library, I wanted to build a small dummy/demo code where I could receive a Python list on the C++ side, cast it to a float pointer object, and then print it.
Though I know I can use the py::list class I haven't figured out the methods available of this class. I looked in the documentation reference, and then in the code (list.h, stl.h) and couldn't figure out which methods where available.
What is the equivalent of __getitem__? Do I have every python method available for py::list?

The code you are looking for is here:
class list : public object {
public:
PYBIND11_OBJECT_CVT(list, object, PyList_Check, PySequence_List)
explicit list(size_t size = 0) : object(PyList_New((ssize_t) size), stolen_t{}) {
if (!m_ptr) pybind11_fail("Could not allocate list object!");
}
size_t size() const { return (size_t) PyList_Size(m_ptr); }
detail::list_accessor operator[](size_t index) const { return {*this, index}; }
detail::list_iterator begin() const { return {*this, 0}; }
detail::list_iterator end() const { return {*this, PyList_GET_SIZE(m_ptr)}; }
template <typename T> void append(T &&val) const {
PyList_Append(m_ptr, detail::object_or_cast(std::forward<T>(val)).ptr());
}
};
Also keep in mind that py::list inherits from py::object, which in turn inherits from py::handle (this also means that you are passing by reference). In my experience, there is very little documentation for this kind of usage, and reading the code is your best bet.
We can see from the class definition that we can use the member functions size, operator[], begin, end (C++ iterators!) and append (templated!). If this is not enough, you can use attr to access python attributes (including methods). Here is an example:
Python code (some_python.py):
import cppimport
cpp = cppimport.imp("some_cpp")
l = [1,2,3,4,5]
cpp.test(l)
print('after C++', l)
cpp.float_cast(l)
C++ code (some_cpp.cpp):
/* <%
setup_pybind11(cfg)
%> */
#include <pybind11/pybind11.h>
#include <iostream>
#include <string>
namespace py = pybind11;
void test(py::list l) {
l.attr("pop")();
std::cout << "List has length " << l.size() << std::endl;
for (py::handle obj : l) { // iterators!
std::cout << " - " << obj.attr("__str__")().cast<std::string>() << std::endl;
}
l.append(10); // automatic casting (through templating)!
}
void float_cast(py::list l) {
float f = l.cast<float>();
}
PYBIND11_MODULE(some_cpp, m) {
m.def("test", &test);
m.def("float_cast", &float_cast);
}
Output:
List has length 4
- 1
- 2
- 3
- 4
after C++ [1, 2, 3, 4, 10]
Traceback (most recent call last):
File "some_python.py", line 9, in <module>
cpp.float_cast(l)
RuntimeError: Unable to cast Python instance to C++ type (compile in debug mode for details)
As you can see I've also included your specific question of casting to a float. Here I've used the cast method of py::handle, which gives a nice exception. You may try to "directly" cast the object (something like float* f = (float*) &l;) but this will give you garbage and I guess this is not what you're looking for.
One more remark: pybind/stl.h enables conversion between Python's standard types and C++ versions. For example, a list may be converted to a std::vector<int>, including typechecks. An important effect of this is that the data is passed as a copy instead of as a reference.

Using a DLL exported from D

I've created a simple encryption program in D, and I had the idea to make a DLL from it and try to import it to, for example, Python.
I've could simply call my main function, becouse it dosn't need any params. But when I get to my encrytion method, it uses dynamic-lenght ubyte[] arrays, but as far as I know, they don't exist in other C/C++ based langs.
For example, there's the first line of one of my funcs:
ubyte[] encode(ubyte[] data, ubyte[] key){
But I can't use an array without fixed lenght in other languages!
How can I import that function, for example, in Python?
EDIT:
I know that I can create a wrapper that takes a pointer and the lenght of the array, but isn't there a more elegant solution?
(Where I don't need to use D to use a lib written in D)

Well tbh. there's no real elegant way other than wrapping a pointer with a length or wrapping to C arrays and then to D.
However you can make a somewhat elegant purpose with the first way using a struct that has a pointer, a length and a property that converts it to a D array.
Then the function you export takes your struct, all that function should do is call an internal function that takes an actual D array and you'd simply pass the array to it and the conversion would happen at that moment through alias this and the conversion property.
An example usage is here:
module main;
import core.stdc.stdlib : malloc;
import std.stdio;
struct DArray(T) {
T* data;
size_t length;
/// This field can be removed, only used for testing purpose
size_t offset;
#property T[] array() {
T[] arr;
foreach(i; 0 .. length) {
arr ~= data[i];
}
return arr;
}
alias array this;
/// This function can be removed, only used for testing purpose
void init(size_t size) {
data = cast(T*)malloc(size * T.sizeof);
length = size;
}
/// This function can be removed, only used for testing purpose
void append(T value) {
data[offset] = value;
offset++;
}
}
// This function is the one exported
void externalFoo(DArray!int intArray) {
writeln("Calling extern foo");
internalFoo(intArray);
}
// This function is the one you use
private void internalFoo(int[] intArray) {
writeln("Calling internal foo");
writeln(intArray);
}
void main() {
// Constructing our test array
DArray!int arrayTest;
arrayTest.init(10);
foreach (int i; 0 .. 10) {
arrayTest.append(i);
}
// Testing the exported function
externalFoo(arrayTest);
}
Here is an absolute minimum version of how to do it
struct DArray(T) {
T* data;
size_t length;
#property T[] array() {
T[] arr;
foreach(i; 0 .. length) {
arr ~= data[i];
}
return arr;
}
alias array this;
}
// This function is the one exported
void externalFoo(DArray!int intArray) {
writeln("Calling extern foo");
internalFoo(intArray);
}
// This function is the one you use
private void internalFoo(int[] intArray) {
writeln("Calling internal foo");
writeln(intArray);
}

C++ OOP: how to create a random number generator object?

So this question is sort of one of translation. I am new to C++, and was looking through the class documentation. However, it looks like finding the answer to my question is a bit hard via the documentation.
I have code for generating a random number between 0 and 1 in C++: (obtained from here, since the rand() function solution for floats is integer based)
#include <random>
#include <iostream>
int main()
{
std::random_device rd;
std::mt19937 gen(rd());
std::uniform_real_distribution<> dis(0, 1); //corrected from 1,2
for (int n = 0; n < 10; ++n) {
std::cout << dis(gen) << ' ';
}
std::cout << '\n';
}
Next, I would like to create a class or struct or something (not really an OOP guy) that has an API like:
float x = my_RandomNumberGenerator.next();
In python, I might write something like:
class my_RNG():
def __init__(self):
self.rd = (the random device object I initialize in c code)
self.gen = (the mersenne_twister engine object)(rd)
self.distribution = (the uniform real distribution object)
def next():
return self.distribution(self.gen)
my_randomNumberGenerator = my_RNG()
print(my_randomNumberGenerator.next())
How would I implement this in C++?
update Here is what I have so far (it does not work... or compile...but there seems to be some strangeness in the way things are initialized in my template code that I got from the reference site that I don't understand):
#include <iostream>
#include <random>
class MyRNG
{
public:
float next(void);
private:
std::random_device randomDevice;
std::mt19937_64 randomGenerator;
std::uniform_real_distribution distribution;
MyRNG(float range_lower,float range_upper);
};
MyRNG::MyRNG(float range_lower, float range_upper)
{
randomGenerator = std::mersenne_twister_engine(randomDevice);
distribution = std::uniform_real_distribution<> distribution(range_lower,range_upper);
}
MyRNG::next(void)
{
return distribution(randomGenerator);
}
int main() {
MyRNG my_rng = MyRNG(0,1);
std::cout << my_rng.next() << std::endl;
return 0;
}

Seems like you just need some form of probability generation class, see below for a basic implementation which meets your question requirements:
template<class Ty = double,
class = std::enable_if_t<std::is_floating_point<Ty>::value>
> class random_probability_generator {
public:
// default constructor uses single random_device for seeding
random_probability_generator()
: mt_eng{std::random_device{}()}, prob_dist(0.0, 1.0) {}
// ... other constructors with custom seeds if necessary
Ty next() { return prob_dist(mt_eng); }
// ... other methods if necessary
private:
std::mt19937 mt_eng;
std::uniform_real_distribution<Ty> prob_dist;
};
Then you can use this simply via:
random_probability_generator<> pgen;
double p = pgen.next(); // double in range [0.0, 1.0]
Or if you want random floats instead (as part of your question seems to imply):
random_probability_generator<float> pgen;
float p = pgen.next(); // float in range [0.0f, 1.0f]
Also, to address why the class you posted isn't compiling, the error in your class is that you try to initialise a std::mt19937_64 type object (randomGenerator) with a std::mersenne_twister_engine instance but they are fundamentally different types. Instead you would need to do
randomGenerator = std::mt19937_64(randomDevice());
in MyRNG constructor, or construct via initialisation list as I have done in the example above.
As pointed out in the comments, a more suitable c++-esque implementation of this is to overload operator() instead of creating a next() method. See below for a better implementation of the above class,
template<class FloatType = double,
class Generator = std::mt19937,
class = std::enable_if_t<std::is_floating_point<FloatType>::value>
> class uniform_random_probability_generator {
public:
typedef FloatType result_type;
typedef Generator generator_type;
typedef std::uniform_real_distribution<FloatType> distribution_type;
// default constructor
explicit uniform_random_probability_generator(Generator&& _eng
= Generator{std::random_device{}()}) : eng(std::move(_eng)), dist() {}
// construct from existing pre-defined engine
explicit uniform_random_probability_generator(const Generator& _eng)
: eng(_eng), dist() {}
// generate next random value in distribution (equivalent to next() in above code)
result_type operator()() { return dist(eng); }
// will always yield 0.0 for this class type
constexpr result_type min() const { return dist.min(); }
// will always yield 1.0 for this class type
constexpr result_type max() const { return dist.max(); }
// resets internal state such that next call to operator()
// does not rely on previous call
void reset_distribution_state() { dist.reset(); }
private:
generator_type eng;
distribution_type dist;
};
Then you can use this similarly to the first class in this answer,
uniform_random_probability_generator<> urpg;
double next_prob = urpg();
Additionally, uniform_random_probability_generator can use a different Generator type as a template parameter so long as this type meets the requirements of UniformRandomBitGenerator. For example, if for any reason you needed to use std::knuth_b instead of std::mt19937 then you can do so as follows:
uniform_random_probability_generator<double, std::knuth_b> urpg_kb;
double next_prob = urpg_kb();

You can create a class that holds a random number generator as a private member variable (like std::mt19937) and seeds it in the constructor. Your next function could just invoke the stored generator to get the next value (applying whatever distribution you want (if any) of course).
This is not very complicated, so I'm afraid I'm missing the real point of your question..

OUT argument internally allocated to return an array of structs

I'm new to swig and I have the following function which i cant fix:
int get_list(IN const char * string, OUT struct entry ** results);
where struct entry is defined:
struct flux_entry
{
char * addr_str;
char cc[2];
};
the entry struct is properly converted to a python class.
I googled but couldn't find any explanation i could use.
I want to make it return a tuple of: (original get_list int return value, python list of entry python objects, based on the results buffer), but don't know how to convert the C entry to a python object in the argout code snippet.
I've managed to get thus far:
%typemap(argout) struct entry **
{
PyObject *o = PyList_New(0);
int i;
for(i=0; $1[i] ; i++)
{
PyList_Append(o, SWIG_HOW_TO_CONVERT_TO_PYOBJECT($1[i]));
}
$result = o;
}
what should i replace SWIG_HOW_TO_CONVERT_TO_PYOBJECT with?
passed results is supposed to be a pointer to a (struct entry *) type, set to NULL before calling get_list and should be set to an allocated array of struct entry * pointers. maybe a small wrapper function could make that easier?
the struct entry array is allocated within the C function using malloc, after calculating (inside get_list) how many elements are needed, and ends with a NULL pointer to indicate the end of the array.
i'd also like to make sure it's freed somewhere :)
thanks!

This should at least give you a starting point that works. I still wasn't sure how the data was returned, since to return an array of pointers so that the final one was NULL I'd think you'd need a struct entry ***, so I just set addr_str = NULL on the last one as a sentinel, and just put some dummy data partially based on the input string into the fields. Modify as needed to suit your needs:
%module example
// Insert the structure definition and function to wrap into the wrapper code.
%{
struct entry {
char* addr_str;
char cc[2];
};
int get_list(const char* string, struct entry** results)
{
*results = malloc(3 * sizeof(struct entry));
(*results)[0].addr_str = malloc(10);
strcpy((*results)[0].addr_str,"hello");
(*results)[0].cc[0] = string[0];
(*results)[0].cc[1] = string[1];
(*results)[1].addr_str = malloc(10);
strcpy((*results)[1].addr_str,"there");
(*results)[1].cc[0] = string[2];
(*results)[1].cc[1] = string[3];
(*results)[2].addr_str = NULL;
return 0;
}
%}
#include <typemaps.i>
// Define the structure for SWIG
struct entry {
char* addr_str;
char cc[2];
};
// Define a set of typemaps to be used for an output parameter.
// This typemap suppresses requiring the parameter as an input.
// A temp variable is created and passed instead.
%typemap(in,numinputs=0) struct entry **OUTPUT (struct entry* temp) %{
$1 = &temp;
%}
// Build a list of tuples containing the two entries from the struct.
// Append the new Python list object to the existing "int" result.
%typemap(argout) struct entry **OUTPUT {
int i = 0;
PyObject* out = PyList_New(0);
while((*$1)[i].addr_str != NULL)
{
//PyObject* t = PyTuple_New(2);
//PyTuple_SET_ITEM(t,0,PyBytes_FromString((*$1)[i].addr_str));
//PyTuple_SET_ITEM(t,1,PyBytes_FromStringAndSize((*$1)[i].cc,2));
//PyList_Append(out,t);
//Py_DECREF(t);
PyObject* s = SWIG_NewPointerObj(*$1+i,$descriptor(struct entry*),0);
PyList_Append(out,s);
Py_DECREF(s);
++i;
}
$result = SWIG_AppendOutput($result,out);
}
// Since a Python object was created and the data copied for each entry struct,
// free the memory returned in the structure.
//%typemap(freearg) struct entry **OUTPUT {
// int i=0;
// while((*$1)[i].addr_str != NULL) {
// free((*$1)[i].addr_str);
// ++i;
// }
// free(*$1);
//}
// Apply the OUTPUT typemap set to the "results" parameter.
%apply struct entry **OUTPUT {struct entry** results};
// Finally, define the function for SWIG
int get_list(const char* string, struct entry** results);
Demo (Python 3.3):
>>> import example
>>> example.get_list('abcd')
[0, [(b'hello', b'ab'), (b'there', b'cd')]]
Hope that helps.
Edit:
I commented out the tuple creation and just save the entry* proxy instead. This doesn't leak Python objects, but the memory malloced for use by an entry* is not freed. I'm not sure where to put that, although I'm experimenting with %extend.