I am learning about C++, swig, and numpy for a lab I am working in. It has been specified that I must use swig (no scypy), I must use numpy arrays, and I may not "bring C code into the python world" such as by using %import "std_vector" in my interface file and just having the python user create one to send in. That being said, I am trying to get a 1d numpy array (if they need more dimensions I will just flatten it) to be passed into my C code by pointer exclusively - my boss doesn't want to have to take the time to copy everything because efficiency is very important. I believe we use c++ 14, python 2.7, and the latest version of swig, and I am using numpy.i as well.
I will provide the code below that I am currently using (just trying to get a minimum viable going here) but I am pretty new, and while it does work, I am not sure that is actually passing a pointer and not copying anything like I would like. Could someone please either confirm that it is, or show me how to make it do so? Thanks x10^99
//The C++ file I am wrapping:
#ifndef _np_array_to_array_h
#define _np_array_to_array_h
using namespace std;
double getMid(double* myArray, int size){
int half = size / 2;
return myArray[half];
}
#endif
///\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\//\/\
//The interface file:
%module np_array_to_array
%{
#define SWIG_FILE_WITH_INIT
#include "np_array_to_array.h"
#include <numpy/arrayobject.h>
%}
%include "numpy.i"
%init %{
import_array();
%}
%apply (double* IN_ARRAY1, int DIM1){(double* myArray, int size)};
%include "np_array_to_array.h"
//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
//How I compile on terminal:
swig -python -c++ np_array_to_array.i
g++ -fpic -c np_array_to_array_wrap.cxx -I/usr/include/python2.7
-I/home/sean/Desktop/SerangLab/Swig/numpy/numpy/core/include/ -I/home/sean/.local/lib/python2.7/site-packages/numpy/core/include/numpy/
g++ -shared np_array_to_array_wrap.o -o _np_array_to_array.so
This will compile and run, and create a successfully working python module that I import with (when in the same directory) "from np_array_to_array import *" and I can successfully run the getMid method, passing in a numpyArray and getting a double back out. As stated above, I am simply not sure if this is actually pass by pointer (not making any copies) or not, as I have not found anything that says one way or the other. Could someone please tell me, and if it is not, explain how one would do this? I believe it should be possible as I think numpy array uses c types and stores memory contiguously like c does.
You can investigate this fairly easily in the code that SWIG generates. There are two parts to this - a .py file and a .cxx file. In each of these there's some code generate for your getMid() function. The Python code just directly passes everything into the C++ code, which on my system ended up looking like this:
SWIGINTERN PyObject *_wrap_getMid(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
double *arg1 = (double *) 0 ;
int arg2 ;
PyArrayObject *array1 = NULL ;
int is_new_object1 = 0 ;
PyObject * obj0 = 0 ;
double result;
if (!PyArg_ParseTuple(args,(char *)"O:getMid",&obj0)) SWIG_fail;
{
npy_intp size[1] = {
-1
};
array1 = obj_to_array_contiguous_force_conversion(obj0, NPY_DOUBLE,
&is_new_object1);
if (!array1 || !require_dimensions(array1, 1) ||
!require_size(array1, size, 1)) SWIG_fail;
arg1 = (double*) array_data(array1);
arg2 = (int) array_size(array1,0);
}
result = (double)getMid(arg1,arg2);
resultobj = SWIG_From_double(static_cast< double >(result));
{
if (is_new_object1 && array1)
{
Py_DECREF(array1);
}
}
return resultobj;
fail:
...
It won't change much between SWIG and Python versions, although some of the SWIG options will change it a little bit.
The important point from the perspective of your question though seems to be the call to obj_to_array_contiguous_force_conversion. It has an argument that's used as an output parameter to indicate if a new object was allocated. If that ends up being set to true then after the call an object ends up getting released also.
From that alone it's pretty safe to conclude that the answer to your question is that it depends what input you pass to the function. If it already satisfies the constraints (i.e. it's contiguous) then you won't end up making a copy. Otherwise it will, since your C++ function requires a contiguous region.
It should also be a safe bet that if you use any of the numpy double types then you'll end up meeting this requirement and not making a copy, but for other data types that's less likely unless you've gone to a bit of effort.
According to Swig docs and the marvelous explanation at SWIG in typemap works, but argout does not by #Flexo, the argout typemap turns reference arguments into return values in Python.
I have a scenario, in which I pass a dict, which then is converted to an unordered_map in typemap(in), which then gets populated in the C++ lib. Stepping through the code, I can see the mapping changed after it returned from C++, so I wonder why there is not a possibility to just convert the unordered_map back in place in to the dict that was passed. Or is it possible by now and I'm just overlooking something?
Thanks!
I am a little confused as to what exactly you are asking, but my understanding is:
You have an "in" typemap to convert a Python dict to a C++ unordered_map for some function argument.
The function then modifies the unordered_map.
After completion of the function, you want the Python dict updated to the current unordered_map, and are somehow having trouble with this step.
Since you know how to convert a dict to an unordered_map, I assume you basically do know how to convert the unordered_map back to the dict using the Python C-API, but are somehow unsure into which SWIG typemap to put the code.
So, under these assumptions, I'll try to help:
"the argout typemap turns reference arguments into return values in Python". Not really, although it is mostly used for this purpose. An "argout" typemap simply supplies code to deal with some function argument (internally referred to as $1) that is inserted into the wrapper code after the C++ function is called. Compare this with an "in" typemap that supplies code to convert the supplied Python argument $input to a C++ argument $1, which is obviously inserted into the wrapper code before the C++ function is called.
The original passed Python dict can be referred to in the "argout" typemap as $input, and the modified unordered_map as $1 (see the SWIG docs linked above).
Therefore, all you need to do is write an "argout" typemap for the same argument signature as the "in" typemap that you already have, and insert the code (using the Python C-API) to update the contents of the Python dict ($input) to reflect the contents of the unordered_map ($1).
Note that this is different from the classical use of "argout" typemaps, which would typically convert the $1 back to a new Python dict and append this to the Python return object, which you can refer to by $result.
I hope this helps. If you are still stuck at some point, please edit your question to make clear at which point you are having trouble.
I am well aware of that the user has already solved his issue, but here goes a solution. Some validation of inputs may be introduced to avoid non-string values of the input dictionary.
Header file
// File: test.h
#pragma once
#include <iostream>
#include <string>
#include <unordered_map>
void method(std::unordered_map<std::string, std::string>* inout) {
for( const auto& n : (*inout) ) {
std::cout << "Key:[" << n.first << "] Value:[" << n.second << "]\n";
}
(*inout)["BLACK"] = "#000000";
};
Interface file
// File : dictmap.i
%module dictmap
%{
#include "test.h"
%}
%include "typemaps.i"
%typemap(in) std::unordered_map<std::string, std::string>* (std::unordered_map<std::string, std::string> temp) {
PyObject *key, *value;
Py_ssize_t pos = 0;
$1 = &temp;
temp = std::unordered_map<std::string, std::string>();
while (PyDict_Next($input, &pos, &key, &value)) {
(*$1)[PyString_AsString(key)] = std::string(PyString_AsString(value));
}
}
%typemap(argout) std::unordered_map<std::string, std::string>* {
$result = PyDict_New();
for( const auto& n : *$1) {
PyDict_SetItemString($result, n.first.c_str(),
PyString_FromString(n.second.c_str()));
}
}
%include "test.h"
Test
import dictmap
out = dictmap.method({'WHITE' : '#FFFFFF'})
Output is an updated dictionary
In[2]: out
Out[3] : {'BLACK': '#000000', 'WHITE': '#FFFFFF'}
The following from c structure has been wrapped with swig so that I can access the data, acquired from another swig wrapped program, in python.
struct RadarParm {
struct {
char major;
char minor;
} revision;
struct {
char code;
char *time;
char *command;
} origin;
int16 cp;
int16 stid;
int16 *pulse;
int16 *lag[2];
char *combf;
};
My main issue is with int16 *lag[2]. When trying to read the data in my python code with:
p_data['lag'] = prm.lag
Where prm is of the type RadarParm returned from the rtserver.c file thru the method struct RadarParm* getRadarParm() and p_data is a python dictionary that stores every part of RadarParm.
Here is my current output when accessing prm.lag:
<Swig Object of type 'int16 **' at 0x1fd55d0>
I have found that there seems to be a numpy.i file that handles arrays and int16 data types. From my limited understanding I would think that this would allow me to access this data. If I could implement it correctly.
Here is what I have so far:
%init %{
import_array();
%}
%apply (int16** ARGOUTVIEW_ARRAY1, DIM_TYPE* DIM1) {(int16* lag,2)};
When compiling this I get:
Error: Syntax error in input(1).
Any ideas on how to make this work?
In this instance numpy.i did not work. The main problem was numpy.i does not support (out) typemaps which in this case was precisely what I needed.
Instead using the command -debug-tmsearch when compiling the swig rtserver.i file. Using the output from the command I was able to determine the defined swig type for RadarParm::lag and what it should be. Here is the code that works.
%typemap(out) int16*[2] {
int i;
$result = PyList_New(2);
for (i = 0; i < 2; i++) {
PyObject *o = PyInt_FromLong($1[i]);
PyList_SetItem($result,i,o);
}
}
%apply int16*{lag[2]};
Most of this was taken from this answer
So I'm porting a program from Python to c++ and I have this block of code here:
opcodes = [
[0x1,'0x1',['b','b',]],
[0x2,'call',['d',]],
[0x3,'0x3',['w',]],
[0x4,'0x4-return',[]],
[0x5,'0x5',[]],
[0x6,'0x6-condjump',['d']],
[0x7,'0x7-condjump',['d']],
[0x8,'0x8',[]],
[0x9,'0x9',[]],
[0xa,'0xa',['d',]],
[0xb,'0xb',['w',]],
[0xc,'0xc',['b',]],
[0xe,'string',['str']],
[0xf,'0xf',['w',]],
[0x10,'0x10',['b',]],
[0x11,'0x11',['w',]],
[0x12,'0x12',['b',]],
[0x14,'0x14',[]],
[0x15,'0x15',['w',]],
[0x16,'0x16',['b',]],
[0x17,'0x17',['w',]],
[0x18,'0x18',['b']],
[0x19,'0x19',[]],
[0x1a,'0x1a',[]],
]
I was wondering what would be the best way to convert this to a C++ array. I'm not too familiar with python sorry and heard this is called a nested list?
Thanks in advance for all answers, this is probably the biggest hurdle of the python code.
This is C++11:
#include <string>
#include <vector>
struct OpCode {
int code;
const char* str;
std::vector<const char*> extras;
};
OpCode opcodes[] = {
{0x1,"0x1",{"b","b",}},
{0x2,"call",{"d",}},
{0x3,"0x3",{"w",}},
{0x4,"0x4-return",{}},
{0x5,"0x5",{}},
{0x6,"0x6-condjump",{"d"}},
{0x7,"0x7-condjump",{"d"}},
{0x8,"0x8",{}},
{0x9,"0x9",{}},
{0xa,"0xa",{"d",}},
{0xb,"0xb",{"w",}},
{0xc,"0xc",{"b",}},
{0xe,"string",{"str"}},
{0xf,"0xf",{"w",}},
{0x10,"0x10",{"b",}},
{0x11,"0x11",{"w",}},
{0x12,"0x12",{"b",}},
{0x14,"0x14",{}},
{0x15,"0x15",{"w",}},
{0x16,"0x16",{"b",}},
{0x17,"0x17",{"w",}},
{0x18,"0x18",{"b"}},
{0x19,"0x19",{}},
{0x1a,"0x1a",{}},
};
Make a struct or a class that will hold [0x1,'0x1',['b','b',]], something like
struct shmizzle {
int forpult;
char *yorgole;
char **flubbo;
};
Probably easier with a class, since you get to initialise it more easily. I'm not a C++ expert though.