We Keep Coding

How to use basic string operations with PyObject* strings?

I would not like to keep converting every Python String Object from PyObject* to std::string or char* with PyUnicode_DecodeUTF8 and PyUnicode_AsUTF8 because it is an expensive operation.
On my last question How to extend/reuse Python C Extensions/API implementation?, I managed to use the Python open function, to directly give me a PyObject* string. Once doing that, it is very simple to send the string back to the Python program because I can just pass its PyObject* pointer back, instead of doing a full char-by-char copy as PyUnicode_DecodeUTF8 or PyUnicode_AsUTF8 do.
On the regex implementation of CPython API, I can find a function like this:
static void* getstring(PyObject* string, Py_ssize_t* p_length,
int* p_isbytes, int* p_charsize,
Py_buffer *view)
{
/* given a python object, return a data pointer, a length (in
characters), and a character size. return NULL if the object
is not a string (or not compatible) */
/* Unicode objects do not support the buffer API. So, get the data directly. */
if (PyUnicode_Check(string)) {
if (PyUnicode_READY(string) == -1)
return NULL;
*p_length = PyUnicode_GET_LENGTH(string);
*p_charsize = PyUnicode_KIND(string);
*p_isbytes = 0;
return PyUnicode_DATA(string);
}
/* get pointer to byte string buffer */
if (PyObject_GetBuffer(string, view, PyBUF_SIMPLE) != 0) {
PyErr_SetString(PyExc_TypeError, "expected string or bytes-like object");
return NULL;
}
*p_length = view->len;
*p_charsize = 1;
*p_isbytes = 1;
if (view->buf == NULL) {
PyErr_SetString(PyExc_ValueError, "Buffer is NULL");
PyBuffer_Release(view);
view->buf = NULL;
return NULL;
}
return view->buf;
}
It does not seem to be using PyUnicode_DecodeUTF8 or PyUnicode_AsUTF8 to work with the PyObject* coming from the Python Interpreter.
How can I use basic string operations with PyObject* strings without conversion then to std::string or char*?
I would call basic operations the following examples: (Just for exemplifying, I am using Py_BuildValue to build a PyObject* string from a string as a char* or std::string)
static PyObject* PyFastFile_do_concatenation(PyFastFile* self)
{
PyObject* hello = Py_BuildValue( "s", "Hello" );
PyObject* word = Py_BuildValue( "s", "word" );
// I am just guessing the `->value` property
PyObject* hello_world = hello->value + word->value;
hello_world; // return the `PyObject*` string `Hello word`
}
static PyObject* PyFastFile_do_substring(PyFastFile* self)
{
PyObject* hello = Py_BuildValue( "s", "Hello word" );
PyObject* hello_world = hello->value[5:];
hello_world; // return the `PyObject*` string `word`
}
static PyObject* PyFastFile_do_contains(PyFastFile* self)
{
PyObject* hello = Py_BuildValue( "s", "Hello word" );
if( "word" in hello->value ) {
Py_BuildValue( "p", true ); // return the `PyObject*` boolean `true`
}
Py_BuildValue( "p", false ); // return the `PyObject*` boolean `false`
}

Python c++ wrapper : Convert multi-type struct to it's python representation (preferable dictionary)

I've chosen setuptools to use my C/C++ code from python scripts.
One of the phases when building such wrapper is to convert the C/C++ return value into python object.
So far I was able to convert simple primitive values and list of primitive values. However, I wish to extend it to multi-value struct, as shown in the example below.
My main challenge right now is how do I create the python struct representation (PyObject* ret = PyList_New(...);) and I do I set it's values properly with the different types.
I tried to create list of items from the same types (such as std::vector<float>) and manage to set the values properly using Py_BuildValue and PyList_SetItem, but I'm still struggling with the multi types...
typedef struct _fileParams
{
bool valid;
int index;
std::string key;
std::value value;
} fileParams;
FileDataBase * db;
static PyObject *searchFileInDB(PyObject *self, PyObject *args)
{
if (db == NULL)
{
PyErr_SetString(PyExc_RuntimeError, "DB could not be initialized");
return NULL;
}
char* fileName = NULL;
int fileNameSize = 0;
PyArg_ParseTuple(args, "s#", &fileName, &fileNameSize);
try
{
fileParams p;
bool res = db->lookup(fileName, fileNameSize, p);
PyObject* ret = PyList_New(...);
if (res)
{
PyObject* r1 = Py_BuildValue("b", p.valid);
PyList_SetItem(ret, 0, r1);
PyObject* r2 = Py_BuildValue("i", p.index);
PyList_SetItem(ret, 1, r2);
PyObject* r1 = Py_BuildValue("s", p.key);
PyList_SetItem(ret, 2, r3);
PyObject* r1 = Py_BuildValue("s", p.value);
PyList_SetItem(ret, 3, r4);
}
return ret;
} catch (...) {
PyErr_SetString(PyExc_RuntimeError, "failed with C exception");
return NULL;
}
}

You probably want to look into the Dictionary Object: Dictionary Objects
I'm guessing you'd want to set values with PyDict_SetItemString() as per that doc.
HTH

SWIG: How to return a struct using %apply? 'No typemaps defined' warning

I currently have a function that uses a struct as a buffer to return some information, like so:
int example_reader(int code, void* return_struct);
My goal is to make it so that when I wrap this function using SWIG so that it can be used in Python, I will return the struct along with the function's regular return value. Thus far, I have been doing so using the %apply command like so:
%apply struct ret_struct *OUTPUT {void* return_struct};
However, when I add the above line to my .i file and try to run SWIG, I get the following warning:
"warning 453: Can't apply (struct ret_struct *OUTPUT. No typemaps are defined"
I believe I'm including the .h file that defines the struct I'm trying to return, so I've had trouble pinpointing the issue. Please correct me if the issue seems to involve the improper inclusion of the struct. I've tried reading through the SWIG documentation as well as other Stack Overflow posts to get some inkling of what the problem might be, but I haven't been able to figure it out thus far. The problem is made slightly trickier because I am trying to return a void pointer to a struct, and the code I'm trying to wrap could have multiple kinds of structs for me to return. What would be a wise way of handling the return of this struct? Thank you!

I have given here a full C example, where an interface is used for returning a struct to the target language together with a return value. In this way you can make a proper interface, where no implementation is given in the header. That is no default implementation of a virtual destructor. If you don't want to use an interface, you can let SWIG and Python know how data are represented.
Interface header: foo.h
typedef struct _Foo Foo;
int foo_new(Foo **obj);
int foo_free(Foo *obj);
int foo_get_value_a(Foo *obj, int *result);
int foo_set_value_a(Foo *obj, int value);
int foo_get_value_b(Foo *obj, char **result);
int foo_set_value_b(Foo *obj, char *value);
SWIG interface: foo.i
%module foo
%{
#include "foo.h"
%}
%include "typemaps.i"
%typemap(in, numinputs=0) Foo ** (Foo *temp) {
$1 = &temp;
}
%typemap(argout) Foo ** {
PyObject* temp = NULL;
if (!PyList_Check($result)) {
temp = $result;
$result = PyList_New(1);
PyList_SetItem($result, 0, temp);
}
temp = SWIG_NewPointerObj(*$1, SWIGTYPE_p__Foo, SWIG_POINTER_NEW);
PyList_Append($result, temp);
Py_DECREF(temp);
}
%delobject foo_free; // Protect for double deletion
struct _Foo {};
%extend _Foo {
~_Foo() {
foo_free($self);
}
};
%ignore _Foo;
Some implementation of the interface: foo.c
%include "foo.h"
#include "foo.h"
#include "stdlib.h"
#include "string.h"
struct FooImpl {
char* c;
int i;
};
int foo_new(Foo **obj)
{
struct FooImpl* f = (struct FooImpl*) malloc(sizeof(struct FooImpl));
f->c = NULL;
*obj = (Foo*) f;
return 0;
}
int foo_free(Foo *obj)
{
struct FooImpl* impl = (struct FooImpl*) obj;
if (impl) {
if (impl->c) {
free(impl->c);
impl->c = NULL;
}
}
return 0;
}
int foo_get_value_a(Foo *obj, int *result)
{
struct FooImpl* impl = (struct FooImpl*) obj;
*result = impl->i;
return 0;
}
int foo_set_value_a(Foo *obj, int value)
{
struct FooImpl* impl = (struct FooImpl*) obj;
impl->i = value;
return 0;
}
int foo_get_value_b(Foo *obj, char **result)
{
struct FooImpl* impl = (struct FooImpl*) obj;
*result = impl->c;
return 0;
}
int foo_set_value_b(Foo *obj, char *value)
{
struct FooImpl* impl = (struct FooImpl*) obj;
int len = strlen(value);
if (impl->c) {
free(impl->c);
}
impl->c = (char*)malloc(len+1);
strcpy(impl->c,value);
return 0;
}
Script for building
#!/usr/bin/env python
from distutils.core import setup, Extension
import os
os.environ['CC'] = 'gcc';
setup(name='foo',
version='1.0',
ext_modules =[Extension('_foo',
['foo.i','foo.c'])])
Usage:
import foo
OK, f = foo.foo_new()
OK = foo.foo_set_value_b(f, 'Hello world!')
OK = foo.foo_free(f)
OK, f = foo.foo_new()
# Test safe to double delete
del f

What do I use instead of PyString_AsString when loading a python module in 3.3

I am trying to load a function from python in one of my c++ programs using this function
char * pyFunction(void)
{
char *my_result = 0;
PyObject *module = 0;
PyObject *result = 0;
PyObject *module_dict = 0;
PyObject *func = 0;
PyObject *pArgs = 0;
module = PyImport_ImportModule("testPython");
if (module == 0)
{
PyErr_Print();
printf("Couldn't find python module");
}
module_dict = PyModule_GetDict(module);
func = PyDict_GetItemString(module_dict, "helloWorld");
result = PyEval_CallObject(func, NULL);
//my_result = PyString_AsString(result);
my_result = strdup(my_result);
return my_result;
}
What should I use instead of PyString_AsString?

Depending on the returned type from your helloWorld() function, it
could vary so it's best to check it.
To handle a returned str (Python 2 unicode), then you will need to
encode it. The encoding will depend on your use case but I'm going to
use UTF-8:
if (PyUnicode_Check(result)) {
PyObject * temp_bytes = PyUnicode_AsEncodedString(result, "UTF-8", "strict"); // Owned reference
if (temp_bytes != NULL) {
my_result = PyBytes_AS_STRING(temp_bytes); // Borrowed pointer
my_result = strdup(my_result);
Py_DECREF(temp_bytes);
} else {
// TODO: Handle encoding error.
}
}
To handle a returned bytes (Python 2 str), then you can get the
string directly:
if (PyBytes_Check(result)) {
my_result = PyBytes_AS_STRING(result); // Borrowed pointer
my_result = strdup(my_result);
}
Also, if you receive a non-string object, you can convert it
using PyObject_Repr(), PyObject_ASCII(), PyObject_Str(), or PyObject_Bytes().
So in the end you probably want something like:
if (PyUnicode_Check(result)) {
// Convert string to bytes.
// strdup() bytes into my_result.
} else if (PyBytes_Check(result)) {
// strdup() bytes into my_result.
} else {
// Convert into your favorite string representation.
// Convert string to bytes if it is not already.
// strdup() bytes into my_result.
}

If you were using PyString_AsString to get a C-compatible const char* representation of what you know to be a str object, then in Py3 you can simply use PyUnicode_AsUTF8:
func = PyDict_GetItemString(module_dict, "helloWorld");
result = PyEval_CallObject(func, NULL);
const char* my_result = PyUnicode_AsUTF8(result);

extending global c variable array with swig and python

I have a global variable array in c that I'd like to pull into python. And I'm having difficulties with varout typemap:
/* example.c */
int foo[] = {0, 1};
And here is the very vanilla interface:
/* example.i */
%module example
%{
extern int foo[2];
%}
%typemap(varout) int foo[] {
int i;
//$1, $1_dim0, $1_dim1
$result = PyList_New($1_dim0);
for (i = 0; i < $1_dim0; i++) {
PyObject *o = PyInt_FromLong((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
%include "example.c"
When I try to build it with the following SConstruct:
import distutils.sysconfig
env = Environment(SWIGFLAGS='-python -shadow -Wall'.split(),
CPPPATH=[distutils.sysconfig.get_python_inc()],
SHLIBPREFIX="")
env.SharedLibrary('_example.so', ['example.c', 'example.i'])
$1_dim0 special variable is not populated, resulting in the following non-compilable code in example_wrap.c:
SWIGINTERN PyObject *Swig_var_foo_get(void) {
PyObject *pyobj = 0;
{
int i;
//foo, , foo_dim1
pyobj = PyList_New();
for (i = 0; i < ; i++) {
PyObject *o = PyInt_FromLong((double) foo[i]);
PyList_SetItem(pyobj,i,o);
}
}
return pyobj;
}
So clearly the typemap match has happened, but dimensionality of array is missing. What am I missing? Hard coding the dimension does works.
In general, is there any way to extend global cvar variables with swig?
$ swig -version
SWIG Version 2.0.4
Compiled with g++ [i686-pc-linux-gnu]
Configured options: +pcre
Please see http://www.swig.org for reporting bugs and further information

You're almost there with your varout typemap. You need to make two minor changes:
You need to add the size ANY to the int foo[] typemap:
%typemap(varout) int foo[ANY] {
int i;
//$1, $1_dim0, $1_dim1
$result = PyList_New($1_dim0);
for (i = 0; i < $1_dim0; i++) {
PyObject *o = PyInt_FromLong((double) $1[i]);
PyList_SetItem($result,i,o);
}
}
This makes sure your typemap is a match for arrays of (any) known size, not just equivalent to int *foo.
You need to modify example.c to make the size of foo clearer. It's legal and correct C as it stands but tricky to deduce the size of the array unless you happen to be a complete C compiler. Changing it to:
int foo[2] = {0, 1};
is sufficient to make sure that it matches the varout typemap.
With those two changes the generated code works as you'd hope:
SWIGINTERN PyObject *Swig_var_foo_get(void) {
PyObject *pyobj = 0;
{
int i;
//foo, 2, foo_dim1
pyobj = PyList_New(2);
for (i = 0; i < 2; i++) {
PyObject *o = PyInt_FromLong((double) foo[i]);
PyList_SetItem(pyobj,i,o);
}
}
return pyobj;
}
is what gets generated on my machine with those changes.

For those like me who ponders what to do with arrays of non-simple types -- here is one way to do it:
The non-simple type:
typedef struct {
int a;
float b;
} Foo;
and a global array:
extern Foo *foov[40];
%typemap(varout) Foo *foov[ANY] {
int i;
$result = PyList_New($1_dim0);
for (i = 0; i < $1_dim0; i++) {
PyObject *o = SWIG_NewPointerObj($1[i], SWIGTYPE_p_Foo, 0);
PyList_SetItem($result, i, o);
}
}
Just shared this since it took me forever to find out, and this article helped. Just needed to find out how to allocate the SWIG version of my non-simple type -- found that buried here:
http://www.swig.org/Doc2.0/Python.html#Python_nn64