I'm trying to call a C++ function from my Python code, if I pass a Boolean or an int it works perfectly, but if I send a string, it only prints the first character.
I am compiling with:
g++ -c -fPIC foo.cpp -Wextra -Wall -o foo.o
g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
python3 fooWrapper.py
Here is the C++ and Python code:
Python:
from ctypes import cdll
lib = cdll.LoadLibrary("./libfoo.so")
lib.Foo_bar("hello")
c++:
#include <iostream>
#include <string>
#include <unistd.h>
void bar(char* string){
printf("%s", string);
}
extern "C" {
void Foo_bar(char* aString){
bar(aString);
}
}
I'm aware of the Boost Library, but i couldn't manage to download it, and this way works well excepts for strings.
Thank you for your help
The problem is that strings are passed as pointers to wchar_t wide characters in Python 3. And in little-endian system your string can be coded in binary as
"h\0\0\0e\0\0\0l\0\0\0l\0\0\0o\0\0\0\0\0\0\0"
Which, when printed with %s will stop at the first null terminator.
For UTF-8-encoded byte strings (char *) you need a bytes object. For example:
lib.Foo_bar("hello".encode())
or use bytes literals:
lib.Foo_bar(b"hello")
Even better if you had specified the correct argument types:
from ctypes import cdll, c_char_p
foo_bar = cdll.LoadLibrary("./libfoo.so").Foo_bar
foo_bar.argtypes = [c_char_p]
foo_bar(b"hello\n")
foo_bar("hello\n")
when run will output the following:
hello
Traceback (most recent call last):
File "foo.py", line 5, in <module>
foo_bar("hello\n")
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
i.e. the latter call that uses a string instead of bytes would throw.
You may also process Python3 strings in C++ directly using the wchar_t type. In that case, you need to do any necessary conversions in C++ like this:
#include <iostream>
#include <locale>
#include <codecvt>
void bar(wchar_t const* aString)
{
// Kudos: https://stackoverflow.com/a/18374698
std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t> convert;
std::cout << convert.to_bytes(aString) << std::endl;
}
extern "C" {
void Foo_bar(wchar_t const* aString)
{
bar(aString);
}
}
You will lose Python2 compatibility, however.
Related
Creating a python module with swig is the easy part.
But what if this module has to interact with my C++ application that imports it from its embedded python instance?
For example #Flexo has made a pretty good example on how to expose the python module to the application here: How can I implement a C++ class in Python, to be called by C++?
All I want is that the application and the module are able to interact with each other like sharing variables and calling callbacks and so on so I found this answer a bit too complex for the task.
I couldn't really find much else on how to achieve this and the swig documentation doesn't explain this either ( I guess it has more to do with python and not with SWIG in particular that's why it is not mentioned in the SWIG documentation ).
I then found out that I can also just include the *_wrap.cxx file created by SWIG in my C++ application and achieve the same results (see code below)
I just started with SWIG and Python thats why my question now is, is there a better way than mine or an official way to achieve the same result? Have I overlooked something?
Can I import my swig module before initializing the python instance without having to include the Swigtest_wrap.cxx file?
That's how I made it:
1. Files
following three files contain everything needed for my python module. Containing only a class which I later want to use a as base class for a new python class.
Swigtest.h - header for python module
#ifndef SWIGTEST_H
#define SWIGTEST_H
class Callback
{
public:
Callback(){}
virtual ~Callback() {}
// we want to override this function in python
virtual void Exec() {}
static void callFunc();
static void setCallback(Callback* callback);
private:
static Callback* m_callback;
};
#endif // SWIGTEST_H
Swigtest.cpp - for python module
Callback* Callback::m_callback = nullptr;
void Callback::callFunc()
{
if(m_callback != nullptr)
{
m_callback->Exec();
return;
}
std::cout << "callback not set" << std::endl;
}
void Callback::setCallback(Callback* callback)
{
m_callback = callback;
}
Swigtest.i - interface file for SWIG
the only thing to note here is the activation of the "director" feature
%module(directors="1") mymodule
// We need to include Swigtest.h in the SWIG generated C++ file
%{
#include <iostream>
#include "Swigtest.h"
%}
// Enable cross-language polymorphism in the SWIG wrapper.
%feature("director") Callback;
// Tell swig to wrap everything in Swigtest.h
%include "Swigtest.h"
Switest.py
this python script creates a new Class derived from our Callback class and sets the callback to
this class.
import mymodule
# lets create a new class derived from callback
class MyPyCallbackFromC(mymodule.Callback):
def Exec(self):
print("this class was created in python - It worked!")
callback = mymodule.Callback()
mycallback = MyPyCallbackFromC()
# set callback to this new class
callback.setCallback(mycallback)
# now lets call it from our c++ application
main.cpp
#include <Python.h>
#include <iostream>
#include "Swigtest.h"
// we include this file to be able to append the python
// module table with our own swig wrapped module
#include "Swigtest_wrap.cxx"
int main()
{
// *_wrap.cxx has to be included for PyInit__mymodule;
// must be added before the Python instance is initialized!
PyImport_AppendInittab("_mymodule", PyInit__mymodule);
Py_Initialize();
// create Callback class
Callback* callback = new Callback();
// call Exec() function of linked class
// should return error because no class is set as Callback yet
callback->callFunc();
// execute our script which sets a new class as our callback
FILE* fp;
const char* filename;
filename = "Swigtest.py";
fp = _Py_fopen(filename, "r");
PyRun_SimpleFile(fp, filename);
// if we now call the callback from our application
// the Exec function that was defined in our python script should be executed
callback->callFunc();
delete callback;
Py_Finalize();
}
2. Building
Building the module
#swig
swig -c++ -python Swigtest.i
#compile
g++ -fpic -c Swigtest.cpp Swigtest_wrap.cxx -I/pathTo/python
#build
g++ -Wall -Wextra -shared Swigtest.o Swigtest_wrap.o -o _mymodule.so
Building the application
#compile
g++ -fpic -c Swigtest.cpp
g++ -fpic -c main.cpp -I/pathTo/python
#build
g++ main.o Swigtest.o -o libmyapp.so -I/pathTo/python -lpython3
3. Execution:
starting the application from the terminal
$ ./libmyapp.so
output
callback not set
this class was created in python - It worked!
That's it.
I have recently bumped into a swig limitation related to the size of C++ std::string.
I have some C++ code returning a pair. I noticed that when the size of the string in the pair is smaller that 2*1024*1024*1024-64 (2GB) the pair properly returns and the string is mapped to a python native string. However if the string is greater than 2GB, then in python the string is not mapped anymore to a native python string. For example using the code below, and mapping through swig to python you can reproduce my error.
Environment:: SWIG Version 3.0.8, Ubuntu 16.04.3 LTS, g++ 5.4.0; Python 2.7.12
/////////// bridge.h
#include <vector>
#include <utility>
#include <string>
#include <iostream>
#include <fstream>
using namespace std;
pair<int, string> large_string(long sz);
long size_pstring(pair<int,string>& p);
void print_pstring(pair<int,string>& p);
string save_pstring(pair<int,string>& p);
//////////bridge.cc
#include "bridge.h"
pair<int, string> large_string(long sz){
pair<int, string> pis;
pis.first=20;
pis.second=string(sz,'A');
return pis;
}
long size_pstring(pair<int,string>& p){
return p.second.size();
}
void print_pstring(pair<int,string>& p){
cout<<"PSTRING: first="<<p.first<<" second.SZ="<<p.second.size()<<"\n";
cout<<"First 100 chars: \n"<<p.second.substr(0,100)<<"\n";
}
string save_pstring(pair<int,string>& p){
string fname="aloe.txt";
std::ofstream ofile(fname.c_str());
ofile<<p.second;
ofile.close();
return fname;
}
////////// bridge.i
%module graphdb
%include stl.i
%include "std_vector.i"
%{
#include "bridge.h"
%}
%include "bridge.h"
namespace std {
%template(p_string) pair<int,string>;
};
//////// makefile
all:
swig -c++ -python bridge.i
g++ -std=c++11 -fpic -c bridge.cc bridge_wrap.cxx -I/usr/include/python2.7/
g++ -shared *.o -o _graphdb.so
Bellow I include a session in python showing that it is probably just a matter of how string is mapped and that most probably an int rather long is used to represent the size of string in swig bridge code.
>>> s=graphdb.large_string(12)
>>> print s
(20, 'AAAAAAAAAAAA')
>>> s=graphdb.large_string(2*1024*1024*1024)
>>> print s
(20, <Swig Object of type 'char *' at 0x7fd4205a6090>)
>>> l=graphdb.size_pstring(s)
>>> print l
2147483648
>>> fname = graphdb.save_pstring(s)
Saving the string to a file is correct and next I can load the file to a python string correctly.
So my question: does anybody know what swig config option I should change to allow large strings to be properly mapped to native python ?
--Thx
I am developing a C++ library in which SWIG is used to generate its Python wrapper. Some of my C++ files use <inittypes.h> to call PRId64 and other macros in sprintf.
I was able to compile my library with Python 2.6 and GCC 4.4.7 on Scientific Linux 6 (RHEL6 clone), but Python 2.7 and GCC 4.8.2 on Scientific Linux 7 (RHEL7 clone) generated many errors like below.
/home/oxon/libTARGET/inc/target/T2EvalBoard.h:562:145: warning: too many arguments for format [-Wformat-extra-args]
In file included from /home/oxon/libTARGET_build/src/targetPYTHON_wrap.cxx:3117:0:
/home/oxon/libTARGET/inc/target/BaseCameraModule.h: In member function ‘virtual void TARGET::BaseCameraModule::ReceiveEvent(uint32_t&, uint8_t**)’:
/home/oxon/libTARGET/inc/target/BaseCameraModule.h:211:66: error: expected ‘)’ before ‘PRIu32’
sprintf(str, "Cannot read event data. Requested length is %" PRIu32 " bytes, but only %" PRId64 " bytes were read.", length, fBytesReturned);
I know that I have to add the following lines in header files first in order to use PRId64 and other.
#define __STDC_FORMAT_MACROS
#include <inttypes.h>
But targetPYTHON_wrap.cxx, which is a source file generated by SWIG, includes <Python.h> in the beginning of the file, and so the above lines are ignored. Indeed, the following code cannot be compiled, because <Python.h> includes <inttypes.h> in it.
#include <Python.h>
#define __STDC_FORMAT_MACROS
#include <inttypes.h>
#include <stdio.h>
int main()
{
printf("Output: " PRIu32 "\n", 100);
return 0;
}
How do I use PRId64 and other macros with <Python.h> and SWIG?
In SWIG, the following adds lines to the very top of the SWIG wrapper, so it will be defined before Python.h:
%begin %{
#define __STDC_FORMAT_MACROS
#include <inttypes.h>
%}
I added -D__STDC_FORMAT_MACROS in CXX_FLAGS, but looking for a better solution if exists.
I have created a c program which requires an input (through scanf). Then I created the .so file and called that in a python script, so that when I run the script, input will be asked in the terminal. But when I run the python program, the terminal hangs.
Please note:
1. My c code
#include <stdio.h>
#include <string.h>
int open(void)
{
char input[20];
scanf("input = %s\n",&input);
printf("\n%s\n","input");
}
2. Command I used for compiling the code
gcc -c usb_comm.c
3.Creating .so file
gcc -shared -o libhello.so usb_comm.o
4.Relevant section of python program
Loading the .so file
from ctypes import cdll
mydll = cdll.LoadLibrary('/home/vineeshvs/Dropbox/wel/workspace/Atmel/libhello.so')
Calling the function defined in the c program
mydll.scanf("%c",mydll.open())
Thanks for listening :)
mydll.open() call scanf. Why do you call scanf in Python?
Just call mydll.open() only:
mydll.open()
UPDATE
#include <stdio.h>
void open(void)
{
char input[20];
printf("input = "); // If you want prompt
scanf("%s", input);
printf("\n%s\n", input);
}
I am building a R extension which has an embedded python in it.
Everything goes well now, except that the python cannot find the encoding I needed. It keeps throwing LookupError when I do something involving 'big5'. However, if I build a stand alone c++ application, the python interpreter does find the encoding and stop throwing errors.
test.cpp for normal stand alone example in c:
#include <Python.h>
int main(int argc, char* argv[]) {
Py_SetProgramName("test"); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString(
"import codecs\n"
"f = codecs.open('big5_encoded_file', encoding='big5', mode='r')"
);
Py_Finalize();
return 0;
}
testr.cpp for R extension:
#include <R.h>
#include <Rdefines.h>
#include <Python.h>
extern "C" SEXP testpy();
SEXP testpy() {
Py_SetProgramName("test"); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString(
"import codecs\n"
"f = codecs.open('big5_encoded_file', encoding='big5', mode='r')"
);
Py_Finalize();
return R_NilValue;
}
A Makefile on ubuntu 12.10:
all: test testr.so
test: test.cpp
g++ test.cpp -o test -I/usr/include/python2.7 -lpython2.7
testr.so: testr.cpp
R CMD SHLIB testr.cpp
The ./test runs normally, but Rscript -e "dyn.load('testr.so');.Call('testpy')" produces a "LookupError: unknown encoding: big5"
Thanks
-- edit --
To build the testr.so, please set:
export PKG_CXXFLAGS=-I/usr/include/python2.7
export PKG_LIBS=-lpython2.7
I notice that it is a linking issue.
I tried to import encodings.big5 in the embedded python, but the error of undefined reference occurred. The solution in http://bugs.python.org/issue4434 works for me:
before PyInitialize() I can call dlopen("libpython2.7.so", RTLD_LAZY | RTLD_GLOBAL);