We Keep Coding

Return array in Python from shared library built in Go

I have the following Go file:
package main
import "C"
//export stringList
func stringList(a, b string) []string {
var parserString[] string
parserString = append(parserString, a)
parserString = append(parserString, b)
return parserString
}
func main() {}
which I then build using go build -o stringlist.so -buildmode=c-shared stringlist.go
Then, I try to call it in Python using
from ctypes import *
lib = cdll.LoadLibrary("./stringlist.so")
lib.stringList.argtypes = [c_wchar_p, c_wchar_p]
lib.stringList("hello", "world")
but receive the error
panic: runtime error: cgo result has Go pointer
goroutine 17 [running, locked to thread]:
main._cgoexpwrap_0de9d34d4a40_stringList.func1(0xc00005ce90)
_cgo_gotypes.go:46 +0x5c
main._cgoexpwrap_0de9d34d4a40_stringList(0x7f9fdfa8eec0, 0x7f9fddaffab0, 0x7f9fdd8f46cf, 0x7ffc501ce460, 0xc00000e040, 0x2, 0x2)
_cgo_gotypes.go:48 +0x11b
Aborted (core dumped)
What is the problem? How can I fix it? Is stringList not returning a proper type?

As mentioned in the comments, you need to convert to C types before returning any value, this means that your function should return **C.char. The Go documentation has functions that convert from *C.chart to C.GoString.
Cgo Wiki
CGO Documentation
Here's a simple albeit probably leaky function; in Cgo you are responsible of
memory management, this particular function allocates the result memory but if you know before calling the Cgo function how much memory or space you need it's possible to preallocate it.
All that it does is convert to upper case the provided strings.
package main
/*
#include <stdlib.h>
*/
import "C"
import (
"fmt"
"strings"
"unsafe"
)
//export stringList
func stringList(a, b *C.char) **C.char {
// convert to Go strings
goStringA := C.GoString(a)
goStringB := C.GoString(b)
//... do something with the strings ...
fmt.Println("CGO: ", goStringA)
fmt.Println("CGO: ", goStringB)
goStringA = strings.ToUpper(goStringA)
goStringB = strings.ToUpper(goStringB)
// Convert back to C strings
// This strings _WILL NOT_ be garbage collect
// it may be that we want to free them.
// https://github.com/golang/go/wiki/cgo#go-strings-and-c-strings
ra := C.CString(goStringA)
rb := C.CString(goStringB)
// Allocate memory for our result pointer
resultMem := C.malloc(C.size_t(2) * C.size_t(unsafe.Sizeof(uintptr(0))))
// Assign to the results var
// https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices
result := (*[1<<30 - 1]*C.char)(resultMem)
(*result)[0] = ra
(*result)[1] = rb
return (**C.char)(resultMem)
}
func main() {}
Because I'm in a macOS computer I have to build it like so:
go build -o stringlist.dylib -buildmode=c-shared main.go
Then I can run it with the following Python code:
#!/usr/bin/env python3
from ctypes import *
lib = cdll.LoadLibrary('./stringlist.dylib')
lib.stringList.argtypes = [c_char_p, c_char_p]
lib.stringList.restype = POINTER(c_char_p)
result = lib.stringList("hello".encode(), "world".encode())
for word in result:
if word:
print(word.decode('utf-8'))
NOTE: The python code produces a segfault; I'm guessing that it's because I'm not using correctly the ctypes module as I'm not that familiar with. However the following C code does not segfault.
#include <stdlib.h>
#include "stringlist.h"
#include <dlfcn.h>
int main(int n, char **args) {
char lib_path[1000];
sprintf(lib_path, "%s/stringlist.dylib", args[1]);
void *handle = dlopen(lib_path, RTLD_LAZY);
char** (*stringList)(char*, char*) = dlsym(handle, "stringList");
const char *a = "hello";
const char *b = "world";
char **result = stringList(a, b);
for (size_t i = 0; i< 2; i++) {
printf("%s\n", result[i]);
}
return 0;
}
Build it like this:
clang main.c -o crun && ./crun

How to properly release Python C API GIL from main thread

I'm trying to embed Python in a C++ multithreaded program.
What I do is calling two statistical functions from the Python C API to perform the Two Sample Kolmogorov-Smirnov Test and the Two Sample Anderson-Darling Test on some data that I collect. So I'm just embedding Python in my code, I'm not extending it or using my own Python functions.
I recently found out that in order to run a multithreaded program that uses the Python C API you need to handle properly the Global Interpreter Lock (GIL) and when ever you use a Python C API function you need to acquire the GIL and then release it when you're done using the API functions.
The thing that I still don't understand is how to properly release the GIL from the main thread in order to let the others execute the Python code.
I tried this (option 1):
int main(int argc, const char * argv[]) {
int n = 4;
std::thread threads[n];
Py_Initialize();
PyEval_InitThreads();
PyEval_SaveThread();
for (int i = 0; i < n; i++) {
threads[i] = std::thread(exec, i);
}
for (int i = 0; i < n; i++) {
threads[i].join();
}
Py_Finalize();
return 0;
}
But it gives me a segmentation fault when calling Py_Finalize().
So I tried this (option 2):
int main(int argc, const char * argv[]) {
int n = 4;
std::thread threads[n];
Py_Initialize();
PyEval_InitThreads();
PyThreadState * Py_UNBLOCK_THREADS
for (int i = 0; i < n; i++) {
threads[i] = std::thread(exec, i);
}
for (int i = 0; i < n; i++) {
threads[i].join();
}
Py_BLOCK_THREADS
Py_Finalize();
return 0;
}
and this (option 3):
int main(int argc, const char * argv[]) {
int n = 4;
std::thread threads[n];
Py_Initialize();
PyEval_InitThreads();
Py_BEGIN_ALLOW_THREADS
for (int i = 0; i < n; i++) {
threads[i] = std::thread(exec, i);
}
for (int i = 0; i < n; i++) {
threads[i].join();
}
Py_END_ALLOW_THREADS
Py_Finalize();
return 0;
}
With both these last two options the code runs but ends with this error:
Exception ignored in: <module 'threading' from '/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py'>
Traceback (most recent call last):
File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 1289, in _shutdown
assert tlock.locked()
AssertionError:
EDIT:
The code that is executed by the spawned threads is this:
double limited_rand(double lower_bound, double upper_bound) {
return lower_bound + (rand() / (RAND_MAX / (upper_bound-lower_bound) ) );
}
double exec_1(std::vector<int> &left_sample, std::vector<int> &right_sample) {
PyGILState_STATE gstate = PyGILState_Ensure(); // Acquiring GIL for thread-safe usage Python C API
PyObject* scipy_stats_module = PyImport_ImportModule("scipy.stats"); // importing "scipy.stats" module
import_array();
npy_intp left_nparray_shape[] = {(npy_intp)left_sample.size()}; // Size of left nparray's first dimension
PyObject* left_sample_nparray = PyArray_SimpleNewFromData(1, left_nparray_shape, NPY_INT, &left_sample[0]); // Creating numpy array with 1 dimension, taking "dim" as a dummy, elements are integers, and the data is taken from "sample1" as a int* pointer
npy_intp right_nparray_shape[] = {(npy_intp)right_sample.size()}; // Size of right nparray's first dimension
PyObject* right_sample_nparray = PyArray_SimpleNewFromData(1, right_nparray_shape, NPY_INT, &right_sample[0]);
PyObject* ks_2samp = PyObject_GetAttrString(scipy_stats_module, "ks_2samp");
Py_DecRef(scipy_stats_module);
PyObject* ks_2samp_return_val = PyObject_CallFunctionObjArgs(ks_2samp, left_sample_nparray, right_sample_nparray, NULL);
Py_DecRef(ks_2samp);
Py_DecRef(right_sample_nparray);
Py_DecRef(left_sample_nparray);
double p_value = PyFloat_AsDouble(PyTuple_GetItem(ks_2samp_return_val, 1));
Py_DecRef(ks_2samp_return_val);
PyGILState_Release(gstate); // Releasing GIL
return p_value;
}
void initialize_c_2d_int_array(int*& c_array, unsigned long row_length_c_array, std::vector<int> &row1, std::vector<int> &row2) {
for (unsigned int i = 0; i < row_length_c_array; i++) {
c_array[i] = row1[i];
c_array[row_length_c_array + i] = row2[i];
}
}
double exec_2(std::vector<int> &left_sample, std::vector<int> &right_sample){
PyGILState_STATE gstate = PyGILState_Ensure(); // Acquiring GIL for thread-safe usage Python C API
PyObject* scipy_stats_module = PyImport_ImportModule("scipy.stats"); // importing "scipy.stats" module
// import_array();
unsigned long n_cols = std::min(left_sample.size(), right_sample.size());
int* both_samples = (int*) (malloc(2 * n_cols * sizeof(int)));
initialize_c_2d_int_array(both_samples, n_cols, left_sample, right_sample);
npy_intp dim3[] = {2, (npy_intp) n_cols};
PyObject* both_samples_nparray = PyArray_SimpleNewFromData(2, dim3, NPY_INT, both_samples);
PyObject* anderson_ksamp = PyObject_GetAttrString(scipy_stats_module, "anderson_ksamp");
Py_DecRef(scipy_stats_module);
PyObject* anderson_2samp_return_val = PyObject_CallFunctionObjArgs(anderson_ksamp, both_samples_nparray, NULL);
Py_DecRef(anderson_ksamp);
Py_DecRef(both_samples_nparray);
free(both_samples);
double p_value = PyFloat_AsDouble(PyTuple_GetItem(anderson_2samp_return_val, 2));
Py_DecRef(anderson_2samp_return_val);
PyGILState_Release(gstate); // Releasing GIL
return p_value;
}
void exec(int thread_id) {
std::vector<int> left_sample;
std::vector<int> right_sample;
int n = 50;
for (int j = 0; j < n; j++) {
int size = 100;
for (int i = 0; i < size; i++) {
left_sample.push_back(limited_rand(0, 100));
right_sample.push_back(limited_rand(0, 100));
}
exec_1(left_sample, right_sample);
exec_2(left_sample, right_sample);
}
}
The functions where I use the Python C API are only exec_1 and exec_2, while exec has just the job to call the repeatedly on new random data. This is the simplest code I could think of that mimics the behavior of my real code. I've also left out every type of error checking when using the Python APIs for a better readability.
Without any other choice I'll run my code like option 2 or option 3 and forget about the error, but I would really like to understand what's going on. Can you help me?
P.S. I'm running Python 3.6.1 under a macOS 10.12.5 system using Xcode 8.3.3. If you need more details let me know.

option1:
I think is giving you a segmentation fault because you called PyEval_SaveThread() (which releases the gil, returns a saved thread state, and sets the current thread state to NULL).
Py_Finalize will try to free all memory associated with the interpreter, and I guess this included the main thread state. So you can either capture this state with:
PyEval_InitThreads(); //initialize and aquire the GIL
//release the GIL, store thread state, set the current thread state to NULL
PyThreadState *mainThreadState = PyEval_SaveThread();
*main code segment*
//re-aquire the GIL (re-initialize the current thread state)
PyEval_RestoreThread(mainThreadState);
Py_Finalize();
return 0;
Or you can immediately call PyEval_ReleaseLock() after calling PyEval_InitThreads() since it looks like the main code segment does not use any embedded python. I had a similar problem and that seemed to fix it.
NOTE: Other threads will still need to aquire/release the GIL wherever necessary

PyObject_CallMethod returns Null

I have an image processing application that I am using OpenCV within Python and then have it embedded in Embarcadero C++ XE6 IDE for Windows.
I followed all the great exampled and was able to get the embedded Python code working in the bare application. However when I try to use numpy (loadImage) I get the following error:
<type 'exception.AttributeError'>:'NoneType' object has no attribute 'dtype'
If I simply return out of the Python function with a True (commented out now) it returns a valid PyObject. If I try to average the intensity of the pixels with numpy it returns a NULL object.
I am thinking I don't have the imports setup correctly. The stand-alone Python application works as expected but not when it's embedded into my C++ application.
Python Code:
import numpy
import cv2
class Camera:
def __init__(self):
print 'OpenCV Version:', cv2.__version__
def loadImage(self):
'''
Load image from file
'''
global img
img = cv2.imread('cameraImageBlob.png',cv2.IMREAD_GRAYSCALE)
#return True
return numpy.average(img)
if __name__ == '__main__':
dc = Camera()
print dc.loadImage()
C++ Code:
#include "Python.h"
int main() {
PyObject *pName, *pModule, *pDict, *pFunc, *pValue, *pClass, *pInstance;
double dValue;
// Initilize the Python interpreter
Py_Initialize();
// Set runtime paths
PyRun_SimpleString("import sys");
PyRun_SimpleString("import numpy");
PyRun_SimpleString("import cv2");
// Build the name object - create a new reference
pName = PyString_FromString((char*)"OpenCVTest");
// Load the module object
pModule = PyImport_Import(pName);
if(pModule != NULL) {
//pDict is a borrowed reference so no DECREF
pDict = PyModule_GetDict(pModule);
// Get the Camera Class
pClass = PyDict_GetItemString(pDict, "Camera");
if(PyCallable_Check(pClass)) {
pInstance = PyObject_CallObject(pClass, NULL);
} else {
return;
}
// Load Image and get intensity
pValue = PyObject_CallMethod(pInstance, "loadImage", NULL);
if(pValue == NULL) {
GetError();
} else {
dValue = PyFloat_AsDouble(pValue);
}
}
Py_DecRef(pName);
Py_DecRef(pModule);
Py_DecRef(pValue);
Py_Finalize();
return 0;
}

Using Qt-DLL in Python

I am creating a DLL which uses Qt. I need to access this DLL from Python.
Here is an example code:
deploydll.pro:
QT += core gui \
xml \
declarative
greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
TEMPLATE = lib
CONFIG += console
TARGET = DeployDll
DEFINES += DEPLOY_LIBRARY
SOURCES += \
deploydll.cpp
HEADERS += \
deploydll.h
deploydll.h:
#ifndef DEPLOYDLL_H
#define DEPLOYDLL_H
#include <iostream>
#if defined DEPLOY_LIBRARY
#define DEPLOY_EXPORT __declspec(dllexport)
#else
#define DEPLOY_EXPORT __declspec(dllimport)
#endif
class DEPLOY_EXPORT DeployDll
{
public:
DeployDll();
bool showMessage();
};
#endif // DEPLOYDLL_H
#deploydll.cpp
#include "deploydll.h"
#include <functional>
#define NOMINMAX
#include <Windows.h>
#include <QApplication>
#include <QMessageBox>
#include <QtConcurrent/QtConcurrent>
QApplication* a = 0;
int* argc = 0;
BOOL WINAPI DllMain( HANDLE hDll, DWORD dwReason, LPVOID lpReserved )
{
switch( dwReason )
{
case DLL_PROCESS_ATTACH:
{
argc = new int( 0 );
QApplication* a = new QApplication( *argc, 0 );
QtConcurrent::run( &QApplication::exec );
}
case DLL_PROCESS_DETACH:
if( argc != 0 )
{
delete argc;
argc = 0;
}
if( a != 0 )
{
delete a;
a = 0;
}
break;
case DLL_THREAD_ATTACH:
break;
case DLL_THREAD_DETACH:
break;
}
return TRUE;
}
DeployDll::DeployDll()
{
std::cout << "Constructor called!\n";
}
bool DeployDll::showMessage()
{
std::cout << "Method called!\n";
QMessageBox msgBox;
msgBox.setText("Method called!");
msgBox.exec();
return true;
}
Here is an example python code:
from ctypes import *
if __name__ == '__main__':
print "Started main!"
cdll.LoadLibrary("DeployDll")
I added the Qt platforms folder into the C:\python27-folder.
The generated DLL is in the folder of the python project.
If I use the DLL in a simple C++ program it works, but when I execute the python script I get the following error message:
Started main!
QApplication::exec: Must be called from the main thread
QWaitCondition: Destroyed while threads are still waiting
I am using Windows 7 64 Bit, Python 2.7.3 and Qt 5.2.1 with MSVC2012 64 Bit compiler.

Meybe you should use QApplication::exec() in the main thread. Why are you use QtConcurrent::run?

If you just call QApplication::exec() in the DllMain this would be blocking the main thread completely. So QApplication::exec() is wrapped by an asynchronous call of QtConcurrent::run.

How to return output from pyrun_simplefile in c code

The code is
{
char name[MAX_JSON_FIELD];
FILE *fp;
copy_cJSON(name,objs[0]);
if ( (fp= fopen(name, "r")) != 0 )
{
Py_Initialize();
PyRun_SimpleFile(fp, name);
Py_Finalize();
fclose(fp);
}
return(clonestr("return string"));
}
How can I get it to return the output of the python file instead of printing it?

I achieved this using a huge workaround. I made both C and Python read and write into a file. I didn't find a better option yet.
I found an actual solution. It consists of 2 files: "main.c" that opens the script-file "script.py" which compares two strings (here: "Hello" and "Mars") and returns the longer one. I still find it strange that it takes ~20 commands to achieve this, maybe there's a better solution out there.
[main.c]
//compile me with "gcc main.c -I/usr/include/python2.7 -lpython2.7"
//original source: "http://python.haas.homelinux.net/python_kapitel_26_003.htm"
//owner is Peter Kaiser and Johannes Ernesti who published the Book "Python" under Galileo Computing
//Translation from german with many additional (and very unprofessional) comments and slight adaption by Cupacoffee, 17.02.2015.
//bugs, grammar mistakes and wrong explainations are my contribution
#include <Python.h>
int main (int argc, char *argv[])
{
char *result;//This char will receive the return value.
PyObject *module, *func, *prm, *ret;//These are some helping variables i don't understand.
Py_Initialize();
PySys_SetPath(".");//Sets the working path to the current path
module = PyImport_ImportModule("script");//Import of the script-file, note that the actual script name is "script.py"!
if (module != 0)//Asks if the script was loaded at all.
{
func = PyObject_GetAttrString(module, "compare_function");//Opens a function within the python script. Notice that you must use a function within the python script, because otherwise you can't return anything.
prm = Py_BuildValue("(ss)", "Hello", "Mars");//The "(ss)" means two strings are passed (replace with "i" for integer for instance), the "Hello" and "Mars" are the strings i pass to the script.
ret = PyObject_CallObject(func, prm);//Returns some python object i have literally no idea about ...
result = PyString_AsString(ret);// ... but luckily there's a function to cast it back to a c-compatible char*!
printf("The Script decdided that '%s' is longer!",result);
Py_DECREF(module);//cleanup?
Py_DECREF(func);//cleanup?
Py_DECREF(prm);//cleanup?
Py_DECREF(ret);//cleanup?
}
else//No script found
{
printf("Error: No script file named \"script.py\" was found!\n");
}
Py_Finalize();
return 0;
}
[script.py]
def compare_function(a, b):#this function takes 2 parameters, they are strings
return (a if min(a) < min(b) else b)#they get compared and returned to the c-program
Good luck.
*Grumble, took me over 2 hours to format this text so I could post it.*

A friend of mine gave me some code snippets which answer the problem. I didn't want to edit the old post because these two programs are entirely new approaches; One in C and one in C++. Both use the same Python Script.
He also pointed me to the manual page of "system" [Unix: "man system"] and "popen" [Unix: "man popen"]. The second one allows interactive communication, which might be useful later.
[C-File:]
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
int callScript()
{
char *cmd = "./script.py hello world";
return WEXITSTATUS(system(cmd));
}
int main(int argc, char *argv[])
{
printf("main - argc: %d, arguments:\n", argc);
for (int i = 0; i < argc; i++)
printf("\targv[%d]: %s\n", i, argv[i]);
int ret = callScript();
printf("exit code of script %d\n", ret);
return 0;
}
[C++-File:]
#include <string>
#include <sstream>
#include <iostream>
#include <stdlib.h>
#include <sys/wait.h>
int callScript(std::string args)
{
std::string cmd = "./script.py" + args;
int ret = system(cmd.c_str());
return WEXITSTATUS(ret);
}
int main(int argc, char *argv[])
{
std::cout << "main - argc: " << argc << ", arguments:" << std::endl;
std::stringstream args;
for (int i = 0; i < argc; i++)
{
std::cout << "\targv[" << i << "]: " << argv[i] << std::endl;
if (i)
args << " " << argv[i];
}
int ret = callScript(args.str());
std::cout << "exit code of script " << ret << std::endl;
return 0;
}
[Python-Script:]
#!/usr/bin/env python
import sys
def Hello(person):
print "Hello " + person
def PrintArgs(argv):
for arg in argv:
print arg
if __name__ == "__main__":
Hello("World!")
PrintArgs(sys.argv[1:])
sys.exit(2)