I have two processes in python that share a single boolean flag:
from multiprocessing import Process, Value
class MyProcess(Process):
def __init__(self):
self.flag = Value('B',false)
# [...]
def run(self):
while self.active:
# do_something()
if some_condition:
self.work_to_be_extended__()
def work_to_be_extended__(self) -> bool:
while some_internal_loop_condition:
if self.flag.value:
# do something
return result
if __name__ == '__main__':
my_proc = MyProcess()
my_proc_flag = my_proc.flag
my_proc.start()
# [...] Some work
if condition:
my_proc_flag.value = True
I need to put MyProcess.work_to_be_extended in an extension module to be executed in C code. Something like:
bool extended_work(void):
{
while (some_condition) {
if (my_proc_flag) {
do_something()
}
return result
}
I've not designed the extension yet, since I'd need to understand first how to share the MyProcess.flag variable. Please, note that I don't need to pass the variable value, I need to pass its reference in order for the extension to see a change in the flag value operated in the main process where the extension does not live`.
Hope I've been quite clear**
Multiprocessing has a sharedctypes submodule for ctypes array and values. You can use it to create a shared ctypes (a int in my example). And then use ctypes.byref to send a pointer to that int.
Since the underlying mechanism is SHM (not some hidden piping under the hood), the pointed memory by this reference is really the same in both process. shval.value is *p pointed by the p argument passed, that is byref(shval).
So, no need for the size 1 array of my previous answer, and, more importantly, for the disclaimer accompanying it.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stdint.h>
void myf(volatile uint32_t *p){
for(;;){
printf("<c>%d</c>\n", *p);
if(*p==100) return;
(*p)++;
sleep(1);
}
}
import multiprocessing as mp
import multiprocessing.sharedctypes as st
import ctypes
mylib=ctypes.CDLL("libtoto.so")
mylib.myf.argtypes=[ctypes.c_void_p]
shval=st.RawValue(st.ctypes.c_uint32,12)
class MyProcess(mp.Process):
def __init__(self):
super().__init__()
def run(self):
mylib.myf(st.ctypes.byref(shval))
if __name__=="__main__":
myproc=MyProcess()
myproc.start()
while True:
i=int(input("val>"))
shval.value=i
So, short answer to your question is: use multiprocessing.sharedctypes and pass byref(sharedval) to your function.
Premise
This answer comes from an adaptation of the good solution given by #chrslg. This extends that usage to apply to other paradigm of Python/C programming, such as C Extension API, Cython and Boost::Python.
Please, read that answer first for a deeper background.
Overview and core summary:
Using a sharedctypes.RawValue as the required variable, it is possible to access the underlying data address by means of the method sharedctypes.ctypes.addressof.
Therefore, one can pass the address of the variable as a long long int (64 bit) and cast it into a pointer to the required data. For example, for a uint8_t variable, one has into the C extension
int64_t address; // This is initialized in some way, depending on the C interface to python
// Pointer to shared data
uint8_t* pointer = (uint8_t*)(address);
printf("Current value of shared data: %u\n", pointer);
Working example for different Python - C/C++ interfaces
Common C shared library
Let's create a base, simple C library that just read for 1 time per second the value of the variable being shared:
// cshare_data/cshare_data.c
#include "cshare_data.h"
#include <time.h>
#include <unistd.h>
#include <stdio.h>
void cshare_data(uint8_t* data, char from_where_called) {
char *s = NULL;
if (from_where_called == 0) {
s = "cTypes CDLL";
} else if (from_where_called == 1)
{
s = "Python C Extension";
} else if (from_where_called == 2)
{
s = "Boost::Python";
} else if (from_where_called == 3)
{
s = "Cython";
}
for (int i = 0; i < 10; i++) {
printf("C code read from %s a value of: %u\n", s, *data);
sleep(1);
}
}
The header:
// cshare_data/cshare_data.h
#ifndef CSHARE_DATA_H
#define CSHARE_DATA_H
#include <stdint.h>
#include <stddef.h>
extern void cshare_data(uint8_t*, char);
#endif
Python shared data editing process
For the rest of the examples, I'll refer to the following Python process that is modifying the shared data (unsigned char in the example):
from multiprocessing.sharedctypes import RawValue, Value
import multiprocessing.sharedctypes as st
from multiprocessing import Process
class MyProcess(Process):
def __init__(self):
Process.__init__(self)
self.int_val = RawValue(st.ctypes.c_ubyte, 0)
def run(self) -> None:
import time
for _ in range(10):
print('Value in Python Process: ', self.int_val.value)
self.int_val.value += 1
time.sleep(1)
my_proc = MyProcess()
my_proc.start()
NOTE: This will not be repeated hereinafter.
Python C Extension
A Python C Extension API that makes use of the above pattern follows:
#include <Python.h>
#include <stdio.h>
#include <time.h>
#include "cshare_data.h"
static PyObject *cshare_data_wrapper(PyObject *self, PyObject *args)
{
PyObject *val = NULL;
// This will store the address of the uchar variable being passed from Python
int64_t address = 0;
// Convert the single element tuple into a 8-byte int (address)
if(!PyArg_ParseTuple(args, "L", &address)) {
printf("Error parsing Tuple\n");
return NULL;
}
// Now address is reinterpreted as the shared variable pointer
uint8_t *pointer = (uint8_t *)(address);
// Call the library function
cshare_data(pointer, 1);
return Py_None;
}
static PyMethodDef CShapreDataMethods[] = {
{"cshare_data", cshare_data_wrapper, METH_VARARGS, "Python interface for sharedata C library function"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef cshareddata_module = {
PyModuleDef_HEAD_INIT,
"csharedata_module",
"Python interface for the fputs C library function",
-1,
CShapreDataMethods
};
PyMODINIT_FUNC PyInit_cshare_data_pyext(void) {
return PyModule_Create(&cshareddata_module);
}
Please, refer to official documentation and this very good tutorial for deeper insight about Python C-API
Boost::Python
Very similar to what done for the Python C-API, the boost wrapper looks like:
extern "C" {
#include "cshare_data.h"
}
#include <boost/python.hpp>
void cshare_data_boost_wrapper(long long int data_address) {
uint8_t* data = reinterpret_cast<uint8_t*>(data_address);
cshare_data(data, 2);
}
BOOST_PYTHON_MODULE(ctrigger) {
using namespace boost::python;
def("cshare_data", cshare_data_boost_wrapper);
}
CMake - Library buildings
Moving from a project with the following tree structure:
```
project_root
| cshare_data.py
|---clibs
| | cshare_data_boost.so
| | cshare_data_pyext.so
| | cshare_data.so
|
|---cshare_data
| | cshare_data.c
| | cshare_data.h
|
| CMakeList.txt
```
The following compilation CMake script was used:
cmake_minimum_required (VERSION 2.6)
project (cshare_data)
set(CMAKE_SHARED_MODULE_PREFIX "")
set(CMAKE_SHARED_LIBRARY_PREFIX "")
# Common C shared library
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR}/clibs)
include_directories(${CMAKE_SOURCE_DIR}/cshare_data)
link_directories(${CMAKE_SOURCE_DIR}/clibs)
# --- Common C shared library ---
add_library(cshare_data SHARED cshare_data/cshare_data.c)
# Needed for Python C Extension Module and Boost::Python
include_directories("/usr/include/python3.8")
# --- Python C Extension Module library ---
add_library(cshare_data_pyext MODULE cshare_data_pyinterface/cshare_data_pyext.c)
target_link_libraries(cshare_data_pyext python3.8)
target_link_libraries(cshare_data_pyext cshare_data)
# --- Python C Extension Module library ---
include_directories("/home/buzz/boost_1_80_0")
link_directories("/home/buzz/boost_1_80_0/build/lib")
add_library(cshare_data_boost MODULE cshare_data_pyinterface/cshare_data_boost.cpp)
target_link_libraries(cshare_data_boost python3.8)
target_link_libraries(cshare_data_boost boost_python38)
target_link_libraries(cshare_data_boost cshare_data)
Python - Calling C wrappers
Just for the purpose of demonstration, I've written 3 different processes that share the same int_val (handled by the above MyProcess) and call the C function to print the value of this variable. Note that, though the lines of code are the same, the address must be withdrawn at each process call since multiprocessing.sharedctypes wraps IPC synchronizing architecture for int_val under the hood, meaning that each actual int_val lives in the proper process.
my_proc = MyProcess()
my_proc.start()
l = []
class FromWhere(IntEnum):
ctype = 0
python_c_extension = 1
boost_python = 2
def from_ctype_import_dll(int_val: RawValue):
import ctypes
reference = st.ctypes.byref(my_proc.int_val)
mylib=ctypes.CDLL("clibs/cshare_data.so")
mylib.cshare_data.argtypes=[ctypes.c_void_p, ctypes.c_char]
mylib.cshare_data(reference, FromWhere.ctype.value)
def from_python_c_extension(int_val: RawValue):
from clibs import cshare_data_pyext
address = st.ctypes.addressof(int_val)
cshare_data_pyext.cshare_data(address)
def from_boost_python(int_val: RawValue):
from clibs import cshare_data_boost
address = st.ctypes.addressof(int_val)
cshare_data_boost.cshare_data(address)
ps: List[Process] = []
ps.append(Process(target=from_ctype_import_dll, args=(my_proc.int_val,)))
ps.append(Process(target=from_python_c_extension, args=(my_proc.int_val,)))
ps.append(Process(target=from_boost_python, args=(my_proc.int_val,)))
for p in ps:
p.start()
for p in ps:
p.join()
The result achieved:
Value in Python Process: 0
C code read from cTypes CDLL a value of: 1
C code read from Python C Extension a value of: 1
C code read from Boost::Python a value of: 1
Value in Python Process: 1
C code read from cTypes CDLL a value of: 2
C code read from Boost::Python a value of: 2
C code read from Python C Extension a value of: 2
Value in Python Process: 2
C code read from cTypes CDLL a value of: 3
C code read from Boost::Python a value of: 3
C code read from Python C Extension a value of: 3
C code read from cTypes CDLL a value of: 3
Value in Python Process: 3
C code read from Boost::Python a value of: 4
C code read from Python C Extension a value of: 4
C code read from cTypes CDLL a value of: 4
Value in Python Process: 4
C code read from Boost::Python a value of: 5
C code read from Python C Extension a value of: 5
C code read from cTypes CDLL a value of: 5
Value in Python Process: 5
C code read from Boost::Python a value of: 6
C code read from Python C Extension a value of: 6
C code read from cTypes CDLL a value of: 6
Value in Python Process: 6
C code read from Python C Extension a value of: 7
C code read from Boost::Python a value of: 7
C code read from cTypes CDLL a value of: 7
Value in Python Process: 7
C code read from Python C Extension a value of: 8
C code read from Boost::Python a value of: 8
C code read from cTypes CDLL a value of: 8
Value in Python Process: 8
C code read from Python C Extension a value of: 9
C code read from Boost::Python a value of: 9
C code read from cTypes CDLL a value of: 9
Value in Python Process: 9
C code read from Python C Extension a value of: 10
C code read from Boost::Python a value of: 10
Related
I'm converting a C code into a Python code that uses a .dll file.
The syntax for accessing the commands from the DLL is given below:
cnc_rdmacro(unsigned short FlibHndl, short number, short length, ODBM *macro);
C code
Pointer to the odbm data structure is as follows:
typedef struct odbm {
short datano ; /* custom macro variable number */
short dummy ; /* (not used) */
long mcr_val ; /* value of custom macro variable */
short dec_val ; /* number of places of decimals */
} ODBM ;
C code used to access the dll command:
short example( short number )
{
ODBM macro ;
char strbuf[12] ;
short ret ;
ret = cnc_rdmacro( h, number, 10, ¯o ) ;
The python code that I converted according to the above C code is as follows:
import ctypes
fs = ctypes.cdll.LoadLibrary(r".dll filepath")
ODBM = (ctypes.c_short * 4)() #the datatype conversion code from the above C code
ret = fs.cnc_rdmacro(libh, macro_no, 10, ctypes.byref(ODBM))
I can get the output without any errors in the above code.
The actual data structure of the ODBM has declared 4 variables of datatypes short, short, long and short which are implemented in the C code. But I had declared the ODBM data structure in python as ctypes.c_short * 4 i.e, 4 variables of short data types.
But my necessity is to declare the ODBM structure the same as in the C code and pass it to the ctypes.byref().
The ultimate solution is to include multiple data types in a single variable as a ctypes instance. Kindly help me out.
A ctypes.Structure should be used here:
import ctypes
class ODBM(ctypes.Structure):
_fields_ = [("datano", ctypes.c_short),
("dummy", ctypes.c_short),
("mcr_val", ctypes.c_long),
("dec_val", ctypes.c_short)]
fs = ctypes.cdll.LoadLibrary(r".dll filepath")
odbm = ODBM()
ret = fs.cnc_rdmacro(libh, macro_no, 10, ctypes.byref(odbm))
print(odbm.mcr_val)
These are my example source code:
C
#include <stdio.h>
#include <stdlib.h>
__declspec(dllexport)
char* sys_open(char* file_name)
{
char *file_path_var = (char *) malloc(100*sizeof(char));
FILE *wrt = fopen(file_name, "r");
fscanf(wrt, "%s", file_path_var);
fclose(wrt);
return file_path_var;
}
Test.txt
test
Python
from ctypes import *
libcdll = CDLL("c.dll")
taken_var = libcdll.sys_open("test.txt")
print("VAR: ", taken_var)
Result
VAR: 4561325
So I'm just getting a random number. What should i do?
I'm not a C developer, but isn't sys_open returning a pointer?. Last time I checked pointers are WORD sized memory addresses in HEX, so it might make sense that python sees a numerical value in HEX and converts it to a decimal?. Maybe what you want to return from your C funcion is &file_path_var
I found the true one.
The python file was wrong, must be:
from ctypes import *
libcdll = CDLL("c.dll")
taken_var = libcdll.sys_open("test.txt")
print("VAR: ", c_char_p(taken_var).value)
I'm writing a python extension module in C. Python stops running when I declare an array of structs greater than 4 elements in a function of that module.
I'm writing the module to increase performance.
I've declared 3 structs( "SKU", "Cromosoma", "Aptitud" ) and I want to create an array of Cromosoma, but when I try to create the array with more than 4 elements it breaks.
// need_for_speed.c extension module code
#include <Python.h>
#include <stdlib.h>
#define MAX_GENES_SIZE 2000
typedef struct{
char codigo[30];
double venta;
char nombre[100];
double categoria;
double peso;
double ubicacion_especifica;
double ubicacion_actual;
double ubicacion_a_mover;
double stock;
} SKU;
typedef struct{
double ubicaciones_rompe_regla;
double cercania_medio;
double desv_std_picks_x_seccion;
double peso_x_ubicacion;
} Aptitud;
typedef struct{
SKU genes[MAX_GENES_SIZE];
Aptitud aptitud;
int genes_size;
int edad;
}Cromosoma;
static PyObject* prueba(PyObject* self, PyObject* args){
Cromosoma a;
SKU s;
strcpy(s.codigo,"1212");
a.genes[0] = s;
Cromosoma poblacion[] = {a,a,a,a,a};
printf("codigo %s ", poblacion[0].genes[0].codigo);
return PyLong_FromDouble(1);
}
static PyMethodDef Methods[] = {
{"prueba", prueba, METH_NOARGS, "Prueba general"},
{ NULL, NULL, 0, NULL }
};
// Module Definition struct
static struct PyModuleDef need_for_speed = {
PyModuleDef_HEAD_INIT,
"need_for_speed",
"Modulo para aumento de la velocidad de procesamiento para el algoritmo genético",
-1,
Methods
};
// Initialize module
PyMODINIT_FUNC PyInit_need_for_speed(void)
{
PyObject *m;
m = PyModule_Create(&need_for_speed);
return m;
}
the setup.py to build this module:
from distutils.core import setup, Extension
setup(name = 'need_for_speed', version = '1.0',ext_modules = [Extension('need_for_speed', ['need_for_speed.c'])])
command to build module:
python setup.py build
when I call the function prueba:
import need_for_speed
i = need_for_speed.prueba()
python stops running without printing or returning anything, but if modify the array named "poblacion" in the "prueba" function to have only 4 elements it runs perfectly, returning 1 and printing "codigo 1212".
I'm using Windows BTW.
It's probably a stack overflow.
Let's see how large your structs are assuming they only take the size of the individual members (neglecting padding, etc.):
SKU: 7 doubles and 130 chars -> 7 * 8 bytes + 130 bytes -> 186 bytes
Aptitud: 4 doubles -> 4 * 8 bytes -> 32 bytes
Cromosoma: 2 ints, 1 Aptitud and 2000 SKU -> 2 * 4 bytes + 32 bytes + 2000 * 186 bytes -> 372040 bytes
So one instance of Chromosoma will take ~370kB. And you create 5/6 of them; one with Cromosoma a; and one for each slot in the array: 4/5.
A typical stack will be only a few megabytes. With 6 * 370kB ~ 2.1MB it's at least reasonable that you exhausted your stack. For example MSVC (Windows Visual Studio C/C++ compiler) by default uses just 1 MB. Given that it fails with an array of size 5 but works with one of size 4 it seems that you have ~2 MB of stack.
To avoid this problem you could increase your stack size (how you do that depends on your compiler). However increasing the stack size will again lead to problems when you need more Chromosoma or change the number of SKUs.
Another alternative (probably better than increasing the stack size) would be to allocate all big arrays on the heap. For example you could allocate the poblacion on the heap - and/or make the Chromosoma.genes a pointer to an array of SKU.
I have code for both Python and C that need to communicate to each other through a pipe created by Popen. I have a test struct in C that needs to be passed back to Python but I can't seem to reconstruct that struct on the Python side. This is a much more complicated project but the struct I created below is just an example to get the code to work, and I can try to figure out the more advanced things later. I am not an expert in C, pointers and piping, and I do not have a clear understanding of it. Most of the C code below is just from my readings.
Python:
testStruct = struct.Struct('< i')
cProg = Popen("./cProg.out", stdin=PIPE, stdout=PIPE)
data = ""
dataRead = cProg.stdout.read(1)
while dataRead != "\n":
data += dataRead
dataRead = cProg.stdout.read(1)
myStruct = testStruct.unpack(data)
print myStruct.i
C:
typedef struct{
int i;
} TestStruct;
int main(void)
{
int wfd = fileno(stdout);
TestStruct t;
t.i = 5;
char sendBack[sizeof(t)];
memcpy(sendBack, &t, sizeof(t));
write(wfd, sendBack, sizeof(sendBack));
write(wfd, "\n", 1);
}
But when I run the Python code I get the error:
unpack requires a string argument of length 4
Like I said I do not understand how structs and C. If there's any suggestion on refining this code, or better yet another suggestion on passing a C struct back to Python to unpack and grab the data. I can read and write through the pipe, the code I have posted are just snippets from my actual code. I know that the issue has to do with sending of the struct back to Python through stdout.
Here's an example of reading data in Python from a C program through a pipe.
C Program
#include <stdio.h>
typedef struct{
int i;
int j;
} TestStruct;
int main() {
TestStruct ts = {11111, 22222};
fwrite(&ts, sizeof ts, 1, stdout);
return 0;
}
Python 2.7 Program
from subprocess import Popen, PIPE
from struct import calcsize, unpack
cprog = Popen("cprog", stdout=PIPE)
fmt = "#ii"
str = cprog.stdout.read(calcsize(fmt))
cprog.stdout.close()
(i, j) = unpack(fmt, str)
print i, j
I have the following DLL ('arrayprint.dll') function that I want to use in Python via ctypes:
__declspec(dllexport) void PrintArray(int* pArray) {
int i;
for(i = 0; i < 5; pArray++, i++) {
printf("%d\n",*pArray);
}
}
My Python script is as follows:
from ctypes import *
fiveintegers = c_int * 5
x = fiveintegers(2,3,5,7,11)
px = pointer(x)
mydll = CDLL('arrayprint.dll')
mydll.PrintArray(px)
The final function call outputs the following:
2
3
5
7
11
2226984
What is the 2226984 and how do I get rid of it? It doesn't look to be the decimal value for the memory address of the DLL, x, or px.
Thanks,
Mike
(Note: I'm not actually using PrintArray for anything; it was just the easiest example I could find that generated the same behavior as the longer function I'm using.)
mydll.PrintArray.restype = None
mydll.PrintArray(px)
By default ctypes assumes the function returns an integral type, which causes undefined behavior (reading a garbage memory location).