I'm writing a Python extension for the functions provided by a GPIO driver. I made progress pretty easily on the simple functions like set_bit() and clear_bit(). But now I need to implement wait_int(), which sleeps until an event is sensed on an input pin and I'm not sure the right way to orchestrate this between c and python. Here's a stripped down example of using the function in c:
main(int argc, char *argv[])
{
int c;
//some setup like testing port availability, clearing interrupts, etc
...
while(1)
{
printf("**\n");
c = wait_int(1);//sleeps until an interrupt occurs on chip 1
if(c > 0) {
printf("Event sense occured on Chip 1 bit %d\n",c);
++event_count;
}
else
break;
}
printf("Event count = %05d\r",event_count);
printf("\nExiting Now\n");
}
Do I just expose wait_int pretty much directly and then do whatever the python equivalent idiom of the infinite loop is? There's also some debouncing that needs to be done. I've done it in c but maybe it could be moved to the python side.
You don't need to do anything on the Python side, you can just treat it as a synchronous function. On the C side, you just block until the event occurs, possibly allowing interrupts. For example, take a look at the implementation of the time.sleep function:
/* LICENSE: http://docs.python.org/license.html */
/* Implement floatsleep() for various platforms.
When interrupted (or when another error occurs), return -1 and
set an exception; else return 0. */
static int
floatsleep(double secs)
{
/* XXX Should test for MS_WINDOWS first! */
#if defined(HAVE_SELECT) && !defined(__BEOS__) && !defined(__EMX__)
struct timeval t;
double frac;
frac = fmod(secs, 1.0);
secs = floor(secs);
t.tv_sec = (long)secs;
t.tv_usec = (long)(frac*1000000.0);
Py_BEGIN_ALLOW_THREADS
if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
#ifdef EINTR
if (errno != EINTR) {
#else
if (1) {
#endif
Py_BLOCK_THREADS
PyErr_SetFromErrno(PyExc_IOError);
return -1;
}
}
Py_END_ALLOW_THREADS
#elif defined(__WATCOMC__) && !defined(__QNX__)
...
All it does is use the select function to sleep for the given period of time. select is used so that if any signal is received (such as SIGINT from hitting Ctrl+C at the terminal), the system call is interrupted and control returns to Python.
Hence. your implementation can just call the C wait_int function. If it supports being interrupted by signals, than great, that will allow the user to interrupt it by hitting Ctrl+C, but make sure to react appropriately such that an exception will be thrown (I'm not certain of how this works, but it looks like returning NULL from the top-level function (time_sleep in this example) will do the trick).
Likewise, for better multithreaded performance, surround the wait call with a pair of Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS macros, but this is not required, especially if you're not using multithreading at all.
Related
In a c++ program, I used "Python.h" to implement a c++ function which can be used in python.
In python, I want this c++ function to run in a limited time.
So, I used a function decorator in python to let the c++ function run in a limited time. If the function exceed the given time, it will raise a RuntimeError, then I just let the function return.
But I the result seems not good, after the c++ function being called many times, the program runs slower and slower, and finally crash.
This is the decorator:
def set_timeout(num, callback):
def wrap(func):
def handle(signum, frame):
raise RuntimeError()
def to_do(*args, **kwargs):
try:
signal.signal(signal.SIGALRM, handle)
signal.alarm(num)
print('start alarm signal.')
r = func(*args, **kwargs)
print('close alarm signal.')
signal.alarm(0)
return r
except RuntimeError as e:
callback()
return to_do
return wrap
def after_timeout():
return
This is the python invoked c++ function with decorator:
#set_timeout(3,after_timeout)
def pytowr_run(gait,tgt,time_interval,tm,posture=None,init_dic={}):
pos,cost,varDict = pytowr.run(gait,tgt[0],tgt[1],time_interval,tm,posture,init_dic)
return pos
Is there any way to let the python invoked c++ function stop running in python?
To stop the c++ function you'll need to send it a signal. But just sending a signal is not enough. In the c++ function you will probably need to set some interrupt points to check if a signal was sent. Example:
#include <csignal>
#include <atomic>
constexpr int SIGALRM = 14;
std::atomic_bool signal_received;
void signal_handler(int signal) {
// check 'signal' if necessary
signal_received.store(true);
}
void config() { //call it at some start point...
signal_received.store(false);
std::signal(SIGALRM, signal_handler);
}
void your_function(/*args*/) {
/* do some work */
if (signal_received.load()) return; //interrupt point. Exiting your c++ function...
// the interrupt point must be placed in a strategic place
/* maybe more work */
}
I think a better solution if i'm understanding your goal, is invoke a C/C++ function and pass in the allotted time into the C/C++ function instead of trying kill the function from python which is doable but makes the code harder to read and debug.
#include datetime.h
PyDateTime_IMPORT
bool your_function(passed_in_vars, start_time * PyDateTime_DateTime allotted_time* int )
{
kill_time = PyDateTime_DATE_GET_MICROSECOND(PyDateTime_DateTime *o) + allotted
/* do some work */
if ( allotted_time > kill_time )
return; //interrupt point. Exiting your c++ function...
//do more work...
}
This makes the code easier no loops or wait states on the python side and makes its easier on the C++ side to know when to kill the function and fire off any clean up code.
And finding Bottles necks easier on the C++ side..
I'm reading APUE.
#include "apue.h"
int globvar = 6; /* external variable in initialized data */
int main(void)
{
int var; /* automatic variable on the stack */
pid_t pid;
var = 88;
printf("before vfork\n"); /* we don′t flush stdio */
if ((pid = vfork()) < 0) {
err_sys("vfork error");
} else if (pid == 0) { /* child */
globvar++; /* modify parent′s variables */
var++;
_exit(0); /* child terminates */
}
/* parent continues here */
printf("pid = %ld, glob = %d, var = %d\n", (long)getpid(), globvar, var);
exit()
}
It's said that vfork() creates a child process without copying the address space the parent process has. And parent will wait until child process calls exit or exec.
Is there anything similar in Python give this kind of low level control? How can this be achieved in Python if possible?
vfork is a mostly-obsolete optimization of fork, intended solely for uses where it is immediately followed by an exec. It was designed back when fork didn't use copy-on-write, and has been rendered almost entirely pointless by copy-on-write.
Your use of vfork is undefined behavior; modifying variables like globvar or var in the child is not allowed. Almost the only thing the child is allowed to do is call one of the exec* functions. The details are in the man page.
If your goal is to share memory between Python processes, you should do that with the multiprocessing module and multiprocessing.sharedctypes.
If your goal is to call vfork, go ahead and call it with ctypes, and watch your program immediately fall apart because it's impossible to call vfork safely from Python:
$ cat asdf.py
import ctypes
import os
lib = ctypes.CDLL('libc.so.6')
vfork = lib.vfork
if not vfork():
os.execv('/usr/bin/echo', ['echo', 'hello', 'world'])
print("Should print in parent, but doesn't due to memory corruption.")
$ python3.6 asdf.py
hello world
Segmentation fault (core dumped)
I have a problem with a shared library function called from Python.
Consider this simplified C program that acquires an image from a camera:
#include <stdio.h>
#include <someproprietarylibraryheader.h>
int myfunction() {
CameraHandle handle;
printf("Running... \n");
if(!camera_init(&handle) { // initialize the camera handle
return -1;
}
if(!camera_precapture(handle)) { // prepare for capture
printf("ERROR precapture\n");
return -1;
}
else {
printf("precapture OK\n");
}
if(!camera_capture(handle)) { // start frame capture (returns immediately)
printf("ERROR capture\n");
return -1;
}
else {
printf("capture OK\n");
}
if(!camera_wait(handle)) { // wait for capture to be completed
printf("ERROR wait\n");
return -1;
}
else {
printf("wait OK\n");
}
if(!camera_close(handle)) { // close the camera
printf("ERROR close\n");
return -1;
}
printf("Done!!\n");
return 0;
}
If I compile this code into a shared library and call myfunction() from a C program linking to it, everything works as expected.
However, consider what happens if one loads the libray and calls myfunction() from Python like this:
from ctypes import *
mylib = cdll.LoadLibrary("mylib.so")
mylib.myfunction()
In this case the program hangs indefinitely at the line camera_capture() in the C code. However, something interesting happens by sending a KeyboardInterrupt with CTRL+C: just before this exception is handled by the interpreter, the program is able to resume and myfunction() continues and terminates normally.
This looks like a hung thread. Indeed by running the above Python script with gdb, I have found out that the proprietary camera API does create some threads. And by examining the backtrace, the program seems to be stuck at a call to nanosleep() somewhere within the proprietary code. Apparently, the nanosleep() function is not properly interrupted, but only when run in Python.
Another hint that this is a thread / interrupt issue is the fact that if I run the Python script in gdb, I can do CTRL+C followed by continue indefinitely at the point where the program hangs. However, if I place a breakpoint with b and then continue, the program resumes and terminates correctly.
Does anybody have a clue about what might prevent this simple program to run smoothly when called from Python, and why don't the threads created by the C library terminate properly when run from Python? Thanks a lot.
I have a toolset which has been written in C++, and given Boost bindings for Python.
Initially, this code was all C++, and I caught a CTRL+C interrupt with:
signal( SIGINT, signalCallbackHandler );
and
void signalCallbackHandler(int /*signum*/)
{
g_myObject->stop();
}
This worked fine.
However, now I've added the Python bindings in, I'm using Python to initialise the objects.
My initial thought was to do it like this:
import signal
def handle_interrupt( signum, frame ) :
g_myObject.stop()
signal.signal( signal.SIGINT, handle_interrupt )
g_myObject = MyObject()
g_myObject.start()
However, this signal handler is never called.
How should I be handling an interrupt like this? Do I need to do it inside C++, and then call the Python function from there?
Your python signal handler is not called because python defers execution of signal handlers until after the next bytecode instruction is to be executed - see the library documentation for signal, section 18.8.1.1:
A Python signal handler does not get executed inside the low-level (C) signal handler. Instead, the low-level signal handler sets a flag which tells the virtual machine to execute the corresponding Python signal handler at a later point(for example at the next bytecode instruction). This has consequences:
It makes little sense to catch synchronous errors like SIGFPE or SIGSEGV that are caused by an invalid operation in C code. Python will return from the signal handler to the C code, which is likely to raise the same signal again, causing Python to apparently hang. From Python 3.3 onwards, you can use the faulthandler module to report on synchronous errors.
A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes.
The reason for this is that a signal can arrive at any time, potentially half way through the execution of a python instruction. It would not be safe for the VM to begin executing the signal handler, because the VM is in an unknown state. Therefore, the actual signal handler installed by python merely sets a flag telling the VM to call the signal handler after the current instruction is complete.
If the signal arrives during execution of your C++ function, then the signal handler sets the flag and returns back to your C++ function.
If the main purpose of the signal handler is to allow the C++ function to be interrupted, then I suggest you dispense with the Python signal handler and install a C++ signal handler that sets a flag which triggers an early exit in your C++ code (presumably returning a value that indicates it was interrupted).
That approach would allow you to use the same code regardless of whether you are calling your code from python, C++ or perhaps another binding.
I have a solution that works for this, although it'd be cleaner if I could get the signal caught in Python rather than C++.
One thing I didn't mention before is that MyObject is a singleton, so I'm getting it with MyObject.getObject()
In Python, I've got:
def signalHandler( signum ) :
if signum == signal.SIGINT :
MyObject.getObject().stop()
def main() :
signal.signal( signal.SIGINT, handle_interrupt )
myObject = MyObject.getObject()
myObject.addSignalHandler( signal.SIGINT, signalHandler )
myObject.start()
In my C++ code, in an area that shouldn't know anything about Python, I have:
class MyObject
{
public :
void addSignalHandler( int signum, void (*handler)( int, void* ), void *data = nullptr );
void callSignalHandler( int signum );
private :
std::map<int, std::pair<void (*)( int, void* ), void*> > m_signalHandlers;
}
void signalCallbackHandler( int signum )
{
MyObject::getObject()->callSignalHandler( signum );
}
void MyObject::addSignalHandler( int signum, void (*handler)( int, void* ), void *data )
{
m_signalHandlers.insert( std::pair<int, std::pair<void (*)( int, void* ), void *> >( signum, std::make_pair( handler, data ) ) );
signal( signum, signalCallbackHandler );
}
void MyObject::callSignalHandler( int signum )
{
std::map<int, std::pair<void (*)( int, void* ), void*> >::iterator handler = m_signalHandlers.find( signum );
if( handler != m_signalHandlers.end() )
{
handler->second.first( signum, handler->second.second );
}
}
And then in my Python bindings:
void signalHandlerWrapper( int signum, void *data )
{
if( nullptr == data )
{
return;
}
PyObject *func = (PyObject*)( data );
if( PyFunction_Check( func ) )
{
PyObject_CallFunction( func, "i", signum );
}
}
void addSignalHandlerWrapper( MyObject *o, int signum, PyObject *func )
{
Py_INCREF( func );
if( PyFunction_Check( func ) )
{
o->addSignalHandler( signum, &signalHandlerWrapper, func );
}
}
What I don't have, which I should add, is something in addSignalHandlerWrapper() that will check if there was already something fo that signal number, and if so, get it and decrement the reference before adding the new one. I haven't done this yet, as this functionality is only used for ending the program, but for completeness, it should be put in place.
Anyway, as I said at the start, this is more involved than it could be. It also only works as I have a singleton that can keep track of the function pointers.
I realize "fast" is a bit subjective so I'll explain with some context. I'm working on a Python module called psutil for reading process information in a cross-platform way. One of the functions is a pid_exists(pid) function for determining if a PID is in the current process list.
Right now I'm doing this the obvious way, using EnumProcesses() to pull the process list, then interating through the list and looking for the PID. However, some simple benchmarking shows this is dramatically slower than the pid_exists function on UNIX-based platforms (Linux, OS X, FreeBSD) where we're using kill(pid, 0) with a 0 signal to determine if a PID exists. Additional testing shows it's EnumProcesses that's taking up almost all the time.
Anyone know a faster way than using EnumProcesses to determine if a PID exists? I tried OpenProcess() and checking for an error opening the nonexistent process, but this turned out to be over 4x slower than iterating through the EnumProcesses list, so that's out as well. Any other (better) suggestions?
NOTE: This is a Python library intended to avoid third-party lib dependencies like pywin32 extensions. I need a solution that is faster than our current code, and that doesn't depend on pywin32 or other modules not present in a standard Python distribution.
EDIT: To clarify - we're well aware that there are race conditions inherent in reading process iformation. We raise exceptions if the process goes away during the course of data collection or we run into other problems. The pid_exists() function isn't intended to replace proper error handling.
UPDATE: Apparently my earlier benchmarks were flawed - I wrote some simple test apps in C and EnumProcesses consistently comes out slower and OpenProcess (in conjunction with GetProcessExitCode in case the PID is valid but the process has stopped) is actually much faster not slower.
OpenProcess could tell you w/o enumerating all. I have no idea how fast.
EDIT: note that you also need GetExitCodeProcess to verify the state of the process even if you get a handle from OpenProcess.
Turns out that my benchmarks evidently were flawed somehow, as later testing reveals OpenProcess and GetExitCodeProcess are much faster than using EnumProcesses after all. I'm not sure what happened but I did some new tests and verified this is the faster solution:
int pid_is_running(DWORD pid)
{
HANDLE hProcess;
DWORD exitCode;
//Special case for PID 0 System Idle Process
if (pid == 0) {
return 1;
}
//skip testing bogus PIDs
if (pid < 0) {
return 0;
}
hProcess = handle_from_pid(pid);
if (NULL == hProcess) {
//invalid parameter means PID isn't in the system
if (GetLastError() == ERROR_INVALID_PARAMETER) {
return 0;
}
//some other error with OpenProcess
return -1;
}
if (GetExitCodeProcess(hProcess, &exitCode)) {
CloseHandle(hProcess);
return (exitCode == STILL_ACTIVE);
}
//error in GetExitCodeProcess()
CloseHandle(hProcess);
return -1;
}
Note that you do need to use GetExitCodeProcess() because OpenProcess() will succeed on processes that have died recently so you can't assume a valid process handle means the process is running.
Also note that OpenProcess() succeeds for PIDs that are within 3 of any valid PID (See Why does OpenProcess succeed even when I add three to the process ID?)
There is an inherent race condition in the use of pid_exists function: by the time the calling program gets to use the answer, the process may have already disappeared, or a new process with the queried id may have been created. I would dare say that any application that uses this function is flawed by design and that optimizing this function is therefore not worth the effort.
I'd code Jay's last function this way.
int pid_is_running(DWORD pid){
HANDLE hProcess;
DWORD exitCode;
//Special case for PID 0 System Idle Process
if (pid == 0) {
return 1;
}
//skip testing bogus PIDs
if (pid < 0) {
return 0;
}
hProcess = handle_from_pid(pid);
if (NULL == hProcess) {
//invalid parameter means PID isn't in the system
if (GetLastError() == ERROR_INVALID_PARAMETER) {
return 0;
}
//some other error with OpenProcess
return -1;
}
DWORD dwRetval = WaitForSingleObject(hProcess, 0);
CloseHandle(hProcess); // otherwise you'll be losing handles
switch(dwRetval) {
case WAIT_OBJECT_0;
return 0;
case WAIT_TIMEOUT;
return 1;
default:
return -1;
}
}
The main difference is closing the process handle (important when the client of this function is running for a long time) and the process termination detection strategy. WaitForSingleObject gives you the opportunity to wait for a while (changing the 0 to a function parameter value) until the process ends.