I have a program that block reading with select,but i want it to return gracefully with sending a SIGINT signal. and process that signal with a handler. unfortunately, the block program does not return from select system call, instead it receives SIGINT and interupt.
can anybody help me with figuring out what is the problem? thanks a lot.
import select
import socket
import signal
class Test(object):
def __init__(self):
self._read, self._write = socket.socketpair()
def wait(self):
readable = [self._read.fileno()]
return select.select(readable, [], readable)
def wakeup(self):
self._write.send('1')
t = Test()
def handler(signum, frame):
t.wakeup()
signal.signal(signal.SIGINT, handler)
if __name__ == '__main__':
print t.wait()
print 'over.'
You misinterpreted the program's behaviour: it did return from the select system call, but with the error 'Interrupted system call' that you have to handle.
import errno
import select
import socket
import signal
class Test(object):
def __init__(self):
self._read, self._write = socket.socketpair()
def wait(self):
readable = [self._read.fileno()]
try:
return select.select(readable, [], readable)
except select.error as e:
if e.args[0] == errno.EINTR:
return
else:
pass
def wakeup(self):
self._write.send('1')
t = Test()
def handler(signum, frame):
t.wakeup()
signal.signal(signal.SIGINT, handler)
if __name__ == '__main__':
print t.wait()
print 'over.'
Related
Here is my script. When I run it in a shell it just hangs indefinitely whereas I would expect it to terminate cleanly.
import logging
from logging import StreamHandler
import pymsteams
import queue
import threading
import atexit
class TeamsHandler(StreamHandler):
def __init__(self, channel_url):
super().__init__()
self.channel_url = channel_url
self.queue = queue.Queue()
self.thread = threading.Thread(target=self._worker)
self.thread.start()
atexit.register(self.queue.put, None)
def _worker(self):
while True:
record = self.queue.get()
if record is None:
break
msg = self.format(record)
print(msg)
def emit(self, record):
# enqueue the record to log and return control to the caller
self.queue.put(record)
if __name__ == "__main__":
my_logger = logging.getLogger('TestLogging')
my_logger.setLevel(logging.DEBUG)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
my_logger.addHandler(console_handler)
CHANNEL_ID = "not_used_anyway"
teamshandler = TeamsHandler(CHANNEL_ID)
teamshandler.setFormatter(logging.Formatter('%(levelname)s %(message)s'))
teamshandler.setLevel(logging.DEBUG)
my_logger.addHandler(teamshandler)
for i in range(1, 2):
my_logger.error(f"this is an error [{i}]")
my_logger.info(f"this is an info [{i}]")
The None record that should be sent by atexit (line 28) never arrives so the thread stays open forever.
How to make sure that the program exits cleanly by modifying the TeamsHandler only ?
I got something working, have a look:
import queue
import threading
class Worker:
def __init__(self):
self.queue = queue.Queue()
threading.Thread(target=self._worker).start()
def _worker(self):
print("starting thread")
while True:
record = self.queue.get()
if record is None:
print("exiting")
break
print(f"Got message: {record}")
def emit(self, record):
self.queue.put(record)
class Wrapper:
def __init__(self):
self._worker = Worker()
def __del__(self):
print("Wrapper is being deleted")
self._worker.emit(None)
def emit(self, record):
self._worker.emit(record)
def main():
worker = Wrapper()
worker.emit("foo")
worker.emit("bar")
print("main exits")
if __name__ == "__main__":
main()
The point here is that when main exits, worker (which is an instance of Wrapper) goes out of scope, and its __del__ method is called, and it sends stop message to a real worker object.
The results of running this code ("Got message" lines can be in different places, of course):
starting thread
main exits
Wrapper is being deleted
Got message: foo
Got message: bar
exiting
As pointed out by avysk, the problem is likely that atexit handlers fire too late, after the waiting for the non-daemon threads is already (supposed to be) done, which leads to deadlock.
If I were you, I'd just add a call like TeamsHandler.finish() at the end of if __name__ == '__main__' block, and modify TeamsHandler along these lines (untested):
_queues = []
class TeamsHandler(StreamHandler):
def __init__(self, channel_url):
super().__init__()
self.channel_url = channel_url
self.queue = queue.Queue()
self.thread = threading.Thread(target=self._worker)
self.thread.start()
_queues.append(self.queue)
def _worker(self):
while True:
record = self.queue.get()
if record is None:
break
msg = self.format(record)
print(msg)
def emit(self, record):
# enqueue the record to log and return control to the caller
self.queue.put(record)
#staticmethod
def finish(self):
for q in _queues:
q.put(None)
del _queues[:]
I am trying to end a thread execution without directly referencing the thread. because it is not possible to do that in the full program.
for reference the main program is for the Raspberry Pi and I need it to stop executing a function/thread immediately once a button is pressed.
I have tried raising an exception from main but the other do not catch it for some reason.
Here is the scrap program that I have been testing on:
import threading
import time
class Thread_Exception(Exception):
def __init__(self, msg):
return super().__init__(msg)
def thread_function(index):
bool = True
try:
while bool:
print("Print from thread #", index)
time.sleep(4)
except Thread_Exception:
print('Exception thrown, thread #', index)
bool = False
if __name__ == "__main__":
try:
for index in range(3):
x = threading.Thread(target=thread_function, args=(index,))
x.start()
time.sleep(20)
raise Thread_Exception("intr")
while True:
continue
except KeyboardInterrupt:
print('Interrupted main')
an example of how it can be done:
import threading
import ctypes
import time
class thread_with_exception(threading.Thread):
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run(self):
# target function of the thread class
try:
while True:
print('running ' + self.name)
finally:
print('ended')
def get_id(self):
# returns id of the respective thread
if hasattr(self, '_thread_id'):
return self._thread_id
for id, thread in threading._active.items():
if thread is self:
return id
def raise_exception(self):
thread_id = self.get_id()
res = ctypes.pythonapi.PyThreadState_SetAsyncExc(thread_id,
ctypes.py_object(SystemExit))
if res > 1:
ctypes.pythonapi.PyThreadState_SetAsyncExc(thread_id, 0)
print('Exception raise failure')
t1 = thread_with_exception('Thread 1')
t1.start()
time.sleep(2)
t1.raise_exception()
t1.join()
The article this came from can currently be found here:
https://www.geeksforgeeks.org/python-different-ways-to-kill-a-thread/
I am exploring Klein and Deferred. In the following example I am trying to increment a number using a child process and return it via Future. I am able to receive the Future call back.
The problem is that deferred object never calls the cb() function and the request made to endpoint never returns. Please help me identify the problem.
Following is my server.py code
from klein import Klein
from twisted.internet.defer import inlineCallbacks, returnValue
import Process4
if __name__ == '__main__':
app = Klein()
#app.route('/visit')
#inlineCallbacks
def get_num_visit(request):
try:
resp = yield Process4.get_visitor_num()
req.setResponseCode(200)
returnValue('Visited = {}'.format(resp))
except Exception as e:
req.setResponseCode(500)
returnValue('error {}'.format(e))
print('starting server')
app.run('0.0.0.0', 5005)
Following is Process4.py code
from multiprocessing import Process
from concurrent.futures import Future
from time import sleep
from twisted.internet.defer import Deferred
def foo(x):
result = x+1
sleep(3)
return result
class MyProcess(Process):
def __init__(self, target, args):
super().__init__()
self.target = target
self.args = args
self.f = Future()
self.visit = 0
def run(self):
r = foo(self.visit)
self.f.set_result(result=r)
def cb(result):
print('visitor number {}'.format(result))
return result
def eb(err):
print('error occurred {}'.format(err))
return err
def future_to_deferred(future):
d = Deferred()
def callback(f):
e = f.exception()
if e:
d.errback(e)
else:
d.callback(f.result())
future.add_done_callback(callback)
return d
def get_visitor_num():
p1 = MyProcess(target=foo, args=None)
d = future_to_deferred(p1.f)
p1.start()
d.addCallback(cb)
d.addErrback(eb)
sleep(1)
return d
Edit 1
Adding callbacks before starting the process p1 solves the problem of calling cb() function. But still the http request made to the endpoint does not return.
It turns out that setting future result self.f.set_result(result=r) in the run() method triggers the callback() method in the child process, where no thread is waiting for the result to be returned!
So to get the callback() function triggered in the MainProcess I had to get the result from the child-process using a multiprocess Queue using a worker thread in the MainProcess and then set the future result.
#notorious.no Thanks for reply. One thing which I noticed is that reactor.callFromThread does switches result from worker thread to MainThread in my modified code however d.callback(f.result()) works just fine but returns result from worker thread.
Following is the modified working code
server.py
from klein import Klein
from twisted.internet.defer import inlineCallbacks, returnValue
import Process4
if __name__ == '__main__':
app = Klein()
visit_count = 0
#app.route('/visit')
#inlineCallbacks
def get_num_visit(req):
global visit_count
try:
resp = yield Process4.get_visitor_num(visit_count)
req.setResponseCode(200)
visit_count = resp
returnValue('Visited = {}'.format(resp))
except Exception as e:
req.setResponseCode(500)
returnValue('error {}'.format(e))
print('starting server')
app.run('0.0.0.0', 5005)
Process4.py
from multiprocessing import Process, Queue
from concurrent.futures import Future
from time import sleep
from twisted.internet.defer import Deferred
import threading
from twisted.internet import reactor
def foo(x, q):
result = x+1
sleep(3)
print('setting result, {}'.format(result))
q.put(result)
class MyProcess(Process):
def __init__(self, target, args):
super().__init__()
self.target = target
self.args = args
self.visit = 0
def run(self):
self.target(*self.args)
def future_to_deferred(future):
d = Deferred()
def callback(f):
e = f.exception()
print('inside callback {}'.format(threading.current_thread().name))
if e:
print('calling errback')
d.errback(e)
# reactor.callFromThread(d.errback, e)
else:
print('calling callback with result {}'.format(f.result()))
# d.callback(f.result())
reactor.callFromThread(d.callback, f.result())
future.add_done_callback(callback)
return d
def wait(q,f):
r = q.get(block=True)
f.set_result(r)
def get_visitor_num(x):
def cb(result):
print('inside cb visitor number {} {}'.format(result, threading.current_thread().name))
return result
def eb(err):
print('inside eb error occurred {}'.format(err))
return err
f = Future()
q = Queue()
p1 = MyProcess(target=foo, args=(x,q,))
wait_thread = threading.Thread(target=wait, args=(q,f,))
wait_thread.start()
defr = future_to_deferred(f)
defr.addCallback(cb)
defr.addErrback(eb)
p1.start()
print('returning deferred')
return defr
I am trying to write a class to handle signals using the signal python module. The reason for having a class is to avoid the use of globals. This is the code I came up with, but unfortunately it is not working:
import signal
import constants
class SignalHandler (object):
def __init__(self):
self.counter = 0
self.break = False
self.vmeHandlerInstalled = False
def setVmeHandler(self):
self.vmeBufferFile = open('/dev/vme_shared_memory0', 'rb')
self.vmeHandlerInstalled = True
signal.signal(signal.SIGUSR1, self.traceHandler)
signal.siginterrupt(signal.SIGUSR1, False)
#...some other stuff...
def setBreakHandler(self):
signal.signal(signal.SIGINT, self.newBreakHandler)
signal.siginterrupt(signal.SIGINT, False)
def newBreakHandler(self, signum, frame):
self.removeVMEHandler()
self.break = True
def traceHandler(self, signum, frame):
self.counter += constants.Count
def removeVMEHandler(self):
if not self.vmeHandlerInstalled: return
if self.vmeBufferFile is None: return
signal.signal(signal.SIGUSR1, signal.SIG_DFL)
self.vmeHandlerInstalled = False
On the main program I use this class in the following way:
def run():
sigHandler = SignalHandler()
sigHandler.setBreakHandler()
sigHandler.setVmeHandler()
while not sigHandler.break:
#....do some stuff
if sigHandler.counter >= constants.Count:
#...do some stuff
This solution is not working, as it appears that the handler for the signal.SIGUSR1 installed in the setVmeHandler method never gets called.
So my question is: is it possible to handle signal inside a class or shall I use globals?
To answer your question, I created the following simple code:
import signal
import time
class ABC(object):
def setup(self):
signal.signal(signal.SIGUSR1, self.catch)
signal.siginterrupt(signal.SIGUSR1, False)
def catch(self, signum, frame):
print("xxxx", self, signum, frame)
abc = ABC()
abc.setup()
time.sleep(20)
If I run it:
python ./test.py
Then in another window send a USR1 signal:
kill -USR1 4357
The process prints the expected message:
('xxxx', <__main__.ABC object at 0x7fada09c6190>, 10, <frame object at 0x7fada0aaf050>)
So I think the answer is Yes, it possible to handle signal inside a class.
As for why you code doesn't work, sorry, I have no idea.
I got a similar problem as toti08, referring to setVmeHandler(self), and found out the handler must have matching parameters i.e. (self, signum,frame).
I'm having problem with QThreads in python.
I would like to start my multi QThread when I push on button Run.
But the compiler outputs following error:
"QThread: Destroyed while thread is still running"
I don't know what is wrong with my code.
Any help would be appreciated.
Here is my code:
# -*- coding: utf-8 -*-
from PySide import QtCore, QtGui
from Ui_MainWindow import Ui_MainWindow
from queue import Queue
import sys, re, random
import time, random, re, urllib.request
from urllib.parse import urljoin
class Worker(QtCore.QThread):
def __init__(self,threadID, name, q, delay):
QtCore.QThread.__init__(self)
self.threadID = threadID
self.name = name
self.q = q
self.delay = delay
self._running = False
def run(self):
self._running = True
print ("start - %s" %self.name)
while self._running:
req = self.request(self.name, self.q, self.delay)
def stop(self, wait=False):
print (self.name)
self._running = False
def request(self, threadName, q1, delay):
while not self.q.empty():
time.sleep(delay)
q = q1.get()
print ("%s: %s %s %s" % (threadName, time.ctime(time.time()), q, delay))
if self.q.empty():
print ("queue empty")
self.stop()
class MainWindow(QtGui.QMainWindow):
def __init__(self, parent=None):
QtGui.QMainWindow.__init__(self, parent)
self.ui = Ui_MainWindow()
self.ui.setupUi(self)
self.backend = Queue()
self.connect(self.ui.actionStart, QtCore.SIGNAL('triggered()'),self.start)
def start(self):
try :
f1 = open('./payload/backend.log')
except FileNotFoundError as e:
return
threadList = ["Thread-1", "Thread-2", "Thread-3", "Thread-4", "Thread-5"]
self.url = "http://test.com"
self.threads = []
threadID = 1
for payload in f1.read().splitlines() :
full_url = urljoin(self.url, payload)
self.backend_dir.put(full_url)
for tName in threadList:
ran_int = random.randint(1, 2)
downloader = Worker(threadID, tName, self.backend, ran_int)
downloader.start()
self.threads.append(downloader)
if __name__ == "__main__":
app = QtGui.QApplication(sys.argv)
window = MainWindow()
window.show()
sys.exit(app.exec_())
log
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
QThread: Destroyed while thread is still running
You are trying to do something that is a built-in of Qt: a QThreadPool.
I would advice you to read the doc and use it instead.
If you really want to use QThread:
You should not subclass QThread. Instead you should subclass the basic QObject to create your worker and use the moveToThread method:
class WorkerSignals(QObject):
finished = pyqtSignal()
class Worker(QObject):
def __init__():
self.signal = WorkerSignals()
def run(self):
# Your stuff
print('running')
self.signal.finished.emit()
Then somewhere else:
thread = QThread()
worker = Worker(...)
worker.moveToThread(thread)
thread.started.connect(worker.run)
worker.finished.connect(thread.quit)
worker.finished.connect(worker.deleteLater)
thread.finished(thread.deleteLater)
thread.start()
The solution is a rough translation of this one in C++:
https://mayaposch.wordpress.com/2011/11/01/how-to-really-truly-use-qthreads-the-full-explanation/
Hope this helps!
The problem is caused by the way you are using the queue.
All the threads start and begin their tasks normally, up until the queue becomes empty. At that point, the first thread to finish terminates, but the other four threads are left waiting for an item to be returned from the queue, which never comes.
This is because you use get with no arguments, which will block indefinitely until an item becomes available. Instead, you should use get_nowait, and also call stop() unconditionally at the end of request():
from queue import Queue, Empty
...
class Worker(QtCore.QThread):
...
def request(self, threadName, q1, delay):
while not q1.empty():
time.sleep(delay)
try:
q = q1.get_nowait()
except Empty:
break
else:
print ("%s: %s %s %s" % (threadName, time.ctime(time.time()), q, delay))
print ("queue empty")
self.stop()
I believe that you need to call self.threads.append(downloader) before downloader.start() so that the thread doesn't go out of scope and get garbage collected.