I'm working on a Python 2.7 script using threading.
There is one global connection object, which has to be used by each thread.
Code Example:
from threading import Thread
import time
class Connection:
def __init__(self):
self.connected = True
def send_command(self, command):
return str(command)+' result'
class Config:
def __init__(self):
self.conn = Connection()
def do_remote_config(self):
time.sleep(2)
return self.conn.send_command('my config')
def do_other_remote_config(self):
time.sleep(2)
return self.conn.send_command('my other config')
class Executor:
def execute(self):
config = Config()
worker1 = Worker(config.do_remote_config)
worker1.start()
worker1.join()
print(worker1.result)
worker2 = Worker(config.do_other_remote_config)
worker2.start()
worker2.join()
print(worker2.result)
class Worker(Thread):
def __init__(self, method):
super(Worker, self).__init__()
self.result = None
self.method = method
def run(self):
try:
self.result = self.method()
except Exception as ex:
self.result = ex
if __name__ == "__main__":
e = Executor()
e.execute()
In order to ensure that none of the threads runs for more than 10 minutes, I wanted to kill each thread in case the time limit is reached. Unfortunately, it turns out that Python threads cannot be killed.
Thread Kill Pill Option:
Due to the actual complexity of the worker threads, it is unfortunately not possible to build some kind of kill-trigger, which lets the worker thread end himself. So, it seems that I really need to get rid of threading here because threads by nature cannot be killed.
Multiprocess Option:
Using the multiprocess module, different processes could be used. Those could then be killed after a certain time. However I did not find a way to pass on my connection object in such a way that it can be used by several processes.
Remote Procedure Calls (RPC) option:
RPCs seem to introduce an unnecessary level of complexity and the kill switch could presumably still not be implemented.
Question:
Which Python technologies would work best in order to being able to use the connection object with all workers while ensuring that each worker can reliably be killed after 10 minutes?
Thanks very much!
Too long for a comment, too abstract for an answer.
I would say that multiprocessing is the way to go if you wish to be able to interrupt processing in a random moment without revamping the whole processing. All other methods demand some sort of cooperation from threads being interrupted.
Certainly splitting the whole process into pieces demand some processing changes as well. All shared file-like resources (opened files, sockets, pipes) are to be opened before the forking of processes and carefully orchestrated. Probably the safest approach would be like this:
you have a master socket being listen()ed by a master process. The master also runs a workers pool. It's essential to create the master socket before the pool, to make the socket available to the workers.
The master delivers new job tasks to the workers and receives results if needed via multiprocessing primitives.
When a new client arrives, the master orders a selected worker from the pool to accept() the connection and returns to back to waiting for new clients and other master's activities. The worker accept()s the connection thus creating a private socket to communicate with a particular client, no other workers can and should access the client socket.
If workers need to communicate with each other, all necessary communication primitives must be created before the pool and distributed among the workers by the master.
Related
I've been stuck on this issue all day, and I have not been able to find any solutions relating to what I am trying to accomplish.
I am trying to pass Queues to threads spawned in sub-processes. The Queues were created in the entrance file and passed to each sub-process as a parameter.
I am making a modular program to a) run a neural network b) automatically update the network models when needed c) log events/images from the neural network to the servers. My former program idolized only one CPU-core running multiple threads and was getting quite slow, so I decided I needed to sub-process certain parts of the program so they can run in their own memory spaces to their fullest potential.
Sub-process:
Client-Server communication
Webcam control and image processing
Inferencing for the neural networks (there are 2 neural networks with their own process each)
4 total sub-processes.
As I develop, I need to communicate across each process so they are all on the same page with events from the servers and whatnot. So Queue would be the best option as far as I can tell.
(Clarify: 'Queue' from the 'multiprocessing' module, NOT the 'queue' module)
~~ However ~~
Each of these sub-processes spawn their own thread(s). For example, the 1st sub-process will spawn multiple threads: One thread per Queue to listen to the events from the different servers and hand them to different areas of the program; one thread to listen to the Queue receiving images from one of the neural networks; one thread to listen to the Queue receiving live images from the webcam; and one thread to listen to the Queue receiving the output from the other neural network.
I can pass the Queues to the sub-processes without issue and can use them effectively. However, when I try to pass them to the threads within each sub-process, I get the above error.
I am fairly new to multiprocessing; however, the methodology behind it looks to be relatively the same as threads except for the shared memory space and GIL.
This is from Main.py; the program entrance.
from lib.client import Client, Image
from multiprocessing import Queue, Process
class Main():
def __init__(self, server):
self.KILLQ = Queue()
self.CAMERAQ = Queue()
self.CLIENT = Client((server, 2005), self.KILLQ, self.CAMERAQ)
self.CLIENT_PROCESS = Process(target=self.CLIENT.do, daemon=True)
self.CLIENT_PROCESS.start()
if __name__ == '__main__':
m = Main('127.0.0.1')
while True:
m.KILLQ.put("Hello world")
And this is from client.py (in a folder called lib)
class Client():
def __init__(self, connection, killq, cameraq):
self.TCP_IP = connection[0]
self.TCP_PORT = connection[1]
self.CAMERAQ = cameraq
self.KILLQ = killq
self.BUFFERSIZE = 1024
self.HOSTNAME = socket.gethostname()
self.ATTEMPTS = 0
self.SHUTDOWN = False
self.START_CONNECTION = MakeConnection((self.TCP_IP, self.TCP_PORT))
# self.KILLQ_THREAD = Thread(target=self._listen, args=(self.KILLQ,), daemon=True)
# self.KILLQ_THREAD.start()
def do(self):
# The function ran as the subprocess from Main.py
print(self.KILLQ.get())
def _listen(self, q):
# This is threaded multiple times listening to each Queue (as 'q' that is passed when the thread is created)
while True:
print(self.q.get())
# self.KILLQ_THREAD = Thread(target=self._listen, args=(self.KILLQ,), daemon=True)
This is where the error is thrown. If I leave this line commented, the program runs fine. I can read from the queue in this sub-process without issue (i.e. the function 'do') not in a thread under this sub-process (i.e. the function '_listen').
I need to be able to communicate across each process so they can be in step with the main program (i.e. in the case of a neural network model update, the inference sub-process needs to shut down so the model can be updated without causing errors).
Any help with this would be great!
I am also very open to other methods of communication that would work as well. In the event that you believe a better communication process would work; it would need to be fast enough to support real-time streaming of 4k images sent to the server from the camera.
Thank you very much for your time! :)
The queue is not the problem. The ones from the multiprocessing package are designed to be picklable, so that they can be shared between processes.
The issue is, that your thread KILLQ_THREAD is created in the main process. Threads are not to be shared between processes. In fact, when a process is forked following POSIX standards, threads that are active in the parent process are not part of the process image that is cloned to the new child's memory space. One reason is that the state of mutexes at the time of the call to fork() might lead to deadlocks in the child process.
You'll have to move the creation of your thread to your child process, i.e.
def do(self):
self.KILLQ_THREAD = Thread(target=self._listen, args=(self.KILLQ,), daemon=True)
self.KILLQ_THREAD.start()
Presumably, KILLQ is supposed to signal the child processes to shut down. In that case, especially if you plan to use more than one child process, a queue is not the best method to achieve that. Since Queue.get() and Queue.get_nowait() remove the item from the queue, each item can only be retrieved and processed by one consumer. Your producer would have to put multiple shutdown signals into the queue. In a multi-consumer scenario, you also have no reasonable way to ensure that a specific consumer receives any specific item. Any item put into the queue can potentially be retrieved by any of the consumers reading from it.
For signalling, especially with multiple recipients, better use Event
You'll also notice, that your program appears to hang quickly after starting it. That's because you start both, your child process and the thread with daemon=True.
When your Client.do() method looks like above, i.e. creates and starts the thread, then exits, your child process ends right after the call to self.KILLQ_THREAD.start() and the daemonic thread immediately ends with it. Your main process does not notice anything and continues to put Hello world into the queue until it eventually fills up and queue.Full raises.
Here's a condensed code example using an Event for shutdown signalling in two child processes with one thread each.
main.py
import time
from lib.client import Client
from multiprocessing import Process, Event
class Main:
def __init__(self):
self.KILLQ = Event()
self._clients = (Client(self.KILLQ), Client(self.KILLQ))
self._procs = [Process(target=cl.do, daemon=True) for cl in self._clients]
[proc.start() for proc in self._procs]
if __name__ == '__main__':
m = Main()
# do sth. else
time.sleep(1)
# signal for shutdown
m.KILLQ.set()
# grace period for both shutdown prints to show
time.sleep(.1)
client.py
import multiprocessing
from threading import Thread
class Client:
def __init__(self, killq):
self.KILLQ = killq
def do(self):
# non-daemonic thread! We want the process to stick around until the thread
# terminates on the signal set by the main process
self.KILLQ_THREAD = Thread(target=self._listen, args=(self.KILLQ,))
self.KILLQ_THREAD.start()
#staticmethod
def _listen(q):
while not q.is_set():
print("in thread {}".format(multiprocessing.current_process().name))
print("{} - master signalled shutdown".format(multiprocessing.current_process().name))
Output
[...]
in thread Process-2
in thread Process-1
in thread Process-2
Process-2 - master signalled shutdown
in thread Process-1
Process-1 - master signalled shutdown
Process finished with exit code 0
As for methods of inter process communication, you might want to look into a streaming server solution.
Miguel Grinberg has written an excellent tutorial on Video Streaming with Flask back in 2014 with a more recent follow-up from August 2017.
I need to receive raw packets from a network interface within Twisted code. The packets will not have the correct IP or MAC address, nor valid headers, so I need the raw thing.
I have tried looking into twisted.pair, but I was not able to figure out how to use it to get at the raw interface.
Normally, I would use scapy.all.sniff. However, that is blocking, so I can't just use it with Twisted. (I also cannot use scapy.all.sniff with a timeout and busy-loop, because I don't want to lose packets.)
A possible solution would be to run scapy.all.sniff in a thread and somehow call back into Twisted when I get a packet. This seems a bit inelegant (and also, I don't know how to do it because I am a Twisted beginner), but I might settle for that if I don't find anything better.
You could run a distributed system and pass the data through a central queuing system. Take the Unix philosophy and create a single application that does a few tasks and does them well. Create one application that sniffs the packets (you can use scapy here since it won't really matter if you block anything) then sends them to a queue (RabitMQ, Redis, SQS, etc) and have another application process the packet from the queue. This method should give you the least amount of headache.
If you need to run everything in a single application, then threads/multiprocessing is the only option. But there are some design patterns you'll want to follow. You can also break up the following code into separate functions and use a dedicated queuing system.
from threading import Thread
from time import sleep
from twisted.internet import defer, reactor
class Sniffer(Thread):
def __init__(self, _reactor, shared_queue):
super().__init__()
self.reactor = _reactor
self.shared_queue = shared_queue
def run(self):
"""
Sniffer logic here
"""
while True:
self.reactor.callFromThread(self.shared_queue.put, 'hello world')
sleep(5)
#defer.inlineCallbacks
def consume_from_queue(_id, _reactor, shared_queue):
item = yield shared_queue.get()
print(str(_id), item)
_reactor.callLater(0, consume_from_queue, _id, _reactor, shared_queue)
def main():
shared_queue = defer.DeferredQueue()
sniffer = Sniffer(reactor, shared_queue)
sniffer.daemon = True
sniffer.start()
workers = 4
for i in range(workers):
consume_from_queue(i+1, reactor, shared_queue)
reactor.run()
main()
The Sniffer class starts outside of Twisted's control. Notice the sniffer.daemon = True, this is so that the thread will stop when the main thread has stopped. If it were set to False (default) then the application will exit only if all the threads have come to an end. Depending on the task at hand this may or may not always be possible. If you can take breaks from sniffing to check a thread event, then you might be able to stop the thread in a safer way.
self.reactor.callFromThread(self.shared_queue.put, 'hello world') is necessary so that the item being put into the queue happens in the main reactor thread as opposed to the thread the Sniffer executes. The main benefit of this would be that there would be some sort of synchronization of the messages coming from the threads (assuming you plan to scale to sniffing multiple interfaces). Also, I wasn't sure of DeferredQueue objects are thread safe :) I treated them like they were not.
Since Twisted isn't managing the threads in this case, it's vital that the developer does. Notice the worker loop and consume_from_queue(i+1, reactor, shared_queue). This loop ensures only the desired number of workers are handling tasks. Inside the consume_from_queue() function, shared_queue.get() will wait (non-blocking) until an item is put into the queue, prints the item, then schedule another consume_from_queue().
I am creating a custom job scheduler with a web frontend in python 3.4 on linux. This program creates a daemon (consumer) thread that waits for jobs to come available in a PriorityQueue. These jobs can manually be added through the web interface which adds them to the queue. When the consumer thread finds a job, it executes a program using subprocess.run, and waits for it to finish.
The basic idea of the worker thread:
class Worker(threading.Thread):
def __init__(self, queue):
self.queue = queue
# more code here
def run(self):
while True:
try:
job = self.queue.get()
#do some work
proc = subprocess.run("myprogram", timeout=my_timeout)
#do some more things
except TimeoutExpired:
#do some administration
self.queue.add(job)
However:
This consumer should be able to receive some kind of signal from the frontend (main thread) that it should stop the current job and instead work on the next job in the queue (saving the state of the current job and adding it to the end of the queue again). This can (and will most likely) happen while blocked on subprocess.run().
The subprocesses can simply be killed (the program that is executed saves sme state in a file) but the worker thread needs to do some administration on the killed job to make sure it can be resumed later on.
There can be multiple such worker threads.
Signal handlers are not an option (since they are always handled by the main thread which is a webserver and should not be bothered with this).
Having an event loop in which the process actively polls for events (such as the child exiting, the timeout occurring or the interrupt event) is in this context not really a solution but an ugly hack. The jobs are performance-heavy and constant context switches are unwanted.
What synchronization primitives should I use to interrupt this thread or to make sure it waits for several events at the same time in a blocking fashion?
I think you've accidentally glossed over a simple solution: your second bullet point says that you have the ability to kill the programs that are running in subprocesses. Notice that subprocess.call returns the return code of the subprocess. This means that you can let the main thread kill the subprocess, and just check the return code to see if you need to do any cleanup. Even better, you could use subprocess.check_call instead, which will raise an exception for you if the returncode isn't 0. I don't know what platform you're working on, but on Linux, killed processes generally don't return a 0 if they're killed.
It could look something like this:
class Worker(threading.Thread):
def __init__(self, queue):
self.queue = queue
# more code here
def run(self):
while True:
try:
job = self.queue.get()
#do some work
subprocess.check_call("myprogram", timeout=my_timeout)
#do some more things
except (TimeoutExpired, subprocess.CalledProcessError):
#do some administration
self.queue.add(job)
Note that if you're using Python 3.5, you can use subprocess.run instead, and set the check argument to True.
If you have a strong need to handle the cases where the worker needs to be interrupted when it isn't running the subprocess, then I think you're going to have to use a polling loop, because I don't think the behavior you're looking for is supported for threads in Python. You can use a threading.Event object to pass the "stop working now" pseudo-signal from your main thread to the worker, and have the worker periodically check the state of that event object.
If you're willing to consider using multiple processing stead of threads, consider switching over to the multiprocessing module, which would allow you to handle signals. There is more overhead to spawning full-blown subprocesses instead of threads, but you're essentially looking for signal-like asynchronous behavior, and I don't think Python's threading library supports anything like that. One benefit though, would be that you would be freed from the Global Interpreter Lock(PDF link), so you may actually see some speed benefits if your worker processes (formerly threads) are doing anything CPU intensive.
I'd like to create a set of processes with the following structure:
main, which dequeues requests from an external source. main generates a variable number of worker processes.
worker which does some preliminary processing on job requests, then sends data to gpuProc.
gpuProc, which accepts job requests from worker processes. When it has received enough requests, it sends the batch to a process that runs on the GPU. After getting the results back, it has to then send back the completed batch of requests back to the worker processes such that the worker that requested it receives it back
One could envision doing this with a number of queues. Since the number of worker processes is variable, it would be ideal if gpuProc had a single input queue into which workers put their job request and their specific return queue as a tuple. However, this isn't possible--you can only share vanilla queues in python via inheritance, and manager.Queues() fail with:
RemoteError:
---------------------------------------------------------------------------
Unserializable message: ('#RETURN', ('Worker 1 asked proc to do some work.', <Queue.Queue instance at 0x7fa0ba14d908>))
---------------------------------------------------------------------------
Is there a pythonic way to do this without invoking some external library?
multiprocessing.Queue is implemented with a pipe, a deque and a thread.
When you call queue.put() the objects ends up in the deque and the thread takes care of pushing it into the pipe.
You cannot share threads within processes for obvious reasons. Therefore you need to use something else.
Regular pipes and sockets can be easily shared.
Nevertheless I'd rather use a different architecture for your program.
The main process would act as an orchestrator routing the tasks to two different Pools of processes, one for CPU bound jobs and the other to GPU bound ones. This would imply you need to share more information within the workers but it's way more robust and scalable.
Here you get a draft:
from multiprocessing import Pool
def cpu_worker(job_type, data):
if job_type == "first_computation":
results do_cpu_work()
elif job_type == "compute_gpu_results":
results = do_post_gpu_work()
return results
def gpu_worker(data):
return do_gpu_work()
class Orchestrator:
def __init__(self):
self.cpu_pool = Pool()
self.gpu_pool = Pool()
def new_task(self, task):
"""Entry point for a new task. The task will be run by the CPU workers and the results handled by the cpu_job_done method."""
self.cpu_pool.apply_async(cpu_worker, args=["first_computation", results], callback=self.cpu_job_done)
def cpu_job_done(self, results):
"""Once the first CPU computation is done, send its results to a GPU worker. Its results will be handled by the gpu_job_done method."""
self.gpu_pool.apply_async(gpu_worker, args=[results], callback=self.gpu_job_done)
def gpu_job_done(self, results):
"""GPU computation done, send the data back for the last CPU computation phase. Results will be handled by the task_done method."""
self.cpu_pool.apply_async(cpu_worker, args=["compute_gpu_results", results], callback=self.task_done)
def task_done(self, results):
"""Here you get your final results for the task."""
print(results)
I'm trying hard to figure out how I can process a directed acyclic graph in parallel. Each node should only be able to "execute" when all its input nodes have been processed beforehand. Imagine a class Task with the following interface:
class Task(object):
result = None
def inputs(self):
''' List all requirements of the task. '''
return ()
def run(self):
pass
I can not think of a way to process the graph that could be represented
by this structure asynchronously with a maximum number of workers at the
same time, except for one method.
I think the optimal processing would be achieved by creating a thread
for each task, waiting for all inputs to be processed. But, spawning
a thread for each task immediately instead of consecutively (i.e. when the
task is ready to be processed) does not sound like a good idea to me.
import threading
class Runner(threading.Thread):
def __init__(self, task):
super(Runner, self).__init__()
self.task = task
self.start()
def run(self):
threads = [Runner(r) for r in self.task.inputs()]
[t.join() for t in threads]
self.task.run()
Is there a way to mimic this behaviour more ideally? Also, this approach
does currently not implement a way to limit the number of running tasks at
a time.
Have one master thread push items to a queue once they are ready for being processsed. Then have a pool of workers listen on the queue for tasks to work on. (Python provides a synchronized queue in the Queue module, renamed to lower-case queue in Python 3).
The master first creates a map from dependencies to dependent tasks. Every task that doesn't have any dependcies can go into the queue. Everytime a task is completed, the master uses the dictionary to figure out which dependent tasks there are, and puts them into the queue if all their depndencies are met now.
Celery (http://www.celeryproject.org/) is the leading task management tool for Python. It should be able to help you with this.