What's the recommended way of sending message from a worker process to another randomly selected (worker or master) process? One approach that I can think of is using Pipes, but since it can only create a pipe between two selected processes, I need to create a pipe for each process pair. This doesn't seem so practical. What I want is to create a complete graph between processes and select one of the pipes randomly.
You could use Queue in order to communicate among your processes by maintaining some convention in your queue.Your could find the details on using Queue here.
P.S :- As mentioned here Queues are thread and process safe.
Related
I am trying to pass messages between my processes. I have a process that is a client to a server. I want other processes to pass information to the client to pass it to the server. Think of the server as a data logger. I am using concurrent.futures.ProcessPoolExecutor. The processes are working continuously in a while True.
Any hint in how to connect the processes is appreciated,
Thank you.
Edit: I found the answer. All credit goes to whomever posted the following post here.
import multiprocessing
qm = multiprocessing.Manager()
pqueue = qm.Queue()
Processes in general are not able to talk to each other through python's standard library. The memory space for the process is allocated by the OS and in general does not overlap. Parent processes (like your original python process) have some limited visibility of child processes ( like the processes created using ProcessPoolExecutor), but not the ability to pass messages between them.
You have a couple options here.
1) Use threads. If you don't actually need parallelism (ex using all cores of a multi-core system), this option might work. Threads all run within a single process and have shared memory space. So you can write to an object from one thread and read from it in another. concurrent.futures.ThreadPoolExecutor will allow you to do this.
2)Use the underlying OS. In general, when two processes running on the same system need to communicate, they do it over ports. The process that is receiving messages will bind to a port. The process that is sending messages will then establish a socket connection over that port, and use this connection to communicate. Refer to https://docs.python.org/3.7/howto/sockets.html for more details.
3)Use a message broker like rabbitMQ or reddis. These are external processes that facilitate communication between processes.
I have python GUI application which can kick off any number of computation-intensive long-running tasks that naturally belong in multiprocessing.Pool workers.
However, I'd like to be able to cancel these tasks, because later GUI input (such as changing a configuration variable) might render these tasks irrelevant.
Is there a popular pattern in Python for keeping track of which workers are working on what task, and interrupting them as needed?
The solutions I can think of are:
When a worker starts on a task it "announces" through some shared state that it is working on that particular task; if we need to cancel that task we look up which process is working on it and .terminate() it. There are many complexities here though.
Use raw multiprocessing.Processes and write a Pool-like manager that does exactly what we want.
Use some alternative library such as Celery. A huge list is here.
I am using PyQt to create the gui for my application and ran into some trouble using threads for seperate processes, so started to use the multiprocessing.Process class. I was, before, using Signals and slots to communicate between the worker process and the gui, but the SignalInstance class can not be pickled and as far as I know cant be used with Process so I am having to find another way to send a progress report (percent done etc) from the worker process to update a progress bar in the gui. what is the best way of doing this?
Please see this answer, you can share memory between processes using the multiprocessing library. Documentation here (see 16.6.1.4. Sharing state between processes).
Is it possible to create a long running process in NodeJs to handle many background operations without interrupting the main thread; something like Celery in Python.
Hint, it's highly preferable to be able to manage that long-running process, in case of failure, or need to be restarted, away from the main process.
http://nodejs.org/api/child_process.html is the right API to create long-running processes, you will have complete control over the child processes (access to stdin/out/err, can send signals etc). This approach however requires that your node process is parent of those children.. If you want the child to outlive the parent, take a look at options.detached during child creation (and following child.unref()).
Please note, however, that Node.js is suited extremely well to avoid such architecture. Typically node.js do all the background stuff in the main thread. I've been writing apps with lots of traffic (like thousands requests per second), with DB, Redis and RabbitMQ access all from the main thread and without any child processes - and it was worked fine, as it should, thanks to Node's evented IO system.
I'm generally using child_process api only to launch separate executables (e.g. ffmpeg to transcode some video file), apart of such scenarios separate processes are probably not what you want.
There is also cluster api which allow single master to handle numerous worker processes, though I think it isn't what you look for, either.
You can create child process to handle your background operations. And then use messages to pass data between the new process and your main thread.
http://nodejs.org/api/child_process.html
Update
It looks like you need to use the server queues, sort of beanstalkd http://kr.github.io/beanstalkd/ + https://www.npmjs.com/package/fivebeans.
hi lets assume i have a simple programm in python. This programm is running every five minutes throught cron. but i dont know how to write it so the programm will allow to run multiple processes of its self simultaneously. i want to speed things up ...
I'd handle the forking and process control inside your main python program. Let the cron spawn only a single process and that process be a master for (possible multiple) worker processes.
As for how you can create multiple workers, there's the threading module for multi threading and multiprocessing module for multi processing. You can also keep your actual worker code as separate files and use the subprocess module.
Now that I think about it, maybe you should use supervisord to do the actual process control and simply write the actual work code.