I have aa few scripts written in python.
I am trying to multi thread them.
When Script A starts. I would like scripts B, C, and D to start.
After A runs, I would A2 to run.
After B runs, I would B2 to run, then B3.
C and D have no follow up scripts.
I have checked that the scripts are independent of each other.
I planning on using "exec" to launch them, and would like to use this "launcher" on Linux and Windows."
I have other multi thread scripts mainly do a procedure A with five threads. This throwing me because all procedures are different but could start and run at the same time.
Ok I'm still not sure where exactly your problem is, but that's the way I'd solve the problem:
#Main.py
from multiprocessing import Process
import ScriptA
# import all other scripts as well
def handle_script_a(*args):
print("Call one or several functions from Script A or calculate some stuff beforehand")
ScriptA.foo(*args)
if __name__ == '__main__':
p = Process(target=handle_script_a, args=("Either so", ))
p1 = Process(target=ScriptA.foo, args=("or so", ))
p.start()
p1.start()
p.join()
p1.join()
# ScriptA.py:
def foo(*args):
print("Function foo called with args:")
for arg in args:
print(arg)
You can either call a function directly or if you want to call several functions in one process use a small wrapper for it. No platform dependent code, no ugly execs and you can create/join processes easily in whatever way fancies you.
And a small example of a queue for interprocess communication - pretty much stolen from the python API but well ;)
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print(q.get()) # prints "[42, None, 'hello']"
p.join()
Create the queue and give it one or more processes. Note that get() blocks, if you want non blocking you can use get_nowait() or specify a timeout as 2nd argument. If you want shared objects there'd be multiprocessing.Array or multiprocessing.Value, just read the documentation for specific information doc link
If you've got more questions relative to IPC create a new question - a extremely large topic in itself.
So it doesn't have to be a Python launcher? Back when I was doing heavy sys admin, I wrote a Perl script using the POE framework to run scripts or whatever with a limited concurrency. Worked great. for example when we had to run a script over a thousand user accounts or a couple of hundred data bases. Limit it to just 4 jobs at a time on an 4-cpu box, 16 on a 16-way server, or any arbitrary number. POE does use fork() to create child procs, but on Windows boxes that works fine under cygwin, FWIW.
A while back I was looking for an equivalent event framework for Python. Looking again today I see Twisted--and some posts indicating that it runs even faster than POE--but maybe Twisted is mostly for network client/server? POE's incredibly flexible. It's tricky at first if you're not used to event driven scripting, and even if you are, but events are a lot easier to grock than threads. (Maybe over-kill for your needs? It's years later I'm still surprised there's not a simple utility to control throughput on multi-cpu machines.)
Related
I have 24 cores on my machine, but I just can't get them all running. When I top, only 3 processes are running, and usually only one hits 100% CPU, the other two ~30%.
I've read all the related threads on this site, but still can't figure out what's wrong with my code.
Pseudocode of how I used pool is as follows
import multiprocessing as mp
def Foo():
pool = mp.Pool(mp.cpu_count())
def myCallbackFun():
pool.map(myFunc_wrapper, myArgs)
optimization(callback=myCallbackFun) # scipy optimization that has a callback function.
Using pdb, I stopped before optimization, and checked I indeed have 24 workers.
But when I resume the program, top tells me I only have three Python processes running. Another thing is, when I ctrl-c to terminate my program, it has soooo many workers to interrupt (e.g., PoolWorker-367) -- I've pressing ctrl-c for minutes, but there are still workers out there. Shouldn't there be just 24 workers?
How to make my program use all CPUs?
With multiprocessing Python starts new processes. With a script like yours it will fork infinitely. You need to wrap the script part of your module like this:
import multiprocessing as mp
if __name__ == '__main__':
pool = mp.Pool(24)
pool.map(myFunc_wrapper, myArgs)
For future readers --
As #mata correctly points out,
You may be running into an IO bottleneck if your involved arguments
are very big
This is indeed my case. Try to minimize the size of the arguments passed to each process.
I have a python script that has to take many permutations of a large dataset, score each permutation, and retain only the highest scoring permutations. The dataset is so large that this script takes almost 3 days to run.
When I check my system resources in windows, only 12% of my CPU is being used and only 4 out of 8 cores are working at all. Even if I put the python.exe process at highest priority, this doesn't change.
My assumption is that dedicating more CPU usage to running the script could make it run faster, but my ultimate goal is to reduce the runtime by at least half. Is there a python module or some code that could help me do this? As an aside, does this sound like a problem that could benefit from a smarter algorithm?
Thank you in advance!
There are a few ways to go about this, but check out the multiprocessing module. This is a standard library module for creating multiple processes, similar to threads but without the limitations of the GIL.
You can also look into the excellent Celery library. This is a distrubuted task queue, and has a lot of great features. Its a pretty easy install, and easy to get started with.
I can answer a HOW-TO with a simple code sample. While this is running, run /bin/top and see your processes. Simple to do. Note, I've even included how to clean up afterwards from a keyboard interrupt - without that, your subprocesses will keep running and you'll have to kill them manually.
from multiprocessing import Process
import traceback
import logging
import time
class AllDoneException(Exception):
pass
class Dum(object):
def __init__(self):
self.numProcesses = 10
self.logger = logging.getLogger()
self.logger.setLevel(logging.INFO)
self.logger.addHandler(logging.StreamHandler())
def myRoutineHere(self, processNumber):
print "I'm in process number %d" % (processNumber)
time.sleep(10)
# optional: raise AllDoneException
def myRoutine(self):
plist = []
try:
for pnum in range(0, self.numProcesses):
p = Process(target=self.myRoutineHere, args=(pnum, ))
p.start()
plist.append(p)
while 1:
isAliveList = [p.is_alive() for p in plist]
if not True in isAliveList:
break
time.sleep(1)
except KeyboardInterrupt:
self.logger.warning("Caught keyboard interrupt, exiting.")
except AllDoneException:
self.logger.warning("Caught AllDoneException, Exiting normally.")
except:
self.logger.warning("Caught Exception, exiting: %s" % (traceback.format_exc()))
for p in plist:
p.terminate()
d = Dum()
d.myRoutine()
You should spawn new processes instead of threads to utilize cores in your CPU. My general rule is one process per core. So you split your problem input space into the number of cores available, each process getting part of the problem space.
Multiprocessing is best for this. You could also use Parallel Python.
Very late to the party - but in addition to using multiprocessing module as reptilicus said, also make sure to set "affinity".
Some python modules fiddle with it, effectively lowering the number of cores available to Python:
https://stackoverflow.com/a/15641148/4195846
Due to Global Interpreter Lock one Python process cannot take advantage of multiple cores. But if you can somehow parallelize your problem (which you should do anyway), then you can use multiprocessing to spawn as many Python processes as you have cores and process that data in each subprocess.
Here is the pseudo code for what I want to do.
import time
def run():
while x < 10000000:
x += 1
if __name__ == "__main__":
p = Process(run)
p.start()
time.sleep(3)
#some code that I don't know that will give me the current value of x
Pythons threading module seems to be the way to go however I have yet to successfully implement this example.
Everything you need is in the multiprocessing module. Perhaps a shared memory object would help here?
Note that threading in Python is affected by the Global Interpreter Lock, which essentially prevents multithreaded Python code.
Well here it is
from multiprocessing import Process, Pipe
import time
def f(conn):
x = 0
while x < 10000000:
if conn.poll():
if conn.recv() == "get":
conn.send(x)
x += 1
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
time.sleep(2)
parent_conn.send("get")
print(parent_conn.recv())
p.join()
turned out to be a duplicate, my version is just more generic.
It really depends on what you're trying to accomplish and the frequency of creation and memory usage of your subprocesses. A few long-lived ones, and you can easily get away with multiple OS-level processes (see the subprocess module`). If you're spawning a lot of little ones, threading is faster and has less memory overhead. But with threading you run into problems like "thread safety", the global interpreter lock, and nasty, boring stuff like semaphores and deadlocks.
Data sharing strategies between two processes or threads can be roughly divided into two categories: "Let's share a block of memory" (using Locks and Mutexes) and "Let's share copies of data" (using messaging, pipes, or sockets). The sharing method is light on memory, but difficult to manage because it means ensuring that one thread doesn't read the same part of shared memory as another thread is writing to it, which is not trivial and hard to debug. The copying method is heavier on memory, but easier to make sense of. Also, it has the distinct advantage of being able to be pretty trivially ported to a network, allowing for distributed computing.
You'll also have to think about the underlying OS. I don't know the specifics, but some are better than others at different approaches.
I'd say start with something like RabbitMQ.
I want to start, from Python, some other Python code, preferably a function, but in another process.
It is mandatory to run this in another process, because I want to run some concurrency tests, like opening a file that was opened exclusively by the parent process (this has to fail).
Requirements:
multiplatform: linux, osx, windows
compatible with Python 2.6-3.x
I would seriously take a look at the documentation for multiprocessing library of Python. From the first sentence of the package's description:
multiprocessing is a package that supports spawning processes using an API similar to the threading module.
It then goes on to say that it side-steps the GIL, which is what it sounds like you're trying to avoid. See their example of a trivial set up:
from multiprocessing import Process
def f(name):
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
That's a function call being done in another process separate from the process you're inside. Again, all this from the documentation.
Is there any easy way to make 2 methods, let's say MethodA() and MethodB() run in 2 different cores? I don't mean 2 different threads. I'm running in Windows, but I'd like to know if it is possible to be platform independent.
edit: And what about
http://docs.python.org/dev/library/multiprocessing.html
and
parallel python ?
You have to use separate processes (because of the often-mentioned GIL). The multiprocessing module is here to help.
from multiprocessing import Process
from somewhere import A, B
if __name__ == '__main__':
procs = [ Process(target=t) for t in (A,B) ]
for p in procs:
p.start()
for p in procs:
p.join()
Assuming you use CPython (the reference implementation) the answer is NO because of the Global Interpreter Lock. In CPython threads are mainly used when there is much IO to do (one thread waits, another does computation).
In general, running different threads is the best portable way to run on multiple cores. Of course, in Python, the global interpreter lock makes this a moot point -- only one thread will make progress at a time.
Because of the global interpreter lock, Python programs only ever run one thread at a time. If you want true multicore Python programming, you could look into Jython (which has access to the JVM's threads), or the brilliant stackless, which has Go-like channels and tasklets.