This question already has answers here:
Jupyter notebook never finishes processing using multiprocessing (Python 3)
(5 answers)
Closed 4 years ago.
I tried to use multiprocessing for a simple code in windows 10, python2.7 and jupyter notebook. When I run the code the kernel got stack without any errors thrown. I checked the task manager for performance and saw 8 (the number of cores in my CPU) processes running with 0% use of the CPU for each and everyone of them.
I looked almost everywhere but didn't find anything. I also tried it in the anaconda prompt but got endless loop of errors.
Here is my code:
import multiprocessing
n_cpu = multiprocessing.cpu_count()
def foo(x):
return x**2
if __name__ == '__main__':
pool = multiprocessing.Pool(processes = n_cpu)
res = pool.map(foo, [1,2,3])
pool.close()
Is this your actual code? With foo being so simple, this code will run so quickly that it's likely to look like 0% CPU usage.
If you see that there are sub-processes spawned, then pool.map is doing it's job. If you want to see that all of them actually use full CPU, give them some real meat to chew on (maybe something like sum([sum(range(x)) for x in range(1000)]).
Related
I am running a very, very simple script in a Jupyter pyspark notebook, but it is not printing results as it runs, it just spits out the output when it's done. Here is the code:
import time
import sys
for i in range(10):
print(i)
time.sleep(1)
This waits 10 seconds and then prints:
0
1
2
3
4
5
6
7
8
9
I would like to print results as they happen. I have tried to flush them using
for i in range(10):
print(i)
sys.stdout.flush()
and print(i, flush=True) to no avail. Any suggestions?
It is a buffering issue. You could also use python -u command or set PYTHONUNBUFFERED envvar. python uses line-buffering if it is run interactively (in a terminal) and block buffering (e.g., ~4K bytes buffers) if the output is redirected
Depending on what you are doing, if you are running pyspark code and are hoping to see results before the job is complete, this may not work.
You may be running into an issue with how spark/pyspark runs your code. Spark is designed to efficiently divide up your task into parts, and distribute those parts to the nodes of your EMR cluster.
This means that the actual work does not happen on the machine where your notebook is running. The main node, where your notebook is running, sends tasks out to all the worker nodes, then collects the results as they are sent back, and only displays them once the job is complete. This can be troublesome for someone used to debugging plain python, but is a big part of what makes pyspark so fast when working with large amounts of data.
I'm running a script like
def test0():
start = time()
for i in range(int(1e8)):
i += 1
print(time() - start)
When I run this on my machine which has 4 CPUs I get the following trace for CPU usage using the Ubuntu 20.04 system monitor.
In the image you can see I ran two experiments separated by some time. But in each experiment, the activity of 2 of the CPUs peaks. Why?
This seems normal to me. The process that is running your python code is not, at least by default, pinned to a specific core. This means that the process can be switched between different cores, which is what is happening in this case. Those spikes are not simultaneous, it indicates that the process was switched from one core to another.
On Linux, you can observe this using
watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p 172810
where 172810 is PID of the python process (which you can get, for example, from the output of top)
If you want to pin the process to a particular core, you can use psutils in your code.
import psutil
p = psutil.Process()
p.cpu_affinity([0]) # pinning the process to cpu 0
Now, you should see only one core spiking. (but avoid doing this if you don't have a good reason for it).
I'm working in Jupyter (Anaconda) with Python 2.7.
I'm trying to get an odeint function I wrote to run multiple times, however it takes an incredible amount of time.
While trying to figure out how to decrease the run time, I realized that when I ran it only took up about 12% of my CPU.
I operate off of an Intel Core i7-3740QM # 2.70GHz:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-3740QM+%40+2.70GHz&id=1481
So, I'm assuming this is because of Python's GIL causing my script to only run off of one core.
After some research on parallel processing in Python, I thought I found the answer by using this code:
import sys
import multiprocessing as mp
Altitude = np.array([[550],[500],[450],[400],[350],[300]])
if __name__ == "__main__":
processes = 4
p = mp.Pool(processes)
mp_solutions = p.map(calc, Altitude)
This doesn't seem to work though. Once I run it, Jupyter just becomes constantly busy. My first thought was that it was just a high computation level so it was taking a long time, but then I looked at my CPU usage and although there were multiple instances of Python processes, none of them were using any CPU.
I can't figure out what the reasoning for this is. I found this post as well and tried using their code but it simply did the same thing:
Multiple scipy.integrate.ode instances
Any help would be much appreciated.
Thanks!
I cannot make even the simplest of examples of parallel processing using the multiprocessing package run in python 2.7 (using spyder as a UI on windows) and I need help figuring out the issue. I have run conda update so all of the packages should be up to date and compatible.
Even the first example in the multiprocessing package documentation (given below) wont work, it generates 4 new processes but the console just hangs. I have tried everything I can find over the last 3 days but none of the code that runs without hanging will allocate more than 25% of my computing power to this task (I have a 4 core computer).
I have given up on running the procedure I have designed and need parallel processing for at this point and I am only trying to get proof of concept so I can build from there. Can someone explain and point me in the right direction? Thanks
Example 1 from https://docs.python.org/2/library/multiprocessing.html
#
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
p = Pool()
print(p.map(f, [1, 2, 3]))
Example 2 (modified from original) from http://chriskiehl.com/article/parallelism-in-one-line/
from multiprocessing import Pool
def fn(i):
return [i,i*i,i*i*i]
test = range(10)
if __name__ == '__main__':
pool = Pool()
results = map(fn,test)
pool.close()
pool.join()
I apologize if there is indeed an answer to this as it seems as though I should be able to manage such a modest task but I am not a programmer and the resources I have found have been less than helpful given my very limited level of knowledge. Please let me know what further information is needed.
Thank you.
After installing spyder on my virtualmachine, it seems to be a spyder specific bug. Example 1 works in IDLE, executed via the command line, executed from within spyder (first saved and then executed), but not when executed line by line in spyder.
I would suggest simply to create a new file in spyder, add the lines of code, save it, and then run it..
For related reports see:
https://groups.google.com/forum/#!topic/spyderlib/LP5d8QZTXd0
QtConse in Spyder cannot use multiprocessing.Manager
Multiprocessing working in Python but not in iPython
https://github.com/spyder-ide/spyder/issues/1900
I am attempting to write a wrapper that iterates through running another program with different input files. The program (over which I have no control, but need to use) needs to be run out of the same directory as the input file(s). So far my method is to use OS-module to change/create directory structure, use apply-async to run the program given a sub-directory, and each child in apply-async changes directory, creates the file, and the first 8 processes run successfully (i have 8 virtual cores)
However, I am queueing up to 100 of these processes (they run a simulation which takes a few minutes, I'm looking to optimize). I use "call" on the outside executable I am running. I thought everything was going great, but then after the 8th simulation runs, everything stops, I check 0 processes are running. It is as if the queue forgot about the other processes.
What can I do to fix this? I know my RAM only goes up about 300 MB out of 8GB.
Do I need to look into implementing some sort of queue myself that waits for the exit code of the simulation executable?
Thank you in advance.
Maybe better than nothing. This shows a correct way to use apply_async(), and demonstrates that - yup - there's no problem creating many more tasks than processes. I'd tell you how this differs from what you're doing, but I have no idea what you're doing ;-)
import multiprocessing as mp
def work(i):
from time import sleep
sleep(2.0 if i & 1 else 1.0)
return i*i
if __name__ == "__main__":
pool = mp.Pool(4)
results = [pool.apply_async(work, (i,)) for i in range(100)]
results = [r.get() for r in results]
print len(results), results
pool.close()
pool.join()