I have a problem when I run a script with python. I haven't done any parallelization in python and don't call any mpi for running the script. I just execute "python myscript.py" and it should only use 1 cpu.
However, when I look at the results of the command "top", I see that python is using almost 390% of my cpus. I have a quad core, so 8 threads. I don't think that this is helping my script to run faster. So, I would like to understand why python is using more than one cpu, and stop it from doing so.
Interesting thing is when I run a second script, that one also takes up 390%. If I run a 3rd script, the cpu usage for each of them drops to 250%. I had a similar problem with matlab a while ago, and the way I solved it was to launch matlab with -singlecompthread, but I don't know what to do with python.
If it helps, I'm solving the Poisson equation (which is not parallelized at all) in my script.
UPDATE:
My friend ran the code on his own computer and it only takes 100% cpu. I don't use any BLAS, MKL or any other thing. I still don't know what the cause for 400% cpu usage is.
There's a piece of fortran algorithm from the library SLATEC, which solves the Ax=b system. That part I think is using a lot of cpu.
Your code might be calling some functions that uses C/C++/etc. underneath. In that case, it is possible for multiple thread usage.
Are you calling any libraries that are only python bindings to some more efficiently implemented functions?
You can always set your process affinity so it run on only one cpu. Use "taskset" command on linux, or process explorer on windows.
This way, you should be able to know if your script has same performance using one cpu or more.
Could it be that your code uses SciPy or other numeric library for Python that is linked against Intel MKL or another vendor provided library that uses OpenMP? If the underlying C/C++ code is parallelised using OpenMP, you can limit it to a single thread by setting the environment variable OMP_NUM_THREADS to 1:
OMP_NUM_THREADS=1 python myscript.py
Intel MKL for sure is parallel in many places (LAPACK, BLAS and FFT functions) if linked with the corresponding parallel driver (the default link behaviour) and by default starts as many compute threads as is the number of available CPU cores.
Related
I'm using jupyterlab and I know that I have 12 cores available.
At the moment I use only 1 and I would like to use more.
I have tried to changed the number I use by write this in the terminal:
export JULIA_NUM_THREADS=7
but then when I print:
import threading
threading.activeCount()
>>>5
how can I make more CPU available for my jupyterlab notebook?
This is really not my field so I'm sorry if is smething really simple I just don't understand what am I doing wrong and where to start from.
TLDD; No configuration needed. It is available to you, just need to code explicitely what you want to run in parallel.
JULIA_ACTIVE_THREADS is a configuration option for the Julia Kernel in Jupyter, not for the Python Kernel (the process that runs your notebook code).
Unless you run Jupyter inside a container, you can use out of the box all cores available in your system. If Jupyter is in a container or a virtual machine, it will use what you allocate and nothing more.
Just remember that by default you use 1 core when you run your Jupyter kernel.
When you run threading.active_count() and get 1, this means you are using one running thread on your code. Moden processors can use several threads for each available core. The bad news is that this is not a measure about how good you are using the cpu.
Python can act as an orchestrator for libraries that work in paraller behind the scenes (think numpy, pandas, tensorflow...).
If you want to code Python code that use more than 1 thread and/or 1 CPU, take a look at the multiprocess module.
The multipreocessing module is part of the standard library, and you can use it inside without trouble inside Jupyter. Probably you will find the Process and Pool methods useful (if you want to work with deep learning, there is a pytorch.multiprocessing module with the same interface but with support for working with GPUs in different threads).
A few thoughts, but to long for a comment, i am not familiar with jupyter, only "normal python", so maybe this all gets in the wrong direction ;):
As far as i know, the à ctive_count (in my opinion you should not use the old camelCase name) only returns the amount of active threads, not the available. So try to add more threads. I have a Quadcore and jupyter starts with 5 threads, but i can add more.
Multithreading is not the same as multiprocessing (If you want to run on different Cores you have to use multiprocessing) (python thread vs. multiproccess), maybe you are looking for the wrong thing?
I have a fitting procedure written in pymc3 which utilizes dot products of matrices which are functions of random variables. On the original machine on which I wrote the program, it would run and by default use all available cores, as expected.
However, I recently started using a brand new machine, including a clean installation of python and pymc3. Unfortunately, this means many aspects of the system differ from the previous machine, so it is difficult to pin down what may be causing differences. However, I am befuddled by the fact that the new machine does not use all cores available or specified (CPU usage is capped at 100%, when it should freely float up to 800%).
My ~/.theanorc has openmp=True and OMP_NUM_THREADS=8. My .bashrc sets OMP, MKL, and GOTO threads to 8 just for good measure. Running check_blas.py in theano/misc/ shows 800% CPU usage. What could be causing the pymc3 application to fail to use all cores in a machine-dependent way?
Thank you.
I am trying to restrict the number of CPUs used by Python (for benchmarking & to see if it speeds up my program).
I have found a few Python modules for achieving this ('os', 'affinity', 'psutil') except that their methods for changing affinity only works with Linux (and sometimes Windows). There is also a suggestion to use the 'taskset' command (Why does multiprocessing use only a single core after I import numpy?) but this command not available on macOS as far as I know.
Is there a (preferable clean & easy) way to change affinity while running Python / iPython on macOS? It seems like changing processor affinity in Mac is not as easy as in other platforms (http://softwareramblings.com/2008/04/thread-affinity-on-os-x.html).
Not possible. See Thread Affinity API Release Notes:
OS X does not export interfaces that identify processors or control thread placement—explicit thread to processor binding is not supported. Instead, the kernel manages all thread placement. Applications expect that the scheduler will, under most circumstances, run its threads using a good processor placement with respect to cache affinity.
Note that thread affinity is something you'd consider fairly late when optimizing a program, there are a million things to do which have a larger impact on your program.
Also note that Python is particularly bad at multithreading to begin with.
Does Enthought Canopy support parallel code execution on CPU using perhaps openMPI or on GPU using openCV or CUDA
I am looking into switching from C++ to python as i want to make GUI for my parallel code.
Is this a good idea. Does python support parallel computation?
Yes, Python does support this. There are three layers to processes with Python:
subprocess: which simply starts a process within the same thread
threading: which starts a new thread and leaves the old on alone. There are some frequent stories that this not necessarily leads to better performance.
multiprocessing: which is what you are after
Here is an intro to parallel processing on Python.
The official docs for the multiprocessing are here.
The ever so useful discussions on the Python Module of the Week are also worth a look.
Edit:
The python libraries mentioned by HT #jonathan are likely to be:
Cuda:
http://mathema.tician.de/software/pycuda
OpenCV:
http://code.google.com/p/pyopencv/
There is a nice tutorial for this here.
And Message Passing Interface:
http://mpi4py.scipy.org/docs/usrman/intro.html
In order to do quality assurance in a critical multicore (8) workstation, I want to run the same code in different processors but not in parallel or concurrently.
I need to run it 8 times, one run for each processor.
What I don't know is how to select the processor I want.
How can this be accomplished in Python?
In Linux with schedutils, I believe you'd use taskset -c X python foo.py to run that specific Python process on CPU X (exactly how you identify your CPUs may vary, but I believe numbers such as 1, 2, 3, ... should work anywhere). I'm sure Windows, BSD versions, etc, have similar commands to support direct processor assignment, but I don't know them.
Which process goes on which core is normaly decided by your OS. On linux there is taskset from the schedutils package to explicitly run a program on a processor.
Python 2.6 has a multiprocessing module that takes python functions and runs them in seperate processes, probably moving each new process to a different core.