We Keep Coding

Python Using Multiprocessing

I am trying to use multiprocessing in python 3.6. I have a for loopthat runs a method with different arguments. Currently, it is running one at a time which is taking quite a bit of time so I am trying to use multiprocessing. Here is what I have:
def test(self):
for key, value in dict.items():
pool = Pool(processes=(cpu_count() - 1))
pool.apply_async(self.thread_process, args=(key,value))
pool.close()
pool.join()
def thread_process(self, key, value):
# self.__init__()
print("For", key)
I think what my code is using 3 processes to run one method but I would like to run 1 method per process but I don't know how this is done. I am using 4 cores btw.

You're making a pool at every iteration of the for loop. Make a pool beforehand, apply the processes you'd like to run in multiprocessing, and then join them:
from multiprocessing import Pool, cpu_count
import time
def t():
# Make a dummy dictionary
d = {k: k**2 for k in range(10)}
pool = Pool(processes=(cpu_count() - 1))
for key, value in d.items():
pool.apply_async(thread_process, args=(key, value))
pool.close()
pool.join()
def thread_process(key, value):
time.sleep(0.1) # Simulate a process taking some time to complete
print("For", key, value)
if __name__ == '__main__':
t()

You're not populating your multiprocessing.Pool with data - you're re-initializing the pool on each loop. In your case you can use Pool.map() to do all the heavy work for you:
def thread_process(args):
print(args)
def test():
pool = Pool(processes=(cpu_count() - 1))
pool.map(thread_process, your_dict.items())
pool.close()
if __name__ == "__main__": # important guard for cross-platform use
test()
Also, given all those self arguments I reckon you're snatching this off of a class instance and if so - don't, unless you know what you're doing. Since multiprocessing in Python essentially works as, well, multi-processing (unlike multi-threading) you don't get to share your memory, which means your data is pickled when exchanging between processes, which means anything that cannot be pickled (like instance methods) doesn't get called. You can read more on that problem on this answer.

I think what my code is using 3 processes to run one method but I would like to run 1 method per process but I don't know how this is done. I am using 4 cores btw.
No, you are in fact using the correct syntax here to utilize 3 cores to run an arbitrary function independently on each. You cannot magically utilize 3 cores to work together on one task with out explicitly making that a part of the algorithm itself/ coding that your self often using threads (which do not work the same in python as they do outside of the language).
You are however re-initializing the pool every loop you'll need to do something like this instead to actually perform this properly:
cpus_to_run_on = cpu_count() - 1
pool = Pool(processes=(cpus_to_run_on)
# don't call a dictionary a dict, you will not be able to use dict() any
# more after that point, that's like calling a variable len or abs, you
# can't use those functions now
pool.map(your_function, your_function_args)
pool.close()
Take a look at the python multiprocessing docs for more specific information if you'd like to get a better understanding of how it works. Under python, you cannot utilize threading to do multiprocessing with the default CPython interpreter. This is because of something called the global interpreter lock, which stops concurrent resource access from within python itself. The GIL doesn't exist in other implementations of the language, and is not something other languages like C and C++ have to deal with (and thus you can actually use threads in parallel to work together on a task, unlike CPython)
Python gets around this issue by simply making multiple interpreter instances when using the multiprocessing module, and any message passing between instances is done via copying data between processes (ie the same memory is typically not touched by both interpreter instances). This does not however happen in the misleadingly named threading module, which often actually slow processes down because of a process called context switching. Threading today has limited usefullness, but provides an easier way around non GIL locked processes like socket and file reads/writes than async python.
Beyond all this though there is a bigger problem with your multiprocessing. Your writing to standard output. You aren't going to get the gains you want. Think about it. Each of your processes "print" data, but its all being displayed in one terminal/output screen. So even if your processes are "printing" they aren't really doing that independently, and the information has to be coalesced back into another processes where the text interface lies (ie your console). So these processes write whatever they were going to to some sort of buffer, which then has to be copied (as we learned from how multiprocessing works) to another process which will then take that buffered data and output it.
Typically dummy programs use printing as a means of showing how there is no order between execution of these processes, that they can finish at different times, they aren't meant to demonstrate the performance benefits of multi core processing.

I have experimented a bit this week with multiprocessing. The fastest way that I discovered to do multiprocessing in python3 is using imap_unordered, at least in my scenario. Here is a script you can experiment with using your scenario to figure out what works best for you:
import multiprocessing
NUMBER_OF_PROCESSES = multiprocessing.cpu_count()
MP_FUNCTION = 'imap_unordered' # 'imap_unordered' or 'starmap' or 'apply_async'
def process_chunk(a_chunk):
print(f"processig mp chunk {a_chunk}")
return a_chunk
map_jobs = [1, 2, 3, 4]
result_sum = 0
if MP_FUNCTION == 'imap_unordered':
pool = multiprocessing.Pool(processes=NUMBER_OF_PROCESSES)
for i in pool.imap_unordered(process_chunk, map_jobs):
result_sum += i
elif MP_FUNCTION == 'starmap':
pool = multiprocessing.Pool(processes=NUMBER_OF_PROCESSES)
try:
map_jobs = [(i, ) for i in map_jobs]
result_sum = pool.starmap(process_chunk, map_jobs)
result_sum = sum(result_sum)
finally:
pool.close()
pool.join()
elif MP_FUNCTION == 'apply_async':
with multiprocessing.Pool(processes=NUMBER_OF_PROCESSES) as pool:
result_sum = [pool.apply_async(process_chunk, [i, ]).get() for i in map_jobs]
result_sum = sum(result_sum)
print(f"result_sum is {result_sum}")
I found that starmap was not too far behind in performance, in my scenario it used more cpu and ended up being a bit slower. Hope this boilerplate helps.

What are the python equivalent of sem_init, sem_post, sem_wait provided by semaphore.h?

I'm trying to lock multiple variables in a piece of python code that I have, one lock per variable. Variables can be easily locked in C using sem_wait, sem_post, sem_init routines available in semaphore.h. What are the equivalent in python?
Addendum:
Currently using a code structure equivalent to
lock1 = threading.lock()
lock2 = threading.lock()
Thread 1
lock1.acquire()
var1 = func1()
lock1.release()
lock2.acquire()
var2 = func2()
lock2.release()
Thread 2
lock1.acquire()
var1 = func1()
lock1.release()
lock2.acquire()
var2 = func2()
lock2.release()

Look at the "threading" module in the standard library. It provides semaphores, locks and other parallel processing niceties.
https://docs.python.org/3.4/library/threading.html

In most python implementations, you have a Global Interpreter Lock that prevents more than one thread at a time from executing python code.
Additionally, in most implementations, the GIL is only released after certain numbers of complete python bytecode operations have executed, meaning that, unlike C, you don't have a situation where load-increment-store operations could run in parallel and be done incorrectly -- in general, changes to variables are atomic.
Hence you usually don't need to protect each variable, and you'll find that a lot of code actually 'just works', depending on how you are using python threads.
Having said that, the Threading module (particularly condition variables) is the way to go if you need tighter grained control beyond what the GIL implicitly provides.

how to safely pass a variables value between threads in python

I read on the python documentation that Queue.Queue() is a safe way of passing variables between different threads. I didn't really know that there was a safety issue with multithreading. For my application, I need to develop multiple objects with variables that can be accessed from multiple different threads. Right now I just have the threads accessing the object variables directly. I wont show my code here because there's way too much of it, but here is an example to demonstrate what I'm doing.
from threading import Thread
import time
import random
class switch:
def __init__(self,id):
self.id=id
self.is_on = False
def self.toggle():
self.is_on = not self.is_on
switches = []
for i in range(5):
switches[i] = switch(i)
def record_switch():
switch_record = {}
while True:
time.sleep(10)
current = {}
current['time'] = time.srftime(time.time())
for i in switches:
current[i.id] = i.is_on
switch_record.update(current)
def toggle_switch():
while True:
time.sleep(random.random()*100)
for i in switches:
i.toggle()
toggle = Thread(target=toggle_switch(), args = ())
record = Thread(target=record_switch(), args = ())
toggle.start()
record.start()
So as I understand, the queue object can be used only to put and get values, which clearly won't work for me. Is what I have here "safe"? If not, how can I program this so that I can safely access a variable from multiple different threads?

Whenever you have threads modifying a value other threads can see, then you are going to have safety issues. The worry is that a thread will try to modify a value when another thread is in the middle of modifying it, which has risky and undefined behavior. So no, your switch-toggling code is not safe.
The important thing to know is that changing the value of a variable is not guaranteed to be atomic. If an action is atomic, it means that action will always happen in one uninterrupted step. (This differs very slightly from the database definition.) Changing a variable value, especially a list value, can often times take multiple steps on the processor level. When you are working with threads, all of those steps are not guaranteed to happen all at once, before another thread starts working. It's entirely possible that thread A will be halfway through changing variable x when thread B suddenly takes over. Then if thread B tries to read variable x, it's not going to find a correct value. Even worse, if thread B tries to modify variable x while thread A is halfway through doing the same thing, bad things can happen. Whenever you have a variable whose value can change somehow, all accesses to it need to be made thread-safe.
If you're modifying variables instead of passing messages, you should be using aLockobject.
In your case, you'd have a global Lock object at the top:
from threading import Lock
switch_lock = Lock()
Then you would surround the critical piece of code with the acquire and release functions.
for i in switches:
switch_lock.acquire()
current[i.id] = i.is_on
switch_lock.release()
for i in switches:
switch_lock.acquire()
i.toggle()
switch_lock.release()
Only one thread may ever acquire a lock at a time (this kind of lock, anyway). When any of the other threads try, they'll be blocked and wait for the lock to become free again. So by putting locks around critical sections of code, you make it impossible for more than one thread to look at, or modify, a given switch at any time. You can put this around any bit of code you want to be kept exclusive to one thread at a time.
EDIT: as martineau pointed out, locks are integrated well with the with statement, if you're using a version of Python that has it. This has the added benefit of automatically unlocking if an exception happens. So instead of the above acquire and release system, you can just do this:
for i in switches:
with switch_lock:
i.toggle()

python Threadsafe lock on loop

Can someone please tell me if the following is threadsafe or not and if it isnt what must I do to make it so.
Note: this only a small sample, not sure if it runs.
TIMER = True
time_lock = threading.Lock()
def timing():
while TIMER:
# some logic will be here for now print time
print time.time()
timer = threading.Thread(target=timing)
timer2 = threading.Thread(target=timing)
timer.start()
timer2.start()
while True:
time_lock.aquire()
if doSomeStuff():
TIMER = True
if otherThings():
break
time_lock.aquire()
TIMER = False
time_lock.release()
time_lock.aquire()
TIMER = False
time_lock.release()

It depends a bit on which implementation you are using, and what you are doing.
First, in the de facto standard implementation of python ("cpython"), only one thread is allowed to run at a time since some internals of the python interpreter aren't thread-safe. This is controlled by the Global Interpreter Lock (aka "GIL"). So in cpython, you don't really need extra locks at all; the GIL makes sure that only one thread at a time is running python code and possibly changing variables. This is a feature of the implementation, not of the language.
Second, if only one thread writes to a simple variable and others only read it you don't need a lock either, for obvious reasons. It is however up to you as the programmer to make sure that this is the case, and it is easy to make mistakes with that.
Even assinging to a simple variable might not need a lock. In python variables are more like labels used to refer to an object, rather than boxes where you can put something in. So simple assignments are atomic (in the sense that they cannot be interrupted halfway), as you can see when you look at the generated python bytecode:
In [1]: import dis
In [2]: x = 7
In [3]: def setx():
global x
x = 12
...:
In [4]: dis.dis(setx)
3 0 LOAD_CONST 1 (12)
3 STORE_GLOBAL 0 (x)
6 LOAD_CONST 0 (None)
9 RETURN_VALUE
The only code that changes x is the single STORE_GLOBAL opcode. So the variable is either changed or it isn't; there is no inconsistent state inbetween.
But if you e.g. want to test a variable for a certain value, and perform an action while that test still holds true, you do need a lock. Because another thread could have changed the variable just after you tested it.
Things like appending to a list or swapping variables are not atomic. But again in cpython these would be protected by the GIL. In other implementations you'd need a lock around such operations to protect against possible inconsistencies.

If I understand you correctly, you want to signal your thread when to stop. Since this is a situation where only one thread writes to a shared variable and only once, you do not need a lock.
Locks are necessary when you are doing concurrent modification of a shared datastructure that cannot be read atomically.

Is the += operator thread-safe in Python?

I want to create a non-thread-safe chunk of code for experimentation, and those are the functions that 2 threads are going to call.
c = 0
def increment():
c += 1
def decrement():
c -= 1
Is this code thread safe?
If not, may I understand why it is not thread safe, and what kind of statements usually lead to non-thread-safe operations.
If it is thread-safe, how can I make it explicitly non-thread-safe?

No, this code is absolutely, demonstrably not threadsafe.
import threading
i = 0
def test():
global i
for x in range(100000):
i += 1
threads = [threading.Thread(target=test) for t in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
assert i == 1000000, i
fails consistently.
i += 1 resolves to four opcodes: load i, load 1, add the two, and store it back to i. The Python interpreter switches active threads (by releasing the GIL from one thread so another thread can have it) every 100 opcodes. (Both of these are implementation details.) The race condition occurs when the 100-opcode preemption happens between loading and storing, allowing another thread to start incrementing the counter. When it gets back to the suspended thread, it continues with the old value of "i" and undoes the increments run by other threads in the meantime.
Making it threadsafe is straightforward; add a lock:
#!/usr/bin/python
import threading
i = 0
i_lock = threading.Lock()
def test():
global i
i_lock.acquire()
try:
for x in range(100000):
i += 1
finally:
i_lock.release()
threads = [threading.Thread(target=test) for t in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
assert i == 1000000, i

(note: you would need global c in each function to make your code work.)
Is this code thread safe?
No. Only a single bytecode instruction is ‘atomic’ in CPython, and a += may not result in a single opcode, even when the values involved are simple integers:
>>> c= 0
>>> def inc():
... global c
... c+= 1
>>> import dis
>>> dis.dis(inc)
3 0 LOAD_GLOBAL 0 (c)
3 LOAD_CONST 1 (1)
6 INPLACE_ADD
7 STORE_GLOBAL 0 (c)
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
So one thread could get to index 6 with c and 1 loaded, give up the GIL and let another thread in, which executes an inc and sleeps, returning the GIL to the first thread, which now has the wrong value.
In any case, what's atomic is an implementation detail which you shouldn't rely on. Bytecodes may change in future versions of CPython, and the results will be totally different in other implementations of Python that do not rely on a GIL. If you need thread safety, you need a locking mechanism.

To be sure I recommend to use a lock:
import threading
class ThreadSafeCounter():
def __init__(self):
self.lock = threading.Lock()
self.counter=0
def increment(self):
with self.lock:
self.counter+=1
def decrement(self):
with self.lock:
self.counter-=1
The synchronized decorator can also help to keep the code easy to read.

It's easy to prove that your code is not thread safe. You can increase the likelyhood of seeing the race condition by using a sleep in the critical parts (this simply simulates a slow CPU). However if you run the code for long enough you should see the race condition eventually regardless.
from time import sleep
c = 0
def increment():
global c
c_ = c
sleep(0.1)
c = c_ + 1
def decrement():
global c
c_ = c
sleep(0.1)
c = c_ - 1

Short answer: no.
Long answer: generally not.
While CPython's GIL makes single opcodes thread-safe, this is no general behaviour. You may not assume that even simple operations like an addition is a atomic instruction. The addition may only be half done when another thread runs.
And as soon as your functions access a variable in more than one opcode, your thread safety is gone. You can generate thread safety, if you wrap your function bodies in locks. But be aware that locks may be computationally costly and may generate deadlocks.

If you actually want to make your code not thread-safe, and have good chance of "bad" stuff actually happening without you trying like ten thousand times (or one time when you real don't want "bad" stuff to happen), you can 'jitter' your code with explicit sleeps:
def íncrement():
global c
x = c
from time import sleep
sleep(0.1)
c = x + 1

Single opcodes are thread-safe because of the GIL but nothing else:
import time
class something(object):
def __init__(self,c):
self.c=c
def inc(self):
new = self.c+1
# if the thread is interrupted by another inc() call its result is wrong
time.sleep(0.001) # sleep makes the os continue another thread
self.c = new
x = something(0)
import threading
for _ in range(10000):
threading.Thread(target=x.inc).start()
print x.c # ~900 here, instead of 10000
Every resource shared by multiple threads must have a lock.

Are you sure that the functions increment and decrement execute without any error?
I think it should raise an UnboundLocalError because you have to explicitly tell Python that you want to use the global variable named 'c'.
So change increment ( also decrement ) to the following:
def increment():
global c
c += 1
I think as is your code is thread unsafe. This article about thread synchronisation mechanisms in Python may be helpful.