I have a generator (or, a list of generators). Let's call them gens
Each generator in gens is a complicated function that returns the next value of a complicated procedure. Fortunately, they are all independent of one another.
I want to call gen.__next__() for each element gen in gens, and return the resulting values in a list. However, multiprocessing is unhappy with pickling generators.
Is there a fast, simple way to do this in Python? I would like it such that gens of length m is mapped to n cores locally on my machine, where n could be larger or smaller than m. Each generator should run on a separate core.
If this is possible, can someone provide a minimal example?
You can't pickle generators. Read more about it here.
There is a blog post which explains it in much more detail. Referring a quote from it:
Let’s ignore that problem for a moment and look what we would need to do to pickle a generator. Since a generator is essentially a souped-up function, we would need to save its bytecode, which is not guarantee to be backward-compatible between Python’s versions, and its frame, which holds the state of the generator such as local variables, closures and the instruction pointer. And this latter is rather cumbersome to accomplish, since it basically requires to make the whole interpreter picklable. So, any support for pickling generators would require a large number of changes to CPython’s core.
Now if an object unsupported by pickle (e.g., a file handle, a socket, a database connection, etc) occurs in the local variables of a generator, then that generator could not be pickled automatically, regardless of any pickle support for generators we might implement. So in that case, you would still need to provide custom getstate and setstate methods. This problem renders any pickling support for generators rather limited.
He also suggests a solution, to use simple iterators.
the best solution to this problem to the rewrite the generators as simple iterators (i.e., one with a __next__ method). Iterators are easy and efficient space-wise to pickle because their state is explicit. You would still need to handle objects representing some external state explicitly however; you cannot get around this.
Another offered solution (which I haven't tried) suggests this
Convert the generator to a class in which the generator code is the __iter__ method
Add __getstate__ and __setstate__ methods to the class, to handling pickling. Remember that you can’t pickle file objects. So __setstate__ will have to re-open files, as necessary.
If your subtasks are truly parallel (do not rely on any shared state), you can do this with multiprocesing.Pool().
Take a look at
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool
This requires you to make the arguments of pool.map() serializable. You can't pass in a generator to your worker, but you can achieve something similar by defining your generator inside the target function, and pass in initialization arguments to the multiprocessing library:
import multiprocessing as mp
import time
def worker(value):
# The generator is defined inside the multiprocessed function
def gen():
for k in range(value):
time.sleep(1) # Simulate long running task
yield k
# Execute the generator
for x in gen():
print(x)
# Do something with x?
pass
pool = mp.Pool()
pool.map(worker, [2, 5, 2])
pool.join() # Wait for all the work to be finished.
pool.close() # Clean up system resources
The output will be:
0
0
0
1
1
1
2
3
4
Note that this solution only really works if you build your generators, then use them only once, as their final state is lost at the end of the worker function.
Keep in mind that anytime you want to use multiprocessing, you have to use for serializable objects due to the limitations of inter-process communication; this can often prove limiting.
If your process is not CPU bound but instead I/O bound (disk access, network access, etc), you'll have a much easier time using threads.
You don't need to pickle the generator, just send an index of the generator to the processing pool.
M = len(gens)
N = multiprocessing.cpu_count()
def proc(gen_idx):
return [r for r in gens[gen_idx]()]
if __name__ == "__main__":
with multiprocessing.Pool(N) as p:
for r in p.imap_unordered(proc, range(M)):
print(r)
Note that I don't call/initialize the generator until within the processing function.
Using imap_unordered will allow you to process the results as each generator completes.
It's quite easy to implement, just dont block the threads sincronusly, just constantly loop thru the states and join them on complition. This template shuld be good enough to give an idea, self.done alwais needs to be set last on thread complition and las on thread reuse.
import threading as th
import random
import time
class Gen_thread(th.Thread):
def is_done(self):
return self.done
def get_result(self):
return self.work_result
def __init__(self, *args, **kwargs):
self.g_id = kwargs['id']
self.kwargs = kwargs
self.args = args
self.work_result = None
self.done = False
th.Thread.__init__(self)
def run(self):
# time.sleep(*self.args) to pass variables
time.sleep(random.randint(1, 4))
self.work_result = 'Thread {0} done'.format(self.g_id + 1)
self.done = True
class Gens(object):
def __init__(self, n):
self.n_needed = 0
self.n_done = 0
self.n_loop = n
self.workers_tmp = None
self.workers = []
def __iter__(self):
return self
def __next__(self):
if self.n_needed == 0:
for w in range(self.n_loop):
self.workers.append(Gen_thread(id=w))
self.workers[w].start()
self.n_needed += 1
while self.n_done != self.n_needed:
for w in range(self.n_loop):
if self.workers[w].is_done():
self.workers[w].join()
self.workers_tmp = self.workers[w].get_result()
self.workers.pop(w)
self.n_loop -= 1
self.n_done += 1
return self.workers_tmp
raise StopIteration()
if __name__ == '__main__':
for gen in Gens(4):
print(gen)
Related
So I was writing a function where a lot of processing happens in the body of a loop, and occasionally it may be of interest to the caller to have the answer to some of the computations.
Normally I would just put the results in a list and return the list, but in this case the results are too large (a few hundred MB on each loop).
I wrote this without really thinking about it, expecting Python's dynamic typing to figure things out, but the following is always created as a generator.
def mixed(is_generator=False):
for i in range(5):
# process some stuff including writing to a file
if is_generator:
yield i
return
From this I have two questions:
1) Does the presence of the yield keyword in a scope immediately turn the object its in into a generator?
2) Is there a sensible way to obtain the behaviour I intended?
2.1) If no, what is the reasoning behind it not being possible? (In terms of how functions and generators work in Python.)
Lets go step by step:
1) Does the presence of the yield keyword in a scope immediately turn the object its in into a generator? Yes
2) Is there a sensible way to obtain the behaviour I intended? Yes, see example below
The thing is to wrap the computation and either return a generator or a list with the data of that generator:
def mixed(is_generator=False):
# create a generator object
gen = (compute_stuff(i) for i in range(5))
# if we want just the generator
if is_generator:
return gen
# if not we consume it with a list and return that list
return list(gen)
Anyway, I would say this is a bad practice. You should have it separated, usually just have the generator function and then use some logic outside:
def computation():
for i in range(5):
# process some stuff including writing to a file
yield i
gen = computation()
if lazy:
for data in gen:
use_data(data)
else:
data = list(gen)
use_all_data(data)
I have some complex class A that computes data (large matrix calculations) while consuming input data from class B.
A itself uses multiple cores. However, when A needs the next chunk of data, it waits for quite some time since B runs in the same main thread.
Since A mainly uses the GPU for computations, I would like to have B collecting data concurrently on the CPU.
My latest approach was:
# every time *A* needs data
def some_computation_method(self):
data = B.get_data()
# start computations with data
...and B looks approximately like this:
class B(object):
def __init__(self, ...):
...
self._queue = multiprocessing.Queue(10)
loader = multiprocessing.Process(target=self._concurrent_loader)
def _concurrent_loader(self):
while True:
if not self._queue.full():
# here: data loading from disk and pre-processing
# that requires access to instance variables
# like self.path, self.batch_size, ...
self._queue.put(data_chunk)
else:
# don't eat CPU time if A is too busy to consume
# the queue at the moment
time.sleep(1)
def get_data(self):
return self._queue.get()
Could this approach be considered a "pythonic" solution?
Since I have not much experience with Python's multiprocessing module, I've built an easy/simplistic approach. However, it looks kind of "hacky" to me.
What would be a better solution to have a class B loading data from disk concurrently and supplying it via some queue, while the main thread runs heavy computations and consumes data from the queue from time to time?
While your solution is perfectly OK, especially for "small" projects, it has the downside of the threading getting tightly coupled with the class B. Hence if you (for example) for some reason wanted to use B in a non-threaded manner, your out of luck.
I would personally write the class in a thread safe manner, and then call it using threads from outside:
class B(object):
def __init__(self):
self._queue = multiprocessing.Queue(10)
...
if __name__ == '__main__':
b = B()
loader = multiprocessing.Process(target=b._concurrent_loader)
loader.start()
This makes B more flexible, separates dependencies better and is easier to test. It also makes the code more readable by being explicit about the thread creation, as compared to it happening implicitly on class creation.
I have a large number of CPU-bounded tasks that I want to run in parallel. Most of those tasks will return similar results and I only need to store unique results and count non-unique ones.
Here's how it is currently designed: I use two managed dictionaries - one for results and another one for result counters. My tasks are checking those dictionaries using unique result keys for the results they found and either write into both dictionaries or only increase the counters for non-unique results (if I have to write I acquire the lock and check again to avoid inconsistency).
What I am concerned about: since Pool.map should actually return result object, even though I do not save a reference to it, results will still pile up in memory until they are garbage collected. Even though I will have millions of just None's there (since I am processing results in a different manner and all my tasks just return None) I can not rely on specific garbage collector behavior so the program might eventually run out of memory. I still want to keep nice features of the pool but leave out this built-in result handling. Is my understanding correct and is my concern valid? If so, are there any alternatives?
Also, now that I laid it out on paper it looks really clumsy :) Do you see a better way to design such thing?
Thanks!
Question: I still want to keep nice features of the pool
Remove return result from multiprocessing.Pool.
Copy class MapResult and inherit from mp.pool.ApplyResult.
Add, replace ,comment the following:
import multiprocessing as mp
from multiprocessing.pool import Pool
class MapResult(mp.pool.ApplyResult):
def __init__(self, cache, chunksize, length, callback, error_callback):
super().__init__(cache, callback, error_callback=error_callback)
...
#self._value = [None] * length
self._value = None
...
def _set(self, i, success_result):
...
if success:
#self._value[i*self._chunksize:(i+1)*self._chunksize] = result
Create your own class myPool(Pool) inherit from multiprocessing.Pool.
Copy def _map_async(... from multiprocessing.Pool.
Add, replace, comment the following:
class myPool(Pool):
def __init__(self, processes=1):
super().__init__(processes=processes)
def _map_async(self, func, iterable, mapper, chunksize=None, callback=None,
error_callback=None):
...
#if self._state != RUN:
if self._state != mp.pool.RUN:
...
#return result
Tested with Python: 3.4.2
I have a thread which is updating a list called l. Am I right in saying that it is thread-safe to do the following from another thread?
filter(lambda x: x[0] == "in", l)
If its not thread safe, is this then the correct approach:
import threading
import time
import Queue
class Logger(threading.Thread):
def __init__(self, log):
super(Logger, self).__init__()
self.log = log
self.data = []
self.finished = False
self.data_lock = threading.Lock()
def run(self):
while not self.finished:
try:
with self.data_lock:
self.data.append(self.log.get(block=True, timeout=0.1))
except Queue.Empty:
pass
def get_data(self, cond):
with self.data_lock:
d = filter(cond, self.data)
return d
def stop(self):
self.finished = True
self.join()
print("Logger stopped")
where the get_data(self, cond) method is used to retrieve a small subset of the data in the self.data in a thread safe manner.
First, to answer your question in the title: filter is just a function. Hence, its thread-safety will rely on the data-structure you use it with.
As pointed out in the comments already, list operations themselves are thread-safe in CPython and protected by the GIL, but that is arguably only an implementation detail of CPython that you shouldn't really rely on. Even if you could rely on it, thread safety of some of their operations probably does not mean the kind of thread safety you mean:
The problem is that iterating over a sequence with filter is in general not an atomic operation. The sequence could be changed during iteration. Depending on the data-structure underlying your iterator this might cause more or less weird effects. One way to overcome this problem is by iterating over a copy of the sequence that is created with one atomic action. Easiest way to do this for standard sequences like tuple, list, string is with the slice operator like this:
filter(lambda x: x[0] == "in", l[:])
Apart from this not necessarily being thread-safe for other data-types, there's one problem with this though: it's only a shallow copy. As your list's elements seem to be list-like as well, another thread could in parallel do del l[1000][:] to empty one of the inner lists (which are pointed to in your shallow copy as well). This would make your filter expression fail with an IndexError.
All that said, it's not a shame to use a lock to protect access to your list and I'd definitely recommend it. Depending on how your data changes and how you work with the returned data, it might even be wise to deep-copy the elements while holding the lock and to return those copies. That way you can guarantee that once returned the filter condition won't suddenly change for the returned elements.
Wrt. your Logger code: I'm not 100 % sure how you plan to use this and if it's critical for you to run several threads on one queue and join them. What looks weird to me is that you never use Queue.task_done() (assuming that its self.log is a Queue). Also your polling of the queue is potentially wasteful. If you don't need the join of the thread, I'd suggest to at least turn the lock acquisition around:
class Logger(threading.Thread):
def __init__(self, log):
super(Logger, self).__init__()
self.daemon = True
self.log = log
self.data = []
self.data_lock = threading.Lock()
def run(self):
while True:
l = self.log.get() # thread will sleep here indefinitely
with self.data_lock:
self.data.append(l)
self.log.task_done()
def get_data(self, cond):
with self.data_lock:
d = filter(cond, self.data)
# maybe deepcopy d here
return d
Externally you could still do log.join() to make sure that all of the elements of the log queue are processed.
If one thread writes to a list and another thread reads that list, the two must be synchronized. It doesn't matter for that aspect whether the reader uses filter(), an index or iteration or whether the writer uses append() or any other method.
In your code, you achieve the necessary synchronization using a threading.Lock. Since you only access the list within the context of with self.data_lock, the accesses are mutually exclusive.
In summary, your code is formally correct concerning the list handling between threads. But:
You do access self.finished without the lock, which is problematic. Assigning to that member will change self, i.e. the mapping of the object to the according members, so this should be synced. Effectively, this won't hurt, because True and False are global constants, at the worst you will have a short delay between setting the state in one thread and seeing the state in the other. It remains bad, because it is habit-forming.
As a rule, when you use a lock, always document which objects this lock protects. Also, document which object is accessed by which thread. The fact that self.finished is shared and requires synchronization would have been obvious. Also, making a visual distinction between public functions and data and private ones (beginning with an _underscore, see PEP 8) helps keeping track of this. It also helps other readers.
A similar issue is your baseclass. In general, inheriting from threading.Thread is a bad idea. Rather, include an instance of the thread class and give it a function like self._main_loop to run on. The reason is that you say that your Logger is a Thread and that all of its baseclass' public members are also public members of your class, which is probably a much wider interface than what you intended.
You should never block with a lock held. In your code, you block in self.log.get(block=True, timeout=0.1) with the lock on the mutex. In that time, even if nothing actually happens, no other thread will be able to call and complete a call to get_data(). There is actually just a tiny window between unlocking the mutex and locking it again where a caller of get_data() does not have to wait, which is very bad for performance. I could even imagine that your question is motivated by the really bad performance this causes. Instead, call log.get(..) without lock, it shouldn't need one. Then, with the lock held, append data to self.data and check self.finished.
I'm trying to get my head around what is the most efficient and less memory consuming way to share the same data source between different process.
Imagine the following code, that simplify my problem.
import pandas as pd
import numpy as np
from multiprocessing import Pool
# method #1
def foo(i): return data[i]
if __name__ == '__main__':
data = pd.Series(np.array(range(100000)))
pool = Pool(2)
print pool.map(foo,[10,134,8,1])
# method #2
def foo((data,i)): return data[i]
if __name__ == '__main__':
data = pd.Series(np.array(range(100000)))
pool = Pool(2)
print pool.map(foo,[(data,10),(data,134),(data,8),(data,1)])
In the first method will use the global variable (won't work on Windows, only on Linux/OSX) which will then access by the function. In the second method I'm passing "data" as part of the arguments.
In terms of memory used during the process, there will be a difference between the two methods?
# method #3
def foo((data,i)): return data[i]
if __name__ == '__main__':
data = pd.Series(np.array(range(100000)))
pool = Pool(2)
# reduce the size of the argument passed
data1 = data[:1000]
print pool.map(foo,[(data1,10),(data1,134),(data1,8),(data1,1)])
A third method, rather than passing all the "data", since we know we'll be using only the first records, I'm only passing the first 1000 records. Will this make any difference?
Background
The problem I'm facing I have a big dataset of about 2 millions rows (4GB in memory) which will then by four subprocess to do some elaboration. Each elaboration only affect a small portion of the data (20000 rows) and I'd like to minimize the memory use by each concurrent process.
I'm going to start with the second and third methods, because they're easier to explain.
When you pass the arguments to pool.map or pool.apply, the arguments will be pickled, sent to the child process using a pipe, and then unpickled in the child. This of course requires two completely distinct copies of the data structures you're passing. It also can lead to slow performance with large data structures, since pickling/unpickling large objects can take quite a while.
With the third method, you're just passing smaller data structures than method two. This should perform better, since you don't need to pickle/unpickle as much data.
One other note - passing data multiple times is definitely a bad idea, because each copy will be getting pickled/unpickled repeatedly. You want to pass it to each child once. Method 1 is a good way to do that, or you can use the initializer keyword argument to explicitly pass data to the child. This will use fork on Linux and pickling on Windows to pass data to the child process:
import pandas as pd
import numpy as np
from multiprocessing import Pool
data = None
def init(_data):
global data
data = _data # data is now accessible in all children, even on Windows
# method #1
def foo(i): return data[i]
if __name__ == '__main__':
data = pd.Series(np.array(range(100000)))
pool = Pool(2, initializer=init, initargs=(data,))
print pool.map(foo,[10,134,8,1])
Using the first method, you're leveraging the behavior of fork to allow the child process to inherit the data object. fork has copy-on-write semantics, which means that the memory is actually shared between the parent and its children, until you try to write to it in the child. When you try to write, the memory page that the data you're trying to write is contained in must be copied, to keep it separate from the parent version.
Now, this sounds like a slam dunk - no need to copy anything as long as we don't write to it, which is surely faster than the pickle/unpickle method. And that's usually the case. However, in practice, Python is internally writing to its objects, even when you wouldn't really expect it to. Because Python uses reference counting for memory management, it needs to increment the internal reference counter on each object every time its passed to a method, or assigned to variable, etc. So, that means the memory page containing the reference count for each object passed to your child process will end up getting copied. This will definitely be faster and use less memory than pickling data multiple times, but isn't quite completely shared, either.