We Keep Coding

Call many object's methods in parallel in python

I have two classes. One called algorithm and the other called Chain. In algorithm, I create multiple chains, which are going to be a sequence of sampled values. I want to run the sampling in parallel at the chain level.
In other words, the algorithm class instantiates n chains and I want to run the _sample method, which belongs to the Chain class, for each of the chains in parallel within the algorithm class.
Below is a sample code that attempts what I would like to do.
I have seen a similar questions here: Apply a method to a list of objects in parallel using multi-processing, but as shown in the function _sample_chains_parallel_worker, this method does not work for my case (I am guessing it is because of the nested class structure).
Question 1: Why does this not work for this case?
The method in _sample_chains_parallel also does not even run in parallel.
Question 2: Why?
Question 3: How do I sample each of these chains in parallel?
import time
import multiprocessing
class Chain():
def __init__(self):
self.thetas = []
def _sample(self):
for i in range(3):
time.sleep(1)
self.thetas.append(i)
def clear_thetas(self):
self.thetas = []
class algorithm():
def __init__(self, n=3):
self.n = n
self.chains = []
def _init_chains(self):
for _ in range(self.n):
self.chains.append(Chain())
def _sample_chains(self):
for chain in self.chains:
chain.clear_thetas()
chain._sample()
def _sample_chains_parallel(self):
pool = multiprocessing.Pool(processes=self.n)
for chain in self.chains:
chain.clear_thetas()
pool.apply_async(chain._sample())
pool.close()
pool.join()
def _sample_chains_parallel_worker(self):
def worker(obj):
obj._sample()
pool = multiprocessing.Pool(processes=self.n)
pool.map(worker, self.chains)
pool.close()
pool.join()
if __name__=="__main__":
import time
alg = algorithm()
alg._init_chains()
start = time.time()
alg._sample_chains()
end = time.time()
print "sequential", end - start
start = time.time()
alg._sample_chains_parallel()
end = time.time()
print "parallel", end - start
start = time.time()
alg._sample_chains_parallel_worker()
end = time.time()
print "parallel, map and worker", end - start

In _sample_chains_parallel you are calling chain._sample() instead of just passing the function: pool.apply_async(chain._sample()). So you are passing the result as an argument instead of letting apply_async calculate it.
But removing () won't help you much, because Python 2 cannot pickle instance methods (possible for Python +3.5). It wouldn't raise the error unless you call get() on the result objects so don't rejoice if you see low times for this approach, that's because it immidiately quits with an unraised exception.
For the parallel versions you would have to relocate worker to the module level and call it pool.apply_async(worker (chain,)) respectively pool.map(worker, self.chains).
Note that you forgot clear_thetas() for _sample_chains_parallel_worker. The better solution would be anyway to let let Chain._sample take care of calling self._clear_thetas().

Timing python function

(Python 2.7.8, Windows)
there is so many questions already about this particular subject, but I cannot seem to get any of them working.
So, what I'm trying to accomplish is timing how long a function takes to execute.
I have functions.py and main.py in following fashion:
#functions.py
def function(list):
does something
return list
...
#main.py
import functions
...stuff...
while:
list = gets list from file
functions.function(list) <--- this needs to get timed
Now I tried time.time() the start and end points first, but it's not accurate enough (difference tends to be 0.0), and after some googling it seems that this isn't the way to go anyway. Apparently what I should use(?) is timeit module. However I cannot understand how to get the function into it.
Any help?

As you mentioned, there's a Python module made for this very task, timeit. Its syntax, while a little idiosyncratic, is quite easy to understand:
timeit.timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000)
stmt is the function call to be measured, in your case: functions.function(list)
setup is the code you need to create the context necessary for stmt to execute, in your case: import functions; list = gets list from file
number is how many time timeit would run stmt to find its average execution time. You might want to change the number, since calling your function a million times might take a while.
tl;dr:
timeit.timeit(stmt='functions.function(list)', setup='import functions; list = gets list from file', number=100)

you see this demo: time.time
>>> def check(n):
... start = time.time()
... for x in range(n):
... pass
... stop = time.time()
... return stop-start
...
>>> check(1000)
0.0001239776611328125
>>> check(10000)
0.0012159347534179688
>>> check(100)
1.71661376953125e-05
the above function returns hum much time in sec taken by for for n loops.
so the algorithm is:
start = time.time()
# your stuff
stop = time.time()
time_taken = stop - start

start = timeit.default_timer()
my_function()
elapsed = timeit.default_timer() - start

Speed improvements of python object creation

This question is academic in nature and typically if I had these issues in the real world I'd probably do things differently. However I was curious about whether the speed at which python objects are created could be improved.
#!/usr/bin/env python
import time
# boring class
class boring:
def __init__(self, a, b, c):
self.a = a
self.b = b
self.c = c
def timefunc(f):
def f_timer(*args, **kwargs):
start = time.time()
result = f(*args, **kwargs)
end = time.time()
print f.__name__, 'took', end - start, 'time'
return result
return f_timer
#timefunc
def standard_boring():
for x in range(10000000):
b = boring(1,2,3)
del(b)
if __name__ == "__main__":
standard_boring()
On my machine this takes about 4 seconds. So curious what Python is doing I dug deeper using strace.
What I see is many tens of thousands of calls to brk e.g. -
brk(0xe35000) = 0xe35000
brk(0xe34000) = 0xe34000
brk(0xe33000) = 0xe33000
brk(0xe32000) = 0xe32000
brk(0xe31000) = 0xe31000
Man describes brk as
brk() and sbrk() change the location of the program break, which
defines the end of the process's data segment (i.e., the program break
is the first location after the end of the uninitialized data segment).
Increasing the program break has the effect of allocating memory to the
process; decreasing the break deallocates memory.
Each call to this presumably some number of cycles, as such the very fact there are loads of calls to this makes me think this is what takes the time.
So is there a way to tell python "Yo, I'm going to be using lots of memory so allocate loads up front". Can I do anything like that? I know why you wouldn't want to, like I said this is academic.
I've experimented with slots, hypothesizing that if I use less memory then this should improve my speed by the very nature of reducing the reoccurring memory allocation requests. I found the results from this quite "lumpy".
Any ideas?

Memoization Handler [duplicate]

This question already has answers here:
What is memoization and how can I use it in Python?
(14 answers)
Closed 6 months ago.
Is it "good practice" to create a class like the one below that can handle the memoization process for you? The benefits of memoization are so great (in some cases, like this one, where it drops from 501003 to 1507 function calls and from 1.409 to 0.006 seconds of CPU time on my computer) that it seems a class like this would be useful.
However, I've read only negative comments on the usage of eval(). Is this usage of it excusable, given the flexibility this approach offers?
This can save any returned value automatically at the cost of losing side effects. Thanks.
import cProfile
class Memoizer(object):
"""A handler for saving function results."""
def __init__(self):
self.memos = dict()
def memo(self, string):
if string in self.memos:
return self.memos[string]
else:
self.memos[string] = eval(string)
self.memo(string)
def factorial(n):
assert type(n) == int
if n == 1:
return 1
else:
return n * factorial(n-1)
# find the factorial of num
num = 500
# this many times
times = 1000
def factorialTwice():
factorial(num)
for x in xrange(0, times):
factorial(num)
return factorial(num)
def memoizedFactorial():
handler = Memoizer()
for x in xrange(0, times):
handler.memo("factorial(%d)" % num)
return handler.memo("factorial(%d)" % num)
cProfile.run('factorialTwice()')
cProfile.run('memoizedFactorial()')

You can memoize without having to resort to eval.
A (very basic) memoizer:
def memoized(f):
cache={}
def ret(*args):
if args in cache:
return cache[args]
else:
answer=f(*args)
cache[args]=answer
return answer
return ret
#memoized
def fibonacci(n):
if n==0 or n==1:
return 1
else:
return fibonacci(n-1)+fibonacci(n-2)
print fibonacci(100)

eval is often misspelt as evil primarily because the idea of executing "strings" at runtime is fraught with security considerations. Have you escaped the code sufficiently? Quotation marks? And a host of other annoying headaches. Your memoise handler works but it's really not the Python way of doing things. MAK's approach is much more Pythonic. Let's try a few experiments.
I edited up both the versions and made them run just once with 100 as the input. I also moved out the instantiation of Memoizer.
Here are the results.
>>> timeit.timeit(memoizedFactorial,number=1000)
0.08526921272277832h
>>> timeit.timeit(foo0.mfactorial,number=1000)
0.000804901123046875
In addition to this, your version necessitates a wrapper around the the function to be memoised which should be written in a string. That's ugly. MAK's solution is clean since the "process of memoisation" is encapsulated in a separate function which can be conveniently applied to any expensive function in an unobtrusive fashion. This is not very Pythonic. I have some details on writing such decorators in my Python tutorial at http://nibrahim.net.in/self-defence/ in case you're interested.

Accurate timing of functions in python

I'm programming in python on windows and would like to accurately measure the time it takes for a function to run. I have written a function "time_it" that takes another function, runs it, and returns the time it took to run.
def time_it(f, *args):
start = time.clock()
f(*args)
return (time.clock() - start)*1000
i call this 1000 times and average the result. (the 1000 constant at the end is to give the answer in milliseconds.)
This function seems to work but i have this nagging feeling that I'm doing something wrong, and that by doing it this way I'm using more time than the function actually uses when its running.
Is there a more standard or accepted way to do this?
When i changed my test function to call a print so that it takes longer, my time_it function returns an average of 2.5 ms while the cProfile.run('f()') returns and average of 7.0 ms. I figured my function would overestimate the time if anything, what is going on here?
One additional note, it is the relative time of functions compared to each other that i care about, not the absolute time as this will obviously vary depending on hardware and other factors.

Use the timeit module from the Python standard library.
Basic usage:
from timeit import Timer
# first argument is the code to be run, the second "setup" argument is only run once,
# and it not included in the execution time.
t = Timer("""x.index(123)""", setup="""x = range(1000)""")
print t.timeit() # prints float, for example 5.8254
# ..or..
print t.timeit(1000) # repeat 1000 times instead of the default 1million

Instead of writing your own profiling code, I suggest you check out the built-in Python profilers (profile or cProfile, depending on your needs): http://docs.python.org/library/profile.html

You can create a "timeme" decorator like so
import time
def timeme(method):
def wrapper(*args, **kw):
startTime = int(round(time.time() * 1000))
result = method(*args, **kw)
endTime = int(round(time.time() * 1000))
print(endTime - startTime,'ms')
return result
return wrapper
#timeme
def func1(a,b,c = 'c',sleep = 1):
time.sleep(sleep)
print(a,b,c)
func1('a','b','c',0)
func1('a','b','c',0.5)
func1('a','b','c',0.6)
func1('a','b','c',1)

This code is very inaccurate
total= 0
for i in range(1000):
start= time.clock()
function()
end= time.clock()
total += end-start
time= total/1000
This code is less inaccurate
start= time.clock()
for i in range(1000):
function()
end= time.clock()
time= (end-start)/1000
The very inaccurate suffers from measurement bias if the run-time of the function is close to the accuracy of the clock. Most of the measured times are merely random numbers between 0 and a few ticks of the clock.
Depending on your system workload, the "time" you observe from a single function may be entirely an artifact of OS scheduling and other uncontrollable overheads.
The second version (less inaccurate) has less measurement bias. If your function is really fast, you may need to run it 10,000 times to damp out OS scheduling and other overheads.
Both are, of course, terribly misleading. The run time for your program -- as a whole -- is not the sum of the function run-times. You can only use the numbers for relative comparisons. They are not absolute measurements that convey much meaning.

If you want to time a python method even if block you measure may throw, one good approach is to use with statement. Define some Timer class as
import time
class Timer:
def __enter__(self):
self.start = time.clock()
return self
def __exit__(self, *args):
self.end = time.clock()
self.interval = self.end - self.start
Then you may want to time a connection method that may throw. Use
import httplib
with Timer() as t:
conn = httplib.HTTPConnection('google.com')
conn.request('GET', '/')
print('Request took %.03f sec.' % t.interval)
__exit()__ method will be called even if the connection request thows. More precisely, you'd have you use try finally to see the result in case it throws, as with
try:
with Timer() as t:
conn = httplib.HTTPConnection('google.com')
conn.request('GET', '/')
finally:
print('Request took %.03f sec.' % t.interval)
More details here.

This is neater
from contextlib import contextmanager
import time
#contextmanager
def timeblock(label):
start = time.clock()
try:
yield
finally:
end = time.clock()
print ('{} : {}'.format(label, end - start))
with timeblock("just a test"):
print "yippee"

Similar to #AlexMartelli's answer
import timeit
timeit.timeit(fun, number=10000)
can do the trick.