If I am creating a program that does some complex calculations on a data set and I already know what some of the values should be, should I still calculate them? For example if I know that 0 or 1 would always be themselves should I just check if the value is 0 or 1 or actually do the calculations?
Edit:
I don't have code because I was asking as a concept. I was creating a program to return the base 10 log of each number in a data set and I was wondering if it would be more efficient to return values I already knew like 0 for 1, "undefined" for 0, and the number of zeros for numbers divisible by 10. I wasn't sure if it was more efficient and if it would be efficient on a larger scale.
Let's try this simple example
$ python3 -m timeit -s "from math import log; mylog=lambda x: log(x)" "mylog(1)"
10000000 loops, best of 3: 0.152 usec per loop
$ python3 -m timeit -s "from math import log; mylog=lambda x: 0.0 if x==1 else log(x)" "mylog(1)"
10000000 loops, best of 3: 0.0976 usec per loop
So there is some speedup, however. All the non special cases run slower
$ python3 -m timeit -s "from math import log; mylog=lambda x: log(x)" "mylog(2)"
10000000 loops, best of 3: 0.164 usec per loop
$ python3 -m timeit -s "from math import log; mylog=lambda x: 0.0 if x==1 else log(x)" "mylog(2)"
1000000 loops, best of 3: 0.176 usec per loop
And in this case, it's better just to leave the wrapper function out altogether
$ python3 -m timeit -s "from math import log" "log(2)"
10000000 loops, best of 3: 0.0804 usec per loop
I have a list, but the numbers in it are strings so I can't find the sum of the list, so I need help in converting the numbers in the list to int.
This is my code
def convertStr(cals):
ret = float(cals)
return ret
TotalCal = sum(cals)
So basically there is list called cals
and it looks like this
(20,45,...etc)
But the numbers in it are strings so when I try finding the sum like this
TotalCal = sum(cals)
And then run it shows an error saying that the list needs to be an int format
so the question is how do I convert all numbers in the list to int format?
If you have a different way of finding the sum of lists it will be good too.
You can use either the python builtin map or a list comprehension for this
def convertStr(cals):
ret = [float(i) for i in (cals)]
return ret
or
def convertStr(cals):
return map(float,cals)
Here are the timeit results for both the approaches
$ python -m timeit "cals = ['1','2','3','4'];[float(i) for i in (cals)]"
1000000 loops, best of 3: 0.804 usec per loop
$ python -m timeit "cals = ['1','2','3','4'];map(float,cals)"
1000000 loops, best of 3: 0.787 usec per loop
As you can see map is faster and more pythonic as compared to the list comprehension. This is discussed in full length here
map may be microscopically faster in some cases (when you're NOT making a lambda for the purpose, but using the same function in map and a listcomp). List comprehensions may be faster in other cases
Another way using itertools.imap. This is the fastest for long lists
from itertools import imap
TotalCal = sum(imap(float,cals)
And using timeit for a list with 1000 entries.
$ python -m timeit "import random;cals = [str(random.randint(0,100)) for r in range(1000)];sum(map(float,cals))"
1000 loops, best of 3: 1.38 msec per loop
$ python -m timeit "import random;cals = [str(random.randint(0,100)) for r in range(1000)];[float(i) for i in (cals)]"
1000 loops, best of 3: 1.39 msec per loop
$ python -m timeit "from itertools import imap;import random;cals = [str(random.randint(0,100)) for r in range(1000)];imap(float,cals)"
1000 loops, best of 3: 1.24 msec per loop
As Padraic mentions below, The imap way is the best way to go! It is fast1 and looks great! Inclusion of a library function has it's bearing on small lists only and not on large lists. Thus for large lists, imap is better suited.
1 List comprehension is still slower than map by 1 micro second!!! Thank god
sum(map(float,cals))
or
sum(float(i) for i in cals)
So I have written down the codes for evaluating polynomial using three different methods. Horner's method should be the fastest, while the naive method should be the slowest, right? But how come the time for computing it is not what I expect? And the time for calculation sometimes turns out to be exactly the same for itera and naive method. What's wrong with it?
import numpy.random as npr
import time
def Horner(c,x):
p=0
for i in c[-1::-1]:
p = p*x+i
return p
def naive(c,x):
n = len(c)
p = 0
for i in range(len(c)):
p += c[i]*x**i
return p
def itera(c,x):
p = 0
xi = 1
for i in range(len(c)):
p += c[i]*xi
xi *= x
return p
c=npr.uniform(size=(500,1))
x=-1.34
start_time=time.time()
print Horner(c,x)
print time.time()-start_time
start_time=time.time()
print itera(c,x)
print time.time()-start_time
start_time=time.time()
print naive(c,x)
print time.time()-start_time
here are some of the results:
[ 2.58646959e+69]
0.00699996948242
[ 2.58646959e+69]
0.00600004196167
[ 2.58646959e+69]
0.00600004196167
[ -3.30717922e+69]
0.00899982452393
[ -3.30717922e+69]
0.00600004196167
[ -3.30717922e+69]
0.00600004196167
[ -2.83469309e+69]
0.00999999046326
[ -2.83469309e+69]
0.00999999046326
[ -2.83469309e+69]
0.0120000839233
Your profiling can be much improved. Plus, we can make your code run 200-500x faster.
(1) Rinse and repeat
You can't run just one iteration of a performance test, for two reasons.
Your time resolution might not be good enough. This is why you sometimes got the same time for two implementations: the time for one run was near the resolution of your timing mechanism, so you recorded only one "tick".
There are all sorts of factors that affect performance. Your best bet for a meaningful comparison will be a lot of iterations.
You don't need gazillions of runs (though, of course, that doesn't hurt), but you estimate and adjust the number of iterations until the variance is within a level acceptable to your purpose.
timeit is a nice little module for profiling Python code.
I added this to bottom of your script.
import timeit
n = 1000
print 'Horner', timeit.timeit(
number = n,
setup='from __main__ import Horner, c, x',
stmt='Horner(c,x)'
)
print 'naive', timeit.timeit(
number = n,
setup='from __main__ import naive, c, x',
stmt='naive(c,x)',
)
print 'itera', timeit.timeit(
number = n,
setup='from __main__ import itera, c, x',
stmt='itera(c,x)',
)
Which produces
Horner 1.8656351566314697
naive 2.2408010959625244
itera 1.9751169681549072
Horner is the fastest, but it's not exactly blowing the doors off the other two.
(2) Look at what is happening...very carefully
Python has operator overloading, so it's easy to miss seeing this.
npr.uniform(size=(500,1)) is giving you a 500 x 1 numpy structure of random numbers.
So what?
Well, c[i] isn't a number. It's a numpy array with one element. Numpy overloads the operators so you can do things like multiply an array by a scalar.
That's fine, but using an array for every element is a lot of overhead, so it's harder to see the difference between the algorithms.
Instead, let's try a simple Python list:
import random
c = [random.random() for _ in range(500)]
And now,
Horner 0.034661054611206055
naive 0.12771987915039062
itera 0.07331395149230957
Whoa! All the time times just got faster (by 10-60x). Proportionally, the Horner implementation got even faster than the other two. We removed the overhead on all three, and can now see the "bare bones" difference.
Horner is 4x faster than naive and 2x faster than itera.
(3) Alternate runtimes
You're using Python 2. I assume 2.7.
Let's see how Python 3.4 fares. (Syntax adjustment: you'll need to put parenthesis around the argument list to print.)
Horner 0.03298933599944576
naive 0.13706714100044337
itera 0.06771054599812487
About the same.
Let's try PyPy, a JIT implementation of Python. (The "normal" Python implementation is called CPython.)
Horner 0.006507158279418945
naive 0.07541298866271973
itera 0.005059003829956055
Nice! Each implementation is now running 2-5x faster. Horner is now 10x the speed of naive, but slightly slower than itera.
JIT runtimes are more difficult to profile than interpreters. Let's increase the number of iterations to 50000, and try it just to make sure.
Horner 0.12749004364013672
naive 3.2823100090026855
itera 0.06546688079833984
(Note that we have 50x the iterations, but only 20x the time...the JIT hadn't taken full effect for many of the first 1000 runs.) Same conclusions, but the differences are even more pronounced.
Granted, the idea of JIT is to profile, analyze, and rewrite the program at runtime, so if your goal is to compare algorithms, this is going to add a lot of non-obvious implementation detail.
Nonetheless, comparing runtimes can be useful in giving a broader perspective.
There are a few more things. For example, your naive implementation computes a variable it never uses. You use range instead of xrange. You could try iterating backwards with an index rather than a reverse slice. Etc.
None of these changed the results much for me, but they were worth considering.
You cannot obtain accurate result by measuring things like that:
start_time=time.time()
print Horner(c,x)
print time.time()-start_time
Presumably most of the time is spend in the IO function involved by the print function. In addition, to have something significant, you should perform the measure on a large number of iteration in order to smooth errors. In the general case, you might want to perform your test on various input data as well -- as depending your algorithm, some case might coincidentally be solved more efficiently than others.
You should definitively take a look at the timeit module. Something like that, maybe:
import timeit
print 'Horner',timeit.timeit(stmt='Horner(c,x)',
setup='from __main__ import Horner, c, x',
number = 10000)
# ^^^^^
# probably not enough. Increase that once you will
# be confident
print 'naive',timeit.timeit(stmt='naive(c,x)',
setup='from __main__ import naive, c, x',
number = 10000)
print 'itera',timeit.timeit(stmt='itera(c,x)',
setup='from __main__ import itera, c, x',
number = 10000)
Producing this on my system:
Horner 23.3317809105
naive 28.305519104
itera 24.385917902
But still with variable results from on run to the other:
Horner 21.1151690483
naive 23.4374330044
itera 21.305426836
As I said before, to obtain more meaningful results, you should definitively increase the number of tests, and run that on several test case in order to smooth results.
If you are doing a lot of benchmarking, scientific computing, numpy related work and many more things using ipython will be an extremely useful tool.
To benchmark you can time the code with timeit using ipython magic where you will get more consistent results each run, it is simply a matter of using timeit then the function or code to time :
In [28]: timeit Horner(c,x)
1000 loops, best of 3: 670 µs per loop
In [29]: timeit naive(c,x)
1000 loops, best of 3: 983 µs per loop
In [30]: timeit itera(c,x)
1000 loops, best of 3: 804 µs per loop
To time code spanning more than one line you simply use %%timeit:
In [35]: %%timeit
....: for i in range(100):
....: i ** i
....:
10000 loops, best of 3: 110 µs per loop
ipython can compile cython code, f2py code and do numerous other very helpful tasks using different plugins and ipython magic commands.
builtin magic commands
Using cython and some very basic improvements we can improve the efficiency of Horner by about 25 percent:
In [166]: %%cython
import numpy as np
cimport numpy as np
cimport cython
ctypedef np.float_t DTYPE_t
def C_Horner(c, DTYPE_t x):
cdef DTYPE_t p
for i in reversed(c):
p = p * x + i
return p
In [28]: c=npr.uniform(size=(2000,1))
In [29]: timeit Horner(c,-1.34)
100 loops, best of 3: 3.93 ms per loop
In [30]: timeit C_Horner(c,-1.34)
100 loops, best of 3: 2.21 ms per loop
In [31]: timeit itera(c,x)
100 loops, best of 3: 4.10 ms per loop
In [32]: timeit naive(c,x)
100 loops, best of 3: 4.95 ms per loop
Using the list in #Paul drapers answer our cythonised version runs twice as fast as the original function and much faster then ietra and naive:
In [214]: import random
In [215]: c = [random.random() for _ in range(500)]
In [44]: timeit C_Horner(c, -1.34)
10000 loops, best of 3: 18.9 µs per loop
In [45]: timeit Horner(c, -1.34)
10000 loops, best of 3: 44.6 µs per loop
In [46]: timeit naive(c, -1.34)
10000 loops, best of 3: 167 µs per loop
In [47]: timeit itera(c,-1.34)
10000 loops, best of 3: 75.8 µs per loop
This is mostly an exercise in learning Python. I wrote this function to test if a number is prime:
def p1(n):
for d in xrange(2, int(math.sqrt(n)) + 1):
if n % d == 0:
return False
return True
Then I realized I can make easily rewrite it using any():
def p2(n):
return not any((n % d == 0) for d in xrange(2, int(math.sqrt(n)) + 1))
Performance-wise, I was expecting p2 to be faster than, or at the very least as fast as, p1 because any() is builtin, but for a large-ish prime, p2 is quite a bit slower:
$ python -m timeit -n 100000 -s "import test" "test.p1(999983)"
100000 loops, best of 3: 60.2 usec per loop
$ python -m timeit -n 100000 -s "import test" "test.p2(999983)"
100000 loops, best of 3: 88.1 usec per loop
Am I using any() incorrectly here? Is there a way to write this function using any() so that it's as far as iterating myself?
Update: Numbers for an even larger prime
$ python -m timeit -n 1000 -s "import test" "test.p1(9999999999971)"
1000 loops, best of 3: 181 msec per loop
$ python -m timeit -n 1000 -s "import test" "test.p2(9999999999971)"
1000 loops, best of 3: 261 msec per loop
The performance difference is minimal, but the reason it exists is that any incurs building a generator expression, and an extra function call, compared to the for loop. Both have identical behaviors, though (shortcut evaluation).
As the size of your input grows, the difference won't diminish (I was wrong) because you're using a generator expression, and iterating over it requires calling a method (.next()) on it and an extra stack frame. any does that under the hood, of course.
The for loop is iterating over an xrange object. any is iterating over a generator expression, which itself is iterating over an xrange object.
Either way, use whichever produces the most readable/maintainable code. Choosing one over the other will have little, if any, performance impact on whatever program you're writing.
From interpreter, I get:
>>> timeit.repeat("-".join( str(n) for n in range(10000) ) , repeat = 3, number=10000)
[1.2294530868530273, 1.2298660278320312, 1.2300069332122803] # this is seconds
From commandline, I get:
$ python -m timeit -n 10000 '"-".join(str(n) for n in range(10000))'
10000 loops, best of 3: 1.79 msec per loop # this is milli second
Why this difference in magnitude of timings in the two cases?
The two lines aren't measuring the same thing. In the first snippet, you're timing the calculation 0-1-2-...-9999. while in the second snippet you're timing the string concatenation "-".join(str(n) for n in range(10000)).
In addition, timeit and repeat report the total time, while the CLI averages the time over the number of iterations. So the first code actually takes 12.29 ms "per loop".