Look at the codes below, I use two ways to solve the problem (simple recursive and DP). Why is the DP way slower?
What's your suggestion?
#!/usr/local/bin/python2.7
# encoding: utf-8
Problem: There is an array with positive integer. given a positive integer S,\
find the total number of combinations in Which the numbers' sum is S.
Method I:
def find_sum_recursive(number_list, sum_to_find):
count = 0
for i in range(len(number_list)):
sub_sum = sum_to_find - number_list[i]
if sub_sum < 0:
continue
elif sub_sum == 0:
count += 1
continue
else:
sub_list = number_list[i + 1:]
count += find_sum_recursive(sub_list, sub_sum)
return count
Method II:
def find_sum_DP(number_list, sum_to_find):
count = 0
if(0 == sum_to_find):
count = 1
elif([] != number_list and sum_to_find > 0):
count = find_sum_DP(number_list[:-1], sum_to_find) + find_sum_DP(number_list[:-1], sum_to_find - number_list[:].pop())
return count
Running it:
def main(argv=None): # IGNORE:C0111
number_list = [5, 5, 10, 3, 2, 9, 8]
sum_to_find = 15
input_setup = ';number_list = [5, 5, 10, 3, 2, 9, 8, 7, 6, 4, 3, 2, 9, 5, 4, 7, 2, 8, 3];sum_to_find = 15'
print 'Calculating...'
print 'recursive starting'
count = find_sum_recursive(number_list, sum_to_find)
print timeit.timeit('count = find_sum_recursive(number_list, sum_to_find)', setup='from __main__ import find_sum_recursive' + input_setup, number=10)
cProfile.run('find_sum_recursive(' + str(number_list) + ',' + str(sum_to_find) + ')')
print 'recursive ended:', count
print 'DP starting'
count_DP = find_sum_DP(number_list, sum_to_find)
print timeit.timeit('count_DP = find_sum_DP(number_list, sum_to_find)', setup='from __main__ import find_sum_DP' + input_setup, number=10)
cProfile.run('find_sum_DP(' + str(number_list) + ',' + str(sum_to_find) + ')')
print 'DP ended:', count_DP
print 'Finished.'
if __name__ == '__main__':
sys.exit(main())
I recode the method II, and it's right now:
def find_sum_DP(number_list, sum_to_find):
count = [[0 for i in xrange(0, sum_to_find + 1)] for j in xrange(0, len(number_list) + 1)]
for i in range(len(number_list) + 1):
for j in range(sum_to_find + 1):
if (0 == i and 0 == j):
count[i][j] = 1
elif (i > 0 and j > 0):
if (j > number_list[i - 1]):
count[i][j] = count[i - 1][j] + count[i - 1][j - number_list[i - 1]]
elif(j < number_list[i - 1]):
count[i][j] = count[i - 1][j]
else:
count[i][j] = count[i - 1][j] + 1
else:
count[i][j] = 0
return count[len(number_list)][sum_to_find]
Compare between method I & II:
Calculating...
recursive starting
0.00998711585999
92 function calls (63 primitive calls) in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
30/1 0.000 0.000 0.000 0.000 FindSum.py:18(find_sum_recursive)
30 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
30 0.000 0.000 0.000 0.000 {range}
recursive ended: 6
DP starting
0.00171685218811
15 function calls in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 FindSum.py:33(find_sum_DP)
3 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
9 0.000 0.000 0.000 0.000 {range}
DP ended: 6
Finished.
If you're using iPython, %prun is your friend here.
Take a look at the output for the recursive version:
2444 function calls (1631 primitive calls) in 0.002 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
814/1 0.002 0.000 0.002 0.002 <ipython-input-1-7488a6455e38>:1(find_sum_recursive)
814 0.000 0.000 0.000 0.000 {range}
814 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.002 0.002 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
And now, for the DP version:
10608 function calls (3538 primitive calls) in 0.007 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
7071/1 0.007 0.000 0.007 0.007 <ipython-input-15-3535e3ab26eb>:1(find_sum_DP)
3535 0.001 0.000 0.001 0.000 {method 'pop' of 'list' objects}
1 0.000 0.000 0.007 0.007 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
7071 is quite a bit higher than 814!
Your problem here is that your dynamic programming method isn't dynamic programming! The point of dynamic programming is that, when you have a problem with overlapping subproblems, as you do here, you store the results of each subproblem, and then when if you need the result again, you take it from that store rather than recalculating. Your code doesn't do that: every time you call find_sum_DP, you're recalculating, even if the same calculation has already been done. The result is that your _DP method is actually not only recursive, but recursive with more function calls than your recursive method.
(I'm currently writing a DP version to demonstrate)
Edit:
I need to add the caveat that, while I should know much more about dynamic programming, I very embarrassingly don't. I'm also writing this quickly and late at night, a bit as an exercise for myself. Nevertheless, here is a dynamic programming implementation of the function:
import numpy as np
def find_sum_realDP( number_list, sum_to_find ):
memo = np.zeros( (len(number_list),sum_to_find+1) ,dtype=np.int)-1
# This will store our results. memo[l][n] will give us the result
# for number_list[0:l+1] and a sum_to_find of n. If it hasn't been
# calculated yet, it will give us -1. This is not at all efficient
# storage, but isn't terribly bad.
# Now that we have that, we'll call the real function. Instead of modifying
# the list and making copies or views, we'll keep the same list, and keep
# track of the index we're on (nli).
return find_sum_realDP_do( number_list, len(number_list)-1, sum_to_find, memo ),memo
def find_sum_realDP_do( number_list, nli, sum_to_find, memo ):
# Our count is 0 by default.
ret = 0
# If we aren't at the sum to find yet, do we have any numbers left after this one?
if ((sum_to_find > 0) and nli>0):
# Each of these checks to see if we've already stored the result of the calculation.
# If so, we use that, if not, we calculate it.
if memo[nli-1,sum_to_find]>=0:
ret += memo[nli-1,sum_to_find]
else:
ret += find_sum_realDP_do(number_list, nli-1, sum_to_find, memo)
# This one is a bit tricky, and was a bug when I first wrote it. We don't want to
# have a negative sum_to_find, because that will be very bad; we'll start using results
# from other places in memo because it will wrap around.
if (sum_to_find-number_list[nli]>=0) and memo[nli-1,sum_to_find-number_list[nli]]>=0:
ret += memo[nli-1,sum_to_find-number_list[nli]]
elif (sum_to_find-number_list[nli]>=0):
ret += find_sum_realDP_do(number_list, nli-1, sum_to_find-number_list[nli], memo)
# Do we not actually have any sum to find left?
elif (0 == sum_to_find):
ret = 1
# If we only have one number left, will it get us there?
elif (nli == 0) and (sum_to_find-number_list[nli] == 0 ):
ret = 1
# Store our result.
memo[nli,sum_to_find] = ret
# Return our result.
return ret
Note that this uses numpy. It's very likely that you don't have this installed, but I'm not sure how to write a reasonably-performing dynamic programming algorithm in Python without it; I don't think Python lists have anywhere near the performance of Numpy arrays. Note also that this vs your code deals with zeros differently, so rather than debug this I'll just say that this code is for nonzero positive integers in the number list. Now, with this algorithm, profiling gives us:
243 function calls (7 primitive calls) in 0.001 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
237/1 0.001 0.000 0.001 0.001 <ipython-input-155-4a624e5a99b7>:9(find_sum_realDP_do)
1 0.000 0.000 0.001 0.001 <ipython-input-155-4a624e5a99b7>:1(find_sum_realDP)
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.zeros}
1 0.000 0.000 0.001 0.001 <string>:1(<module>)
2 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
243 is a great deal better than even the recursive version! But your example data is small enough that it doesn't really show off how much better a dynamic programming algorithm is.
Let's try nlist2 = [7, 6, 2, 3, 7, 7, 2, 7, 4, 2, 4, 5, 6, 1, 7, 4, 6, 3, 2, 1, 1, 1, 4,
2, 3, 5, 2, 4, 4, 2, 4, 5, 4, 2, 1, 7, 6, 6, 1, 5, 4, 5, 3, 2, 3, 7,
1, 7, 6, 6], with the same sum_to_find=15. This has 50 values, and 900206 ways to get 15...
With find_sum_recursive:
3335462 function calls (2223643 primitive calls) in 14.137 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1111820/1 13.608 0.000 14.137 14.137 <ipython-input-46-7488a6455e38>:1(find_sum_recursive)
1111820 0.422 0.000 0.422 0.000 {range}
1111820 0.108 0.000 0.108 0.000 {len}
1 0.000 0.000 14.137 14.137 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
And now with find_sum_realDP:
736 function calls (7 primitive calls) in 0.007 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
730/1 0.007 0.000 0.007 0.007 <ipython-input-155-4a624e5a99b7>:9(find_sum_realDP_do)
1 0.000 0.000 0.007 0.007 <ipython-input-155-4a624e5a99b7>:1(find_sum_realDP)
1 0.000 0.000 0.000 0.000 {numpy.core.multiarray.zeros}
1 0.000 0.000 0.007 0.007 <string>:1(<module>)
2 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
So we have less than 1/1000th of the calls, and run in less than 1/2000th of the time. Of course, the bigger a list you use, the better the DP algorithm will work. On my computer, running with sum_to_find of 15 and a list of 600 random numbers from 1 to 8, realDP only takes 0.09 seconds, and has less than 10,000 function calls; it's around this point that the 64-bit integers I'm using start overflowing and we have all sorts of other problems. Needless to say, the recursive algorithm would never be able to handle a list anywhere near that size before the computer stopped functioning, either from the materials inside it breaking down or the heat death of the universe.
One thing is that your code does much list copying. It would be faster if it just passed index or indices to define a “window view” and not to copy the lists all over. For the first method you can easily add a parametr starting_index and use it in your for loop. In the second method, your write number_list[:].pop() and copy whole list just to get the last element which you could simply do as number_list[-1]. You could also add a parameter ending_index and use it in your test (len(number_list) == ending_index instead of number_list != [], btw even just plain number_list is better than testing against empty list).
Related
I'm trying to profile a few lines of Pandas code, and when I run %prun i'm finding most of my time is taken by {isinstance}. This seems to happen a lot -- can anyone suggest what that means and, for bonus points, suggest a way to avoid it?
This isn't meant to be application specific, but here's a thinned out version of the code if that's important:
def flagOtherGroup(df):
try:mostUsed0 = df[df.subGroupDummy == 0].siteid.iloc[0]
except: mostUsed0 = -1
try: mostUsed1 = df[df.subGroupDummy == 1].siteid.iloc[0]
except: mostUsed1 = -1
df['mostUsed'] = 0
df.loc[(df.subGroupDummy == 0) & (df.siteid == mostUsed1), 'mostUsed'] = 1
df.loc[(df.subGroupDummy == 1) & (df.siteid == mostUsed0), 'mostUsed'] = 1
return df[['mostUsed']]
%prun -l15 temp = test.groupby('userCode').apply(flagOtherGroup)
And top lines of prun:
Ordered by: internal time
List reduced from 531 to 15 due to restriction <15>
ncalls tottime percall cumtime percall filename:lineno(function)
834472 1.908 0.000 2.280 0.000 {isinstance}
497048/395400 1.192 0.000 1.572 0.000 {len}
32722 0.879 0.000 4.479 0.000 series.py:114(__init__)
34444 0.613 0.000 1.792 0.000 internals.py:3286(__init__)
25990 0.568 0.000 0.568 0.000 {method 'reduce' of 'numpy.ufunc' objects}
82266/78821 0.549 0.000 0.744 0.000 {numpy.core.multiarray.array}
42201 0.544 0.000 1.195 0.000 internals.py:62(__init__)
42201 0.485 0.000 1.812 0.000 internals.py:2015(make_block)
166244 0.476 0.000 0.615 0.000 {getattr}
4310 0.455 0.000 1.121 0.000 internals.py:2217(_rebuild_blknos_and_blklocs)
12054 0.417 0.000 2.134 0.000 internals.py:2355(apply)
9474 0.385 0.000 1.284 0.000 common.py:727(take_nd)
isinstance, len and getattr are just the built-in functions. There are a huge number of calls to the isinstance() function here; it is not that the call itself takes a lot of time, but the function was used 834472 times.
Presumably it is the pandas code that uses it.
I have a script that finds the sum of all numbers that can be written as the sum of fifth powers of their digits. (This problem is described in more detail on the Project Euler web site.)
I have written it two ways, but I do not understand the performance difference.
The first way uses nested list comprehensions:
exp = 5
def min_combo(n):
return ''.join(sorted(list(str(n))))
def fifth_power(n, exp):
return sum([int(x) ** exp for x in list(n)])
print sum( [fifth_power(j,exp) for j in set([min_combo(i) for i in range(101,1000000) ]) if int(j) > 10 and j == min_combo(fifth_power(j,exp)) ] )
and profiles like this:
$ python -m cProfile euler30.py
443839
3039223 function calls in 2.040 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1007801 1.086 0.000 1.721 0.000 euler30.py:10(min_combo)
7908 0.024 0.000 0.026 0.000 euler30.py:14(fifth_power)
1 0.279 0.279 2.040 2.040 euler30.py:6(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1007801 0.175 0.000 0.175 0.000 {method 'join' of 'str' objects}
1 0.013 0.013 0.013 0.013 {range}
1007801 0.461 0.000 0.461 0.000 {sorted}
7909 0.002 0.000 0.002 0.000 {sum}
The second way is the more usual for loop:
exp = 5
ans= 0
def min_combo(n):
return ''.join(sorted(list(str(n))))
def fifth_power(n, exp):
return sum([int(x) ** exp for x in list(n)])
for j in set([ ''.join(sorted(list(str(i)))) for i in range(100, 1000000) ]):
if int(j) > 10:
if j == min_combo(fifth_power(j,exp)):
ans += fifth_power(j,exp)
print 'answer', ans
Here is the profiling info again:
$ python -m cProfile euler30.py
answer 443839
2039325 function calls in 1.709 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
7908 0.024 0.000 0.026 0.000 euler30.py:13(fifth_power)
1 1.081 1.081 1.709 1.709 euler30.py:6(<module>)
7902 0.009 0.000 0.015 0.000 euler30.py:9(min_combo)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1007802 0.147 0.000 0.147 0.000 {method 'join' of 'str' objects}
1 0.013 0.013 0.013 0.013 {range}
1007802 0.433 0.000 0.433 0.000 {sorted}
7908 0.002 0.000 0.002 0.000 {sum}
Why does the list comprehension implementation call min_combo() 1,000,000 more times than the for loop implementation?
Because on the second one you implemented again the content of min_combo inside the set call...
Do the same thing and you'll have the same result.
BTW, change those to avoid big lists being created:
sum([something for foo in bar]) -> sum(something for foo in bar)
set([something for foo in bar]) -> set(something for foo in bar)
(without [...] they become generator expressions).
I am using np.random.multinomial to sample a multinomial distribution M times (given probabilities [X_0 X_1 .. X_n] it returns counts [C_0 C_1 ... C_n] sampled from the specified multinomial, where \sum_i C_i = M). Given these sampled values (the C_i's), I want to assign them uniformly at random to some objects I have.
Currently what I'm doing is:
draws = np.random.multinomial(M, probs, size=1)
draws = draws[0]
draws_list = []
for idx,num in enumerate(draws):
draws_list += [idx]*num
random.shuffle(draws_list)
Then draws_list is a randomly shuffled list of the sampled values.
The problem is that populating draws_list (the for loop) is very slow. Is there a better/faster way to do this?
Try this code. This strategy is to allocate the memory first, then to fill data.
draws_list1 = np.empty(M, dtype=np.int)
acc = 0
for idx, num in enumerate(draws):
draws_list1[acc:acc+num].fill(idx)
acc += num
Here's the full code for profiling.
import numpy as np
import cProfile
M=10000000
draws = np.random.multinomial(M, [1/6.]*6, size=1)
draws = draws[0]
draws_list1 = np.empty(M, dtype=np.int)
def impl0():
draws_list0 = []
for idx, num in enumerate(draws):
draws_list0 += [idx]*num
return draws_list0
def impl1():
acc = 0
for idx, num in enumerate(draws):
draws_list1[acc:acc+num].fill(idx)
acc += num
return draws_list1
cProfile.run("impl0()")
cProfile.run("impl1()")
Here's the result of cProfile. If the statement np.empty is located in function impl1, 0.020 seconds are elapsed.
3 function calls in 0.095 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.020 0.020 0.095 0.095 <string>:1(<module>)
1 0.076 0.076 0.076 0.076 prof.py:11(impl0)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
9 function calls in 0.017 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.017 0.017 <string>:1(<module>)
1 0.000 0.000 0.017 0.017 prof.py:17(impl1)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
6 0.017 0.003 0.017 0.003 {method 'fill' of 'numpy.ndarray' objects}
I'm trying to time two different functions in python.
The first:
import cProfile
def bin_search(A, first,last, target):
#returns index of target in A, if present
#returns -1 if target is not present in A
if first > last:
return -1
else:
mid = (first+last)/2
if A[mid]==target:
return mid
elif A[mid]>target:
return bin_search(A,first,mid-1,target)
else:
return bin_search(A,mid+1,last,target)
the second
def trin_search(A,first,last,target):
#returns index of target in A, if present
#returns -1 if target is not present in A
if target> last or target<first:
return -1
if first>last:
return -1
else:
one_third=first+(last-first)/3
two_thirds=first+2*(last-first)/3
if A[one_third]==target:
return one_third
elif A[one_third]>target:
#search the left-hand third
return trin_search(A,first, one_third,target)
elif A[two_thirds]==target:
return two_thirds
elif A[two_thirds]>target:
#search the middle third
return trin_search(A,one_third+1,two_thirds-1,target)
else:
#search the right-hand third
return trin_search(A,two_thirds+1,last,target)
I'm trying to time them using the cprofile.run() method. I call:
cprofile.run('trin_search(newlist, newlist[0], newlist[-1], 17)')
and
cprofile.run('bin_search(newlist, newlist[0], newlist[-1], 17)')
with the results for the first:
6 function calls (4 primitive calls) in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
3/1 0.000 0.000 0.000 0.000 Jan 18.py:16(trin_search)
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 0.000 0.000 profile:0(trin_search(newlist, newlist[0], newlist[-1], 17))
and the second
7 function calls (3 primitive calls) in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
5/1 0.000 0.000 0.000 0.000 Jan 18.py:2(bin_search)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
How is it possible that they take 0 time to operate?
Cheers,
As already pointed out by other use the timeit module, here's an example how to time a function with parameters:
import timeit
arg = 10
def foo(arg):
return arg**arg
t=timeit.Timer("foo(arg)","from __main__ import foo, arg")
print t.timeit(5)
Note that you have to import both the function and the variables that you are using in your function call.
Also, I would suggest that you use IPython where you have "magic commands" so you can simply do %timeit foo(arg).
For your example, this should work:
t=timeit.Timer("bin_search(newlist, newlist[0], newlist[-1], 17)",
"from __main__ import bin_search, newlist")
try the timeit module. It is made for benchmarking code snippets.
Based on that answer here are two versions of merge function used for mergesort.
Could you help me to understand why the second one is much faster.
I have tested it for list of 50000 and the second one is 8 times faster (Gist).
def merge1(left, right):
i = j = inv = 0
merged = []
while i < len(left) and j < len(right):
if left[i] <= right[j]:
merged.append(left[i])
i += 1
else:
merged.append(right[j])
j += 1
inv += len(left[i:])
merged += left[i:]
merged += right[j:]
return merged, inv
.
def merge2(array1, array2):
inv = 0
merged_array = []
while array1 or array2:
if not array1:
merged_array.append(array2.pop())
elif (not array2) or array1[-1] > array2[-1]:
merged_array.append(array1.pop())
inv += len(array2)
else:
merged_array.append(array2.pop())
merged_array.reverse()
return merged_array, inv
Here is the sort function:
def _merge_sort(list, merge):
len_list = len(list)
if len_list < 2:
return list, 0
middle = len_list / 2
left, left_inv = _merge_sort(list[:middle], merge)
right, right_inv = _merge_sort(list[middle:], merge)
l, merge_inv = merge(left, right)
inv = left_inv + right_inv + merge_inv
return l, inv
.
import numpy.random as nprnd
test_list = nprnd.randint(1000, size=50000).tolist()
test_list_tmp = list(test_list)
merge_sort(test_list_tmp, merge1)
test_list_tmp = list(test_list)
merge_sort(test_list_tmp, merge2)
Similar answer as kreativitea's above, but with more info (i think!)
So profiling the actual merge functions, for the merging of two 50K arrays,
merge 1
311748 function calls in 15.363 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 15.363 15.363 <string>:1(<module>)
1 15.322 15.322 15.362 15.362 merge.py:3(merge1)
221309 0.030 0.000 0.030 0.000 {len}
90436 0.010 0.000 0.010 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
merge2
250004 function calls in 0.104 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 0.104 0.104 <string>:1(<module>)
1 0.074 0.074 0.103 0.103 merge.py:20(merge2)
50000 0.005 0.000 0.005 0.000 {len}
100000 0.010 0.000 0.010 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
100000 0.014 0.000 0.014 0.000 {method 'pop' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'reverse' of 'list' objects}
So for merge1, it's 221309 len, 90436 append, and takes 15.363 seconds.
So for merge2, it's 50000 len, 100000 append, and 100000 pop and takes 0.104 seconds.
len and append pop are all O(1) (more info here), so these profiles aren't showing what's actually taking the time, since going of just that, it should be faster, but only ~20% so.
Okay the cause is actually fairly obvious if you just read the code:
In the first method, there is this line:
inv += len(left[i:])
so every time that is called, it has to rebuild an array. If you comment out this line (or just replace it with inv += 1 or something) then it becomes faster than the other method. This is the single line responsible for the increased time.
Having noticed this is the cause, the issue can be fixed by improving the code; change it to this for a speed up. After doing this, it will be faster than merge2
inv += len(left) - i
Update it to this:
def merge3(left, right):
i = j = inv = 0
merged = []
while i < len(left) and j < len(right):
if left[i] <= right[j]:
merged.append(left[i])
i += 1
else:
merged.append(right[j])
j += 1
inv += len(left) - i
merged += left[i:]
merged += right[j:]
return merged, inv
You can use the excellent cProfile module to help you solve things like this.
>>> import cProfile
>>> a = range(1,20000,2)
>>> b = range(0,20000,2)
>>> cProfile.run('merge1(a, b)')
70002 function calls in 0.195 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.184 0.184 0.195 0.195 <pyshell#7>:1(merge1)
1 0.000 0.000 0.195 0.195 <string>:1(<module>)
50000 0.008 0.000 0.008 0.000 {len}
19999 0.003 0.000 0.003 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
>>> cProfile.run('merge2(a, b)')
50004 function calls in 0.026 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.016 0.016 0.026 0.026 <pyshell#12>:1(merge2)
1 0.000 0.000 0.026 0.026 <string>:1(<module>)
10000 0.002 0.000 0.002 0.000 {len}
20000 0.003 0.000 0.003 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
20000 0.005 0.000 0.005 0.000 {method 'pop' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'reverse' of 'list' objects}
After looking at the information a bit, it looks like the commenters are correct-- its not the len function-- it's the string module. The string module is invoked when you compare the length of things, as follows:
>>> cProfile.run('0 < len(c)')
3 function calls in 0.000 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {len}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
It is also invoked when slicing a list, but this is a very quick operation.
>>> len(c)
20000000
>>> cProfile.run('c[3:2000000]')
2 function calls in 0.011 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.011 0.011 0.011 0.011 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
TL;DR: Something in the string module is taking 0.195s in your first function, and 0.026s in your second function. : apparently, the rebuilding of the array in inv += len(left[i:]) this line.
If I had to guess, I would say it probably has to do with the cost of removing elements from a list, removing from the end (pop) is quicker than removing from the beginning. the second favors removing elements from the end of the list.
See Performance Notes: http://effbot.org/zone/python-list.htm
"The time needed to remove an item is about the same as the time needed to insert an item at the same location; removing items at the end is fast, removing items at the beginning is slow."