We Keep Coding

Understanding speed in Python through a competition problem exceeding time limits

I recently have been doing some competition problems. In particular this one:
https://open.kattis.com/problems/knightjump
An abridged statement of the problem:
You are given a two dimensional (square) chess board of size N (1-based indexing). Some of the cells on this board are ‘.’ denoting an empty cell. Some of the cells on this board are ‘#’ denoting a blocked cell, which you are not allowed to visit. Exactly one of the cells on this board is ‘K’ denoting the initial position of the knight. Determine the minimum number of steps required for the Knight to reach cell (1,1) while avoiding cells with ‘#’ in the path.
I know this is a simple BFS problem and have written the following code to solve it:
from collections import deque
from sys import stdin
dirs = ((2,1), (2,-1), (-2,1), (-2,-1), (1,2), (1,-2), (-1,2), (-1,-2))
N = int(stdin.readline())
q = deque()
mat = [list(str(stdin.readline())) for _ in range(N)]
for i in range(N):
for j in range(N):
if mat[i][j] == "K":
inX, inY = i+1, j+1
break
q.append((inX,inY,0))
flag = False
visited = [[0]*N for _ in range(N)]
while q:
ix,iy, moves = q.popleft()
visited[ix-1][iy-1] = 1
if (ix,iy) == (1,1):
print(moves)
flag = True
break
for d in dirs:
dx,dy = d
if 1 <= ix + dx < N+1 and 1 <= iy + dy < N+1 and mat[ix+dx-1][iy+dy-1] != "#" and visited[ix+dx-1][iy+dy-1] == 0:
q.append((ix+dx,iy+dy, moves+1))
if not flag:
print(-1)
All test data I've generated has given a correct answer, but I am recieving a time limit error on the site for some cases (and correct on all other). The input limits are N <= 100. I know that C++ is faster, but I was hoping to use this problem to learn a bit about why my solution times out but many Python solutions on a similar concept perform quick, and hopefully teach me something about Python nuts and bolts I didn't know. For example - I found this solution on someone's Github:
from collections import deque
n = int(input())
grid = [list(input()) for _ in range(n)]
for i in range(n):
for j in range(n):
if grid[i][j] == 'K':
k_i, k_j = i, j
break
visited = [[0 for x in range(n)] for y in range(n)]
# time complexity is number of configs (n^2) multiplied by
# work per configs (iterate through 8 deltas)
def k_jumps(i, j):
q = deque()
q.append((0, i, j))
visited[i][j] = 1
valid = False
while len(q) > 0:
d, i, j = q.popleft()
if (i,j) == (0,0):
print(d)
valid = True
break
deltas = {(-2, 1), (-2, -1), (1, 2), (1, -2), (-1, 2), (2, 1), (2, -1), (-1, -2)}
for delta in deltas:
di, dj = delta
if 0 <= i + di < n and 0 <= j + dj < n and visited[i+di][j+dj] == 0 and grid[i+di][j+dj] != '#':
visited[i+di][j+dj] = 1
q.append((d+1, i+di, j+dj))
if not valid:
print(-1)
k_jumps(k_i, k_j)
(Thanks to user Case Pleog).
Superficially the solutions are the same (BFS) - however the second solution runs in 0.07s, while my solutions times out at over 1s.
While there are some minor differences between our solutions, I can't see what explains such a large time discrepancy. I already made some changes to my code to see if that was the issue, e.g. the line:
if 1 <= ix + dx < N+1 and 1 <= iy + dy < N+1 and mat[ix+dx-1][iy+dy-1] != "#" and visited[ix+dx-1][iy+dy-1] == 0:
and indeed that speeded things up, as before I was using:
if min(point) > 0 and max(point) < N+1 and mat[point[0]+dx-1][point[1]+dy-1] != "#" and visited[point[0]+dx-1][point[1]+dy-1] == 0:
where point was a tuple consisting of ix,iy. Any advice would be appreciated - I would like to know more about what's causing the time difference as its > 10x difference. This is just a hobby, so some of my code may have amateur quirks/inefficiencies which hopefully I can explain. Thanks!

Do visited[ix+dx-1][iy+dy-1] = 1 right away when you do q.append((ix+dx,iy+dy, moves+1)), otherwise you might put that same coordinate into the queue multiple times.

Function to sum the first and last item in an array until there's only two items left in the array

I've recently failed a coding challenge and I'd like to know the solution to the problem.
Here's what was asked of me (from memory as I do not have access to the question anymore):
Create a function that takes an array of integers and sums the first and last item, then the second and second to last, and so on. You must do this until there's only two items left in the returning array.
Example:
Input: [1, 4, 2, 3, 6, 10]
Returns: [16, 10]
Because:
[(1 + 10), (4 + 6), (2 + 3)] =
[11, 10, 5] =>
[(11 + 5), 10] =
[16, 10]
Example 2:
Input: [-1, 3, 2, -2, 11, 7, -9]
Returns: [-12, 23]
Because:
[(-1 + (-9)), (3 + 7), (2 + 11), -2] =
[-10, 10, 13, -2] =>
[(-10 + (-2)), (10 + 13)] =
[-12, 23]
Constraints:
-101 < arr[i] < 101
You may assume that each arr[i] is unique
Here's my attempt at solving this problem:
def sumFirstAndLast (array):
if len(array) == 2:
return array
else:
left = 0
right = len(array) - 1
result = []
while left < right:
result.append(array[left] + array[right])
left += 1
right -= 1
array = result
result = sumFirstAndLast(array)
return result
But it's throwing an error:
Traceback (most recent call last):
File "<string>", line 22, in <module>
File "<string>", line 11, in sumFirstAndLast
IndexError: list index out of range
>
Can someone provide me with a solution to this problem? Why am I getting this error? Is my logic to approaching this problem incorrect?

array = result and result = sumFirstAndLast(array) should be put outside the while loop
left could be equal to right if there are odd elements. After while left < right we need an additional condition check if left == right.
The code can be as follows:
def sumFirstAndLast (array):
if len(array) == 2:
return array
else:
left = 0
right = len(array) - 1
result = []
while left < right:
result.append(array[left] + array[right])
left += 1
right -= 1
if left == right:
result.append(array[left])
return sumFirstAndLast(result)

Some issues:
The indentation of two statements is wrong. The recursive call should not be made within the loop, but once the loop has been completed.
When the size of the array is odd, then the middle element is currently ignored and drops out. It should still be added to the result.
Not a huge problem if it is guaranteed, but you should better guard your function for the case the array has fewer than 2 values.
So without changing anything else to your code, it would become:
def sumFirstAndLast (array):
if len(array) <= 2:
return array
else:
left = 0
right = len(array) - 1
result = []
while left < right:
result.append(array[left] + array[right])
left += 1
right -= 1
if left == right: # There is a middle element. It needs to be retained
result.append(array[left])
array = result
result = sumFirstAndLast(array) # recursion outside of the loop
return result
In a more compact, iterative version, it could be:
def sumFirstAndLast (array):
while len(array) > 2:
array = [
array[i] + array[-i-1] for i in range(len(array) // 2)
] + ([array[len(array) // 2]] if len(array) % 2 else [])
return array

I think recursion makes it harder to reason about what's going on (and thus, it's harder to reason about what's going wrong).
I would instead solve this iteratively, repeatedly performing the operation until you have a list with only two elements. The code ends up being pretty similar to the recursive version, but in my opinion it's much more readable (and thus much easier to debug):
def sumFirstAndLast (array):
if len(array) == 2:
return array
result = array.copy()
while len(result) > 2:
left = 0
right = len(result) - 1
new_result = []
while left < right:
new_result.append(result[left] + result[right])
left += 1
right -= 1
if left == right:
new_result.append(result[left])
result = new_result
return result

I'm gonna try to give you some tips as well.
Basically when you are solving a problem, you should break it into smaller problems. Start for the input. As you are working with operations 2 to 2, you should think, well what will happen if the input is minor than 2, that is the minimun required number for you to operate. Second, what happens if my input is odd or even? is it the same? In this case not. Lastly, try to approach a single loop and see the output before looping the whole thing and get a messy output.
With that in mind, you will solve the problem if the provided input doesn't have a minimun required size for operation:
if len(array) < 2:
return "Array must be of size bigger than 2"
Solve a single loop for odd cases:
for i in range(int(len(array) / 2) + len(array) % 2):
if i < len(array) - i - 1:
new_array.append(array[i] + array[len(array) - i - 1])
Solve a single loop for even cases:
for i in range(int(len(array) / 2) + len(array) % 2):
if i < len(array) - i - 1:
new_array.append(array[i] + array[len(array) - i - 1])
if len(array) % 2:
new_array.append(array[int(len(array) / 2)])
Now you have a single case loop, do the complete loop:
while len(array) > 2:
for i in range(int(len(array) / 2) + len(array) % 2):
if i < len(array) - i - 1:
new_array.append(array[i] + array[len(array) - i - 1])
if len(array) % 2:
new_array.append(array[int(len(array) / 2)])
array = new_array
new_array = []
Join all of that together and you have your Solution:
def solve(array):
if len(array) < 2:
return "Array must be of size bigger than 2"
new_array = []
while len(array) > 2:
for i in range(int(len(array) / 2) + len(array) % 2):
if i < len(array) - i - 1:
new_array.append(array[i] + array[len(array) - i - 1])
if len(array) % 2:
new_array.append(array[int(len(array) / 2)])
array = new_array
new_array = []
return array
At the end you can search for a compact solution, because now you at least solved the question in the interview, and got the logic.
def solve(array):
if len(array) < 2:
return "Array must be of size bigger than 2"
while len(array) > 2:
remainder = (array[int(len(array) / 2)]) if len(array) % 2 else False
array = [(array[i] + array[len(array) - i - 1]) for i in range(int(len(array) / 2))]
array.append(remainder) if remainder is not False else None
return array

Is there a python function that returns the first positive int that does not occur in list?

I'm tryin to design a function that, given an array A of N integers, returns the smallest positive integer (greater than 0) that does not occur in A.
This code works fine yet has a high order of complexity, is there another solution that reduces the order of complexity?
Note: The 10000000 number is the range of integers in array A, I tried the sort function but does it reduces the complexity?
def solution(A):
for i in range(10000000):
if(A.count(i)) <= 0:
return(i)

The following is O(n logn):
a = [2, 1, 10, 3, 2, 15]
a.sort()
if a[0] > 1:
print(1)
else:
for i in range(1, len(a)):
if a[i] > a[i - 1] + 1:
print(a[i - 1] + 1)
break
If you don't like the special handling of 1, you could just append zero to the array and have the same logic handle both cases:
a = sorted(a + [0])
for i in range(1, len(a)):
if a[i] > a[i - 1] + 1:
print(a[i - 1] + 1)
break
Caveats (both trivial to fix and both left as an exercise for the reader):
Neither version handles empty input.
The code assumes there no negative numbers in the input.

O(n) time and O(n) space:
def solution(A):
count = [0] * len(A)
for x in A:
if 0 < x <= len(A):
count[x-1] = 1 # count[0] is to count 1
for i in range(len(count)):
if count[i] == 0:
return i+1
return len(A)+1 # only if A = [1, 2, ..., len(A)]

This should be O(n). Utilizes a temporary set to speed things along.
a = [2, 1, 10, 3, 2, 15]
#use a set of only the positive numbers for lookup
temp_set = set()
for i in a:
if i > 0:
temp_set.add(i)
#iterate from 1 upto length of set +1 (to ensure edge case is handled)
for i in range(1, len(temp_set) + 2):
if i not in temp_set:
print(i)
break

My proposal is a recursive function inspired by quicksort.
Each step divides the input sequence into two sublists (lt = less than pivot; ge = greater or equal than pivot) and decides, which of the sublists is to be processed in the next step. Note that there is no sorting.
The idea is that a set of integers such that lo <= n < hi contains "gaps" only if it has less than (hi - lo) elements.
The input sequence must not contain dups. A set can be passed directly.
# all cseq items > 0 assumed, no duplicates!
def find(cseq, cmin=1):
# cmin = possible minimum not ruled out yet
size = len(cseq)
if size <= 1:
return cmin+1 if cmin in cseq else cmin
lt = []
ge = []
pivot = cmin + size // 2
for n in cseq:
(lt if n < pivot else ge).append(n)
return find(lt, cmin) if cmin + len(lt) < pivot else find(ge, pivot)
test = set(range(1,100))
print(find(test)) # 100
test.remove(42)
print(find(test)) # 42
test.remove(1)
print(find(test)) # 1

Inspired by various solutions and comments above, about 20%-50% faster in my (simplistic) tests than the fastest of them (though I'm sure it could be made faster), and handling all the corner cases mentioned (non-positive numbers, duplicates, and empty list):
import numpy
def firstNotPresent(l):
positive = numpy.fromiter(set(l), dtype=int) # deduplicate
positive = positive[positive > 0] # only keep positive numbers
positive.sort()
top = positive.size + 1
if top == 1: # empty list
return 1
sequence = numpy.arange(1, top)
try:
return numpy.where(sequence < positive)[0][0]
except IndexError: # no numbers are missing, top is next
return top
The idea is: if you enumerate the positive, deduplicated, sorted list starting from one, the first time the index is less than the list value, the index value is missing from the list, and hence is the lowest positive number missing from the list.
This and the other solutions I tested against (those from adrtam, Paritosh Singh, and VPfB) all appear to be roughly O(n), as expected. (It is, I think, fairly obvious that this is a lower bound, since every element in the list must be examined to find the answer.) Edit: looking at this again, of course the big-O for this approach is at least O(n log(n)), because of the sort. It's just that the sort is so fast comparitively speaking that it looked linear overall.

Optimizing solution to Three Sum

I am trying to solve the 3 Sum problem stated as:
Given an array S of n integers, are there elements a, b, c in S such that a + b + c = 0? Find all unique triplets in the array which gives the sum of zero.
Note: The solution set must not contain duplicate triplets.
Here is my solution to this problem:
def threeSum(nums):
"""
:type nums: List[int]
:rtype: List[List[int]]
"""
nums.sort()
n = len(nums)
solutions = []
for i, num in enumerate(nums):
if i > n - 3:
break
left, right = i+1, n-1
while left < right:
s = num + nums[left] + nums[right] # check if current sum is 0
if s == 0:
new_solution = [num, nums[left], nums[right]]
# add to the solution set only if this triplet is unique
if new_solution not in solutions:
solutions.append(new_solution)
right -= 1
left += 1
elif s > 0:
right -= 1
else:
left += 1
return solutions
This solution works fine with a time complexity of O(n**2 + k) and space complexity of O(k) where n is the size of the input array and k is the number of solutions.
While running this code on LeetCode, I am getting TimeOut error for arrays of large size. I would like to know how can I further optimize my code to pass the judge.
P.S: I have read the discussion in this related question. This did not help me resolve the issue.

A couple of improvements you can make to your algorithm:
1) Use sets instead of a list for your solution. Using a set will insure that you don't have any duplicate and you don't have to do a if new_solution not in solutions: check.
2) Add an edge case check for an all zero list. Not too much overhead but saves a HUGE amount of time for some cases.
3) Change enumerate to a second while. It is a little faster. Weirdly enough I am getting better performance in the test with a while loop then a n_max = n -2; for i in range(0, n_max): Reading this question and answer for xrange or range should be faster.
NOTE: If I run the test 5 times I won't get the same time for any of them. All my test are +-100 ms. So take some of the small optimizations with a grain of salt. They might NOT really be faster for all python programs. They might only be faster for the exact hardware/software config the tests are running on.
ALSO: If you remove all the comments from the code it is a LOT faster HAHAHAH like 300ms faster. Just a funny side effect of however the tests are being run.
I have put in the O() notation into all of the parts of your code that take a lot of time.
def threeSum(nums):
"""
:type nums: List[int]
:rtype: List[List[int]]
"""
# timsort: O(nlogn)
nums.sort()
# Stored val: Really fast
n = len(nums)
# Memory alloc: Fast
solutions = []
# O(n) for enumerate
for i, num in enumerate(nums):
if i > n - 3:
break
left, right = i+1, n-1
# O(1/2k) where k is n-i? Not 100% sure about this one
while left < right:
s = num + nums[left] + nums[right] # check if current sum is 0
if s == 0:
new_solution = [num, nums[left], nums[right]]
# add to the solution set only if this triplet is unique
# O(n) for not in
if new_solution not in solutions:
solutions.append(new_solution)
right -= 1
left += 1
elif s > 0:
right -= 1
else:
left += 1
return solutions
Here is some code that won't time out and is fast(ish). It also hints at a way to make the algorithm WAY faster (Use sets more ;) )
class Solution(object):
def threeSum(self, nums):
"""
:type nums: List[int]
:rtype: List[List[int]]
"""
# timsort: O(nlogn)
nums.sort()
# Stored val: Really fast
n = len(nums)
# Hash table
solutions = set()
# O(n): hash tables are really fast :)
unique_set = set(nums)
# covers a lot of edge cases with 2 memory lookups and 1 hash so it's worth the time
if len(unique_set) == 1 and 0 in unique_set and len(nums) > 2:
return [[0, 0, 0]]
# O(n) but a little faster than enumerate.
i = 0
while i < n - 2:
num = nums[i]
left = i + 1
right = n - 1
# O(1/2k) where k is n-i? Not 100% sure about this one
while left < right:
# I think its worth the memory alloc for the vars to not have to hit the list index twice. Not sure
# how much faster it really is. Might save two lookups per cycle.
left_num = nums[left]
right_num = nums[right]
s = num + left_num + right_num # check if current sum is 0
if s == 0:
# add to the solution set only if this triplet is unique
# Hash lookup
solutions.add(tuple([right_num, num, left_num]))
right -= 1
left += 1
elif s > 0:
right -= 1
else:
left += 1
i += 1
return list(solutions)

I benchamrked the faster code provided by PeterH but I found a faster solution, and the code is simpler too.
class Solution(object):
def threeSum(self, nums):
res = []
nums.sort()
length = len(nums)
for i in xrange(length-2): #[8]
if nums[i]>0: break #[7]
if i>0 and nums[i]==nums[i-1]: continue #[1]
l, r = i+1, length-1 #[2]
while l<r:
total = nums[i]+nums[l]+nums[r]
if total<0: #[3]
l+=1
elif total>0: #[4]
r-=1
else: #[5]
res.append([nums[i], nums[l], nums[r]])
while l<r and nums[l]==nums[l+1]: #[6]
l+=1
while l<r and nums[r]==nums[r-1]: #[6]
r-=1
l+=1
r-=1
return res
https://leetcode.com/problems/3sum/discuss/232712/Best-Python-Solution-(Explained)

Implementing Median of Median Selection Algorithm in Python

Okay. I give up. I've been trying to implement the median of medians algorithm but I am continually given the wrong result. I know there is a lot of code below, but I can't find my error, and each chunk of code has a fairly process design. Quicksort is what I use to sort the medians I get from the median of medians pivot selection. Should be a straightforward quicksort implementation. getMean simply returns the mean of a given list.
getPivot is what I use to select the pivot. It runs through the list, taking 5 integers at a time, finding the mean of those integers, placing that mean into a list c, then finding the median of c. That is the pivot I use for the dSelect algorithm.
The dSelect algorithm is simple in theory. The base case returns an item when the list is 1 item long. Otherwise, much like in quickSort, I iterate over the list. If the number I am currently on, j, is less than the pivot, I move it to the left of the list, i, and increment i. If it is larger, I move it to the right of the list, i + 1, and do not increment i. After this loops through the entire list, I should have the pivot in its proper index, and print statements indicate that I do. At this point, I recurse to the left or the right depending on whether the pivot is greater than or less than the position I am trying to find.
I am not sure what other print statements to test at this point, so I'm turning to anyone dedicated enough to take a stab at this code. I know there are related topics, I know I could do more print statements, but believe me, I've tried. What should be a simple algo has got me quite stumped.
def quickSort(m, left, right):
if right - left <= 1:
return m
pivot = m[left]
i = left + 1
j = left + 1
for j in range(j, right):
if m[j] < pivot:
m[j], m[i] = m[i], m[j]
i += 1
elif m[j] == pivot:
m[j], m[i] = m[i], m[j]
print m
m[left], m[i-1] = m[i-1], m[left]
m = quickSort(m, left, i-1)
m = quickSort(m, i, right)
print m
return m
def getMedian(list):
length = len(list)
if length <= 1:
return list[0]
elif length % 2 == 0:
i = length/2
return list[i]
else:
i = (length + 1)/2
return list[i]
def getPivot(m):
c = []
i = 0
while i <= len(m) - 1:
tempList = []
j = 0
while j < 5 and i <= len(m) - 1:
tempList.append(m[i])
i = i + 1
j = j + 1
tempList = quickSort(tempList, 0, len(tempList) - 1)
c.append(getMedian(tempList))
c = quickSort(c, 0, len(c) - 1)
medianOfMedians = getMedian(c)
return medianOfMedians
def dSelect(m, position):
pivot = getPivot(m)
i = 0
j = 0
if len(m) <= 1:
return m[0]
for j in range(0, len(m)):
if m[j] < pivot:
m[j], m[i] = m[i], m[j]
i += 1
elif m[j] == pivot:
m[j], m[i] = m[i], m[j]
print "i: " + str(i)
print "j: " + str(j)
print "index of pivot: " + str(m.index(pivot))
print "pivot: " + str(pivot) + " list: " + str(m)
if m.index(pivot) == position:
return pivot
elif m.index(pivot) > position:
return dSelect(m[0:i], position)
else:
return dSelect(m[i:], position - i)

The biggest issue is with this line here:
i = (length + 1)/2
if list = [1, 2, 3, 4, 5, 6, 7] the answer should be 4 which is list[3]. Your version looks like the following:
i = (7 + 1) / 2
and so i is equal to 4 instead of 3. Similar problem with the even length list part although that shouldn't be as big of an issue.