We Keep Coding

Understanding speed in Python through a competition problem exceeding time limits

I recently have been doing some competition problems. In particular this one:
https://open.kattis.com/problems/knightjump
An abridged statement of the problem:
You are given a two dimensional (square) chess board of size N (1-based indexing). Some of the cells on this board are ‘.’ denoting an empty cell. Some of the cells on this board are ‘#’ denoting a blocked cell, which you are not allowed to visit. Exactly one of the cells on this board is ‘K’ denoting the initial position of the knight. Determine the minimum number of steps required for the Knight to reach cell (1,1) while avoiding cells with ‘#’ in the path.
I know this is a simple BFS problem and have written the following code to solve it:
from collections import deque
from sys import stdin
dirs = ((2,1), (2,-1), (-2,1), (-2,-1), (1,2), (1,-2), (-1,2), (-1,-2))
N = int(stdin.readline())
q = deque()
mat = [list(str(stdin.readline())) for _ in range(N)]
for i in range(N):
for j in range(N):
if mat[i][j] == "K":
inX, inY = i+1, j+1
break
q.append((inX,inY,0))
flag = False
visited = [[0]*N for _ in range(N)]
while q:
ix,iy, moves = q.popleft()
visited[ix-1][iy-1] = 1
if (ix,iy) == (1,1):
print(moves)
flag = True
break
for d in dirs:
dx,dy = d
if 1 <= ix + dx < N+1 and 1 <= iy + dy < N+1 and mat[ix+dx-1][iy+dy-1] != "#" and visited[ix+dx-1][iy+dy-1] == 0:
q.append((ix+dx,iy+dy, moves+1))
if not flag:
print(-1)
All test data I've generated has given a correct answer, but I am recieving a time limit error on the site for some cases (and correct on all other). The input limits are N <= 100. I know that C++ is faster, but I was hoping to use this problem to learn a bit about why my solution times out but many Python solutions on a similar concept perform quick, and hopefully teach me something about Python nuts and bolts I didn't know. For example - I found this solution on someone's Github:
from collections import deque
n = int(input())
grid = [list(input()) for _ in range(n)]
for i in range(n):
for j in range(n):
if grid[i][j] == 'K':
k_i, k_j = i, j
break
visited = [[0 for x in range(n)] for y in range(n)]
# time complexity is number of configs (n^2) multiplied by
# work per configs (iterate through 8 deltas)
def k_jumps(i, j):
q = deque()
q.append((0, i, j))
visited[i][j] = 1
valid = False
while len(q) > 0:
d, i, j = q.popleft()
if (i,j) == (0,0):
print(d)
valid = True
break
deltas = {(-2, 1), (-2, -1), (1, 2), (1, -2), (-1, 2), (2, 1), (2, -1), (-1, -2)}
for delta in deltas:
di, dj = delta
if 0 <= i + di < n and 0 <= j + dj < n and visited[i+di][j+dj] == 0 and grid[i+di][j+dj] != '#':
visited[i+di][j+dj] = 1
q.append((d+1, i+di, j+dj))
if not valid:
print(-1)
k_jumps(k_i, k_j)
(Thanks to user Case Pleog).
Superficially the solutions are the same (BFS) - however the second solution runs in 0.07s, while my solutions times out at over 1s.
While there are some minor differences between our solutions, I can't see what explains such a large time discrepancy. I already made some changes to my code to see if that was the issue, e.g. the line:
if 1 <= ix + dx < N+1 and 1 <= iy + dy < N+1 and mat[ix+dx-1][iy+dy-1] != "#" and visited[ix+dx-1][iy+dy-1] == 0:
and indeed that speeded things up, as before I was using:
if min(point) > 0 and max(point) < N+1 and mat[point[0]+dx-1][point[1]+dy-1] != "#" and visited[point[0]+dx-1][point[1]+dy-1] == 0:
where point was a tuple consisting of ix,iy. Any advice would be appreciated - I would like to know more about what's causing the time difference as its > 10x difference. This is just a hobby, so some of my code may have amateur quirks/inefficiencies which hopefully I can explain. Thanks!

Do visited[ix+dx-1][iy+dy-1] = 1 right away when you do q.append((ix+dx,iy+dy, moves+1)), otherwise you might put that same coordinate into the queue multiple times.

Selecting values based on threshold using Python

The present code selects minimum values by scanning the adjoining elements in the same and the succeeding row. However, I want the code to select all the values if they are less than the threshold value. For example, in row 2, I want the code to pick both 0.86 and 0.88 since both are less than 0.9, and not merely minimum amongst 0.86,0.88. Basically, the code should pick up the minimum value if all the adjoining elements are greater than the threshold. If that's not the case, it should pick all the values less than the threshold.
import numpy as np
import numba as nb
Pe = np.random.rand(5,5)
def minValues(arr):
n, m = arr.shape
assert n >= 1 and m >= 2
res = []
i, j = 0, np.argmin(arr[0,:])
res.append((i, j))
iPrev = jPrev = -1
while iPrev < n-1:
cases = [(i, j-1), (i, j+1), (i+1, j)]
minVal = np.inf
iMin = jMin = -1
# Find the best candidate (smallest value)
for (i2, j2) in cases:
if i2 == iPrev and j2 == jPrev: # No cycles
continue
if i2 < 0 or i2 >= n or j2 < 0 or j2 >= m: # No out-of-bounds
continue
if arr[i2, j2] < minVal:
iMin, jMin = i2, j2
minVal = arr[i2, j2]
assert not np.isinf(minVal)
# Store it and update the values
res.append((iMin, jMin))
iPrev, jPrev = i, j
i, j = iMin, jMin
return np.array(res)
T=minValues(Pe)
Path=Pe[T.T[0], T.T[1]]
Current output:
Desired output:

Try this:
def minValues(arr):
n, m = arr.shape
assert n >= 1 and m >= 2
res = []
i, j = 0, np.argmin(arr[0,:])
print(f"Values i, j are: {i}, {j}")
res.append((i, j))
iPrev = jPrev = -1
while iPrev < n-1:
lowerVals = []
print(f"Values iPrev, jPrev are: {iPrev}, {jPrev}")
cases = [(i, j-1), (i, j+1), (i+1, j)]
print(f"Posible cases are: {cases}")
minVal = np.inf
print(f"MinVal is: {minVal}")
iMin = jMin = -1
# Find the best candidate (smallest value)
for (i2, j2) in cases:
if i2 == iPrev and j2 == jPrev: # No cycles
continue
if i2 < 0 or i2 >= n or j2 < 0 or j2 >= m: # No out-of-bounds
continue
if arr[i2, j2] < arr[i, j]:
lowerVals.append((i2, j2))
if arr[i2, j2] < minVal:
iMin, jMin = i2, j2
minVal = arr[i2, j2]
if not lowerVals: lowerVals.append((iMin, jMin))
print(f"Values iMin, jMin are: {iMin}, {jMin}")
print(f"MinVal is: {minVal}")
print(f"Lower values are: {lowerVals}")
assert not np.isinf(minVal)
# Store it and update the values
res += lowerVals
print(f"Final res after current iteration: {res}\n")
iPrev, jPrev = i, j
i, j = iMin, jMin
return np.array(res)
I used the prints for debugging, but I just check in each iteration all the values which are lower than the current value and add them to path at the end of the iteration.
Edit: introducing the extra line of my comment you get the above code, this should work.

This is an interesting problem - looking at hydrology flow/downhill direction?
The issue with the description is that you don't specify which of the neighboring cells to iterate to next, meaning, if you have three neighboring cells that are lower, they all get added to the output array but which is selected? Also, when looking back through the output array how do you know which ones got added with respect to a single neighboring cell? I went about this by selecting the lowest value of the neighboring cells with less than the source cell's value (biggest dropoff), then continued the search from that cell and so on. For each iteration, there is a new nested list of the cell positions for the neighboring cells that are lower. This could readily be updated any number of ways, to include exporting an additional list showing which cells were used for search (essentially the path taken, while there might be highlighted cells (lower) that were not visited per se.
import numpy as np
Pe = np.random.rand(5,5)
def nearestMin(arr, output_arr=[[0,0]], n=0, m=0):
if m==arr.shape[1]:
return output_arr
lower_vals = np.argwhere(arr < arr[n,m])
if lower_vals.size == 0: #No values anywhere less than current cell
return output_arr
#Get offset from current point, as tuple of x and y distance
smaller_adjacents = np.where(Pe < Pe[n, m])
if smaller_adjacents[0].size == 0: #No lower points anywhere
return output_arr
ind = np.where((abs(n - smaller_adjacents[0]) <= 1 ) & \
(abs(m - smaller_adjacents[1]) <= 1))[0]
if ind.size == 0: #No lower points neighboring
return output_arr
xs, ys = smaller_adjacents[0][ind], smaller_adjacents[1][ind]
lower_neighbors = np.vstack((xs, ys)).T.tolist()
output_arr.append(lower_neighbors) #Unbracket to have final output array be flat (2d)
n_depth = [Pe[x,y] for x,y in lower_neighbors] - Pe[n,m]
biggest_drop = np.where(n_depth==n_depth.min())[0][0]
lowest_neighbor = lower_neighbors[biggest_drop]
return nearestMin(arr, output_arr, n=lowest_neighbor[0], m=lowest_neighbor[1])
nearestMin(Pe)
If you want the path, the same code can be modified as shown below to return that as a second array. In this example, the values for the path are listed. Each value corresponds to the "list of lists" item in the first returned array of neighboring lower values. Again, not sure exactly what you're going for.
import numpy as np
Pe = np.random.rand(5,5)
def nearestMin(arr, output_arr=[[0,0]], path=[], n=0, m=0):
#Initialize path variable properly
if output_arr == [[0,0]]:
path.append(arr[0,0])
if m==arr.shape[1]:
return (output_arr, path)
lower_vals = np.argwhere(arr < arr[n,m])
if lower_vals.size == 0: #No values anywhere less than current cell
return (output_arr, path)
#Get offset from current point, as tuple of x and y distance
smaller_adjacents = np.where(Pe < Pe[n, m])
if smaller_adjacents[0].size == 0: #No lower points anywhere
return (output_arr, path)
ind = np.where((abs(n - smaller_adjacents[0]) <= 1 ) & \
(abs(m - smaller_adjacents[1]) <= 1))[0]
if ind.size == 0: #No lower points neighboring
return (output_arr, path)
xs, ys = smaller_adjacents[0][ind], smaller_adjacents[1][ind]
lower_neighbors = np.vstack((xs, ys)).T.tolist()
output_arr.append(lower_neighbors) #Unbracket to have final output array be flat (2d)
n_depth = [Pe[x,y] for x,y in lower_neighbors] - Pe[n,m]
biggest_drop = np.where(n_depth==n_depth.min())[0][0]
lowest_neighbor = lower_neighbors[biggest_drop]
path.append(arr[lowest_neighbor[0], lowest_neighbor[1]])
return nearestMin(arr, output_arr, path, n=lowest_neighbor[0], m=lowest_neighbor[1])
nearestMin(Pe)

Advanced Hailstone - Nested loop / combination

Generating a hailstone sequence that follows the pattern below:
if x is even -> x/2
if x is odd -> [a]x+[b]
where a and b are integer values {0,...,10}, allowing for 121 possible combinations of a and b. I need to list whether the sequence converges for all 1000 x values
I am using python to solve the problem, I'm a beginner at coding with python but am a quick learner and need guidance on how to resolve
`for n in range(1,1001):
for i in a:
for j in b:
while j != 1 & i != 1:
print ("a:", i, "b:", j)
if j % 2 == 0:
j = j / 2
length = length + 1
else:
n = (n * j) + i
if n == 1:
print (n)
'
the above works in that it runs but it doesn't do what I want. It just keeps looping integer one and wont move past it

I see following problems in your code:
range usage, range does not accept list as argument, you should use for i in a: when a is list, rework all your fors accordingly.
broken while syntax: you have too many :s in your while, should be one : (at end)
possibly broken indentation: note that if is not aligned with corresponding else, keep in mind that in Python language indentation is crucial
j = j / b will produce error - you can't divide int by list
you never define n, thus n = (n * a) + 1 and all other lines with n, will produce errors
Keep in mind that my list of problems might be incomplete

'a = [0,1,2,3,4,5,6,7,8,9,10]
b = [0,1,2,3,4,5,6,7,8,9,10]
for n in range(1,1001):
for i in a:
for j in b:
while j != 1 & i != 1:
print ("a:", i, "b:", j)
if j % 2 == 0:
j = j / 2
length = length + 1
else:
n = (n * j) + i
if n == 1:
print (n)'
updates...I want to still be able to understand when it converges and it won't move to the next value

Optimal Search Tree Using Python - Code Analysis

First of all, sorry about the naive question. But I couldn't find help elsewhere
I'm trying to create an Optimal Search Tree using Dynamic Programing in Python that receives two lists (a set of keys and a set of frequencies) and returns two answers:
1 - The smallest path cost.
2 - The generated tree for that smallest cost.
I basically need to create a tree organized by the most accessed items on top (most accessed item it's the root), and return the smallest path cost from that tree, by using the Dynamic Programming solution.
I've the following implemented code using Python:
def optimalSearchTree(keys, freq, n):
#Create an auxiliary 2D matrix to store results of subproblems
cost = [[0 for x in xrange(n)] for y in xrange(n)]
#For a single key, cost is equal to frequency of the key
#for i in xrange (0,n):
# cost[i][i] = freq[i]
# Now we need to consider chains of length 2, 3, ... .
# L is chain length.
for L in xrange (2,n):
for i in xrange(0,n-L+1):
j = i+L-1
cost[i][j] = sys.maxint
for r in xrange (i,j):
if (r > i):
c = cost[i][r-1] + sum(freq, i, j)
elif (r < j):
c = cost[r+1][j] + sum(freq, i, j)
elif (c < cost[i][j]):
cost[i][j] = c
return cost[0][n-1]
def sum(freq, i, j):
s = 0
k = i
for k in xrange (k,j):
s += freq[k]
return s
keys = [10,12,20]
freq = [34,8,50]
n=sys.getsizeof(keys)/sys.getsizeof(keys[0])
print(optimalSearchTree(keys, freq, n))
I'm trying to output the answer 1. The smallest cost for that tree should be 142 (the value stored on the Matrix Position [0][n-1], according to the Dynamic Programming solution). But unfortunately it's returning 0. I couldn't find any issues in that code. What's going wrong?

You have several very questionable statements in your code, definitely inspired by C/Java programming practices. For instance,
keys = [10,12,20]
freq = [34,8,50]
n=sys.getsizeof(keys)/sys.getsizeof(keys[0])
I think you think you calculate the number of items in the list. However, n is not 3:
sys.getsizeof(keys)/sys.getsizeof(keys[0])
3.142857142857143
What you need is this:
n = len(keys)
One more find: elif (r < j) is always True, because r is in the range between i (inclusive) and j (exclusive). The elif (c < cost[i][j]) condition is never checked. The matrix c is never updated in the loop - that's why you always end up with a 0.
Another suggestion: do not overwrite the built-in function sum(). Your namesake function calculates the sum of all items in a slice of a list:
sum(freq[i:j])

import sys
def optimalSearchTree(keys, freq):
#Create an auxiliary 2D matrix to store results of subproblems
n = len(keys)
cost = [[0 for x in range(n)] for y in range(n)]
storeRoot = [[0 for i in range(n)] for i in range(n)]
#For a single key, cost is equal to frequency of the key
for i in range (0,n):
cost[i][i] = freq[i]
# Now we need to consider chains of length 2, 3, ... .
# L is chain length.
for L in range (2,n+1):
for i in range(0,n-L+1):
j = i + L - 1
cost[i][j] = sys.maxsize
for r in range (i,j+1):
c = (cost[i][r-1] if r > i else 0)
c += (cost[r+1][j] if r < j else 0)
c += sum(freq[i:j+1])
if (c < cost[i][j]):
cost[i][j] = c
storeRoot[i][j] = r
return cost[0][n-1], storeRoot
if __name__ == "__main__" :
keys = [10,12,20]
freq = [34,8,50]
print(optimalSearchTree(keys, freq))

Splitting math sums with python

This is a simple question that has been bothering me for a while now.
I am attempting to rewrite my code to be parallel, and in the process I need to split up a sum to be done on multiple nodes and then add those small sums together. The piece that I am working with is this:
def pia(n, i):
k = 0
lsum = 0
while k < n:
p = (n-k)
ld = (8.0*k+i)
ln = pow(16.0, p, ld)
lsum += (ln/ld)
k += 1
return lsum
where n is the limit and i is an integer. Does anyone have some hints on how to split this up and get the same result in the end?
Edit: For those asking, I'm not using pow() but a custom version to do it efficiently with floating point:
def ssp(b, n, m):
ssp = 1
while n>0:
if n % 2 == 1:
ssp = (b*ssp) % m
b = (b**2) % m
n = n // 2
return ssp

Since the only variable that's used from one pass to the next is k, and k just increments by one each time, it's easy to split the calculation.
If you also pass k into pia, then you'll have both a definable starting and ending points, and you can split this up into as many pieces as you want, and at the end, add all the results together. So something like:
# instead of pia(20000, i), use pia(n, i, k) and run
result = pia(20000, i, 10000) + pia(10000, i, 0)
Also, since n is used to both set the limits and in the calculation directly, these two uses need to be split.
from math import pow
def pia(nlimit, ncalc, i, k):
lsum = 0
while k < nlimit:
p = ncalc-k
ld = 8.0*k+i
ln = ssp(16., p, ld)
lsum += ln/ld
k += 1
return lsum
if __name__=="__main__":
i, ncalc = 5, 10
print pia(10, ncalc, i, 0)
print pia(5, ncalc, i, 0) + pia(10, ncalc, i, 5)

Looks like I found a way. What I did was in the sum I had each node calculate a portion (ex. node one calculates k=1, node 2 k=2, node 3 k=3, node 4 k=4, node 1 k=5...) and then gathered them up and added them.