Advice on writing a solver for Vexed levels

Advice on writing a solver for Vexed levels - python

Vexed is a popular puzzle game, with many versions available (some of them GPL free software). It is very suitable for small screen devices; versions are available for Android, iOS, etc. I discovered it on the PalmOS platform.
Just for fun, I'd like to write a solver that will solve Vexed levels.
Vexed is a block-sliding puzzle game. Here are the rules in a nutshell:
0) Each level is a grid of squares, bounded by an impassible border. In any level there will be some solid squares, which are impassible. There are some number of blocks of various colors; these could be resting on the bottom border, resting on solid squares, or resting on other blocks (of a different color). Most levels are 8x8 or smaller.
1) The only action you can take is to slide a block to the left or to the right. Each square traveled by a block counts as one move.
2) There is gravity. If, after you slide a block, it is no longer resting on a solid square or another block, it will fall until it comes to rest on another block, a solid square, or the bottom border. Note that you cannot ever lift it up again.
3) Any time two or more blocks of the same color touch, they disappear. Note that chains are possible: if a supporting block disappears, blocks that rested upon it will fall, which could lead to more blocks of the same color touching and thus disappearing.
4) The goal is to make all blocks disappear in the minimum number of moves. Each level has a "par score" which tells you the minimum number of moves. (In the original PalmOS game, the "par score" wasn't necessarily the minimum, but in the Android version I play these days it is the minimum.)
Here is the SourceForge project with the source for the PalmOS version of the game:
http://sourceforge.net/projects/vexed/
I'm an experienced software developer, but I haven't done really any work in AI sort of stuff (pathfinding, problem-solving, etc.) So I'm looking for advice to get me pointed in the right direction.
At the moment, I can see two basic strategies for me to pursue:
0) Just write a brute-force solver, probably in C for the speed, that cranks through every possible solution for every game and returns a list of all solutions, best one first. Would this be a reasonable approach, or would the total number of possible moves make this too slow? I don't think any levels exist larger than 10x10.
1) Learn some AI-ish algorithms, and apply them in a clever way to solve the problem, probably using Python.
Note that the source for PalmOS Vexed includes a solver. According to the author, "The solver uses A* with pruning heuristics to find solutions."
http://www.scottlu.com/Content/Vexed.html
So, one strategy I could pursue would be to study the A* algorithm and then study the C++ code for the existing solver and try to learn from that.
I'm going to tag this with Python and C tags, but if you think I should be using something else, make your sales pitch and I'll consider it!
Here is ASCII art of a level from "Variety 25 Pack"; level 48, "Dark Lord". I am able to solve most levels but this one has, well, vexed me. Par score for this level is 25 moves, but I have not yet solved it at all!
__________
|## g####|
|## # b##|
|## # p##|
|#g ###|
|bp ###|
|p# p g |
==========
In this picture, the borders are underscores, vertical bars, and equals characters. Filled-in squares are '#'. Open spaces are space characters. Colored blocks are 'g' (green), 'b' (blue) and 'p' (purple).
By the way, I'll likely make the input file format to the solver be ASCII art of the levels, just like this but without the fussy line border characters.
Thanks for any advice!
EDIT:
I have accepted an answer. Thank you to the people who gave me answers.
This is a semi-brute-force solver. It isn't using A* but it is cutting short unprofitable branches of the tree.
It reads in a simple text file with the level data. A letter is a block, a '_' (underscore) is an open space, and a '#' is a filled-in space.
#!/usr/bin/env python
#
# Solve levels from the game Vexed.
from collections import Counter
import sys
level_blocks = set(chr(x) for x in range(ord('a'), ord('z')+1))
level_other = set(['_', '#'])
level_valid = set().union(level_blocks, level_other)
def prn_err(s='\n'):
sys.stderr.write(s)
sys.stderr.flush()
def validate_llc(llc):
if len(llc) == 0:
raise ValueError, "need at least one row of level data"
w = len(llc[0])
if w < 2:
raise ValueError, "level data not wide enough"
for i, row in enumerate(llc):
if len(row) != w:
s = "level data: row %d is different length than row 0"
raise ValueError, s % i
for j, ch in enumerate(row):
if ch not in level_valid:
s = "char '%c' at (%d, %d) is invalid" % (ch, i, j)
raise ValueError, s
class Info(object):
pass
info = Info()
info.width = 0
info.height = 0
info.spaces = set()
info.boom_blocks = set()
info.best_solution = 9999999999
info.title = "unknown"
class Level(object):
"""
Hold the state of a level at a particular move. self.parent points
to the previous state, from a previous move, so the solver builds a
tree representing the moves being considered. When you reach a solution
(a state where there are no more blocks) you can walk up the tree
back to the root, and you have the chain of moves that leads to that
solution."""
def __init__(self, x):
if isinstance(x, Level):
self.blocks = dict(x.blocks)
self.colors = dict(x.colors)
self.parent = x
self.s_move = ''
self.rank = x.rank + 1
else:
if isinstance(x, basestring):
# allow to init from a one-line "data" string
# example: "___;___;r_r"
x = x.split(';')
# build llc: list of rows, each row a list of characters
llc = [[ch for ch in row.strip()] for row in x]
llc.reverse()
info.width = len(llc[0])
info.height = len(llc)
validate_llc(llc)
# Use llc data to figure out the level, and build self.blocks
# and info.spaces. self.blocks is a dict mapping a coordinate
# tuple to a block color; info.spaces is just a set of
# coordinate tuples.
self.blocks = {}
for y in range(info.height):
for x in range(info.width):
loc = (x, y)
c = llc[y][x]
if c == '_':
# it's a space
info.spaces.add(loc)
elif c in level_blocks:
# it's a block (and the block is in a space)
self.blocks[loc] = c
info.spaces.add(loc)
else:
# must be a solid square
assert(c == '#')
# colors: map each block color onto a count of blocks.
self.colors = Counter(self.blocks.values())
# parent: points to the level instance that holds the state
# previous to the state of this level instance.
self.parent = None
# s_move: a string used when printing out the moves of a solution
self.s_move = 'initial state:'
# rank: 0 == initial state, +1 for each move
self.rank = 0
self.validate()
print "Solving:", info.title
print
sys.stdout.flush()
if self._update():
print "level wasn't stable! after updating:\n%s\n" % str(self)
def lone_color(self):
return any(count == 1 for count in self.colors.values())
def is_solved(self):
return sum(self.colors.values()) == 0
def validate(self):
if info.height == 0:
raise ValueError, "need at least one row of level data"
if info.width < 2:
raise ValueError, "level data not wide enough"
if self.lone_color():
raise ValueError, "cannot have just one of any block color"
for x, y in info.spaces:
if not 0 <= x < info.width or not 0 <= y < info.height:
raise ValueError, "Bad space coordinate: " + str(loc)
for x, y in self.blocks:
if not 0 <= x < info.width or not 0 <= y < info.height:
raise ValueError, "Bad block coordinate: " + str(loc)
if any(count < 0 for count in self.colors.values()):
raise ValueError, "cannot have negative color count!"
colors = Counter(self.blocks.values())
for k0 in [key for key in self.colors if self.colors[key] == 0]:
del(self.colors[k0]) # remove all keys whose value is 0
if colors != self.colors:
raise ValueError, "self.colors invalid!\n" + str(self.colors)
def _look(self, loc):
"""
return color at location 'loc', or '_' if empty, or '#' for a solid sqaure.
A bad loc does not raise an error; it just returns '#'.
"""
if loc in self.blocks:
return self.blocks[loc]
elif loc in info.spaces:
return '_'
else:
return '#'
def _lookxy(self, x, y):
loc = x, y
return self._look(loc)
def _board_mesg(self, mesg, loc):
x, y = loc
return "%s %c(%d,%d)" % (mesg, self._look(loc), x, y)
def _blocked(self, x, y):
return self._lookxy(x, y) != '_'
def _s_row(self, y):
return ''.join(self._lookxy(x, y) for x in xrange(info.width))
def data(self, ch_join=';'):
return ch_join.join(self._s_row(y)
for y in xrange(info.height - 1, -1, -1))
# make repr() actually print a representation
def __repr__(self):
return type(self).__name__ + "(%s)" % self.data()
# make str() work
def __str__(self):
return self.data('\n')
def _move_block(self, loc_new, loc_old):
self.blocks[loc_new] = self.blocks[loc_old]
del(self.blocks[loc_old])
def _explode_block(self, loc):
if loc in info.boom_blocks:
return
info.boom_blocks.add(loc)
color = self.blocks[loc]
self.colors[color] -= 1
def _try_move(self, loc, d):
x, y = loc
if not d in ('<', '>'):
raise ValueError, "d value '%c' invalid, must be '<' or '>'" % d
if d == '<':
x_m = (x - 1)
else:
x_m = (x + 1)
y_m = y
loc_m = (x_m, y_m)
if self._blocked(x_m, y_m):
return None # blocked, so can't move there
# Not blocked. Let's try the move!
# Make a duplicate level...
m = Level(self)
# ...try the move, and see if anything falls or explodes...
m._move_block(loc_m, loc)
m._update()
if m.lone_color():
# Whoops, we have only one block of some color. That means
# no solution can be found by considering this board.
return None
# finish the update
m.s_move = self._board_mesg("move:", loc) + ' ' + d
m.parent = self
return m
def _falls(self, loc):
x, y = loc
# blocks fall if they can, and only explode when at rest.
# gravity loop: block falls until it comes to rest
if self._blocked(x, y - 1):
return False # it is already at rest
while not self._blocked(x, y - 1):
# block is free to fall so fall one step
y -= 1
loc_m = (x, y)
self._move_block(loc_m, loc)
return True # block fell to new location
def _explodes(self, loc):
x, y = loc
exploded = False
color = self._look(loc)
# look left, right, up, and down for blocks of same color
for e_loc in [(x-1, y), (x+1, y), (x, y-1)]:
if e_loc in self.blocks and self.blocks[e_loc] == color:
self._explode_block(e_loc)
exploded = True
if exploded:
self._explode_block(loc)
return exploded
def _update(self):
c = 0
while True:
# TRICKY: sum() works on functions that return a bool!
# As you might expect, True sums as 1 and False as 0.
f = sum(self._falls(loc) for loc in self.blocks)
e = sum(self._explodes(loc) for loc in self.blocks)
for loc in info.boom_blocks:
del(self.blocks[loc])
info.boom_blocks.clear()
c += f + e
if (f + e) == 0:
# no blocks fell or exploded; board is stable, update is done
break
return c
def print_moves(self):
lst = [self]
a = self
while a.parent:
a = a.parent
lst.append(a)
lst.reverse()
for i, a in enumerate(lst):
if i:
print "Move %d of %d" % (i, len(lst) - 1)
print a.s_move
print a
print
def solve(self):
c = 0
seen = set()
solutions = []
seen.add(self.data())
q = []
if self.is_solved():
solutions.append(self)
else:
q.append(self)
while q:
a = q.pop(0)
# Show dots while solver is 'thinking' to give a progress
# indicator. Dots are written to stderr so they will not be
# captured if you redirect stdout to save the solution.
c += 1
if c % 100 == 0:
prn_err('.')
if a.rank > info.best_solution:
# We cannot beat or even match the best solution.
# No need to think any more about this possibility.
# Just prune this whole branch of the solution tree!
continue
for loc in a.blocks:
for d in ('<', '>'):
m = a._try_move(loc, d)
if not m or m.data() in seen:
continue
if m.is_solved():
if info.best_solution > a.rank:
print "\nnew best solution: %d moves" % m.rank
info.best_solution = a.rank
else:
print "\nfound another solution: %d moves" % m.rank
solutions.append(m)
else:
seen.add(m.data())
q.append(m)
print
print "Considered %d different board configurations." % c
print
solutions.sort(key=lambda a: a.rank)
for n, a in enumerate(solutions):
print "solution %d): %d moves" % (n, a.rank)
a.print_moves()
if not solutions:
print "no solutions found!"
def load_vex_file(fname):
with open(fname, "rt") as f:
s = f.next().strip()
if s != "Vexed level":
raise ValueError, "%s: not a Vexed level file" % fname
s = f.next().strip()
if not s.startswith("title:"):
raise ValueError, "%s: missing title" % fname
info.title = s[6:].lstrip() # remove "title:"
for s in f:
if s.strip() == "--":
break
return Level(f)
if __name__ == "__main__":
if len(sys.argv) == 1:
print "Usage vexed_solver <vexed_level_file.vex>"
sys.exit(1)
fname = sys.argv[1]
level = load_vex_file(fname)
level.solve()
Here is an example level file:
Vexed level
title: 25-48, "Dark Lord"
--
##_f####
##_#_c##
##_#_p##
#f___###
cp___###
p#_p_f__
On my computer, it solves "Dark Lord" in almost exactly 10 seconds, considering 14252 different board configurations. I wrote in Python 2.x instead of Python 3, because I want to try this with PyPy and see how fast it becomes.
Next I should work on applying A* to this. I guess I can make a metric like "better to move an orange block toward another orange block than away" and try to work that in. But I do want all the solutions to pop out, so maybe I'm done already. (If there are three solutions that are all the minimum number of moves, I want to see all three.)
I welcome comments on this Python program. I had fun writing it!
EDIT: I did try this with PyPy but I never updated this until now. On the computer I used with PyPy, the solver could solve the "Dark Lord" level in 10 seconds using CPython; that dropped to 4 seconds with PyPy. The cool part is that I could see the speedup as the JIT kicked in: this program prints dots as it is working, and under PyPy I can see the dots start out slower and then just accelerate. PyPy is nifty.

Studying Wikipedia may be better than studying the actual source code. A* is written out pretty clearly there. But that feels like cheating, doesn't it?
As all good ideas, A* is actually pretty obvious in retrospective. It's fun trying to work it through, and there are a few nice insights along the way. Here's how you get to it:
Write the brute-force solver. You'll need much of what you write in the more advanced versions: a game state, and a description of getting from one state to another. You'll also end up removing duplicate states. You should have a queue of some sort for states to be considered, a set of states you've already done, and structure to hold the best solution(s) found so far. And a method that takes a state from the queue and generates a state's „neighbor“ states (ones reachable from it). That's the basic structure of classical AI algorithms. Note that you're technically „generating“ or „exploring“ a huge graph here.
After that, add a simple pruning algorithm: if a state has only one block of some color left, there's no need to consider it further. See if you can come up with other pruning algorithms (i.e. ones that mark a state as „unsolvable“). A good pruning algorithm will eliminate lots of pointless states, thus justifying the time it takes to run the pruning itself.
Then, introduce a heuristic score: rank each state with a number that tells you how „good“ the state looks – about how much more solving will it take. Make your queue a priority queue. This will allow you to consider the „best looking“ states first, so the program should come up with a solution faster. But, the first solution found may not actually be the best, so to be sure that you find the best one, you still need to run the whole program.
Store the minimum cost (number of moves) that you took to get to each state. Remember to update it if you find a better path. Take the states with the lowest sum of their cost and their heuristic score first; those will more likely lead to a better solution.
And here comes A*. You need to modify your heuristic function so that it doesn't overestimate the distance to the goal, i.e. it can be lower than the number of moves you will actually need, but not higher. Then, note that if you found a solution, its heuristic score will be 0. And, any state where the sum of its cost and heuristic is more than the cost of a solution can't lead to a better solution. So, you can prune that state. But since you're taking the states in order, once you hit that threshold you can just stop and return, since all other states in the queue would be pruned as well.
All that's left now is perfecting your heuristic: it can never overestimate, but the better estimate it gives the less time A* will take. The better the heuristic, the better your results. Take care that the heuristic doesn't take so much time to complete – you wouldn't want, say, generating the solution by brute force, even though it would give the perfect answer :)
Wikipedia has some more discussion and possible improvements, if you get this far. But the best improvements you can make at this point will likely come from improving the heuristic function.

Maybe translate it into a classical planning problem (using PDDL syntax). Then you can try out some planners that are freely available.
E.g. try Fast Forward.

Related

Improving an algorithm for peak to peak measurements

I've implemented an algorithm to detect the negative part of a given peak. The main problem it gets effected by outliers easily, any recommendations on how to improve it?
def stepdown(self):
peak_location_y = self.spec[self.peak_idx]
peak_indcies = self.peak_idx
spec_y = self.spec
neg_peak = []
for peak_index, peak_y in zip(peak_indcies, peak_location_y):
i = 1
tmp = [0]
try:
while (
peak_y >= spec_y[peak_index + i]
and
spec_y[peak_index + i - 1] >= spec_y[peak_index + i]
):
tmp.append(peak_index + i)
i += 1
neg_peak.append(tmp[-1])
except IndexError:
print("Index Error")
return neg_peak
I know the quality of the code is horrible. I'm just prototyping.
Here is to examples when it works correctly and when it fail.
The upper part of the figure is negative peaks detected by the algorithm, and the lower part is positive peaks.

Would be great if you provided the rest of the class, e.g. what is self.spec?
Also, you could provide some detail on how you intend the algorithm to work.
As you have tagged scipy, I'll recommend you to check out scipy.signal.find_peaks if you haven't already.

Simple Genetic Algorithm meeting local optimum for "Hello World"

My target was simple, using genetic algorithm to reproduce the classical "Hello, World" string.
My code was based on this post. The code mainly contain 4 parts:
Generate the population which has serval different individual
Define the fitness and grade function which evaluate the individual good or bad based on the comparing with target.
Filter the population and leave len(pop)*retain individuals
Add some other individuals and mutate randomly
The parents's DNA will pass over to its children to comprise the whole population.
I modified the code and shows like this:
import numpy as np
import string
from operator import add
from random import random, randint
def population(GENSIZE,target):
p = []
for i in range(0,GENSIZE):
individual = [np.random.choice(list(string.printable[:-5])) for j in range(0,len(target))]
p.append(individual)
return p
def fitness(source, target):
fitval = 0
for i in range(0,len(source)-1):
fitval += (ord(target[i]) - ord(source[i])) ** 2
return (fitval)
def grade(pop, target):
'Find average fitness for a population.'
summed = reduce(add, (fitness(x, target) for x in pop))
return summed / (len(pop) * 1.0)
def evolve(pop, target, retain=0.2, random_select=0.05, mutate=0.01):
graded = [ (fitness(x, target), x) for x in p]
graded = [ x[1] for x in sorted(graded)]
retain_length = int(len(graded)*retain)
parents = graded[:retain_length]
# randomly add other individuals to
# promote genetic diversity
for individual in graded[retain_length:]:
if random_select > random():
parents.append(individual)
# mutate some individuals
for individual in parents:
if mutate > random():
pos_to_mutate = randint(0, len(individual)-1)
individual[pos_to_mutate] = chr(ord(individual[pos_to_mutate]) + np.random.randint(-1,1))
#
parents_length = len(parents)
desired_length = len(pop) - parents_length
children = []
while len(children) < desired_length:
male = randint(0, parents_length-1)
female = randint(0, parents_length-1)
if male != female:
male = parents[male]
female = parents[female]
half = len(male) / 2
child = male[:half] + female[half:]
children.append(child)
parents.extend(children)
return parents
GENSIZE = 40
target = "Hello, World"
p = population(GENSIZE,target)
fitness_history = [grade(p, target),]
for i in xrange(20):
p = evolve(p, target)
fitness_history.append(grade(p, target))
# print p
for datum in fitness_history:
print datum
But it seems that the result can't fit targetwell.
I tried to change the GENESIZE and loop time(more generation).
But the result always get stuck. Sometimes, enhance the loop time can help to find a optimum solution. But when I change the loop time to an much larger number like for i in xrange(10000). The result shows the error like:
individual[pos_to_mutate] = chr(ord(individual[pos_to_mutate]) + np.random.randint(-1,1))
ValueError: chr() arg not in range(256)
Anyway, how to modify my code and get an good result.
Any advice would be appreciate.

The chr function in Python2 only accepts values in the range 0 <= i < 256.
You are passing:
ord(individual[pos_to_mutate]) + np.random.randint(-1,1)
So you need to check that the result of
ord(individual[pos_to_mutate]) + np.random.randint(-1,1)
is not going to be outside that range, and take corrective action before passing to chr if it is outside that range.
EDIT
A reasonable fix for the ValueError might be to take the amended value modulo 256 before passing to chr:
chr((ord(individual[pos_to_mutate]) + np.random.randint(-1, 1)) % 256)
There is another bug: the fitness calculation doesn't take the final element of the candidate list into account: it should be:
def fitness(source, target):
fitval = 0
for i in range(0,len(source)): # <- len(source), not len(source) -1
fitval += (ord(target[i]) - ord(source[i])) ** 2
return (fitval)
Given that source and target must be of equal length, the function can be written as:
def fitness(source, target):
return sum((ord(t) - ord(s)) ** 2 for (t, s) in zip(target, source))
The real question was, why doesn't the code provided evolve random strings until the target string is reached.
The answer, I believe, is it may, but will take a lot of iterations to do so.
Consider, in the blog post referenced in the question, each iteration generates a child which replaces the least fit member of the gene pool if the child is fitter. The selection of the child's parent is biased towards fitter parents, increasing the likelihood that the child will enter the gene pool and increase the overall "fitness" of the pool. Consequently the members of the gene pool converge on the desired result within a few thousand iterations.
In the code in the question, the probability of mutation is much lower, based on the initial conditions, that is the defaults for the evolve function.
Parents that are retained have only a 1% chance of mutating, and one third of the time the "mutation" will not result in a change (zero is a possible result of random.randint(-1, 1)).
Discard parents are replaced by individuals created by merging two retained individuals. Since only 20% of parents are retained, the population can converge on a local minimum where each new child is effectively a copy of an existing parent, and so no diversity is introduced.
So apart from fixing the two bugs, the way to converge more quickly on the target is to experiment with the initial conditions and to consider changing the code in the question to inject more diversity, for example by mutating children as in the original blog post, or by extending the range of possible mutations.

Benefit of converting Python method to C extension?

A relatively simple question:
If I convert a CPU-bound bottleneck method from Python to a C extension (roughly implementing the same algorithm),
How much increase in speed, and performance should I expect?
What factors determine that?
UPDATE:
People seemed to be complaining on the lack of specifics. I was mostly trying to understand what factors would make a piece of Python code a good candidate for being rewritten in C (i.e., when would porting to C actually give you a speed boost if the original Python is CPU-bound).
For specifics, this is the piece of code I'm looking at. Basically it's a recursive method that takes two lists of lists (a list of "columns", where each column contains possible values that could go in that column...basically, a schema), and seeing if it's possible to make less than n (usually 1) change(s) (where a change might be to add a new value to a column, add a new column, remove a column, etc.) such that there's some sequence of values (one value from each column) you could construct out of either schema. It's very similar in spirit to calculating the edit distance between to strings. Here's the code:
def CheckMerge(self, schemai, schemaj, starti, startj, \
changesLeft, path):
# if starti == 0 and startj == 0:
# print '\n'
# print schemai.schema
# print ''
# print schemaj.schema
if starti == len(schemai.schema) and startj == len(schemaj.schema):
return (True, path)
if starti < len(schemai.schema):
icopy = schemai.schema[starti]
else:
icopy = []
if startj < len(schemaj.schema):
jcopy = schemaj.schema[startj]
else:
jcopy = []
intersect = set(icopy).intersection(set(jcopy))
intersect.discard('')
if len(intersect) == 0:
if starti < len(schemai.schema) and \
('' in schemai.schema[starti] or changesLeft > 0):
if not '' in schemai.schema[starti]:
changesLeft -= 1
changesCopy = list(path)
changesCopy.append('skipi')
result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj, \
changesLeft, changesCopy)
if result:
return (result,steps)
elif not '' in schemai.schema[starti]:
changesLeft += 1
if startj < len(schemaj.schema) and \
('' in schemaj.schema[startj] or changesLeft > 0):
if not '' in schemaj.schema[startj]:
changesLeft -= 1
changesCopy = list(path)
changesCopy.append('skipj')
result,steps = self.CheckMerge(schemai, schemaj, starti, startj+1, \
changesLeft, changesCopy)
if result:
return (result, steps)
elif not '' in schemaj.schema[startj]:
changesLeft += 1
if changesLeft > 0:
changesCopy = list(path)
changesCopy.append('replace')
changesLeft -= 1
result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj+1, \
changesLeft, changesCopy)
if result:
return (result, steps)
return (False, None)
else:
changesCopy = list(path)
changesCopy.append('merge')
result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj+1, \
changesLeft, changesCopy)
if result:
return (result, steps)
else:
return (False, None)

That solely and completely depends on your code.
If some piece of your code is supported by the hardware, like, if you're computing the Hamming weight, doing AES encrption, calculating CRC, or have a vectorizable code, there are hardware instructions for them that boosts up the speed, and you can accesss them by C code but not python code.

Python runs pretty fast, so you would need a distinct reason to convert a Python function to C, like to access hardware, which has already been mentioned. But, here is another reason.
Python (C Python) suffers from the Global Interpreter Lock (GIC) problem. Python threads cannot run simultaneously, only one at a time. So, you could put thread-specific code into C, which is not restricted by the GIC problem.
In general, if you believe your Python code is slow and it there is not a specific reason as you have mentioned in your post, then you may need to adapt to more Python-ic coding conventions, like list comprehensions and other features found in Python and not too many other languages.
My final comment is not a reflection on your code sample. Instead I am supplying it as the general wisdom that I've learned listening to a lot of Python presentations.

How to get rid of maximum recursion depth error or better solve this?

Problem: We have a square grid of 5 rows and 4 columns. We need to use these numbers to fill the grid; 1,2,3,4,5,6,7,8,9,10,12,18,20,21,24,27,30,35,36,40. We need to fill the grid in such a way that every horizontal and vertical neighbors should divide other without remainder. For example, 12 and 3 can be neighbors because 12 % 3 == 0, but 5 and 12 can't. Grid 2x2 is given to be 10.
I tried to solve problem using a list of sets. Each set represent possible values for each grid. When every set has only one element, problem is solved. Here are the functions I use to try to solve this problem (I added whole thing just in case, but I think my problem is in solve function.);
class CannotSolveError(Exception):
pass
def suitable_neighbor(a,b):
"return True if a and b can be neighbors."
return (a > b) and (a % b == 0) or (b % a == 0)
def equalize_tables(table1, table2):
"Make two tables equal, by changing first one in-place"
for i in range(len(table1)):
table1[i] = table2[i]
def remove_possibility(table, row, column, value):
"""Remove possibilities that can't be neighbors with value in rowxcolumn grid."""
index = ((row - 1) * num_cols) + column - 1
if len(table[index]) == 1:
return # This is a solved grid, do nothing.
remaining_possibilities = set(
filter(lambda x: suitable_neighbor(x, value), table[index])
)
if not remaining_possibilities:
raise ValueError("Invalid move")
if len(remaining_possibilities) == 1:
"Only one possibility remains, try to put it!"
copy_table = table[:]
try:
"Try it on copy"
put(copy_table, row, column, remaining_possibilities.pop())
except ValueError:
"Cannot put, re-raise and do nothing.."
raise
else:
"Putting is successfull, update original table"
equalize_tables(table, copy_table)
else:
table[index] = remaining_possibilities
def put(table, row, column, value):
"""Put a value on a grid, modifies given table. use with care!"""
index = ((row - 1) * num_cols) + column - 1
"Is this move possible?"
if value not in table[index]:
raise ValueError("Cannot put %d on %dx%d" % (value, row, column))
"Remove possibilities from left neighbor"
if column > 1:
remove_possibility(table, row, column - 1, value)
"Remove possibilities from right neighbor"
if column < num_cols:
remove_possibility(table, row, column + 1, value)
"Remove possibilities from upper neighbor"
if row > 1:
remove_possibility(table, row - 1, column, value)
"Remove possibilities from lower neighbor"
if row < num_rows:
remove_possibility(table, row + 1, column, value)
"Remove this value from every other set."
for i in range(num_rows * num_cols):
if i == index:
continue
table[i].discard(value)
"Put one-item set in place. Have to do this last."
table[index] = set([value])
def solve(table):
"Try to solve the table by trying possible values on grids."
to_try = [(i,len(table[i])) for i in range(num_rows * num_cols) if len(table[i]) > 1]
"Grid with least remaining possibilities will be tried first."
to_try.sort(key = lambda x: x[1])
for index, _ in to_try:
for value in table[index]:
row = index / num_cols + 1
column = index % num_cols + 1
copy_table = table[:]
put(copy_table, row, column, value)
try:
solve(copy_table)
equalize_tables(table, copy_table)
return
except CannotSolveError:
continue
except ValueError:
continue
raise CannotSolveError
I think this algorithm should solve the problem. But I am exceding maximum recursion depth. Any ideas how to work around this, or how should I solve this problem better in Python?
This is not a homework question. I am working on this by myself.

To avoid blowing up your stack, a more robust approach is to devise an encoding for your partial solutions (partially filled in board), and implement the backtracking yourself. That will require a lot less memory than relying on python's stack.
Google's Peter Norvig has written an illuminating article describing how he used such techniques to build an efficient backtracking sudoku solver. It uses a technique he calls "constraint propagation" to limit the space of options, so that the solution can be quickly found via brute force backtracking search (that is, without checking every possible grid of numbers, but only pursuing partial grids that might still lead to a solution). I think you will find it extremely applicable, not only for the general ideas but also for the specifics: Your problem, as you have approached it, is extremely close to a sudoku solver.

There's a way to set custom value for Python recursion limit (which is 1000 by default):
import sys
sys.setrecursionlimit(10000000)
You can add those lines before recursive call and if the problem remains, you have to review your implementation for other possible bugs.

it's a rainy day here, so i wrote a solution. i can post if you want, but perhaps you would rather find it yourself?
here are some hints:
your code doesn't seem to start with 10 at (2,2)
when trying a new value, you could add it to any empty space. the best space to try is one that has lots of neighbours, because that lets you test and reject bad values quickly.
assumed above, or a different way of saying the same thing - my search was over values. so i chose a location for "next move" and tried each value there. the opposite would be to search over locations (chose a "next value" and search with that value at each location), but that is not as efficient (see above).
when backtracking and re-trying, always follow the same pattern of locations. for example, (2,2) is 10 then (2,3) might be 40, then you might find nothing fits (2,4). so you backtrack and remove 40 and try a different number at (2,3). but the second number you try (after 10 and something at (2,2)) is always at (2,3). if you aren't careful in this way you can end up testing many duplicate combinations. sorry not sure this is very clear. basically - choose a "path" that you fill and stick to it when searching and backtracking. since this path is chosen to maximise the number of neighbours (point above) i constructed it as i went on, but kept a cache of the path locations that i used when backtracking. this is easier to explain by showing the code...
for the table i used an array of arrays. when copying i re-used columns that were not changed. this should reduce memory use (i don't know if it is important).
the search only needs to recurse 40 times (once for each value) so the stack is plenty big enough.
a simple search, in python, that tries each value in turn, backtracking on failure, runs ~4 minutes on my laptop (assuming you use the hints above) (without the printing a slightly modified version takes just 8 seconds).
i found it was useful to have a python function that, given a grid and a position, returns a list (well, a generator, with yield) of the coordinates of neighbours. that made writing other functions, like the one that tests whether a move is ok, simpler.
anyway, if you want the code or the solution (i changed my code to print all and there was just one) just ask and i will post. of course, it may have a bug too :o)
solution
i tweaked this some, and it now prints out the (2,2)=10 solution and then searches for all solutions (which is still running for me):
#!/usr/bin/python3
nx, ny = 4, 5
values = [1,2,3,4,5,6,7,8,9,10,12,18,20,21,24,27,30,35,36,40]
# grid[x][y] so it is a list of columns (prints misleadingly!)
grid = [[0 for _ in range(ny)] for _ in range(nx)]
# cache these to avoid re-calculating
xy_moves = {}
debug = False
def neighbours(grid, x, y):
'coordinates of vertical/horizontal neighbours'
for (xx, yy) in [(x-1,y),(x+1,y),(x,y-1),(x,y+1)]:
if xx > -1 and xx < nx and yy > -1 and yy < ny:
yield xx, yy
def filled_neighbours(grid, x, y):
'filter "neighbours" to give only filled cells'
return filter(lambda xy: grid[xy[0]][xy[1]], neighbours(grid, x, y))
def count_neighbours(grid, x, y):
'use this to find most-constrained location'
return sum(1 for _ in filled_neighbours(grid, x, y))
def next_xy(grid, depth):
'''given a certain depth in the search, where should we move next?
choose a place with lots of neighbours so that we have good
constraints (and so can reject bad moves)'''
if depth not in xy_moves:
best, x, y = 0, nx // 2, ny // 2 # default to centre
for xx in range(nx):
for yy in range(ny):
if not grid[xx][yy]:
count = count_neighbours(grid, xx, yy)
if count > best:
best, x, y = count, xx, yy
xy_moves[depth] = (x, y)
if debug: print('next move for %d is %d,%d' % (depth, x, y))
return xy_moves[depth]
def drop_value(value, values):
'remove value from the values'
return [v for v in values if v != value]
def copy_grid(grid, x, y, value):
'copy grid, replacing the value at x,y'
return [[value if j == y else grid[i][j] for j in range(ny)]
if x == i else grid[i]
for i in range(nx)]
def move_ok(grid, x, y, value):
'are all neighbours multiples?'
for (xx, yy) in filled_neighbours(grid, x, y):
g = grid[xx][yy]
if (g > value and g % value) or (g < value and value % g):
if debug:
print('fail: %d at %d,%d in %s' % (value, x, y, grid))
return False
return True
def search(grid, values, depth=0):
'search over all values, backtracking on failure'
if values:
(x, y) = next_xy(grid, depth)
for value in values:
if move_ok(grid, x, y, value):
if debug: print('add %d to %d,%d' % (value, x, y))
for result in search(copy_grid(grid, x, y, value),
drop_value(value, values),
depth+1):
yield result
else:
yield grid
# run the search, knowing that (2,2) (which is (1,1) for zero-indexing)
# has the value 10.
for result in search(copy_grid(grid, 1, 1, 10), drop_value(10, values)):
print(result)
# how many solutions in total?
#xy_moves = {} # reset cache
#for (n, solution) in enumerate(search(grid, values)):
# print('%d: %s' % (n, solution))
it starts by choosing the square where it will add the next number using next_xy(). it chooses a place near as many existing numbers as possible so that it can test and reject numbers efficiently (the position is saved in xy_moves so that it doesn't need to be re-found if we backtrack). for each value it checks to see if putting the value at that position works using move_ok. if so, it calculates a new grid (with the value added) and a new list of values (with the used value removed) and recurses. recursion finishes when there are no values left to add.
and here is the result (each inner list is a column):
> time ./grid.py
[[4, 20, 5, 35, 7], [40, 10, 30, 1, 21], [8, 2, 6, 18, 3], [24, 12, 36, 9, 27]]
real 0m5.909s
[deleted incorrect comment about recursion and generators]
update
it finished the global search - if you don't fix (2,2) at the start there seem to be 12 solutions in total (3 distinct solutions if you ignore simple symmetries).
update 2
final code, including search for all solutions without symmetrical duplicates

How to make this method non-recursive?

Hey. This example is pretty specific but I think it could apply to a broad range of functions.
It's taken from some online programming contest.
There is a game with a simple winning condition. Draw is not possible. Game cannot go on forever because every move takes you closer to the terminating condition. The function should, given a state, determine if the player who is to move now has a winning strategy.
In the example, the state is an integer. A player chooses a non-zero digit and subtracts it from the number: the new state is the new integer. The winner is the player who reaches zero.
I coded this:
from Memoize import Memoize
#Memoize
def Game(x):
if x == 0: return True
for digit in str(x):
if digit != '0' and not Game(x-int(digit)):
return True
return False
I think it's clear how it works. I also realize that for this specific game there's probably a much smarter solution but my question is general. However this makes python go crazy even for relatively small inputs. Is there any way to make this code work with a loop?
Thanks
This is what I mean by translating into a loop:
def fac(x):
if x <= 1: return x
else: return x*fac(x-1)
def fac_loop(x):
result = 1
for i in xrange(1,x+1):
result *= i
return result
## dont try: fac(10000)
print fac_loop(10000) % 100 ## works

In general, it is only possible to convert recursive functions into loops when they are primitive-recursive; this basically means that they call themselves only once in the body. Your function calls itself multiple times. Such a function really needs a stack. It is possible to make the stack explicit, e.g. with lists. One reformulation of your algorithm using an explicit stack is
def Game(x):
# x, str(x), position
stack = [(x,str(x),0)]
# return value
res = None
while stack:
if res is not None:
# we have a return value
if not res:
stack.pop()
res = True
continue
# res is True, continue to search
res = None
x, s, pos = stack.pop()
if x == 0:
res = True
continue
if pos == len(s):
# end of loop, return False
res = False
continue
stack.append((x,s,pos+1))
digit = s[pos]
if digit == '0':
continue
x -= int(digit)
# recurse, starting with position 0
stack.append((x,str(x),0))
return res
Basically, you need to make each local variable an element of a stack frame; the local variables here are x, str(x), and the iteration counter of the loop. Doing return values is a bit tricky - I chose to set res to not-None if a function has just returned.

By "go crazy" I assume you mean:
>>> Game(10000)
# stuff skipped
RuntimeError: maximum recursion depth exceeded in cmp
You could start at the bottom instead -- a crude change would be:
# after defining Game()
for i in range(10000):
Game(i)
# Now this will work:
print Game(10000)
This is because, if you start with a high number, you have to recurse a long way before you reach the bottom (0), so your memoization decorator doesn't help the way it should.
By starting from the bottom, you ensure that every recursive call hits the dictionary of results immediately. You probably use extra space, but you don't recurse far.
You can turn any recursive function into an iterative function by using a loop and a stack -- essentially running the call stack by hand. See this question or this quesstion, for example, for some discussion. There may be a more elegant loop-based solution here, but it doesn't leap out to me.

Well, recursion mostly is about being able to execute some code without losing previous contexts and their order. In particular, function frames put and saved onto call stack during recursion, therefore giving constraint on recursion depth because stack size is limited. You can 'increase' your recursion depth by manually managing/saving required information on each recursive call by creating a state stack on the heap memory. Usually, amount of available heap memory is larger than stack's one. Think: good quick sort implementations eliminate recursion to the larger side by creating an outer loop with ever-changing state variables (lower/upper array boundaries and pivot in QS example).
While I was typing this, Martin v. Löwis posted good answer about converting recursive functions into loops.

You could modify your recursive version a bit:
def Game(x):
if x == 0: return True
s = set(digit for digit in str(x) if digit != '0')
return any(not Game(x-int(digit)) for digit in s)
This way, you don't examine digits multiple times. For example, if you are doing 111, you don't have to look at 110 three times.
I'm not sure if this counts as an iterative version of the original algorithm you presented, but here is a memoized iterative version:
import Queue
def Game2(x):
memo = {}
memo[0] = True
calc_dep = {}
must_calc = Queue.Queue()
must_calc.put(x)
while not must_calc.empty():
n = must_calc.get()
if n and n not in calc_dep:
s = set(int(c) for c in str(n) if c != '0')
elems = [n - digit for digit in s]
calc_dep[n] = elems
for new_elem in elems:
if new_elem not in calc_dep:
must_calc.put(new_elem)
for k in sorted(calc_dep.keys()):
v = calc_dep[k]
#print k, v
memo[k] = any(not memo[i] for i in v)
return memo[x]
It first calculates the set of numbers that x, the input, depends on. Then it calculates those numbers, starting at the bottom and going towards x.
The code is so fast because of the test for calc_dep. It avoids calculating multiple dependencies. As a result, it can do Game(10000) in under 400 milliseconds whereas the original takes -- I don't know how long. A long time.
Here are performance measurements:
Elapsed: 1000 0:00:00.029000
Elapsed: 2000 0:00:00.044000
Elapsed: 4000 0:00:00.086000
Elapsed: 8000 0:00:00.197000
Elapsed: 16000 0:00:00.461000
Elapsed: 32000 0:00:00.969000
Elapsed: 64000 0:00:01.774000
Elapsed: 128000 0:00:03.708000
Elapsed: 256000 0:00:07.951000
Elapsed: 512000 0:00:19.148000
Elapsed: 1024000 0:00:34.960000
Elapsed: 2048000 0:01:17.960000
Elapsed: 4096000 0:02:55.013000
It's reasonably zippy.

We Keep Coding

Python is a programming language that lets you work quickly and integrate systems more effectively.