We Keep Coding

How can I change this to use a q table for reinforcement learning

I am working on learning q-tables and ran through a simple version which only used a 1-dimensional array to move forward and backward. now I am trying 4 direction movement and got stuck on controlling the person.
I got the random movement down now and it will eventually find the goal. but I want it to learn how to get to the goal instead of randomly stumbling on it. So I would appreciate any advice on adding a qlearning into this code. Thank you.
Here is my full code as it stupid simple right now.
import numpy as np
import random
import math
world = np.zeros((5,5))
print(world)
# Make sure that it can never be 0 i.e the start point
goal_x = random.randint(1,4)
goal_y = random.randint(1,4)
goal = (goal_x, goal_y)
print(goal)
world[goal] = 1
print(world)
LEFT = 0
RIGHT = 1
UP = 2
DOWN = 3
map_range_min = 0
map_range_max = 5
class Agent:
def __init__(self, current_position, my_goal, world):
self.current_position = current_position
self.last_postion = current_position
self.visited_positions = []
self.goal = my_goal
self.last_reward = 0
self.totalReward = 0
self.q_table = world
# Update the totoal reward by the reward
def updateReward(self, extra_reward):
# This will either increase or decrese the total reward for the episode
x = (self.goal[0] - self.current_position[0]) **2
y = (self.goal[1] - self.current_position[1]) **2
dist = math.sqrt(x + y)
complet_reward = dist + extra_reward
self.totalReward += complet_reward
def validate_move(self):
valid_move_set = []
# Check for x ranges
if map_range_min < self.current_position[0] < map_range_max:
valid_move_set.append(LEFT)
valid_move_set.append(RIGHT)
elif map_range_min == self.current_position[0]:
valid_move_set.append(RIGHT)
else:
valid_move_set.append(LEFT)
# Check for Y ranges
if map_range_min < self.current_position[1] < map_range_max:
valid_move_set.append(UP)
valid_move_set.append(DOWN)
elif map_range_min == self.current_position[1]:
valid_move_set.append(DOWN)
else:
valid_move_set.append(UP)
return valid_move_set
# Make the agent move
def move_right(self):
self.last_postion = self.current_position
x = self.current_position[0]
x += 1
y = self.current_position[1]
return (x, y)
def move_left(self):
self.last_postion = self.current_position
x = self.current_position[0]
x -= 1
y = self.current_position[1]
return (x, y)
def move_down(self):
self.last_postion = self.current_position
x = self.current_position[0]
y = self.current_position[1]
y += 1
return (x, y)
def move_up(self):
self.last_postion = self.current_position
x = self.current_position[0]
y = self.current_position[1]
y -= 1
return (x, y)
def move_agent(self):
move_set = self.validate_move()
randChoice = random.randint(0, len(move_set)-1)
move = move_set[randChoice]
if move == UP:
return self.move_up()
elif move == DOWN:
return self.move_down()
elif move == RIGHT:
return self.move_right()
else:
return self.move_left()
# Update the rewards
# Return True to kill the episode
def checkPosition(self):
if self.current_position == self.goal:
print("Found Goal")
self.updateReward(10)
return False
else:
#Chose new direction
self.current_position = self.move_agent()
self.visited_positions.append(self.current_position)
# Currently get nothing for not reaching the goal
self.updateReward(0)
return True
gus = Agent((0, 0) , goal)
play = gus.checkPosition()
while play:
play = gus.checkPosition()
print(gus.totalReward)

I have a few suggestions based on your code example:
separate the environment from the agent. The environment needs to have a method of the form new_state, reward = env.step(old_state, action). This method is saying how an action transforms your old state into a new state. It's a good idea to encode your states and actions as simple integers. I strongly recommend setting up unit tests for this method.
the agent then needs to have an equivalent method action = agent.policy(state, reward). As a first pass, you should manually code an agent that does what you think is right. e.g., it might just try to head towards the goal location.
consider the issue of whether the state representation is Markovian. If you could do better at the problem if you had a memory of all the past states you visited, then the state doesn't have the Markov property. Preferably, the state representation should be compact (the smallest set that is still Markovian).
once this structure is set-up, you can then think about actually learning a Q table. One possible method (that is easy to understand but not necessarily that efficient) is Monte Carlo with either exploring starts or epsilon-soft greedy. A good RL book should give pseudocode for either variant.
When you are feeling confident, head to openai gym https://www.gymlibrary.dev/ for some more detailed class structures. There are some hints about creating your own environments here: https://www.gymlibrary.dev/content/environment_creation/

Python implementation of Determining DNA Health algorithm from HackerRank

I am trying to solve Determining DNA Health challenge from Hackerrank using python. (I have to add I am somewhat new to python 3. Still learning the language)
My solution fails for test cases 7, 8 and 9 with a message reading "Wrong Answer".
When I run the following code locally, I can confirm that for these test cases my implementation produces the expected output.
I am wondering what would be the problem.
I am a bit puzzled at the moment. Is there a problem with my implementation? If so how come it produces correct answers for 28 test cases but fails on these 3? Or is it a misleading/confusing result message from Hacker Rank, as I happen to know that people find these 3 test cases (7, 8 and 9) problematic from what I learnt from reading discussions.
Any help would be highly appreciated.
Here is the code I wrote:
from bisect import bisect_left
from bisect import bisect_right
import sys
from unittest.mock import right
class TrieNode(object):
def __init__(self):
self.subnodes = {}
self.isTerminal = False
self.indexList = []
self.healthList = []
def addSubnode(self, aChar):
if (self.subnodes.get(aChar)):
return self.subnodes[aChar]
else:
newNode = TrieNode()
self.subnodes[aChar] = newNode
return newNode
def addIndexAndValue(self, index, health):
self.isTerminal = True
self.indexList.append(index)
lastHealth = 0
healthLength = len(self.healthList)
if (healthLength>0):
lastHealth = self.healthList[healthLength-1]
self.healthList.append(lastHealth + health)
def getSubnodeFor(self, aChar):
return self.subnodes.get(aChar)
def getValueForIndexes(self, startIndex, endIndex):
listSize = len(self.indexList)
if listSize < 1:
return 0
elif listSize == 1:
if startIndex <= self.indexList[0] and endIndex >= self.indexList[0]:
return self.healthList[0]
else:
return 0
else: # listSize > 1
rightInd = bisect_left(self.indexList, endIndex)
if rightInd < listSize and endIndex < self.indexList[0]:
return 0
big = 0
if rightInd >= listSize:
big = self.healthList[listSize - 1]
else:
if endIndex >= self.indexList[rightInd]:
big = self.healthList[rightInd]
else:
big = self.healthList[rightInd-1]
leftInd = bisect_left(self.indexList, startIndex)
small = 0
if leftInd >= listSize:
return 0
else:
if startIndex <= self.indexList[leftInd]:
if (leftInd > 0):
small = self.healthList[leftInd - 1]
else:
small = 0
else:
small = self.healthList[leftInd]
return big - small
class Trie(object):
def __init__(self):
self.root = TrieNode()
def getRoot(self):
return self.root
def createTrie(self, genes, healths):
for i in range(len(genes)):
node = self.root
for c in genes[i]:
node = node.addSubnode(c)
node.addIndexAndValue(i, healths[i])
def calculateHealth(trie, d, first, last):
total = 0
dLength = len(d)
for i in range(0, dLength):
node = trie.getRoot()
for j in range(i, dLength):
node = node.getSubnodeFor(d[j])
if node != None:
if node.isTerminal:
val = node.getValueForIndexes(first, last)
total = total + val
else:
break
return total
def readFromFile(aFileName):
inputArr = None
with open('../hackerRank/src/' + aFileName, encoding='utf-8') as aFile:
inputArr = aFile.read().splitlines()
return inputArr
def runFor(fileName, minimumValue, maximumValue):
inp = readFromFile(fileName)
n = inp[0]
genes = inp[1].rstrip().split()
healths = list(map(int, inp[2].rstrip().split()))
trie = Trie()
trie.createTrie(genes, healths)
s = int(inp[3])
minVal = sys.maxsize
maxVal = -1
for fItr in range(s):
line = inp[fItr+4].split()
first = int(line[0])
last = int(line[1])
d = line[2]
val = calculateHealth(trie, d, first, last)
if val < minVal:
minVal = val
if val > maxVal:
maxVal = val
print (minVal,maxVal)
assert minimumValue == minVal
assert maximumValue == maxVal
# TextX.txt 's are simple text files, which hold test data for regarding test case
# following the file name are real expected numbers for each relevant test case
# I got those from hacker rank
runFor('Test2.txt', 15806635, 20688978289)
runFor('Test7.txt', 0, 7353994)
runFor('Test8.txt', 0, 8652768)
runFor('Test9.txt', 0, 9920592)
runFor('Test33.txt', 11674463, 11674463)

One reference that might assist can be found at:
https://gist.github.com/josephmisiti/940cee03c97f031188ba7eac74d03a4f
Please read the notes he has included.
This is the input I have been using.
6
a b c aa d b
1 2 3 4 5 6
3
1 5 caaab
0 4 xyz
2 4 bcdybc

Python error: int object has no attribute to append?

I can't figure out why my code isn't working, very frustrating. I constantly get the error: int object has no attribute to append (for average.append(i, average//250)). But I can't figure out what exactly is wrong here. Is it not possible to import other definition in append functions?
I hope somebody can help me out!
Any help with my code in general is appreciated :)
def main():
average = []
y_values = []
for x in range(0, 2501, 500):
for i in range(250):
average.append(calculate(x))
average = sum(average)
print("{} euro, {} worpen".format(i, average//250))
y_values.append(average//250)
x_values = [0, 500, 1000, 1500, 2000, 2500]
y_values = []
plt.plot(x_values, y_values)
plt.xlabel("Startgeld")
plt.ylabel("Aantal worpen")
plt.title("Monopoly")
plt.show()
def calculate(game_money):
piece = monopoly.Piece()
board = monopoly.Board()
owns = possession(board)
dice = throw()
throw_count = 0
number = 0
total_throw = 0
while not all(owns.values()):
number == throw()
piece.move(number)
total_throw = total_throw + number
throw_count += 1
if total_throw > 40:
game_money += 200
elif board.values[piece.location] > 0:
if game_money > board.values[piece.location]:
if owns[board.names[piece.location]] == False:
owns[board.names[piece.location]] = True
game_money = game_money - board.values[piece.location]
return total_throw
def throw():
dice = randint(1,6) + randint(1,6)
return dice
def possession(board):
owns = {}
for i in range(40):
if board.values[i] > 0:
owns[board.names[i]] = False
return owns
if __name__ == "__main__":
main()

You done a small mistake in your code. See my comment below and correct your code accordingly. Good Luck :-)
y_values = []
average = []
for x in range(0, 2501, 500):
for i in range(250):
average.append(calculate(x))
#average = sum(average) #This is your mistake. Now onward average will be considered as int object make it like below
average1 = sum(average)
print("{} euro, {} worpen".format(i, average1//250))
y_values.append(average1//250)

Is this a genetic algorithm?

I am trying to make a genetic algorithm that finds the word given in the console input. But I don't know if I succeeded to do a full genetic algorithm.
Here is the code:
main.py:
from population import Population
target = input()
maxPop = 10
mutation = 100
print("\n\n\n")
pop = Population(target, maxPop, mutation)
population.py:
import random
from ADN import genetic
class Population:
def __init__(self, target, maxPop, mut):
adn = genetic()
self.popul = []
i = 0
while i < maxPop:
self.popul.append(adn.genFirst(len(target)))
print(self.popul[i])
i+=1
#oldPop = self.popul
#adn.fitness(oldPop, target)
#"""
while target not in self.popul:
oldPop = self.popul
self.popul = adn.fitness(oldPop, target)
if target in self.popul:
return
#"""
ADN.py:
import random
class genetic:
def genFirst(self, length):
bestGenes = ""
self.letters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890[],. "
word = ""
i = 0
while i < length:
word += random.choice(self.letters)
i+=1
return word
def fitness(self, oldPop, target):
newPop = []
j = 0
for word in oldPop:
newW = ""
for letter in word:
if(letter not in target):
letter = random.choice(self.letters)
else:
if(target.index(letter) != word.index(letter)):
letter = random.choice(self.letters)
newW += letter
newPop.append(newW)
print(newPop)
return newPop
If it is not a full genetic algorithm, what is missing?

No, it's not a genetic algorithm. It is not even an evolutionary algorithm. It misses the fitness function which should calculate how good is every member of the calculation. After that you should decide which code would you want to make: genetic or evolutionary. Being a beginner you should try the evolutionary algorithm, it's easier and it does not contain the crossover function (which is difficult for beginners).
Try this:
import random
genes = "abcdefghijklmnopqrsttuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ+-[]()1234567890;<>?/ "
target = input()
def genPar(length):
parent = []
for i in range(length):
parent.append(random.choice(genes))
return "".join(parent)
def fitness(parent):
total = 0
for i in range(len(parent)):
if(parent[i] == target[i]):
total += 1
return total
def mutate(parent):
index = random.choice(range(len(parent)))
child = []
for i in range(len(parent)):
if(i == index):
letter = random.choice(genes)
else:
letter = parent[i]
child.append(letter)
return "".join(child)
parent = genPar(len(target))
bestPar = parent
bestFitness = fitness(parent)
print(parent)
generations = 1
while True:
child = mutate(bestPar)
childFit = fitness(child)
if(childFit > bestFitness):
bestFitness = childFit
bestPar = child
print(child)
generations += 1
if(child == target):
break
print("\nGenerations: " + str(generations))

Until u see Initialization -> Fitness -> Genetic operators (mutation, crossover) -> Fitness -> Substitution cycle you can't say it is Genetic/Evolutionary algorithm :)...

for the basic genetic algorithm, you need to use some operator selection, fitness, mutation, crossover.
there different types of selection, crossover, and mutation that you can use based on your problem.
a simple example of crossover and mutation.
def single_point_crossover(parent1,parent2):
crossover_point = random.randint(1,9)
#print("crossover point", crossover_point)
child_1 = np.hstack((parent1[0:crossover_point], parent2[crossover_point:]))
child_2 = np.hstack((parent2[:crossover_point],parent1[crossover_point:]))
return child_1,child_2
def mutation(parent1,parent2):
n = len(parent1)
pos_1 = random.randint(0,n-1)
pos_2 = random.randint(0,n-1)
#print(pos_1, pos_2)
def swap(sol, posA, posB):
result = sol.copy()
elA = sol[posA]
elB = sol[posB]
result[posA] = elB
result[posB] = elA
return result
child1 = swap(parent1, pos_1, pos_2)
child2 = swap(parent2, pos_1, pos_2)
return child1,child2

Sympy seems to break down with higher numbers

I've been playing around with sympy and decided to make an arbitrary equations solver since my finance class was getting a little dreary. I wrote a basic framework and started playing with some examples, but some work and some don't for some reason.
from sympy import *
import sympy.mpmath as const
OUT_OF_BOUNDS = "Integer out of bounds."
INVALID_INTEGER = "Invalid Integer."
INVALID_FLOAT = "Invalid Float."
CANT_SOLVE_VARIABLES = "Unable to Solve for More than One Variable."
CANT_SOLVE_DONE = "Already Solved. Nothing to do."
# time value of money equation: FV = PV(1 + i)**n
# FV = future value
# PV = present value
# i = growth rate per perioid
# n = number of periods
FV, PV, i, n = symbols('FV PV i n')
time_value_money_discrete = Eq(FV, PV*(1+i)**n)
time_value_money_continuous = Eq(FV, PV*const.e**(i*n))
def get_sym_num(prompt, fail_prompt):
while(True):
try:
s = input(prompt)
if s == "":
return None
f = sympify(s)
return f
except:
print(fail_prompt)
continue
equations_supported = [['Time Value of Money (discrete)', [FV, PV, i, n], time_value_money_discrete],
['Time Value of Money (continuous)',[FV, PV, i, n], time_value_money_continuous]]
EQUATION_NAME = 0
EQUATION_PARAMS = 1
EQUATION_EXPR = 2
if __name__ == "__main__":
while(True):
print()
for i, v in enumerate(equations_supported):
print("{}: {}".format(i, v[EQUATION_NAME]))
try:
process = input("What equation do you want to solve? ")
if process == "" or process == "exit":
break
process = int(process)
except:
print(INVALID_INTEGER)
continue
if process < 0 or process >= len(equations_supported):
print(OUT_OF_BOUNDS)
continue
params = [None]*len(equations_supported[process][EQUATION_PARAMS])
for i, p in enumerate(equations_supported[process][EQUATION_PARAMS]):
params[i] = get_sym_num("What is {}? ".format(p), INVALID_FLOAT)
if params.count(None) > 1:
print(CANT_SOLVE_VARIABLES)
continue
if params.count(None) == 0:
print(CANT_SOLVE_DONE)
continue
curr_expr = equations_supported[process][EQUATION_EXPR]
for i, p in enumerate(params):
if p != None:
curr_expr = curr_expr.subs(equations_supported[process][EQUATION_PARAMS][i], params[i])
print(solve(curr_expr, equations_supported[process][EQUATION_PARAMS][params.index(None)]))
This is the code I have so far. I guess I can strip it down to a basic example if need be, but I was also wondering if there was a better way to implement this sort of system. After I have this down, I want to be able to add arbitrary equations and solve them after inputting all but one parameter.
For example, if I put in (for equation 0), FV = 1000, PV = 500, i = .02, n is empty I get 35.0027887811465 which is the correct answer. If I redo it and change FV to 4000, it returns an empty list as the answer.
Another example, when I input an FV, PV, and an n, the program seems to hang. When I input small numbers, I got RootOf() answers instead of a simple decimal.
Can anyone help me?
Side note: I'm using SymPy 0.7.6 and Python 3.5.1 which I'm pretty sure are the latest

This is a floating point accuracy issue. solve by default plugs solutions into the original equation and evaluates them (using floating point arithmetic) in order to sort out false solutions. You can disable this by setting check=False. For example, for Hugh Bothwell's code
for fv in range(1870, 1875, 1):
sols = sp.solve(eq.subs({FV:fv}), check=False)
print("{}: {}".format(fv, sols))
which gives
1870: [66.6116466112007]
1871: [66.6386438584579]
1872: [66.6656266802551]
1873: [66.6925950919998]
1874: [66.7195491090752]

I don't have an answer, but I do have a much simpler demonstration case ;-)
import sympy as sp
FV, n = sp.symbols("FV n")
eq = sp.Eq(FV, sp.S("500 * 1.02 ** n"))
# see where it breaks
for fv in range(1870, 1875, 1):
sols = sp.solve(eq.subs({FV:fv}))
print("{}: {}".format(fv, sols))
which produces
1870: [66.6116466112007]
1871: [66.6386438584579]
1872: []
1873: []
1874: []
At a guess this is where the accuracy breaks down enough that it can't find a verifiable solution for n?
Also, while poking at this I did a fairly extensive rewrite which you may find useful. It does pretty much the same as your code but in a much more loosely-coupled fashion.
import sympy as sp
class Equation:
def __init__(self, label, equality_str, eq="=="):
self.label = label
# parse the equality
lhs, rhs = equality_str.split(eq)
self.equality = sp.Eq(sp.sympify(lhs), sp.sympify(rhs))
# index free variables by name
self.vars = {var.name: var for var in self.equality.free_symbols}
def prompt_for_values(self):
# show variables to be entered
var_names = sorted(self.vars, key=str.lower)
print("\nFree variables are: " + ", ".join(var_names))
print("Enter a value for all but one (press Enter to skip):")
# prompt for values by name
var_values = {}
for name in var_names:
value = input("Value of {}: ".format(name)).strip()
if value:
var_values[name] = sp.sympify(value)
# convert names to Sympy variable references
return {self.vars[name]:value for name,value in var_values.items()}
def solve(self):
values = self.prompt_for_values()
solutions = sp.solve(self.equality.subs(values))
# remove complex answers
solutions = [sol.evalf() for sol in solutions if sol.is_real]
return solutions
def __str__(self):
return str(self.equality)
# Define some equations!
equations = [
Equation("Time value of money (discrete)", "FV == PV * (1 + i) ** n"),
Equation("Time value of money (continuous)", "FV == PV * exp(i * n)" )
]
# Create menu
menu_lo = 1
menu_hi = len(equations) + 1
menu_prompt = "\n".join(
[""]
+ ["{}: {}".format(i, eq.label) for i, eq in enumerate(equations, 1)]
+ ["{}: Exit".format(menu_hi)]
+ ["? "]
)
def get_int(prompt, lo=None, hi=None):
while True:
try:
value = int(input(prompt))
if (lo is None or lo <= value) and (hi is None or value <= hi):
return value
except ValueError:
pass
def main():
while True:
choice = get_int(menu_prompt, menu_lo, menu_hi)
if choice == menu_hi:
print("Goodbye!")
break
else:
solutions = equations[choice - 1].solve()
num = len(solutions)
if num == 0:
print("No solutions found")
elif num == 1:
print("1 solution found: " + str(solutions[0]))
else:
print("{} solutions found:".format(num))
for sol in solutions:
print(sol)
if __name__ == "__main__":
main()