We Keep Coding

How can I change this to use a q table for reinforcement learning

I am working on learning q-tables and ran through a simple version which only used a 1-dimensional array to move forward and backward. now I am trying 4 direction movement and got stuck on controlling the person.
I got the random movement down now and it will eventually find the goal. but I want it to learn how to get to the goal instead of randomly stumbling on it. So I would appreciate any advice on adding a qlearning into this code. Thank you.
Here is my full code as it stupid simple right now.
import numpy as np
import random
import math
world = np.zeros((5,5))
print(world)
# Make sure that it can never be 0 i.e the start point
goal_x = random.randint(1,4)
goal_y = random.randint(1,4)
goal = (goal_x, goal_y)
print(goal)
world[goal] = 1
print(world)
LEFT = 0
RIGHT = 1
UP = 2
DOWN = 3
map_range_min = 0
map_range_max = 5
class Agent:
def __init__(self, current_position, my_goal, world):
self.current_position = current_position
self.last_postion = current_position
self.visited_positions = []
self.goal = my_goal
self.last_reward = 0
self.totalReward = 0
self.q_table = world
# Update the totoal reward by the reward
def updateReward(self, extra_reward):
# This will either increase or decrese the total reward for the episode
x = (self.goal[0] - self.current_position[0]) **2
y = (self.goal[1] - self.current_position[1]) **2
dist = math.sqrt(x + y)
complet_reward = dist + extra_reward
self.totalReward += complet_reward
def validate_move(self):
valid_move_set = []
# Check for x ranges
if map_range_min < self.current_position[0] < map_range_max:
valid_move_set.append(LEFT)
valid_move_set.append(RIGHT)
elif map_range_min == self.current_position[0]:
valid_move_set.append(RIGHT)
else:
valid_move_set.append(LEFT)
# Check for Y ranges
if map_range_min < self.current_position[1] < map_range_max:
valid_move_set.append(UP)
valid_move_set.append(DOWN)
elif map_range_min == self.current_position[1]:
valid_move_set.append(DOWN)
else:
valid_move_set.append(UP)
return valid_move_set
# Make the agent move
def move_right(self):
self.last_postion = self.current_position
x = self.current_position[0]
x += 1
y = self.current_position[1]
return (x, y)
def move_left(self):
self.last_postion = self.current_position
x = self.current_position[0]
x -= 1
y = self.current_position[1]
return (x, y)
def move_down(self):
self.last_postion = self.current_position
x = self.current_position[0]
y = self.current_position[1]
y += 1
return (x, y)
def move_up(self):
self.last_postion = self.current_position
x = self.current_position[0]
y = self.current_position[1]
y -= 1
return (x, y)
def move_agent(self):
move_set = self.validate_move()
randChoice = random.randint(0, len(move_set)-1)
move = move_set[randChoice]
if move == UP:
return self.move_up()
elif move == DOWN:
return self.move_down()
elif move == RIGHT:
return self.move_right()
else:
return self.move_left()
# Update the rewards
# Return True to kill the episode
def checkPosition(self):
if self.current_position == self.goal:
print("Found Goal")
self.updateReward(10)
return False
else:
#Chose new direction
self.current_position = self.move_agent()
self.visited_positions.append(self.current_position)
# Currently get nothing for not reaching the goal
self.updateReward(0)
return True
gus = Agent((0, 0) , goal)
play = gus.checkPosition()
while play:
play = gus.checkPosition()
print(gus.totalReward)

I have a few suggestions based on your code example:
separate the environment from the agent. The environment needs to have a method of the form new_state, reward = env.step(old_state, action). This method is saying how an action transforms your old state into a new state. It's a good idea to encode your states and actions as simple integers. I strongly recommend setting up unit tests for this method.
the agent then needs to have an equivalent method action = agent.policy(state, reward). As a first pass, you should manually code an agent that does what you think is right. e.g., it might just try to head towards the goal location.
consider the issue of whether the state representation is Markovian. If you could do better at the problem if you had a memory of all the past states you visited, then the state doesn't have the Markov property. Preferably, the state representation should be compact (the smallest set that is still Markovian).
once this structure is set-up, you can then think about actually learning a Q table. One possible method (that is easy to understand but not necessarily that efficient) is Monte Carlo with either exploring starts or epsilon-soft greedy. A good RL book should give pseudocode for either variant.
When you are feeling confident, head to openai gym https://www.gymlibrary.dev/ for some more detailed class structures. There are some hints about creating your own environments here: https://www.gymlibrary.dev/content/environment_creation/

Python find closest turtle via mouse click

I'm in the process of creating a minesweeper style game using a turtle-based grid setup. I need to find the closest cell within the grid and reveal the icon located under it whether that be a bomb or a number icons. I'm not looking to make it exact, I just need the mouse click to find the nearest cell in the grid even if the click isn't directly on the board. Currently my code only reveals the icon of the last turtle created on the board and then does nothing else with further clicks.
What can I do to make it recognize the real closest click and do it multiple times until the last bomb is found?
import random
import turtle
import cell
class Game:
def __init__(self, size):
registershapes()
self.__boardsize = size
self.__boardlist = []
self.__bombnum = 0
self.__probe = 0
self.__probelist = []
offset = (size-1) * 17
for x in range(size):
for y in range(size):
t = cell.Cell(x,y)
t.up()
t.shape('question.gif')
t.goto(y*34-offset, offset-x*34)
self.__boardlist.append(t)
def hideMines(self, num):
if num > self.__boardsize ** 2:
return False
self.__bombnum = num
self.__rnums = []
i = 0
while i < self.__bombnum:
currentnum = random.randrange(0, (self.__boardsize**2) - 1)
if currentnum not in self.__rnums:
self.__rnums.append(currentnum)
i += 1
return True
def probe(self, x, y):
for t in self.__boardlist:
pos = t.position()
distx = abs(x - pos[0])
disty = abs(y - pos[1])
distfinal = (distx ** 2 + disty ** 2) ** 0.5
curdist = 0
if curdist < distfinal:
curdist = distfinal
closest = t
if closest in self.__probelist:
return (self.__probe, self.__bombnum)
elif closest in self.__rnums:
closest.shape("bomb.gif")
self.__bombnum -= 1
self.__probe += 1
self.__probelist.append(closest)
return (self.__probe, self.__bombnum)
else:
closest.shape("0.gif")
self.__probe += 1
self.__probelist.append(closest)
return (self.__probe, self.__bombnum)
def registershapes():
wn = turtle.Screen()
wn.register_shape('0.gif')
wn.register_shape('1.gif')
wn.register_shape('2.gif')
wn.register_shape('3.gif')
wn.register_shape('4.gif')
wn.register_shape('5.gif')
wn.register_shape('6.gif')
wn.register_shape('7.gif')
wn.register_shape('8.gif')
wn.register_shape('9.gif')
wn.register_shape('bomb.gif')
wn.register_shape('question.gif')

I believe you're approaching this problem the wrong way. You're activating screen.onclick() and trying to map it to a turtle. Instead, activate turtle.onclick() on the individual turtles, deactivating it when a turtle is selected. Then you don't have to search for the turtle in question, and not actively ignore turtles that have already been selected.
Below is my rework of your code from this and your previous question into an example you can run. I had to guess about the definition of the Cell class:
from turtle import Turtle, Screen
import random
class Cell(Turtle):
def __init__(self, number):
super().__init__("question.gif")
self.__number = number
self.penup()
def number(self):
return self.__number
class Game:
def __init__(self, size):
registershapes()
self.__boardsize = size
self.__boardlist = []
self.__bombnum = 0
self.__probe = 0
self.__rnums = None
offset = (size - 1) * 17
for y in range(size):
for x in range(size):
t = Cell(x + y * size)
t.goto(x * 34 - offset, offset - y * 34)
t.onclick(lambda x, y, self=t: closure(self))
self.__boardlist.append(t)
def hideMines(self, num):
if num > self.__boardsize ** 2:
return False
self.__bombnum = num
self.__rnums = []
i = 0
while i < self.__bombnum:
currentnum = random.randrange(0, self.__boardsize ** 2 - 1)
if currentnum not in self.__rnums:
self.__rnums.append(currentnum)
i += 1
return True
def probe(self, closest):
closest.onclick(None)
if closest.number() in self.__rnums:
closest.shape("bomb.gif")
self.__bombnum -= 1
else:
closest.shape("0.gif")
self.__probe += 1
return (self.__probe, self.__bombnum)
def registershapes():
screen.register_shape('0.gif')
# ...
screen.register_shape('bomb.gif')
screen.register_shape('question.gif')
def closure(closest):
_, rem = mine.probe(closest)
if rem == 0:
over = screen.textinput("Text Input", "Would you like to play again? (Y)es or (N)o")
if over.upper() == 'Y':
main()
else:
screen.bye()
def main():
global mine
board = screen.numinput("Numeric Input", "Enter desired board size: ")
mine = Game(int(board))
nummine = screen.numinput("Numeric Input", "Enter desired number of mines: ")
mine.hideMines(int(nummine))
screen = Screen()
mine = None
main()
screen.mainloop()

Recursion in python didn't work out

def make_ship(length):
ship = []
ship_tmp = []
ship_row = random_row(board, length)
ship.append(ship_row)
ship_col = random_col(board, length)
ship.append(ship_col)
i = 0
if randint(0,1) == 0:
while i<length:
ship_tmp.append(ship[:])
ship[0] += 1
for ship in ship_tmp:
if ship in ship_all:
make_ship(length)
else:
ship_all.append(ship)
i+=1
else:
while i<length:
ship_tmp.append(ship[:])
ship[1] += 1
for ship in ship_tmp:
if ship in ship_all:
make_ship(length)
else:
ship_all.append(ship)
i+=1
The aim of this problem is to generate several ship on the board and prevent them from touching each other,but the make_ship() recursion didn't work out, how to solve this problem?

Using gobal variable:
global i
i=0
def make_ship(length):
global i
ship = []
ship_tmp = []
ship_row = random_row(board, length)
ship.append(ship_row)
ship_col = random_col(board, length)
ship.append(ship_col)
if randint(0,1) == 0:
while i<length:
ship_tmp.append(ship[:])
ship[0] += 1
for ship in ship_tmp:
if ship in ship_all:
make_ship(length)
else:
ship_all.append(ship)
i+=1
else:
while i<length:
ship_tmp.append(ship[:])
ship[1] += 1
for ship in ship_tmp:
if ship in ship_all:
make_ship(length)
else:
ship_all.append(ship)
i+=1
You may also try to use class to solve the problem:
class ship_class(object):
def __int__(self,length):
self.i=0
self.length=length
def make_ship(self):
ship = []
ship_tmp = []
ship_row = random_row(board, length)
ship.append(ship_row)
ship_col = random_col(board, length)
ship.append(ship_col)
if randint(0,1) == 0:
while self.i<self.length:
ship_tmp.append(ship[:])
ship[0] += 1
for ship in ship_tmp:
if ship in ship_all:
make_ship(self.length)
else:
ship_all.append(ship)
self.i+=1
else:
while self.i<self.length:
ship_tmp.append(ship[:])
ship[1] += 1
for ship in ship_tmp:
if ship in ship_all:
make_ship(self.length)
else:
ship_all.append(ship)
self.i+=1

Things you should learn:
Global variables are rarely a good idea; they break encapsulation and make debugging hard because you can't tell where changes came from (or didn't come from, sometimes!).
Meaningful comments are a very good thing. Meaningful variable names are even better. Well-factored, meaningful classes are a gift from heaven.
Keep your functions simple - ideally less than 10 or 12 lines. This lets you reason about the whole thing at once, so you can see most problems at a glance. If it's too complicated to think about the whole thing at once, maybe you should break it down further.
Recursion is good for dealing with branching structures. A ship is not a branching structure.
Here is a simple class-based solution:
import random
class Board:
def __init__(self, width, height, empty_char="."):
"""
Create a new gameboard
"""
self.width = width
self.height = height
self.empty_char = empty_char
self.ships = []
def __str__(self):
# make an empty board
board = [[self.empty_char] * self.width for y in range(self.height)]
# add ships
for ship in self.ships:
ship.draw(board)
# return as string
return "\n".join("".join(row) for row in board)
def add_rand_ship(self, length, char, tries=10):
"""
Add a ship at a random location on the board,
not overlapping any existing ships
"""
for try_ in range(tries):
# create a random Ship
new_ship = Ship.rand_ship(length, char, self.width, self.height)
# make sure it doesn't intersect any existing ships
if not any(new_ship.hits(ship) for ship in self.ships):
self.ships.append(new_ship)
return
raise ValueError("Failed creating new ship after {} tries".format(tries))
class Ship:
def __init__(self, x, y, length, char, horiz=True):
self.x = x
self.y = y
self.length = length
self.char = char
self.horiz = horiz
# get every square occupied by this ship
if self.horiz:
self.locations = set((x + dx, y) for dx in range(self.length))
else:
self.locations = set((x, y + dy) for dy in range(self.length))
def draw(self, board):
"""
Draw ship to board
"""
for x,y in self.locations:
board[y][x] = self.char
def hits(self, ship):
"""
Does this ship occupy any of the same locations as the other one?
"""
return bool(self.locations & ship.locations)
#classmethod
def rand_ship(cls, length, char, board_width, board_height):
"""
Create a new Ship at a random board location
"""
horiz = random.choice((False, True))
x = random.randint(0, board_width - (length if horiz else 1))
y = random.randint(0, board_height - (1 if horiz else length))
return cls(x, y, length, char, horiz)
def main():
bd = Board(30, 20)
for length in (2, 2, 3, 4):
for side in ("X", "O"):
bd.add_rand_ship(length, side)
print(bd)
main()
which produces something like:
..............................
..............................
..............................
.X............................
.X............................
.X............................
..............................
..............................
..........OOO.................
...............XX.............
..............................
...X..........................
...X......OOOO................
...X..........................
...X..........................
............OO................
..............................
..............................
...X.....................O....
...X.....................O....

How can I get my user input to run from my class? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
bob = Turtle()
bob.shape("turtle")
class SimpleDraw(Turtle):
def __init__(Turtle):
bob = Turtle
def draw_square(bob, length):
'''This function draws a regular square'''
for i in range(4):
bob.forward(length)
bob.left(90)
def draw_polygon(bob, n, length):
'''This function draws a regular polygon'''
angle = 360.0 / n
for i in range(n):
bob.forward(length)
bob.left(angle)
def draw_circle (t, r):
circumference = 2 * math.pi * r
n = 50
length = circumference / n
polygon (t, n, length)
drawObj = SimpleDraw
drawObj.draw_polygon
drawObj.draw_circle
drawObj.draw_square
def testSimpleDraw():
number = raw_input('Please choose a draw option: \
1-draw square \
2-draw polygon \
3-draw circle \
0-to exit');
if number == 0:
exit()
else:
if number == 1:
drawObj.draw_square(bob, 5)
else:
if number == 2:
drawObj.draw_polygon(bob, 7, 70)
else:
if number == 3:
drawObj.draw_circle(50, 50)
if __name__ == '__main__':
testSimpleDraw()

drawObj = SimpleDraw needs to be changed to drawObj = SimpleDraw()
That's how you instantiate a class. You are assigning the SimpleDraw class to drawObj - drawObj is the same as SimpleDraw class not an instance of SimpleDraw in your code.

In this line:
drawObj = SimpleDraw
you assign the class SimpleDraw to the name drawObj. Instead, you should create an instance of the class to assign:
drawObj = SimpleDraw() # note parentheses
You don't need the __init__ in your SimpleDraw class; you inherit this from Turtle. It is conventional to call the first argument to instance methods self, rather than bob! For example:
def draw_square(self, length):
'''This function draws a regular square'''
for i in range(4):
self.forward(length)
self.left(90)
You will need to adjust your draw_circle method to correctly call the draw_polygon method:
def draw_circle(self, r, n=50):
circumference = 2 * math.pi * r
length = circumference / n
self.draw_polygon(n, length)
Note that you can use elif to simplify your code:
if number == 0:
...
elif number == 1:
...

I'm going to take a guess at what you're trying to do. First, lets rewrite this code and I'll explain the changes as we go along.
class SimpleDraw(Turtle):
# No __init__ need, calls __init__ of subclass implicitly
# So self refers to an instance of Turtle
# Since all your code essentially makes use of the same thing, lets create
# a generic base draw.
def draw_generic(self, angle, length, n):
for i in range(n):
self.forward(length)
self.left(angle)
def draw_square(self, length):
# Notice self instead of bob, this is how you refer to the instance
# of the class that has been created.
"""
This function draws a regular square.
"""
self.draw_generic(90, length, 4) # Note, self implicitly passed
def draw_polygon(self, n, length):
"""
This function draws a regular polygon
"""
angle = 360.0 / n
self.draw_generic(angle, length, n)
def draw_circle(self, t, r):
circumference = 2 * math.pi * r
n = 50
length = circumference / n
self.draw_polygon(t, n, length) # Note: draw_polygon instead of polygon
def test_simple_draw():
# Note the lowercase and underscores, this preferred by PEP 8
# http://stackoverflow.com/questions/8908760/should-i-use-camel-case-or-underscores-in-python
options = ["1 - draw square", "2 - draw polygon", "3 - draw circle", "0 -exit"]
msg = "Please choose a draw option:\n %s" % ("\n".join(options))
# Note that number is a string, so we have to try and coerce it to an int, and
# validate this as well.
number = None
while number is None:
try:
number = raw_input(msg)
number = int(number)
if 0 > number or 3 < number:
raise ValueError # Not a valid option
except ValueError: # If the coercion fails
print "%s is not a valid option. Please re-enter." % number
number = None # Set to None so we loop again
# Now we have the number, and it's actually a number.
# You make a lot of use of else, which can be replaced by elif, which is the
# like else but specifies a condition; a union of if and else
drawObj = SimpleDraw() # We create our object here, note braces for instantiation
if number == 0:
exit()
elif number == 1:
drawObj.draw_square(5)
elif number == 2:
drawObj.draw_polygon(7, 70)
else: # We can have confidence in this value because of our checks above
drawObj.draw_circle(50, 50)
if __name__ == "__main__":
test_simple_draw()
Now This code gets across the idea that you initially wanted, but makes use of the class inheritance that you were originally trying to use.
Further sources for reference:
Classes: http://docs.python.org/2/tutorial/classes.html

import math
import random
import turtle
class SimpleTurtle(turtle.Turtle):
# There is no need to override the default __init__
# Also, making the self argument 'bob' is just irritating -
# everyone expects to see 'self', why mess us up?
def draw_polygon(self, num_sides, side_length):
"""
Draw an n-sided regular polygon
"""
angle = 360. / num_sides
for side in range(num_sides):
self.forward(side_length)
self.left(angle)
def draw_square(self, side_length):
"""
Draw a square
"""
self.draw_polygon(4, side_length)
def draw_circle(self, radius, num_sides=60):
"""
Draw a circle
"""
angle = 360. / num_sides
side_length = 2. * radius * math.sin(math.radians(angle))
self.draw_polygon(num_sides, side_length)
def get_int(prompt, lo=None, hi=None):
while True:
try:
val = int(raw_input(prompt))
if (lo is None or lo <= val) and (hi is None or val <= hi):
return val
except ValueError:
pass
def do_menu(prompt, options):
print(prompt)
for i,opt in enumerate(options, 1):
print(" {:>2}: {}".format(i, opt))
return get_int("? ", 1, len(options)) - 1
def test_SimpleTurtle():
bob = SimpleTurtle(shape="turtle")
while True:
choice = do_menu("Please choose a draw option:", ["draw a square", "draw a polygon", "draw a circle", "quit"])
if choice == 0:
side_length = random.randint(10, 50)
bob.draw_square(side_length)
elif choice == 1:
sides = random.randint(5, 12)
side_length = random.randint(6, 30)
bob.draw_polygon(sides, side_length)
elif choice == 2:
radius = random.randint(8, 40)
bob.draw_circle(radius)
else:
break
if __name__ == '__main__':
test_SimpleTurtle()

Dragon Curve in Python

I created a program to draw a dragon curve using turtle graphics.. but my result doesn't really look like the picture does in the link:
One problem I noticed is that I want to save the produced string into the variable newWord.. but I can't use newWord as a parameter in my function drawit, which actually draws lines based on the string. When I try to do that I get the error "global variable newWord not defined." So in my code I just copied the output of newWord to be drawn, without actually passing the variable that I wanted to pass.
I'm not sure if the problem is with my createWord function or if I'm just not 'drawing enough' in drawit.
import turtle
def createWord(max_it, axiom, proc_rules):
word = axiom
t = 1
while (t < max_it):
word = rewrite(word, proc_rules)
t=t+1
newWord = word
def rewrite(word, proc_rules):
wordList = list(word)
for i in range(len(wordList)):
curChar = wordList[i]
if curChar in proc_rules:
wordList[i] = proc_rules[curChar]
return "".join(wordList)
def drawit(newWord, d, angle):
newWordLs = list(newWord)
for i in range(len(newWordLs)):
cur_Char = newWordLs[i]
if cur_Char == 'F':
turtle.forward(d)
elif cur_Char == '+':
turtle.right(angle)
elif cur_Char == '-':
turtle.left(angle)
else:
i = i+1
#sample test of dragon curve
def main():
createWord(10, 'FX', {'X':'X+YF','Y':'FX-Y'})
drawit('FX+YF+FX-YF+FX+YF-FX-YF+FX+YF+FX-YF-FX+YF-FX-YF', 20, 90)
if __name__=='__main__': main()

newWord is locally scoped inside of createWord(), so after createWord() is finished, newWord disappears.
Consider creating newWord in the global scope so you can modify it with createWord - or better yet, let createWord() return a value, and set newWord to that value.
I would think that printing "word" and then using it as a parameter in drawit would result in the same thing as using a variable.
It does, but if you want to change the length of your dragon curve, you'll have to copy/paste the string every time instead of simply changing the value of max_it.
Edit: My solution with some sexy recursion (=
import turtle
def dragon_build(turtle_string, n):
""" Recursively builds a draw string. """
""" defining f, +, -, as additional rules that don't do anything """
rules = {'x':'x+yf', 'y':'fx-y','f':'f', '-':'-', '+':'+'}
turtle_string = ''.join([rules[x] for x in turtle_string])
if n > 1: return dragon_build(turtle_string, n-1)
else: return turtle_string
def dragon_draw(size):
""" Draws a Dragon Curve of length 'size'. """
turtle_string = dragon_build('fx', size)
for x in turtle_string:
if x == 'f': turtle.forward(20)
elif x == '+': turtle.right(90)
elif x == '-': turtle.left(90)
def main():
n = input("Size of Dragon Curve (int): ")
dragon_draw(n)
if __name__ == '__main__': main()