I am trying to create a q-learning chess engine where the output of the last layer of the neural network (the density is equal to the number of legal moves) is run through a argmax() function which returns an integer that I am using as an index for the array where the legal moves are stored. Here is part of my code:
#imports
env = gym.make('ChessAlphaZero-v0') #builds environment
obs = env.reset()
type(obs)
done = False #game is not won
num_actions = len(env.legal_moves) #array where legal moves are stored
obs = chess.Board()
model = models.Sequential()
def dqn(board):
#dense layers
action = layers.Dense(num_actions)(layer5)
i = np.argmax(action)
move = env.legal_moves[i]
return keras.Model(inputs=inputs, outputs=move)
But when I run the code I get the following error:
TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
Any code examples would be appreciated, thanks.
The correct way to build a model and forward an input in keras is this:
1. Building the model
model = models.Sequential()
model.add(layers.Input(observation_shape))
model.add(layers.Dense(units=128, activation='relu'))
model.add(layers.Dense(units=num_actions, activation='softmax'))
return model
or
inputs = layers.Input(observation_shape)
x = layers.Dense(units=128, activation='relu')(inputs)
outputs = layers.Dense(units=num_actions, activation='softmax')(x)
model = keras.Model(inputs, output)
Both ways are equal.
2. Forward an observation & Get the best possible action
action_values = model.predict(observation)
best_action_index = tf.argmax(action_values)
best_action = action_values[best_action_index]
Implementing DQN by yourself in keras can be quite frustrating. You might wanna use a DRL framework such as tf_agents that has implementations of lots of agents:
https://www.tensorflow.org/agents
This repository contains a clean and easy to understand implementation of DQN for openai gym environments. Also, it contains examples of using tf_agents library as well for more complex agents:
https://github.com/kochlisGit/Tensorflow-DQN
Related
import tensorflow as tf
import keras
import tensorflow.keras.layers as tfl
from tensorflow.keras.layers.experimental.preprocessing import RandomFlip, RandomRotation
I am trying to figure out which I should use for Data Augmentation. In the documentation, there is:
tf.keras.layers.RandomFlip and RandomRotation
Then we have in tf.keras.layers.experimental.preprocessing the same things, randomFlip and RandomRotation.
Which should I use? I've seen guides that use both.
This is my current code:
def data_augmenter():
data_augmentation = tf.keras.Sequential([
tfl.RandomFlip(),
tfl.RandomRotation(0.2)
])
return data_augmentation
and this is a part of my model:
def ResNet50(image_shape = IMG_SIZE, data_augmentation=data_augmenter()):
input_shape = image_shape + (3,)
# Remove top layer in order to put mine with the correct classification labels, get weights for imageNet
base_model = tf.keras.applications.resnet_v2.ResNet50V2(input_shape=input_shape, include_top=False, weights='imagenet')
# Freeze base model
base_model.trainable = False
# Define input layer
inputs = tf.keras.Input(shape=input_shape)
# Apply Data Augmentation
x = data_augmentation(inputs)
I am a bit confused here..
If you find something in an experimental module and something in the same package by the same name, these will typically be aliases of one another. For the sake of backwards compatibility, they don't remove the experimental one (at least not for a few iterations.)
You should generally use the non-experimental one if it exists, since this is considered stable and should not be removed or changed later.
The following page shows Keras preprocessing exerimental. If it redirects to the preprocessing module, it's an alias. https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing
I'm currently working on a reinforcement learning model, and have come across an issue while trying to create a DQN to work within my custom environment.
While instantiating the DQN agent with this line:
dqn = DQNAgent(model=model, memory=memory, policy=policy,
nb_actions=(None,actions), nb_steps_warmup=10, target_model_update=1e-2)
Note that actions = 3 (integer).
I get the error code:
raise ValueError(f'Model output "{model.output}" has invalid shape. DQN expects a model that has one dimension for each action, in this case {self.nb_actions}.')
ValueError: Model output "Tensor("dense_2/BiasAdd:0", shape=(?, 3), dtype=float32)" has invalid shape. DQN expects a model that has one dimension for each action, in this case 3
After digging into the DQN files, I noticed that the error is arising by the fact that
(?,3) != (None,3)
From my understanding the question mark is simply a placeholder to represent an unknown amount of data points. So why does it have a problem with None not being equal to it and how do I fix this?
Thanks
I believe you are using the keras-rl library.
The nb_actions parameter should be an integer that specifies the number of total possible actions that you can perform in your environment.
Taking the Taxi-v2 environment from OpenAI Gym as an example, we can obtain the number of total possible actions via:
import gym
ENV_NAME = "Taxi-v2"
env = gym.make(ENV_NAME)
nb_actions = env.action_space.n
If your actions variable is the number of actions, you can try:
dqn = DQNAgent(model=model, memory=memory, policy=policy, nb_actions=actions, nb_steps_warmup=10, target_model_update=1e-2)
I am trying to use a frozen, pretrained, DeepLabv3 model in a larger tf.keras training pipeline, but have been having trouble figuring out how to use it as a tf.keras Model. I am trying to use tf.keras as I feel there would be a slowdown using a feed_dict (the only way I know of to use a frozen graph) in the middle of multiple forward passes. The deeplab model referenced in the code below is built in regular keras (as opposed to tf.contrib.keras)
from keras import backend as K
# Create, compile and train model...
frozen_graph = freeze_session(K.get_session(),
output_names=[out.op.name for out in deeplab.outputs])
tf.train.write_graph(frozen_graph, "./", "my_model.pb", as_text=False)
graph = load_graph("my_model.pb")
# We can verify that we can access the list of operations in the graph
for op in graph.get_operations():
print(op.name)
# prefix/Placeholder/inputs_placeholder
# ...
# prefix/Accuracy/predictions
# We access the input and output nodes
x = graph.get_tensor_by_name("prefix/input_1:0")
y = graph.get_tensor_by_name("prefix/bilinear_upsampling_2/ResizeBilinear:0")
# We launch a Session
with tf.Session(graph=graph) as sess:
print(graph)
model2 = models.Model(inputs=x,outputs=y)
model2.summary()
and i get an error
ValueError: Input tensors to a Model must come from `tf.layers.Input`. Received: Tensor("prefix/input_1:0", shape=(?, 512, 512, 3), dtype=float32) (missing previous layer metadata).
I feel like I've seen others replace the input tensor with an Input Layer to trick tf.keras into building the graph, but after a few hours I am feeling stuck. Any help would be appreciated!
You can recreate the model object from its config. See the from_config method here https://keras.io/models/about-keras-models/.
The config is stored and loaded back by the save_model/load_model functions. I am not familiar with freeze_session.
I'm trying to implement a CNN using theano/lasagne.
I've made a neural network but can't figure out how to train it with the current state.
This is how I'm trying to get the output of the network with the current_states as input.
train = theano.function([input_var], lasagne.layers.get_output(l.out))
output = train(current_states)
However I get this error:
theano.compile.function_module.UnusedInputError: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 0 is not part of the computational graph needed to compute the outputs: inputs.
To make this error into a warning, you can pass the parameter on_unused_input='warn' to theano.function. To disable it completely, use on_unused_input='ignore'.
Why is current_states not used?
I want to get the output of the model on the current_states. How do I do this?
(the CNN build code: http://pastebin.com/Gd35RncU)
The following code snippet works for me:
import lasagne, theano
import theano.tensor as T
import numpy as np
input_var = theano.tensor.tensor4('inputs')
l_out = build_cnn(input_var)
train = theano.function([input_var], lasagne.layers.get_output(l_out))
x = np.random.randn(10, 4, 80, 80).astype(theano.config.floatX)
train(x)
You didn't post your entire code, but you can check to see if in your script you are passing in the input_var variable to your build_cnn function. If you do not, then input_var will not be part of your computational graph, which is why Theano is raising the error.
In python when I want to get the data from a layer using caffe I have the following code
input_image = caffe.io.load_image(imgName)
input_oversampled = caffe.io.resize_image(input_image, self.net.crop_dims)
prediction = self.net.predict([input_image])
caffe_input = np.asarray(self.net.preprocess('data', prediction))
self.net.forward(data=caffe_input)
data = self.net.blobs['fc7'].data[4] // I want to get this value in lua
Hoever when I'm using torch I'm a bit stuck since I don't know how to perform the same action.
Currently I have the following code
require 'caffe'
require 'image'
net = caffe.Net('/opt/caffe/models/bvlc_reference_caffenet/deploy.prototxt', '/opt/caffe/models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel')
img = image.lena()
dest = torch.Tensor(3, 227,227)
img = image.scale(dest, img)
img = img:resize(10,3,227,227)
output = net:forward(img:float())
conv_nodes = net:findModules('fc7') -- not working
Any help would be appreciated
First of all please note that torch-caffe-binding (i.e the tool you use with require 'caffe') is a direct wrapper around Caffe library thanks to LuaJIT FFI.
This means that it allows you to conveniently do a forward or backward with a Torch tensor, but behind the scenes these operations are made on a caffe::Net and not on a Torch nn network.
So if you want to manipulate a plain Torch network what you should use is the loadcaffe library which fully converts the network into a nn.Sequential:
require 'loadcaffe'
local net = loadcaffe.load('net.prototxt', 'net.caffemodel')
Then you can use findModules. However please note that you cannot use their initial label anymore (like conv1 or fc7) as they are discarded after convert.
Here fc7 (= INNER_PRODUCT) corresponds to the N-1 linear transformation. So you can get it as follow:
local nodes = net:findModules('nn.Linear')
local fc7 = nodes[#nodes-1]
Then you can read the data (weights and biases) via fc7.weight and fc7.bias - these are regular torch.Tensor-s.
UPDATE
As of commit 2516fac loadcaffe now saves layer names in addition. So to retrieve the 'fc7' layer you can now do something like:
local fc7
for _,m in pairs(net:listModules()) do
if m.name == 'fc7' then
fc7 = m
break
end
end