I am trying to compute the input signal "maximizing" the activation of a given neuron of an encoder NN (the goal is to understand what my latent features are modelling).
I wrote a little python script which loads the .h5 file with the trained encoder model and builds a tensorflow graph to compute iteratively the "best activation signal".
It seems like my tensorflow implementation is not right. Despite the fact that I run tf.initialize_all_variables(), a FailedPreconditionError: Attempting to use uninitialized value X error is raised.
I am a little new in the use of tensorflow without using keras so this may be a trivial mistake but I could really use some help on this. Here is my code. Thanks a lot.
import tensorflow as tf
import tensorflow.keras as keras
import numpy as np
import matplotlib.pyplot as plt
input_sequence_size = 20
input_dim = 4
encoding_dim = 10
model_save = 'siple_autoencoder_encoder.h5'
model = keras.models.load_model(model_save)
lambda_param = 0.1
n_steps = 100
X = tf.Variable(tf.random_uniform([1, input_sequence_size * input_dim], -1.0, 1.0), name = 'X')
prediction = model.predict(X, steps = 1)
y = tf.gather_nd(prediction, [[0]], batch_dims=0, name=None)
gradient = tf.gradients(y, [X])[0]
step = tf.assign(X, X + lambda_param * gradient)
init = tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
# output = y.eval()
for i in range(n_steps):
sess.run(step)
activation_signal_1 = X.eval()
Related
I have several DistributionLambda layers as the outputs of one model, and I would like to make a Concatenate-like operation into a new layer, in order to have only one output that is the mix of all the distributions, assuming they are independent. Then, I can apply a log-likelihood loss to the output of the model. Otherwise, I cannot apply the loss over a Concatenate layer, because it lost the log_prob method. I have been trying with the Blockwise distribution, but with no luck so far.
Here an example code:
from tensorflow.keras import layers
from tensorflow.keras import models
from tensorflow.keras import optimizers
from tensorflow_probability import distributions
from tensorflow_probability import layers as tfp_layers
def likelihood_loss(y_true, y_pred):
"""Adding negative log likelihood loss."""
return -y_pred.log_prob(y_true)
def distribution_fn(params):
"""Distribution function."""
return distributions.Normal(
params[:, 0], math.log(1.0 + math.exp(params[:, 1])))
output_steps = 3
...
lstm_layer = layers.LSTM(10, return_state=True)
last_layer, l_h, l_c = lstm_layer(last_layer)
lstm_state = [l_h, l_c]
dense_layer = layers.Dense(2)
last_layer = dense_layer(last_layer)
last_layer = tfp_layers.DistributionLambda(
make_distribution_fn=distribution_fn)(last_layer)
output_layers = [last_layer]
# Get output sequence, re-injecting the output of each step
for number in range(1, output_steps):
last_layer = layers.Reshape((1, 1))(last_layer)
last_layer, l_h, l_c = lstm_layer(last_layer, initial_state=lstm_states)
# Storing state for next time step
lstm_states = [l_h, l_c]
last_layer = tfp_layers.DistributionLambda(
make_distribution_fn=distribution_fn)(dense_layer(last_layer))
output_layers.append(last_layer)
# This does not work
# last_layer = distributions.Blockwise(output_layers)
# This works for the model but cannot compute loss
# last_layer = layers.Concatenate(axis=1)(output_layers)
the_model = models.Model(inputs=[input_layer], outputs=[last_layer])
the_model.compile(loss=likelihood_loss, optimizer=optimizers.Adam(lr=0.001))
The problem is your Input, not your output layer ;)
Input:0 is referenced in your error message.
Could you try to be more specific about your input?
To calculate the second derivative of a neural network output with respect to the input using a keras model, I implemented codes used in those 2 posts :
Keras: using the sum of the first and second derivatives of a model as final output
Second derivative in Keras
However, for both techniques, the second derivative is always equal to 0. Here is the problem reproduced on a simple regression of the quadratic function:
import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.backend import set_session
from tensorflow.keras import datasets, layers, models
from tensorflow.keras import optimizers
import numpy as np
x = np.linspace(0, 1, 1000)
y = x**2
def model_regression():
model = tf.keras.Sequential([
layers.Dense(1024, input_dim=1, activation="relu"),
layers.Dense(1, activation="linear")])
model.compile(optimizer=optimizers.Adam(lr=0.001),
loss='mean_squared_error')
return model
my_model = model_regression()
my_model.fit(x, y, epochs=10, batch_size=20)
y_pred = my_model.predict(x)
#### Technique 1
first_input = K.gradients(my_model.output, my_model.input)
second_input = K.gradients(first_input, my_model.input)
iterate_first = K.function([my_model.input], [first_input])
iterate_second = K.function([my_model.input], [second_input])
#### Technique 2
# def grad(y, x):
# print(y.shape)
# return layers.Lambda(lambda z: K.gradients(z[0], z[1]), output_shape=[1])([y, x])
# derivative1 = grad(my_model.output, my_model.input)
# derivative2 = grad(derivative1, my_model.input)
# iterate_first = K.function([my_model.input], [derivative1])
# iterate_second = K.function([my_model.input], [derivative2])
first_derivative, second_derivative = [], []
for i in range(x.shape[0]):
first_derivative.append(iterate_first(np.array([[x[i]]]))[0][0][0])
second_derivative.append(iterate_second(np.array([[x[i]]]))[0][0][0])
Here is the graphical result, as you can see the second derivative is always equal to 0 while it should approximatively be equal to 2.
How can I compute correctly the second derivative of my neural network in keras ?
I have a very simple Keras model and I want to compute the gradient of the different layers using TensorFlow.
I start by creating the computational graph in the first cell of a Jupyter notebook.
Here is the code of the computational graph:
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.backend as K
import numpy as np
from tensorflow.keras.layers import Dense, Input, Layer
from tensorflow.keras.models import Model
input_tensor = Input(shape=(20,), name="input")
print(input_tensor.name)
hidden = Dense(100, activation='relu')(input_tensor)
out1 = Dense(10, activation='relu', name="out1")(hidden)
model = Model(inputs=input_tensor, outputs=[out1])
grad = []
for i in range(4):
grad.append(tf.gradients(out1, model.trainable_weights[i]))
model.compile(loss={"out1": "mse"},
optimizer=tf.train.AdamOptimizer(learning_rate=0.001))
np.random.seed(0)
X = np.random.random((3, 20)).astype(np.float32)
Y = np.random.random((3, 10)).astype(np.float32)
model.fit(x={'input' : X}, y={'out1' : Y}, batch_size=1, epochs=10)
Then each time I run the tf.gradients operator, I get a different gradient vector (the gradient changes). This result is not reasonable. Where is the problem in my code?
And here is the code for the created Session:
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
out_grad = sess.run(grad, feed_dict={'input:0':X})
print(out_grad)
I solved this problem by using tf.keras.backend.get_session().
I think that the problem is that tf.global_variables_initializer() reinitialize the training weights each time it is called.
When I run this code multiple times, I get consistent results.
sess = tf.keras.backend.get_session({'input:0':X})
out_grad = sess.run(grad, feed_dict={'input:0':X})
I'm trying to get BatchNormalization working properly in Keras (2.2.4), and haven't had luck. Its behavior seems inconsistent across model.fit() and model.predict()/evaluate().
My original problem was in the context of a complex GAN setup with various layers that were switching between frozen and unfrozen. In an attempt to grok why things were failing, I created this toy example, in which there is only a single BatchNormalization layer trying to learn the identity function, and there is no more freezing/unfreezing nonsense:
import numpy as np
import keras
from keras.layers import Input, BatchNormalization
if __name__ == '__main__':
x = np.random.uniform(low=-1.0, high=1.0, size=(20,1,))
y = x
ip = tx =Input(shape=[1,])
tx = BatchNormalization()(tx)
mod = keras.Model(inputs=ip, outputs=tx)
mod.compile(optimizer=keras.optimizers.SGD(lr=0.01), loss="mse")
mod.fit(x, y, epochs=2000)
print("mod evaluate", mod.evaluate(x, y))
I also created a pure tensorflow implementation that tries to do the exact same thing:
import tensorflow as tf
import numpy as np
if __name__ == '__main__':
x = np.random.uniform(low=-1.0, high=1.0, size=(20,1))
y = x
input = tx = tf.Variable(initial_value=x)
is_training = tf.Variable(initial_value=False)
tx = tf.layers.batch_normalization(tx, training=is_training)
output = tx
train_output = tf.placeholder(tf.float32, shape=(None, 1))
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
loss = tf.losses.mean_squared_error(train_output, output)
train_op = tf.train.MomentumOptimizer(learning_rate=0.01, momentum=0.9).minimize(loss)
train_op = tf.group([train_op, update_ops])
sess = tf.Session()
sess.run(tf.global_variables_initializer())
is_training.load(False, sess)
py = sess.run(output)
print("py", py)
print("mse", np.mean(np.square(py - x)))
print("training...")
for i in range(2000):
is_training.load(True, sess)
sess.run(train_op, {train_output: y})
is_training.load(False, sess)
py = sess.run(output)
print("step", i, "mse", np.mean(np.square(py - x)))
is_training.load(False, sess)
py = sess.run(output)
print("mse", np.mean(np.square(py - x)))
I expect the Keras and Tensorflow results to be similar. But the Keras code prints out a MSE error approx 0.0002 at the end, while showing a loss of about 1e-12 during training. The Tensorflow code gives a way lower error, at approx 1e-16 for both training and test. The Tensorflow result is the expected one, since it should be trivial to learn the identity function, and once the moving mean and variance of BN converge, the results should be identical across train/test.
Why is this happening? Why is the behavior inconsistent across Tensorflow and Keras, and how can I get a lower error using pure Keras? Any insight into the situation would be appreciated.
I try to divide my neural network model and restore() function with setting random weights to zero.
Here's the model code: http://pastebin.com/TqN6kkeb
(it works properly).
And here's the function:
from __future__ import print_function
import tensorflow as tf
tf.GraphKeys.VARIABLES = tf.GraphKeys.GLOBAL_VARIABLES
import random
from LogReg import accuracy
from LogReg import W
from LogReg import x,y
# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
def restore(model_file):
with tf.Session() as sess:
new_saver = tf.train.import_meta_graph(model_file + ".meta")
new_saver.restore(sess, model_file)
with tf.variable_scope("foo", reuse=True):
temp_var = tf.get_variable("W")
size_2a = tf.get_variable("b")
size_2 = tf.shape(size_2a).eval()[0]
size_1 = tf.shape(temp_var).eval()[0]
ones_mask = tf.Variable(tf.ones([size_1, size_2]))
arg = random.sample(xrange(size_1), size_1/2)
index_num=tf.convert_to_tensor(arg, dtype=tf.int32)
print("om", ones_mask)
print("index", index_num)
print(W)
zeroes = tf.zeros([size_1/2, size_2])
update = tf.scatter_update(ones_mask, index_num, zeroes)
print(update)
assign_op = W.assign(tf.mul(W, update))
sess.run(update)
sess.run(assign_op)
init_op = tf.global_variables_initializer()
sess.run(init_op)
new_saver.save(sess, model_file)
print("Accuracy_new:", accuracy.eval({x: mnist.test.images, y:mnist.test.labels}))
restore('./MyModel2')
The problems are:
1) is that it keeps writing me
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value Variable in this line:
update = tf.scatter_update(ones_mask, index_num, zeroes)
no matter what. I have read these topics: Prettytensor: Attempting to use uninitialized value and Update a subset of weights in TensorFlow (and many others), but advices from there didn't help to fix my bug.
And I don't understand, what's the problem with the initialization as long as I run tf.global_variables_initializer();
2) all of the weights seem to be setting to zero instead of the half, and I can't understand why.
Please, help, I really stuck.
Just for the record (and others finding this post), the method name has changed, as per the page here: https://www.tensorflow.org/versions/r0.10/how_tos/variables/#initialization
you should run the initialize_all_variables() method like this:
import tensorflow as tf
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)