I am trying to train a CNN in tensorflow (keras) with different learning rates per layer. As this option is not included in tensorflow i am trying to modify an already existing optimizer like suggested in this github comment.
When I simply copy the source code of the SGD optimizer that i found in
"C:\Users\user\Anaconda3\envs\tf_gpu\Lib\site-packages\tensorflow_core\python\keras\optimizers.py"
(without any modifications) into my script and I try to run it I get the following error:
TypeError Traceback (most recent call last)
<ipython-input-2-757ea125a090> in <module>
----> 1 opti = SGD(lr=0.001, momentum=0.9, nesterov=False)
<ipython-input-1-dad895e21911> in __init__(self, lr, momentum, decay, nesterov, **kwargs)
63
64 def __init__(self, lr=0.01, momentum=0., decay=0., nesterov=False, **kwargs):
---> 65 super(SGD, self).__init__(**kwargs)
66 with K.name_scope(self.__class__.__name__):
67 self.iterations = K.variable(0, dtype='int64', name='iterations')
TypeError: __init__() missing 1 required positional argument: 'name'
Do I need to make modifications directly in the source code and compile tensorflow from source or am I just doing something wrong?
After attempting to install tensorflow (with GPU support) and getting a lot of errors with CuDNN and CUDA I ended up installing it using
conda install tensorflow-gpu
which worked perfectly, so I am a bit scared to uninstall the now running tensorflow version and compile it from source.
Also, if you know other ways how to modify the learning rate for each layer individually using tensorflow please let me know.
This could be from a bad implementation.
I solved this by simply adding a name argument name='sgd'.
opti = SGD(lr=0.001, momentum=0.9, nesterov=False, name='sgd')
Let me know if that works
Related
I am trying to build a neural network in Python for solving PDEs, and, as such, I have had to write custom training steps. My training function looks like this:
...
tf.enable_eager_execution()
class PDENet:
...
def train_step():
input = self.input
with tf.GradientTape() as tape, tf.Session() as sess:
tape.watch(input)
output = self.model(input)
self.loss = self.pde_loss(output) # (network does not use training data)
grad = tape.gradient(self.loss, self.model.trainable_weights)
self.optimizer.apply_gradients([(grad, self.model)])
...
Due to my hardware, I have no choice but to use tensorflow==1.12.0 and keras==2.2.4.
When I run this code, I get "RuntimeError: Attempting to capture an EagerTensor without building a function". I have seen other posts about this, but all of the answers say to update tensorflow/keras, which I can't, use "tf.enable_eager_execution()", which I've already done, and "tf.disable_v2_behavior()", which is nonexistent on older versions of tensorflow. Is there anything else I can do to solve this problem? The error makes me think tensorflow wants me to add #tf.function, but that feature also doesn't seem to exist in tensorflow 1.
I'm trying to implement a simple recurrent network using TensorFlow, but am receiving the above error. I've looked through several answers related to the:
"Failed to convert a NumPy array to a Tensor (Unsupported object type ____)"
error, but none so far have addressed "tensorflow.python.framework.ops.EagerTensor" as the unsupported type. I am receiving this error after trying to implement code from this tutorial (albeit with a different data-set).
The error occurs on the history = model.fit line:
# Define the network
epochs_qty = 50
batch_size_qty = 72
model = Sequential()
model.add(LSTM(epochs_qty, input_shape = (train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss = 'mae', optimizer = 'adam')
# Fit the network
history = model.fit(train_X, train_y, epochs = epochs_qty, batch_size = batch_size_qty, validation_data = (test_X, test_y), verbose = 2, shuffle = False)
The data sets have the following shapes:
print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)
>> (1762, 1, 2) (1762,) (588, 1, 2) (588,)
I am running the following versions:
Python 3.7.9
Windows 10
tensorflow-gpu-2.3.1
CUDA Toolkit 10.1 Update 1
cuDNN v8.0.3 for CUDA 10.1
I have tried disabling eager execution, but this leads to a pile of additional errors, and does not seem optimal for future code development.
Also, I have tried running this code both locally and through a jupyter notebook. Both result in the exact same error, so it seems like my software setup is not the issue.
Can anyone please suggest where to look next for the cause of this error?
As per the tensorflow team suggestion, I'm getting used to tensorflow's eager execution with tf.keras. However, whenever I train a model, I receive a warning (EDIT: actually, I receive this warning repeated many times, more than once per training step, flooding my standard output):
E tensorflow/core/common_runtime/bfc_allocator.cc:373] tried to deallocate nullptr
The warning doesn't seem to affect the quality of the training but I wonder what it means and if it is possible to get rid of it.
I use a conda virtual environment with python 3.7 and tensorflow 1.12 running on a CPU. (EDIT: a test with python 3.6 gives the same results.) A minimal code that reproduces the warnings follows. Interestingly, it is possible to comment the line tf.enable_eager_execution() and see that the warnings disappear.
import numpy as np
import tensorflow as tf
tf.enable_eager_execution()
N_EPOCHS = 50
N_TRN = 10000
N_VLD = 1000
# the label is positive if the input is a number larger than 0.5
# a little noise is added, just for fun
x_trn = np.random.random(N_TRN)
x_vld = np.random.random(N_VLD)
y_trn = ((x_trn + np.random.random(N_TRN) * 0.02) > 0.5).astype(float)
y_vld = ((x_vld + np.random.random(N_VLD) * 0.02) > 0.5).astype(float)
# a simple logistic regression
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(1, input_dim=1))
model.add(tf.keras.layers.Activation('sigmoid'))
model.compile(
optimizer=tf.train.AdamOptimizer(),
# optimizer=tf.keras.optimizers.Adam(), # doesn't work at all with tf eager execution
loss='binary_crossentropy',
metrics=['accuracy']
)
# Train model on dataset
model.fit(
x_trn, y_trn,
epochs=N_EPOCHS,
validation_data=(x_vld, y_vld),
)
model.summary()
Quick solutions:
It did not appear when I ran the same script in TF 1.11 while the optimization was performed to reach the same final validation accuracy on a synthetic dataset.
OR
Suppress the errors/warning using the native os module (adapted from https://stackoverflow.com/a/38645250/2374160). ie; by setting the Tensorflow logging environment variable to not show any error messages.
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
More info:
Solving this error in the correct way may require familiarity with MKL library calls and its interfacing on Tensorflow which is written in C (this is beyond my current TF expertise)
In my case, this memory deallocation error occurred whenever the
apply_gradients() method of an optimizer was called. In your script, it is called when the model is being fitted to the training data.
This error is raised from here: tensorflow/core/common_runtime/mkl_cpu_allocator.h
I hope this helps as a temporary solution for convenience.
I am a novice programmer trying to follow this guide.
However, I ran across an issue. The guide says to define the loss function as:
def loss(labels, logits):
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
This gives me the following error:
sparse_categorical_crossentropy() got an unexpected keyword argument
'from_logits'
which I take to mean that from_logits is an argument not specified in the function, which is supported by the documentation, which that tf.keras.losses.sparse_categorical_crossentropy() has only two possible inputs.
Is there a way to specify that logits are being used or is that even necesarry?
I had the same problem while working through the tutorial. I changed the code from
def loss(labels, logits):
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
to
def loss(labels, logits):
return tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits)
and this resolved the issue without having to install tf-nightly.
The from_logits parameter is introduced in Tensorflow 1.13.
You can compare 1.12 and 1.13 with these urls:
https://github.com/tensorflow/tensorflow/blob/r1.12/tensorflow/python/keras/losses.py
https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/python/keras/losses.py
1.13 is not released at the time of writing. This is why the tutorial starts with the line
!pip install -q tf-nightly
I am really new to Keras so forgive me if my query is a bit silly. I installed Keras in my system using the default methods and it works fine. I want to add a new optimizer to Keras so that I can easily mention "optimizer = mynewone " under the model.compile function. How do I go about changing the " optimizer.py " code in Keras and ensuring that the change is reflected on my Keras environment. Here is what I tried:
Suppose I change the optimizer name from rmsprop to rmsprops in the code I get the following error:
model.compile(loss='binary_crossentropy', optimizer='rmsprops', metrics= ['accuracy'])
Traceback (most recent call last):
File "<ipython-input-33-40773d534448>", line 1, in <module>
model.compile(loss='binary_crossentropy', optimizer='rmsprops', metrics=['accuracy'])
File "/home/kiran/anaconda/lib/python3.5/site-packages/keras/models.py", line 589, in compile
**kwargs)
File "/home/kiran/anaconda/lib/python3.5/site-packages/keras/engine/training.py", line 469, in compile
self.optimizer = optimizers.get(optimizer)
File "/home/kiran/anaconda/lib/python3.5/site-packages/keras/optimizers.py", line 614, in get
# Instantiate a Keras optimizer
File "/home/kiran/anaconda/lib/python3.5/site-packages/keras/utils/generic_utils.py", line 16, in get_from_module
str(identifier))
ValueError: Invalid optimizer: rmsprops
Then when I click on optimizers.py I get the code developed by Keras in my environment. After that in the code I replaced all "rmsprop" keywords with "rmsprops" and saved the file. So I thought I must have the updated optimizers.py in my system. But when I go back to my original file and run model.compile it throws the same error.
Any help would be really appreciated. Thanks in advance.
I think your approach is complicated and it doesn't have to be. Let's say you implement your own optimizer by subclassing keras.optimizers.Optimizer:
class MyOptimizer(Optimizer):
optimizer functions here.
Then to instantiate it in your model you can do this:
myOpt = MyOptimizer()
model.compile(loss='binary_crossentropy', optimizer=myOpt, metrics= ['accuracy'])
Just pass an instance of your optimizer as the optimizer parameter of model.compile and that's it, Keras will now use your optimizer.
Are you sure that it is a new optimizer that you want? Not a custom objective function? Objectives can be custom it's easy to define, optimizers are trickier.
There is already a huge number of optimizers with a lot of parameters. However if you really want to go down that road I would advise you to go to tensorflow!
Then you will be able to use this in Keras
It's all I can do for you, but maybe there is another way that I don't know of.