I'm trying to understand how a with block behaves in the above scenarios. I assume to always have one graph and one session only. I understand that I have (at least?) 2 ways to work using a session in a with block:
Example 1 : using as_default() that creates a context manager:
Same documentation says:
Use with the with keyword to specify that calls to
tf.Operation.run or tf.Tensor.eval should be executed in this
session.
# Create session
sess = tf.Session()
# Enter with block on a new context manager
with sess.as_default():
# Train: Following line should result calling tf.Operation.run
sess.run([optimizer, loss], feed_dict={x: x_train, y: y_train)
# Eval: Following line should result calling tf.Tensor.eval
sess.run([loss], feed_dict={x: x_eval, y: y_eval)
Example 2 : with block on session as stated in same documentation in a lower section:
Alternatively, you can use with tf.Session(): to create a session that
is automatically closed on exiting the context, including when an
uncaught exception is raised.
# Enter with block on session instead of Context Manager
with tf.Session() as sess:
# Train: Following line seems calling tf.Operation.run as per my test
sess.run([optimizer, loss], feed_dict={x: x_train, y: y_train)
# Eval: Following is unclear
sess.run([loss], feed_dict={x: x_eval, y: y_eval)
I would like to understand what is the correct usage as I see GitHub examples of both cases but of course without the results. In my tests both Example 1 and 2 works for training. For evaluation it seems there is a difference that I can't understand. It exceeds my knowledge to browse the Tensorflow source. Can someone please explain?
They do slightly different things, so each may or may not be correct depending on the usage. tf.Session.as_default() will just ensure that the session is set as the default ones, so calls to eval and run will use that session by default:
import tensorflow as tf
sess = tf.Session()
with sess.as_default():
print(sess is tf.get_default_session()) # True
However, as stated in the documentation, tf.Session.as_default() will not automatically close the session after the with block. If you want that, you can use the session itself as the context manager:
import tensorflow as tf
with tf.Session() as sess:
# Do something with the session
sess.run([]) # RuntimeError: Attempted to use a closed Session.
However, although (from my point of view) not clearly documented, using a session as context manager also makes it the default session.
import tensorflow as tf
with tf.Session() as sess:
print(sess is tf.get_default_session()) # True
What is the point of tf.Session.as_default(), then? Well, simply when you want to temporarily make a session the default one but you do not one to close it after that (in which case you should manually close it later, or use it as an outer context manager). This is probably most relevant when you have more than one open session. Consider the following case:
import tensorflow as tf
with tf.Session() as sess1, tf.Session() as sess2:
print(sess2 is tf.get_default_session()) # True
Here sess2 is the default because its context was added later (it can be considered to be "inner" to the context created by sess1). But now maybe you want to make sess1 the default for a while. However you can not use sess1 itself as context manager again:
import tensorflow as tf
with tf.Session() as sess1, tf.Session() as sess2:
# Do something with sess2
with sess1:
# RuntimeError: Session context managers are not re-entrant.
# Use `Session.as_default()` if you want to enter
# a session multiple times.
So you can switch between one and other default sessions with tf.Session.as_default():
import tensorflow as tf
with tf.Session() as sess1, tf.Session() as sess2:
with sess1.as_default():
# Do something with sess1
with sess2.as_default():
# Do something with sess2
# This is not really needed because sess2 was the default
# in the outer context but you can add it to be explicit
# Both sessions are closed at the end of the outer context
Of course, you can be extra explicit even with one session, if you want:
import tensorflow as tf
with tf.Session() as sess, sess.as_default():
# ...
Personally, I have never used tf.Session.as_default() in my actual code, but then again I have rarely needed to use multiple sessions, and I prefer to use tf.Session.run() instead of relying on the default session anyway, but that is mostly a matter of personal taste I suppose.
Related
My understandings on Sessions in Tensorflow still seem to be flawed even after reading the official documentation and this tutorial.
In particular, does tf.global_variable_initializer() initialize global variables with regard to a particular session, or for all the sessions in the program? Are there ways to "uninitialize" a variable in / during a session?
Can a tf.variable be used in multiple sessions? The answer seems to be yes (e.g. the following code), but then are there good cases where we want multiple sessions in a program, instead of a single one?
#!/usr/bin/env python
import tensorflow as tf
def main():
x = tf.constant(0.)
with tf.Session() as sess:
print(sess.run(x))
with tf.Session() as sess:
print(sess.run(x))
if __name__ == '__main__':
main()
In particular, does tf.global_variable_initializer() initialize global variables with regard to a particular session, or for all the sessions in the program?
With regards to a particular session. Check this out.
tf.reset_default_graph()
x = tf.Variable(tf.random.normal([1,5]))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
first_sess_out = sess.run(x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
second_sess_out = sess.run(x)
np.testing.assert_array_equal(first_sess_out, second_sess_out)
The assertion fails so it is per session.
Are there ways to "uninitialize" a variable in / during a session?
tf.reset_default_graph()
x = tf.Variable(tf.random.normal([1,5]))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
first_init_out = sess.run(x)
sess.run(tf.global_variables_initializer())
second_init_out = sess.run(x)
np.testing.assert_array_equal(first_init_out, second_init_out)
Apparently there is, after running tf.global_variables_initializer() the variables got reinitialized. Thus, the assertion fails.
Can a tf.Variable be used in multiple sessions? The answer seems to be yes (e.g. the following code), but then are there good cases where we want multiple sessions in a program, instead of a single one?
Yes, it can be used as you can see on the examples above. Good cases are when you want to execute the graph multiple times in a single run.
I want to use tf.print to show tensor value, but it has no result?
This is my code and is there something wrong for that:
from __future__ import print_function
import tensorflow as tf
sess = tf.InteractiveSession()
a = tf.constant([1.0, 3.0])
tf.print(a)
From the documentation of tf.Print - that's deprecated and suggests to use tf.print:
Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators.
This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:
sess = tf.Session()
with sess.as_default():
tensor = tf.range(10)
print_op = tf.print(tensor)
with tf.control_dependencies([print_op]):
out = tf.add(tensor, tensor)
sess.run(out)
Hence, if you enable the eager mode your code will work as you expected, if you want to continue using the static-graph mode you have to use sess.run
import tensorflow as tf
a = tf.constant([1.0, 3.0])
init_op = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init_op)
print(sess.run(a))
is what I'd do. Import tensorflow, set up your variables, set up and run the initializer for them, and then print the session evaluating the constant
I have a question about InteractiveSession in Tensorflow
I know tf.InteractiveSession() is just convenient syntactic
sugar for keeping a default session open and basically work the same like below:
with tf.Session() as sess:
# Do something
However, I have seen some examples online, they did't call close() at the end of the code after using InteractiveSession.
Question:
1. Would it caused any problem without closing the session like session leak?
2. How the GC work for the InteractiveSession if we don't close it?
Yes, tf.InteractiveSession is just convenient syntactic sugar for keeping a default session open.
The Session implementation has a comment
Calling this method frees all resources associated with the session.
A quick test
#! /usr/bin/env python
# -*- coding: utf-8 -*-
import argparse
import tensorflow as tf
import numpy as np
def open_interactive_session():
A = tf.Variable(np.random.randn(16, 255, 255, 3).astype(np.float32))
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
def open_and_close_interactive_session():
A = tf.Variable(np.random.randn(16, 255, 255, 3).astype(np.float32))
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
sess.close()
def open_and_close_session():
A = tf.Variable(np.random.randn(16, 255, 255, 3).astype(np.float32))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--num', help='repeat', type=int, default=5)
parser.add_argument('type', choices=['interactive', 'interactive_close', 'normal'])
args = parser.parse_args()
sess_func = open_and_close_session
if args.type == 'interactive':
sess_func = open_interactive_session
elif args.type == 'interactive_close':
sess_func = open_and_close_interactive_session
for _ in range(args.num):
sess_func()
with tf.Session() as sess:
print("bytes used=", sess.run(tf.contrib.memory_stats.BytesInUse()))
gives
"""
python example_session2.py interactive
('bytes used=', 405776640)
python example_session2.py interactive_close
('bytes used=', 7680)
python example_session2.py
('bytes used=', 7680)
"""
This provokes a session-leak, when not closing the session.Note, even when closing the session, there is currently bug in TensorFlow which keep 1280 bytes per session see Tensorflow leaks 1280 bytes with each session opened and closed?. (This has been fixed now).
Further, there is some logic in the __del__ trying to start the GC.
Interestingly, I never saw the warning
An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call InteractiveSession.close() to release resources held by the other session(s)
which seems to be implemented. It guess the only raison d'être of the InteractiveSession is its usage in Jupyter Notebookfiles or inactive shells in combination with .eval(). But I advised against using eval (see Official ZeroOut gradient example error: AttributeError: 'list' object has no attribute 'eval')
However, I have seen some examples online, they did't call close() at the end of the code after using InteractiveSession.
And I am not surprised by that. Guess how many code snippets are the without a free or delete after some malloc. Bless the OS that it frees up the memory.
Here is a great question on how to find the first occurence of Nan in a tensorflow graph:
Debugging nans in the backward pass
The answer is quite helpful, here is the code from it:
train_op = ...
check_op = tf.add_check_numerics_ops()
sess = tf.Session()
sess.run([train_op, check_op]) # Runs training and checks for NaNs
Apparently, running the training and the numerical check at the same time will result in an error report as soon as Nan is encountered for the first time.
How do I integrate this into Keras ?
In the documentation, I can't find anything that looks like this.
I checked the code, too.
The update step is executed here:
https://github.com/fchollet/keras/blob/master/keras/engine/training.py
There is a function called _make_train_function where an operation to compute the loss and apply updates is created. This is later called to train the network.
I could change the code like this (always assuming that we're running on a tf backend):
check_op = tf.add_check_numerics_ops()
self.train_function = K.function(inputs,
[self.total_loss] + self.metrics_tensors + [check_op],
updates=updates, name='train_function', **self._function_kwargs)
I'm currently trying to set this up properly and not sure whether the code above actually works.
Maybe there is an easier way ?
I've been running into the exact same problem, and found an alternative to the check_add_numerics_ops() function. Instead of going that route, I use the TensorFlow Debugger to walk through my model, following the example in https://www.tensorflow.org/guide/debugger to figure out exactly where my code produces nans. This snippet should work for replacing the TensorFlow Session that Keras is using with a debugging session, allowing you to use tfdbg.
from tensorflow.python import debug as tf_debug
sess = K.get_session()
sess = tf_debug.LocalCLIDebugWrapperSession(sess)
K.set_session(sess)
TensorFlow has two ways to evaluate part of graph: Session.run on a list of variables and Tensor.eval. Is there a difference between these two?
If you have a Tensor t, calling t.eval() is equivalent to calling tf.get_default_session().run(t).
You can make a session the default as follows:
t = tf.constant(42.0)
sess = tf.Session()
with sess.as_default(): # or `with sess:` to close on exit
assert sess is tf.get_default_session()
assert t.eval() == sess.run(t)
The most important difference is that you can use sess.run() to fetch the values of many tensors in the same step:
t = tf.constant(42.0)
u = tf.constant(37.0)
tu = tf.mul(t, u)
ut = tf.mul(u, t)
with sess.as_default():
tu.eval() # runs one step
ut.eval() # runs one step
sess.run([tu, ut]) # evaluates both tensors in a single step
Note that each call to eval and run will execute the whole graph from scratch. To cache the result of a computation, assign it to a tf.Variable.
The FAQ session on tensor flow has an answer to exactly the same question. I will just go ahead and leave it here:
If t is a Tensor object, t.eval() is shorthand for sess.run(t) (where sess is the current default session. The two following snippets of code are equivalent:
sess = tf.Session()
c = tf.constant(5.0)
print sess.run(c)
c = tf.constant(5.0)
with tf.Session():
print c.eval()
In the second example, the session acts as a context manager, which has the effect of installing it as the default session for the lifetime of the with block. The context manager approach can lead to more concise code for simple use cases (like unit tests); if your code deals with multiple graphs and sessions, it may be more straightforward to explicit calls to Session.run().
I'd recommend that you at least skim throughout the whole FAQ, as it might clarify a lot of things.
eval() can not handle the list object
tf.reset_default_graph()
a = tf.Variable(0.2, name="a")
b = tf.Variable(0.3, name="b")
z = tf.constant(0.0, name="z0")
for i in range(100):
z = a * tf.cos(z + i) + z * tf.sin(b - i)
grad = tf.gradients(z, [a, b])
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
print("z:", z.eval())
print("grad", grad.eval())
but Session.run() can
print("grad", sess.run(grad))
correct me if I am wrong
The most important thing to remember:
The only way to get a constant, variable (any result) from TenorFlow is the session.
Knowing this everything else is easy:
Both tf.Session.run() and tf.Tensor.eval() get results from the session where tf.Tensor.eval() is a shortcut for calling tf.get_default_session().run(t)
I would also outline the method tf.Operation.run() as in here:
After the graph has been launched in a session, an Operation can be executed by passing it to tf.Session.run(). op.run() is a shortcut for calling tf.get_default_session().run(op).
Tensorflow 2.x Compatible Answer: Converting mrry's code to Tensorflow 2.x (>= 2.0) for the benefit of the community.
!pip install tensorflow==2.1
import tensorflow as tf
tf.compat.v1.disable_eager_execution()
t = tf.constant(42.0)
sess = tf.compat.v1.Session()
with sess.as_default(): # or `with sess:` to close on exit
assert sess is tf.compat.v1.get_default_session()
assert t.eval() == sess.run(t)
#The most important difference is that you can use sess.run() to fetch the values of many tensors in the same step:
t = tf.constant(42.0)
u = tf.constant(37.0)
tu = tf.multiply(t, u)
ut = tf.multiply(u, t)
with sess.as_default():
tu.eval() # runs one step
ut.eval() # runs one step
sess.run([tu, ut]) # evaluates both tensors in a single step
In tensorflow you create graphs and pass values to that graph. Graph does all the hardwork and generate the output based on the configuration that you have made in the graph.
Now When you pass values to the graph then first you need to create a tensorflow session.
tf.Session()
Once session is initialized then you are supposed to use that session because all the variables and settings are now part of the session. So, there are two ways to pass external values to the graph so that graph accepts them. One is to call the .run() while you are using the session being executed.
Other way which is basically a shortcut to this is to use .eval(). I said shortcut because the full form of .eval() is
tf.get_default_session().run(values)
You can check that yourself.
At the place of values.eval() run tf.get_default_session().run(values). You must get the same behavior.
what eval is doing is using the default session and then executing run().