OutOfRangeError while training StyleGan2 network - python

I've been running into plenty of problems training my first network using the StyleGAN2 repo and after changing to a smaller GPU batch size of 2 because of only 11GB of VRAM available to me the training manages to get through 1-4 ticks before returning these OutOfRangeErrors.
Ryzen 3950x
RTX 2080ti
32GB DDR4 RAM
Windows 10
Tensorflow-gpu 1.4
Building TensorFlow graph...
Initializing logs...
Training for 25000 kimg...
tick 0 kimg 10065.1 lod 0.00 minibatch 32 time 1m 17s sec/tick 77.4 sec/kimg 605.07 maintenance 0.0 gpumem 8.6
Traceback (most recent call last):
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
return fn(*args)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[{{node GPU0/DataFetch/IteratorGetNext}}]]
[[GPU0/DataFetch/UpscaleLOD/Cast/_5109]]
(1) Out of range: End of sequence
[[{{node GPU0/DataFetch/IteratorGetNext}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_training.py", line 202, in <module>
main()
File "run_training.py", line 197, in main
run(**vars(args))
File "run_training.py", line 128, in run
dnnlib.submit_run(**kwargs)
File "C:\ML\stylegan2dv\dnnlib\submission\submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "C:\ML\stylegan2dv\dnnlib\submission\internal\local.py", line 22, in submit
return run_wrapper(submit_config)
File "C:\ML\stylegan2dv\dnnlib\submission\submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "C:\ML\stylegan2dv\training\training_loop.py", line 308, in training_loop
tflib.run(data_fetch_op, feed_dict)
File "C:\ML\stylegan2dv\dnnlib\tflib\tfutil.py", line 31, in run
return tf.get_default_session().run(*args, **kwargs)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
run_metadata)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[node GPU0/DataFetch/IteratorGetNext (defined at C:\ML\stylegan2dv\training\dataset.py:136) ]]
[[GPU0/DataFetch/UpscaleLOD/Cast/_5109]]
(1) Out of range: End of sequence
[[node GPU0/DataFetch/IteratorGetNext (defined at C:\ML\stylegan2dv\training\dataset.py:136) ]]
0 successful operations.
0 derived errors ignored.
Errors may have originated from an input operation.
Input Source operations connected to node GPU0/DataFetch/IteratorGetNext:
Dataset/IteratorV2 (defined at C:\ML\stylegan2dv\training\dataset.py:119)
Input Source operations connected to node GPU0/DataFetch/IteratorGetNext:
Dataset/IteratorV2 (defined at C:\ML\stylegan2dv\training\dataset.py:119)
Original stack trace for 'GPU0/DataFetch/IteratorGetNext':
File "run_training.py", line 202, in <module>
main()
File "run_training.py", line 197, in main
run(**vars(args))
File "run_training.py", line 128, in run
dnnlib.submit_run(**kwargs)
File "C:\ML\stylegan2dv\dnnlib\submission\submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "C:\ML\stylegan2dv\dnnlib\submission\internal\local.py", line 22, in submit
return run_wrapper(submit_config)
File "C:\ML\stylegan2dv\dnnlib\submission\submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "C:\ML\stylegan2dv\training\training_loop.py", line 208, in training_loop
reals_write, labels_write = training_set.get_minibatch_tf()
File "C:\ML\stylegan2dv\training\dataset.py", line 136, in get_minibatch_tf
return self._tf_iterator.get_next()
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 426, in get_next
output_shapes=self._structure._flat_shapes, name=name)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 1974, in iterator_get_next
output_shapes=output_shapes, name=name)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
op_def=op_def)
File "C:\Users\TE 1\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
self._traceback = tf_stack.extract_stack()

Related

InvalidArgumentError: Input to reshape is a tensor with 0 values, but the requested shape has 54912

Very beginner question, I hope that's fine
I'm trying to train this model from GitHub with the MAPS dataset and I made new .tfrecords with this code for the train set. It is based from the code here but I altered some things to make way for a different input (another MIDI file I'm just calling "tempo MIDI").
def create_train_set(tempopath, train_list, outdir, min_length, max_length):
# train_list = list of wav paths selected for
train_file_pairs = []
# find matching midi files
for wav_path in train_list:
midi_file = ''
tempo_midi_file = ''
if os.path.isfile(wav_path + '.mid'):
midi_file = wav_path + '.mid'
if os.path.isfile(wav_path + '.midi'):
midi_file = wav_path + '.midi'
if os.path.isfile(tempopath + os.path.basename(wav_path) + '_tempo.mid'):
tempo_midi_file = tempopath + os.path.basename(wav_path) + '_tempo.mid'
if os.path.isfile(tempopath + os.path.basename(wav_path) + '_tempo.midi'):
tempo_midi_file = tempopath + os.path.basename(wav_path) + '_tempo.midi'
wav_file = wav_path + '.wav'
train_file_pairs.append((wav_file, midi_file, tempo_midi_file))
train_output_name = os.path.join(outdir, 'train.tfrecord')
with tf.python_io.TFRecordWriter(train_output_name) as writer:
for idx, pair in enumerate(train_file_pairs):
print('{} of {}: {}'.format(idx, len(train_file_pairs), pair[0]))
# load the wav data
wav_data = tf.gfile.Open(pair[0], 'rb').read()
# load the midi data and convert to a notesequence
ns = midi_io.midi_file_to_note_sequence(pair[1])
tempo = midi_io.midi_file_to_note_sequence(pair[2])
# aldu = audio_label_data_utils.py
for example in aldu.process_record(
wav_data, ns, tempo, pair[0], min_length, max_length,
sample_rate):
writer.write(example.SerializeToString())
with the tf.Example as follows:
example = tf.train.Example(
features=tf.train.Features(
feature={
'id':
tf.train.Feature(
bytes_list=tf.train.BytesList(
value=[example_id.encode('utf-8')])),
'sequence':
tf.train.Feature(
bytes_list=tf.train.BytesList(
value=[ns.SerializeToString()])),
'audio':
tf.train.Feature(
bytes_list=tf.train.BytesList(value=[wav_data])),
'tempo':
tf.train.Feature(
bytes_list=tf.train.BytesList(
value=[velocity_range.SerializeToString()])),
'velocity_range':
tf.train.Feature(
bytes_list=tf.train.BytesList(
value=[velocity_range.SerializeToString()])),
}))
However, when I try to train the model, I get this error message (I marked the py scripts with a print line so I know where everything's going):
Running wav_to_spec from data.py
Running _wav_to_mel in data.py
Running wav_to_num_frames from data.py
Running wav_to_spec from data.py
Running _wav_to_mel in data.py
Running wav_to_num_frames from data.py
E0611 07:56:55.419340 8436 error_handling.py:70] Error recorded from training_loop: Input to reshape is a tensor with 0 values, but the requested shape has 54912
[[{{node Reshape_8}}]]
[[IteratorGetNext]]
I0611 07:56:55.420338 8436 error_handling.py:96] training_loop marked as finished
W0611 07:56:55.421335 8436 error_handling.py:130] Reraising captured error
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
return fn(*args)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 0 values, but the requested shape has 54912
[[{{node Reshape_8}}]]
[[IteratorGetNext]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "onsets_frames_transcription_train.py", line 128, in <module>
console_entry_point()
File "onsets_frames_transcription_train.py", line 124, in console_entry_point
tf.app.run(main)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "onsets_frames_transcription_train.py", line 120, in main
additional_trial_info=additional_trial_info)
File "onsets_frames_transcription_train.py", line 95, in run
num_steps=FLAGS.num_steps)
File "C:\Users\User\magenta\magenta\models\onsets_frames_transcription\train_util.py", line 134, in train
estimator.train(input_fn=transcription_data, max_steps=num_steps)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\tpu\tpu_estimator.py", line 2876, in train
rendezvous.raise_errors()
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\tpu\error_handling.py", line 131, in raise_errors
six.reraise(typ, value, traceback)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 693, in reraise
raise value
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\tpu\tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
saving_listeners)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
run_metadata=run_metadata)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1353, in run
raise six.reraise(*original_exc_info)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 693, in reraise
raise value
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
return self._sess.run(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1411, in run
run_metadata=run_metadata)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1169, in run
return self._sess.run(*args, **kwargs)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
run_metadata)
File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 0 values, but the requested shape has 54912
[[{{node Reshape_8}}]]
[[IteratorGetNext]]
From that, I figured the problem lies in wav_to_num_frames but this is the only code for it.
def wav_to_num_frames(wav_audio, frames_per_second):
"""Transforms a wav-encoded audio string into number of frames."""
print("Running wav_to_num_frames from data")
w = wave.open(six.BytesIO(wav_audio))
return np.int32(w.getnframes() / w.getframerate() * frames_per_second)
I didn't get this problem back when I tried training the model with tfrecords created with the original code, so I don't know what's wrong.
It turns out that problem wasn't the created .tfrecords itself but rather the size of the tensors I assigned for the newly added data. There isn't a concrete answer for this though since it's very specific to this situation.

Tensorflow object detection api training error "TypeError: Input 'y' of 'Mul' Op has type float32

EDIT2
Ok so far i have tried with python3.5 -tf 1.10 and python 2.7 tf 1.10
I m still getting this error
Traceback (most recent call last):
File "object_detection/model_main.py", line 101, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 455, in train_and_evaluate
return executor.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 594, in run
return self.run_local()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/training.py", line 695, in run_local
saving_listeners=saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1179, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1209, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 1167, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/nvidia/tensorflow/models/research/object_detection/model_lib.py", line 287, in model_fn
prediction_dict, features[fields.InputDataFields.true_image_shape])
File "/home/nvidia/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 686, in loss
keypoints, weights)
File "/home/nvidia/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 859, in _assign_targets
groundtruth_weights_list)
File "/home/nvidia/tensorflow/models/research/object_detection/core/target_assigner.py", line 481, in batch_assign_targets
anchors, gt_boxes, gt_class_targets, unmatched_class_label, gt_weights)
File "/home/nvidia/tensorflow/models/research/object_detection/core/target_assigner.py", line 180, in assign
match = self._matcher.match(match_quality_matrix, **params)
File "/home/nvidia/tensorflow/models/research/object_detection/core/matcher.py", line 239, in match
return Match(self._match(similarity_matrix, **params),
File "/home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py", line 190, in _match
_match_when_rows_are_non_empty, _match_when_rows_are_empty)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2074, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 1920, in BuildCondBranch
original_result = fn()
File "/home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py", line 153, in _match_when_rows_are_non_empty
-1)
File "/home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py", line 203, in _set_values_using_indicator
indicator = tf.cast(1-indicator, x.dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 878, in r_binary_op_wrapper
x = ops.convert_to_tensor(x, dtype=y.dtype.base_dtype, name="x")
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1028, in convert_to_tensor
as_ref=False)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1124, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 228, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 207, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 442, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 353, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected bool, got 1 of type 'int' instead.
Has anybody tried to train on TX2 or is it for my case only and i did something wrong?
ORIGINAL
Trying to train on mobilenet ssd on Jetson TX2 (I know it is not for taining but i have no better option)
followed these guides
https://towardsdatascience.com/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_locally.md
Training runs on my laptop (CPU) fine but i get the following error on my TX2
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
preferred_dtype=default_dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1040, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 883, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("Loss/Loss/huber_loss/Sub_1:0", shape=(24, 1917, 4), dtype=float32)'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "object_detection/model_main.py", line 101, in <module>
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 425, in train_and_evaluate
executor.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 504, in run
self.run_local()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 636, in run_local
hooks=train_hooks)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 355, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 824, in _train_model
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 805, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/nvidia/tensorflow/models/research/object_detection/model_lib.py", line 287, in model_fn
prediction_dict, features[fields.InputDataFields.true_image_shape])
File "/home/nvidia/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 708, in loss
weights=batch_reg_weights)
File "/home/nvidia/tensorflow/models/research/object_detection/core/losses.py", line 74, in __call__
return self._compute_loss(prediction_tensor, target_tensor, **params)
File "/home/nvidia/tensorflow/models/research/object_detection/core/losses.py", line 157, in _compute_loss
reduction=tf.losses.Reduction.NONE
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py", line 444, in huber_loss
math_ops.multiply(delta, linear))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py", line 326, in multiply
return gen_math_ops.mul(x, y, name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 4689, in mul
"Mul", x=x, y=y, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 546, in _apply_op_helper
inferred_from[input_arg.type_attr]))
TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type int32 of argument 'x'.
NOTE:
Used precompiled wheels to install tensorflow
There was an error with protobuf compiler that has been solved by
removing this line
reserved 6; (line number 104)
in ssd.proto on object_detection/protos folder
I found this solution here but i couldnt find the link
Here is the script to start training
PIPELINE_CONFIG_PATH=/home/nvidia/testtraining/models/model/ssd_mobilenet_v1_pets.config
MODEL_DIR=/home/nvidia/testtraining/models/model/
NUM_TRAIN_STEPS=50000
NUM_EVAL_STEPS=2000
python3 object_detection/model_main.py \
--pipeline_config_path=${PIPELINE_CONFIG_PATH} \
--model_dir=${MODEL_DIR} \
--num_train_steps=${NUM_TRAIN_STEPS} \
--num_eval_steps=${NUM_EVAL_STEPS} \
--alsologtostderr
Laptop TF version 1.10.0
Jetson TX2 tf version 1.6.0-rc1
I m new to Ubuntu and Tensorflow so go easy on me :)
Thanks
EDIT:
It seems like line 546, in _apply_op_helper is some sort of error handling line.
I tried to fix this error with following edit. Added these. In /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py Added these to line 236 just after define statement
import tensorflow as tf
y = tf.cast(y, x.dtype)
This created some other error message which is solved by editing /home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py line 203-204 to these
indicator = tf.cast(1-indicator, x.dtype)
return tf.add(tf.multiply(x, indicator), val * indicator)
But i m still getting error
Traceback (most recent call last):
File "object_detection/model_main.py", line 101, in <module>
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/model_main.py", line 97, in main
tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 425, in train_and_evaluate
executor.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 504, in run
self.run_local()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/training.py", line 636, in run_local
hooks=train_hooks)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 355, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 824, in _train_model
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 805, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/nvidia/tensorflow/models/research/object_detection/model_lib.py", line 287, in model_fn
prediction_dict, features[fields.InputDataFields.true_image_shape])
File "/home/nvidia/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 686, in loss
keypoints, weights)
File "/home/nvidia/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 859, in _assign_targets
groundtruth_weights_list)
File "/home/nvidia/tensorflow/models/research/object_detection/core/target_assigner.py", line 481, in batch_assign_targets
anchors, gt_boxes, gt_class_targets, unmatched_class_label, gt_weights)
File "/home/nvidia/tensorflow/models/research/object_detection/core/target_assigner.py", line 180, in assign
match = self._matcher.match(match_quality_matrix, **params)
File "/home/nvidia/tensorflow/models/research/object_detection/core/matcher.py", line 239, in match
return Match(self._match(similarity_matrix, **params),
File "/home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py", line 190, in _match
_match_when_rows_are_non_empty, _match_when_rows_are_empty)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2047, in cond
orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 1897, in BuildCondBranch
original_result = fn()
File "/home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py", line 153, in _match_when_rows_are_non_empty
-1)
File "/home/nvidia/tensorflow/models/research/object_detection/matchers/argmax_matcher.py", line 203, in _set_values_using_indicator
indicator = tf.cast(1-indicator, x.dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py", line 983, in r_binary_op_wrapper
x = ops.convert_to_tensor(x, dtype=y.dtype.base_dtype, name="x")
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 950, in convert_to_tensor
as_ref=False)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1040, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 235, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 214, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 433, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 344, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected bool, got 1 of type 'int' instead.
And this one is out of my leage
I think there is a huge compatability issues and i will just install tf 1.1 instead
I m open to new ideas though
The key to your problem is here(bottom of traceback):
TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type int32 of argument 'x'.
your y has type float32, but argument - x(place where you pass this y to), needs to be int32.
Try using tf.cast(y, tf.int32) or something like that.
Sometimes there are some changes in tf/you use some older model versions. So this may happen from time to time.
So just open
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py"
find line 546, and do that cast. (using sudo vim, i guess)

How can i save tensors object to an numpy array?

I have implemented autoencoder on my custom images data for sign language recognition.Now i want to save tensors object of the output layer to an numpy array. I tried Session.run(tensor) and tensor.eval(). Here is my code.
#define model
x= tf.placeholder(tf.float32,[None,784])
y_=tf.placeholder(tf.float32,[None,6])
k=190
l=180
m=150
n=130
o=100
num_of_epoch=10
w1=tf.Variable(tf.truncated_normal([784,k],stddev=0.1))
b1=tf.Variable(tf.zeros([k]))
w2=tf.Variable(tf.truncated_normal([k,l],stddev=0.1))
b2=tf.Variable(tf.zeros([l]))
w3=tf.Variable(tf.truncated_normal([l,m],stddev=0.1))
b3=tf.Variable(tf.zeros([m]))
w4=tf.Variable(tf.truncated_normal([m,n],stddev=0.1))
b4=tf.Variable(tf.zeros([n]))
w5=tf.Variable(tf.truncated_normal([n,o],stddev=0.1))
b5=tf.Variable(tf.zeros([o]))
w6=tf.Variable(tf.truncated_normal([o,6],stddev=0.1))
b6=tf.Variable(tf.zeros([6]))
y1=tf.nn.relu(tf.matmul(x,w1)+b1)
y2=tf.nn.relu(tf.matmul(y1,w2)+b2)
y3=tf.nn.relu(tf.matmul(y2,w3)+b3)
y4=tf.nn.relu(tf.matmul(y3,w4)+b4)
y5=tf.nn.relu(tf.matmul(y4,w5)+b5)
y=tf.nn.softmax(tf.matmul(y5,w6)+b6)
cross_entropy=tf.reduce_mean(-tf.reduce_sum(y_*tf.log(y),
reduction_indices=[1]))
train_step=tf.train.GradientDescentOptimizer(0.03).minimize(cross_entropy)
init=tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for i in range(num_of_epoch):
train_data = {x:x_train,y_:y_train}
sess.run(train_step,feed_dict=train_data)
currect_prediction=tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
accuracy=tf.reduce_mean(tf.cast(currect_prediction,tf.float32))
sess.run(accuracy,feed_dict={x:x_train,y_:y_train})
currect_prediction=tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
accuracy=tf.reduce_mean(tf.cast(currect_prediction,tf.float32))
sess.run(accuracy,feed_dict= {x:x_test,y_:y_test})
y_p = tf.argmax(y, 1).eval() #this line shows me the error
print(y_p)
I am getting the below error. How can I fix this error and save tensor data to numpy array ?
Traceback (most recent call last):
File "<ipython-input-45-5e38490a3e8e>", line 1, in <module>
runfile('C:/Users/RIFAT/PycharmProjects/tensorflow_autoencoder
/autoencoderreconstruction.py',
wdir='C:/Users/RIFAT/PycharmProjects/tensorflow_autoencoder')
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\spyder\utils
\site\sitecustomize.py", line 880, in runfile
execfile(filename, namespace)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\spyder\utils
\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/RIFAT/PycharmProjects/tensorflow_autoencoder
/autoencoderreconstruction.py", line 112, in <module>
y_p = tf.argmax(y, 1).eval()
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\framework\ops.py", line 606, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\framework\ops.py", line 3928, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python\client
\session.py", line 789, in run
run_metadata_ptr)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python\client
\session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python\client
\session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python\client
\session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
InvalidArgumentError: Shape [-1,784] has negative dimensions
[[Node: Placeholder_62 = Placeholder[dtype=DT_FLOAT, shape=[?,784],
_device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op 'Placeholder_62', defined at:
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\spyder\utils\ipython
\start_kernel.py", line 231, in <module>
main()
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\spyder\utils\ipython
\start_kernel.py", line 227, in main
kernel.start()
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\ipykernel\kernelapp.py",
line 477, in start
ioloop.IOLoop.instance().start()
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\zmq\eventloop
\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tornado\ioloop.py", line
888, in start
handler_func(fd_obj, events)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tornad
\stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\zmq\eventloop
\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\zmq\eventloop
\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\zmq\eventloop
\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tornado
\stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\ipykernel
\kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\ipykernel
\kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\ipykernel
\kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\ipykernel\ipkernel.py",
line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\ipykernel\zmqshell.py",
line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\IPython
\core\interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\IPython
\core\interactiveshell.py", line 2827, in run_ast_nodes
if self.run_code(code, result):
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\IPython
\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-45-5e38490a3e8e>", line 1, in <module>
runfile('C:/Users/RIFAT/PycharmProjects/tensorflow_autoencoder
/autoencoderreconstruction.py', wdir='C:/Users/RIFAT/PycharmProjects
/tensorflow_autoencoder')
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\spyder\utils
\site\sitecustomize.py", line 880, in runfile
execfile(filename, namespace)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\spyder\utils
\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/RIFAT/PycharmProjects/tensorflow_autoencoder
/autoencoderreconstruction.py", line 62, in <module>
x= tf.placeholder(tf.float32,[None,784])
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\ops\array_ops.py", line 1530, in placeholder
return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\ops\gen_array_ops.py", line 1954, in _placeholder
name=name)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\framework\op_def_library.py", line 767, in apply_op
op_def=op_def)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\framework\ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\RIFAT\Anaconda3\lib\site-packages\tensorflow\python
\framework\ops.py", line 1269, in __init__
self._traceback = _extract_stack()
InvalidArgumentError (see above for traceback): Shape [-1,784] has
negative dimensions
[[Node: Placeholder_62 = Placeholder[dtype=DT_FLOAT, shape=[?,784],
_device="/job:localhost/replica:0/task:0/cpu:0"]()]]
That's because y is a tensor in the graph and not a variable. When you run .eval() on a variable, it gives you the current value held by that variable in that session but if you run .eval() on a tensor instead like tf.argmax(y, 1).eval() in your case then tensor flow runs the graph to that node to get the value of that node. And since in your case, it doesn't get the value of the placeholder x and y_ while running the graph, it gives the error. One way to resolve this error is by passing the values of the placeholders in your eval call like this:
tf.argmax(y, 1).eval(feed_dict= {x:x_test,y_:y_test})
But, a more preferred way is giving the context of your session to the eval call in which case it will return the value of the tensor. Ex:
tf.argmax(y, 1).eval(session = sess)
Your question is not 100% clear. But the error you are seeing is caused by the fact that you try to run the graph without the feed dict in place. To see the output of the prediction (that is with the argmax(y, 1) present) you'd simply run:
y_p = sess.run(tf.argmax(y, 1), feed_dict=train_data)
print(y_p)
But that will give you the actual predicted value (on the train data as this is fed, to get this on the test data, simply geed in the test_data). To get the probabilities you'd pull y without the argmax:
y_p = sess.run(y, feed_dict=train_data)
print(y_p)

Reading PNG files using Tensorflow

I want to build a conVNN based on my own image dataset. for that I first I need to read the files here's what I've done :
import tensorflow as tf
# Here generating a tensor of type string that include all the filename with png extention
filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once("test_png/*.png"))
# Initializing a file Reader
image_reader = tf.WholeFileReader()
# Here the file all the files mentioned ie filename queue and
# returns the the file name and the pixelvalue in form of a tensor !
imageName,imagefile= image_reader.read(filename_queue)
image = tf.image.decode_png(imagefile)
tf.global_variables_initializer()
with tf.Session() as sess:
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run([image])
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
to make things easy: the test folder contains 10 png files named 1.png 2.png .... 10.png
when I run the code I get the following :
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.FailedPreconditionError'>, Attempting to use uninitialized value matching_filenames
[[Node: matching_filenames/read = Identity[T=DT_STRING, _class=["loc:#matching_filenames"], _device="/job:localhost/replica:0/task:0/cpu:0"](matching_filenames)]]
Caused by op 'matching_filenames/read', defined at:
File "c:\users\engine\appdata\local\programs\python\python35\lib\runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\engine\appdata\local\programs\python\python35\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
app.launch_new_instance()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelapp.py", line 474, in start
ioloop.IOLoop.instance().start()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tornado\ioloop.py", line 887, in start
handler_func(fd_obj, events)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelbase.py", line 390, in execute_request
user_expressions, allow_stdin)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\zmqshell.py", line 501, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\IPython\core\interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\IPython\core\interactiveshell.py", line 2821, in run_ast_nodes
if self.run_code(code, result):
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-1-110a62c88def>", line 3, in <module>
filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once("test_png/*.png"))
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\training\input.py", line 68, in match_filenames_once
collections=[ops.GraphKeys.LOCAL_VARIABLES])
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\variables.py", line 197, in __init__
expected_shape=expected_shape)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\variables.py", line 316, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1338, in identity
result = _op_def_lib.apply_op("Identity", input=input, name=name)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value matching_filenames
[[Node: matching_filenames/read = Identity[T=DT_STRING, _class=["loc:#matching_filenames"], _device="/job:localhost/replica:0/task:0/cpu:0"](matching_filenames)]]
---------------------------------------------------------------------------
OutOfRangeError Traceback (most recent call last)
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1038 try:
-> 1039 return fn(*args)
1040 except errors.OpError as e:
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1020 feed_dict, fetch_list, target_list,
-> 1021 status, run_metadata)
1022
c:\users\engine\appdata\local\programs\python\python35\lib\contextlib.py in __exit__(self, type, value, traceback)
65 try:
---> 66 next(self.gen)
67 except StopIteration:
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
465 compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466 pywrap_tensorflow.TF_GetCode(status))
467 finally:
OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
During handling of the above exception, another exception occurred:
OutOfRangeError Traceback (most recent call last)
<ipython-input-1-110a62c88def> in <module>()
16
17 # Get an image tensor and print its value.
---> 18 image_tensor = sess.run([image])
19 print(image_tensor)
20
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
776 try:
777 result = self._run(None, fetches, feed_dict, options_ptr,
--> 778 run_metadata_ptr)
779 if run_metadata:
780 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
980 if final_fetches or final_targets:
981 results = self._do_run(handle, final_targets, final_fetches,
--> 982 feed_dict_string, options, run_metadata)
983 else:
984 results = []
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1030 if handle is None:
1031 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032 target_list, options, run_metadata)
1033 else:
1034 return self._do_call(_prun_fn, self._session, handle, feed_dict,
c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1050 except KeyError:
1051 pass
-> 1052 raise type(e)(node_def, op, message)
1053
1054 def _extend_graph(self):
OutOfRangeError: FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
Caused by op 'ReaderReadV2', defined at:
File "c:\users\engine\appdata\local\programs\python\python35\lib\runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\engine\appdata\local\programs\python\python35\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
app.launch_new_instance()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
app.start()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelapp.py", line 474, in start
ioloop.IOLoop.instance().start()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tornado\ioloop.py", line 887, in start
handler_func(fd_obj, events)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tornado\stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\kernelbase.py", line 390, in execute_request
user_expressions, allow_stdin)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\ipykernel\zmqshell.py", line 501, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\IPython\core\interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\IPython\core\interactiveshell.py", line 2821, in run_ast_nodes
if self.run_code(code, result):
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-1-110a62c88def>", line 9, in <module>
imageName,imagefile= image_reader.read(filename_queue)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\io_ops.py", line 193, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 411, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "c:\users\engine\appdata\local\programs\python\python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
OutOfRangeError (see above for traceback): FIFOQueue '_0_input_producer' is closed and has insufficient elements (requested 1, current size 0)
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
To be honest I have no clew how I'm suppose to solve this ?
thanks in advance for any hint
Update
After Ujjwal answer I've the the parameter for tf.train.string_input_producer as followed :
import tensorflow as tf
import os
files = os.listdir('test_png')
# Here generating a tensor of type string that include all the filename with png extention
filename_queue = tf.train.string_input_producer(files)
# Initializing a file Reader
image_reader = tf.WholeFileReader()
# Here the file all the files mentioned ie filename queue and
# returns the the file name and the pixelvalue in form of a tensor !
imageName,imagefile= image_reader.read(filename_queue)
image = tf.image.decode_png(imagefile)
#tf.global_variables_initializer()
with tf.Session() as sess:
tf.global_variables_initializer().run()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run([image])
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
Now it seem that TF can find the files but still can't read them, here's the actual error message :
[Command: python -u D:\Masterarbeit\Tensorflow\Main\convNN\own_DATA.py]
2017-06-08 15:12:31.845015: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.845384: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.845662: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.845949: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.846859: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.847174: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.847464: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-08 15:12:31.847754: W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
return fn(*args)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
status, run_metadata)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Can not get size for: 4.png : The system cannot find the file specified.
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Masterarbeit\Tensorflow\Main\convNN\own_DATA.py", line 22, in <module>
image_tensor = sess.run([image])
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
run_metadata_ptr)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Can not get size for: 4.png : The system cannot find the file specified.
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
Caused by op 'ReaderReadV2', defined at:
File "D:\Masterarbeit\Tensorflow\Main\convNN\own_DATA.py", line 12, in <module>
imageName,imagefile= image_reader.read(filename_queue)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\io_ops.py", line 193, in read
return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 411, in _reader_read_v2
queue_handle=queue_handle, name=name)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\Engine\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
NotFoundError (see above for traceback): Can not get size for: 4.png : The system cannot find the file specified.
[[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]
[Finished in 1.972s]
Update
Based on Ujjwal answer is the code that I'm using now, hopefully it may help someone out there :
import tensorflow as tf
import os
# list files name
files = os.listdir("Test_PNG")
files = ["Test_PNG/" + s for s in files]
files = [os.path.abspath(s) for s in files ]
# Here generating a tensor of type string that include all the filename with png extention
filename_queue = tf.train.string_input_producer(files)
# Initializing a file Reader
image_reader = tf.WholeFileReader()
# Here the file all the files mentioned ie filename queue and
# returns the the file name and the pixelvalue in form of a tensor !
imageName,imagefile= image_reader.read(filename_queue)
image = tf.image.decode_png(imagefile)
#tf.global_variables_initializer()
with tf.Session() as sess:
tf.global_variables_initializer().run()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run([image])
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
You have to initialize the variables. Try moving tf.global_variables_initializer() inside tf.Session() as sess: block and run it as tf.global_variables_initializer().run()

TensorFlow missing CPU Op for FFT (InvalidArgumentError: No OpKernel was registered to support Op 'FFT' with these attrs)

I am new to tensorflow and want to create a graph which performs fft on real data, similar to numpys rfft function:
def rfftOp(_in, name='rfft', graph=tf.get_default_graph()):
with graph.as_default():
with tf.device('/cpu:0'):
with tf.name_scope(name):
cast = tf.complex(tf.cast(_in, tf.float32, name='cast_to_float32'), tf.constant(0.0, dtype=tf.float32), name='cast_to_complex')
fftOp = tf.fft(cast, name='fft')
half, _ = tf.split(0, 2, fftOp, name='split')
double = tf.mul(tf.constant(2.0, dtype=tf.complex64), half)
return double
sess = tf.InteractiveSession()
inp = tf.placeholder(np.float64, shape=(256,), name='input')
fftOp = rfftOp(inp)
print(sess.run(fftOp, feed_dict={inp: d}))
However, I am getting the following error message:
---------------------------------------------------------------------------
InvalidArgumentError Traceback (most recent call last)
<ipython-input-18-0f6d789c912c> in <module>()
6 inp = tf.placeholder(np.float64, shape=(256,), name='input')
7 fftOp = rfftOp(inp)
----> 8 print(sess.run(fftOp, feed_dict={inp: d}))
/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
338 try:
339 result = self._run(None, fetches, feed_dict, options_ptr,
--> 340 run_metadata_ptr)
341 if run_metadata:
342 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
562 try:
563 results = self._do_run(handle, target_list, unique_fetches,
--> 564 feed_dict_string, options, run_metadata)
565 finally:
566 # The movers are no longer used. Delete them.
/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
635 if handle is None:
636 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
--> 637 target_list, options, run_metadata)
638 else:
639 return self._do_call(_prun_fn, self._session, handle, feed_dict,
/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
657 # pylint: disable=protected-access
658 raise errors._make_specific_exception(node_def, op, error_message,
--> 659 e.code)
660 # pylint: enable=protected-access
661
InvalidArgumentError: No OpKernel was registered to support Op 'FFT' with these attrs
[[Node: rfft_4/fft = FFT[_device="/device:CPU:0"](rfft_4/cast_to_complex)]]
Caused by op u'rfft_4/fft', defined at:
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/ipykernel/__main__.py", line 3, in <module>
app.launch_new_instance()
File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 596, in launch_instance
app.start()
File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 442, in start
ioloop.IOLoop.instance().start()
File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py", line 162, in start
super(ZMQIOLoop, self).start()
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 883, in start
handler_func(fd_obj, events)
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 391, in execute_request
user_expressions, allow_stdin)
File "/usr/local/lib/python2.7/dist-packages/ipykernel/ipkernel.py", line 199, in do_execute
shell.run_cell(code, store_history=store_history, silent=silent)
File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2723, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2825, in run_ast_nodes
if self.run_code(code, result):
File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2885, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-18-0f6d789c912c>", line 7, in <module>
fftOp = rfftOp(inp)
File "<ipython-input-17-e44d5219afe4>", line 6, in rfftOp
fftOp = tf.fft(cast, name='fft')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 518, in fft
return _op_def_lib.apply_op("FFT", in_=in_, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2154, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1154, in __init__
self._traceback = _extract_stack()
indicating that the Op for tensorflows fft is missing.
I've found a similar issue, but it focus on the GPU Op.
I am using the tensorflow/tensorflow docker image.
So, is there anything missing in the docker image or do I have to use tensorflows fft another way?
You are forcing TensorFlow to try to run the FFT operation on CPU by calling with tf.device('/cpu:0'). However the FFT operations are currently only implemented for GPU, which is why you end up with an error message.
If you have a GPU available you can simply remove the call to tf.device(). TensorFlow will then automatically run the FFT operation on GPU.
This problem is solved in version 1.3 of TensorFlow.

Categories