We Keep Coding

Slow ResNet50 training time using AWS Sagemaker GPU instance

I am trying to train a ResNet50 model using keras with tensorflow backend. I'm using a sagemaker GPU instance ml.p3.2xlarge but my training time is extremely long. I am using conda_tensorflow_p36 kernel and I have verified that I have tensorflow-gpu installed.
When inspecting the output of nvidia-smi I see the process is on the GPU, but the utilization is never above 0%.
Tensorflow also recognizes the GPU.
Screenshot of training time.
Is sagemaker in fact using the GPU even though the usage is 0%?
Could the long epoch training time be caused by another issue?

Looks like you've completed 8 steps and it just takes very long. What's your step time?
It might be due to data loading. Where ia data stored? Try to take data loading out of the picture by caching and feeding a single image to the DNN repeatedly and see if that helps.

Tensorflow-gpu not utilizing gpu

I am trying to create and train a CNN model. But every time I run the code, the tensorflow is not utilising GPU instead it uses CPU. I have installed the latest version of tensorflow. Attaching the details below.
python => 3.9.5\
Tensorflow-GPU => 2.5.0\
CUDA => 11.3\
cuDNN => 8.2.1
While running I get the following output with a warning message. (Platform: VS code)
2021-07-28 15:35:13.163991: W tensorflow/core/common_runtime/bfc_allocator.cc:337] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
System Performance
Output
Code is accessible here
Note: I had already tried adding the following code to activate gpu and it isn't working.
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
tf.config.set_visible_devices(gpus[0], 'GPU')
Output for suggestion:

Try this:
physical_devices = tf.config.list_physical_devices('GPU')
print("Num GPUs:", len(physical_devices))
Since your model is quite small it is possible that the GPU is so bottlenecked by the rest of your system that it can't use the full GPU.
On another note, Task Manager is not the best tool to check on the GPU load, if available use nvidia-smi.

To specifically do what the warning says I set it as env setting as follows
os.environ['TF_ENABLE_GPU_GARBAGE_COLLECTION'] = 'false'

Evaluation metrics not displayed on Tensorboard

When training a Tensorflow object detection model, I want to view the mean average precision (mAP) of my test data. Running train.py and eval.py in two consoles, I can view the loss rates of the training data, and even the objects detected in the test set images through
tensorboard --logdir=model_dir/
however no precision scalars are being displayed for the test set.
I am using python 3 on windows 10, and successfully installed pycocotools using;
pip install git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI
Cheers

I thought I had this problem, hence the upvote. But I simply wasnt waiting long enough. Leave the eval running for the at least couple of hours or longer alongside the train job.

#Graham Monkman is it a must for the eval and train job to run together so that I can get the evaluation metrics (mAP, etc..) ?

CUDA_ERROR_OUT_OF_MEMORY on Tensorflow#object_detection/train.py

I'm running Tensorflow Object Detection API to train my own detector using the object_detection/train.py script, found here. The problem is that I'm getting CUDA_ERROR_OUT_OF_MEMORY constantly.
I found some suggestions to reduce the batch size so the trainer consumes less memory, but I reduced from 16 to 4 and I'm still getting the same error. The difference is that when using batch_size=16, the error was thrown in step ~18 and now it is been thrown in step ~70. EDIT: setting batch_size=1 didn't solve the problem, as I still got the error at step ~2700.
What can I do to make it run smoothly until I stop the training proccess? I don't really need to get a fast training.
EDIT:
I'm currently using a GTX 750 Ti 2GB for this. The GPU is not being used for anything else than training and providing monitor image. Currently, I'm using only 80 images for training and 20 images for evaluation.

I think is not about batch_size, because you can start the training at first place.
open a terminal ans run
nvidia-smi -l
to check if there are other process kick in when this error happens. if you set batch_size=16, you can find out pretty quick.

Found the solution for my problem. The batch_size was not the problem, but a higher batch_size made the training memory consumption increase faster, because I was using the config.gpu_options.allow_growth = True configuration. This setting allows Tensorflow to increase memory consumption when needed and tries to use until 100% of GPU memory.
The problem was that I was running the eval.py script at the same time (as recommended in their tutorial) and it was using parte of the GPU memory. When the train.py script tried to use all 100%, the error was thrown.
I solved it by settings the maximum use percentage to 70% for the training proccess. It also solved the problem of stuttering while training. This may not be the optimum value for my GPU, but it is configurable using config.gpu_options.per_process_gpu_memory_fraction = 0.7 setting, for example.

Another option is to dedicate the GPU for training and use the CPU for evaluation.
Disadvantage: Evaluation will consume large portion of your CPU, but only for a few seconds every time a training checkpoint is created, which is not often.
Advantage: 100% of your GPU is used for training all the time
To target CPU, set this environment variable before you run the evaluation script:
export CUDA_VISIBLE_DEVICES=-1
You can explicitly set the evaluate batch job size to 1 in pipeline.config to consume less memory:
eval_config: {
metrics_set: "coco_detection_metrics"
use_moving_averages: false
batch_size: 1;
}
If you're still having issues, TensorFlow may not be releasing GPU memory between training runs. Try restarting your terminal or IDE and try again. This answer has more details.

AWS g2.8xlarge performance and out of memory issues when using tensorflow

I am currently training a recurrent net on Tensorflow for a text classification problem and am running into performance and out of memory issues. I am on AWS g2.8xlarge with Ubuntu 14.04, and a recent nightly build of tensorflow (which I downloaded on Aug 25).
1) Performance issue:
On the surface, both the CPU and GPU are highly under-utilized. I've run multiple tests on this (and have used line_profiler and memory_profiler in the process). Train durations scale linearly with number of epochs, so I tested with 1 epoch. For RNN config = 1 layer, 20 nodes, training time = 146 seconds.
Incidentally, that number is about 20 seconds higher/slower than the same test run on a g2.2xlarge!
Here is a snapshot of System Monitor and nvidia-smi (updated every 2 seconds) about 20 seconds into the run:
SnapshotEarlyPartOfRun
As you can see, GPU utilization is at 19%. When I use nvprof, I find that the total GPU process time is about 27 seconds or so. Also, except for one vCPU, all others are very under-utilized. The numbers stay around this level, till the end of the epoch where I measure error across the entire training set, sending GPU utilization up to 45%.
Unless I am missing something, on the surface, each device is sitting around waiting for something to happen.
2) Out of memory issue:
If I increase the number of nodes to 200, it gives me an Out of Memory error which happens on the GPU side. As you can see from the above snapshots, only one of the four GPUs is used. I've found that the way to get tensorflow to use the GPU has to do with how you assign the model. If you don't specify anything, tensorflow will assign it to a GPU. If you specify a GPU, only it will be used. Tensorflow does not like it when I assign it to multiple devices with a "for d in ['/gpu:0',...]". I get into an issue with re-using the embedding variable. I would like to use all 4 GPUs for this (without setting up distributed tensorflow). Here is the snapshot of the Out of memory error:
OutofMemoryError
Any suggestions you may have for both these problems would be greatly appreciated!

Re (1), to improve GPU utilization did you try increasing the batch size and / or shortening the sequences you use for training?
Re (2), to use multiple GPUs you do need to manually assign the ops to GPU devices, I believe. The right way is to place ops on specific GPUs by doing
with g.Device("/gpu:0"):
...
with g.Device("/gpu:1"):
...