We Keep Coding

(Tensorflow) Stuck at Epoch 1 during model.fit()

I've been trying to make Tensorflow 2.8.0 work with my Windows GPU (GeForce GTX 1650 Ti), and even though it detects my GPU, any model that I make will be stuck at Epoch 1 indefinitely when I try to use the fit method till the kernel (I've tried on jupyter notebook and spyder) hangs and restarts.
Based on Tensorflow's website, I've downloaded the respective cuDNN and CUDA versions, for which I've further verified (together with tensorflow's detection of my GPU) by running the various commands:
CUDA (Supposed to be 11.2)
(on command line)
nvcc --version
Build cuda_11.2.r11.2/compiler.29373293_0
(In python)
import tensorflow.python.platform.build_info as build
print(build.build_info['cuda_version'])
Output: '64_112'
cuDNN (Supposed to be 8.1)
import tensorflow.python.platform.build_info as build
print(build.build_info['cuda_version'])
Output: '64_8' # Looks like v8 but I've actually installed v8.1 (cuDNN v8.1.1 (Feburary 26th, 2021), for CUDA 11.0,11.1 and 11.2) so I think it's fine?
GPU Checks
tf.config.list_physical_devices('GPU')
Output: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
tf.test.is_gpu_available()
Output: True
tf.test.gpu_device_name()
Output: This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Created device /device:GPU:0 with 2153 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1650 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
When I then try to fit any sort of model, it just fails following what I described above. What is surprising is that even though it can't load code such as that described in Tensorflow's CNN Tutorial, the only time it ever works is if I run the chunk of code from this stackoverflow question. This chunk of code looks almost the same as every other chunk that failed.
Can someone help me with this issue? I've been desperately testing TensorFlow with every chunk of code that I came across for the past couple of hours, and the only time where it does not get stuck at Epoch 1 is with the link above.
**(I've also tried running only on my CPU via os.environ['CUDA_VISIBLE_DEVICES'] = '-1' and everything seems to work fine)

Update (Solution)
It seems like the suggestions from this post helped - I've copied the following files from the zipped cudnn bin sub folder (cudnn-11.2-windows-x64-v8.1.1.33\cuda\bin) into my cuda bin folder (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin)
cudnn_adv_infer64_8.dll
cudnn_adv_train64_8.dll
cudnn_cnn_infer64_8.dll
cudnn_cnn_train64_8.dll
cudnn_ops_infer64_8.dll
cudnn_ops_train64_8.dll
It seems like I initially misinterpreted the copy all cudnn*.dll files as only copying over the cudnn64_8.dll file, rather than copying every other file listed above.

Python running slower than before some system change

I have two computers at home for my python projects. One is a PC with an Intel i5-7400 CPU and the other is a laptop with an Intel i7-10750H CPU. Presumably the laptop is faster running the same python code than the PC. This was the case before I made some changes to the laptop in an attempt to leverage its Nvida GPU for training DNN model.
I followed the instruction from Tensorflow GPU support webpage to upgrade Nvida GPU driver, install Cuda toolkit and cuDNN with the recommanded version. After the installation, I created a new conda environment and installed latest tensorflow. With all this I could detect my GPU with tf.config.list_physical_devices() and run some test code on the GPU. However, the performance was not lifted and even worse, the laptop became noticeably slower running the same code on its CPU. I tested the following simple code on both machines:
from datetime import datetime
import numpy as np
t0 = datetime.now()
for i in range(1000):
a = np.random.rand(1000, 1000)
b = np.random.rand(1000, 1000)
c = np.matmul(a, b)
t1 = datetime.now()
print(t1 - t0)
The PC ran it in 32s but the laptop needed 45s. I tried a few things to resolve this, including uninstalling Cuda toolkit/cuDNN and reinstalling anaconda (tried different anaconda versions). But the issue still remains. Anyone has any insights about why this happens and what to try in order to address it? Many thanks.
Update: I notice that the same python code uses about 30% CPU if running on the PC's intel i5-7400 CPU but uses over 90% CPU and is slower if on the laptop intel i7-10750H CPU. Is it normal?

This is probably not your main problem however, your laptop run with battery? The laptop can decrease performance for saving battery life

There are many reasons to consider. Firstly, The code you are ruining
doesn't use GPU at all. It's all about CPU's holding the throttle.
Basically, as a part of "Thermal management" Laptops CPU power limit
throttle is constantly controlled. Fact is in CPU's runs code faster
than GPU's. So, maybe your laptop reaching thermal limitations and
throttles down to slower for most of the time it takes to run the
program. Maybe your PC CPU is able to withhold that throttle so it's
finishing bit faster.
Once check Benchmarking your code. A wonderful instructions return here
https://stackoverflow.com/a/1593034/15358800

Why do I get CUDA out of memory when running PyTorch model [with enough GPU memory]?

I am asking this question because I am successfully training a segmentation network on my GTX 2070 on laptop with 8GB VRAM and I use exactly the same code and exactly the same software libraries installed on my desktop PC with a GTX 1080TI and it still throws out of memory.
Why does this happen, considering that:
The same Windows 10 + CUDA 10.1 + CUDNN 7.6.5.32 + Nvidia Driver 418.96 (comes along with CUDA 10.1) are both on laptop and on PC.
The fact that training with TensorFlow 2.3 runs smoothly on the GPU on my PC, yet it fails allocating memory for training only with PyTorch.
PyTorch recognises the GPU (prints GTX 1080 TI) via the command : print(torch.cuda.get_device_name(0))
PyTorch allocates memory when running this command: torch.rand(20000, 20000).cuda() #allocated 1.5GB of VRAM.
What is the solution to this?

Most of the people (even in the thread below) jump to suggest that decreasing the batch_size will solve this problem. In fact, it does not in this case. For example, it would have been illogical for a network to train on 8GB VRAM and yet to fail to train on 11GB VRAM, considering that there were no other applications consuming video memory on the system with 11GB VRAM and the exact same configuration is installed and used.
The reason why this happened in my case was that, when using the DataLoader object, I set a very high (12) value for the workers parameter. Decreasing this value to 4 in my case solved the problem.
In fact, although at the bottom of the thread, the answer provided by Yurasyk at https://github.com/pytorch/pytorch/issues/16417#issuecomment-599137646 pointed me in the right direction.
Solution: Decrease the number of workers in the PyTorch DataLoader. Although I do not exactly understand why this solution works, I assume it is related to the threads spawned behind the scenes for data fetching; it may be the case that, on some processors, such an error appears.

plt.plot() cause blue screen on windows

This code cause a blue screen on windows on my computer :
import matplotlib.pyplot as plt
plt.plot(range(10),range(10)) # This is the line that cause the crash
WhoCrashed tells me this :
This was probably caused by the following module: nt_wrong_symbols.sys
(nt_wrong_symbols) Bugcheck code: 0x124 (0x0, 0xFFFFB60A6AF4D028,
0xB2000000, 0x70005) Error: WHEA_UNCORRECTABLE_ERROR
Here is a link to the full Minidump
What I have done:
Fully tested the CPU with a CPU-Z stress test
Fully tested the RAM with memtest86+
Tested the GPU with Assassin's creed origin in full ultra
Tested the same code on Ubuntu (double boot) : works fine
This lead me to believe this is a windows specific error.
Hardware configuration :
i9-7940X
GTX 1080 Ti
64 Gb RAM #2400Mhz (CPU frequency)
Software :
Windows 10, fresh install (I've always had this issue)
Python 2.7 installed through Anaconda ( I tested the code with Jupyter and IPython with the same results)
Windows and graphic drivers up to date
This is the only thing that causes blue screen on my computer, and I'm out of ideas on how to solve this, any advice would be greatly appreciated.
NOTE : I asked this question here as it appears to be matplotlib related, I hope this is the right place
EDIT : Correction : it does not happens all the time, but more like 95% of the time.

I updated the BIOS and it seems to work now. As i9-7940X is very recent (Q3'17), my old BIOS version was supposed to work with it but was released before the CPU (06/17) so that might have been the issue.
I'll post again if blue screens come back.

I had the same problem on an Alienware Area 51 machine. Fixed it by disabling processor's "hyperthreading" on the BIOS configuration. Also, I had a similar crashing issue on another machine with Ubuntu when trying to use multithreading.
In conclusion Matplotlib and multithreading don't get along well.

vtk crashes with opengl needs update using windows 10 and vulkan

I am using python wrapping to use VTK. I recently upgraded to Windows 10 64 bit, installed my laptop drivers from the relevant manufacturers.I built VTK 7.0 using MSVSC 2010 64 bit with python wrappings. I have switchable graphics card between NVDIA GeForce GT 555M and  Intel HD 3000 Graphics. If I use NVDIA I don't get any vtk object at the renderwin but only the background. If I use Intel Graphics and  run any VTK examples I get the following error:
ERROR: In ......\src\Rendering\OpenGL2\vtkOpenGLRenderWindow.cxx, line 545vtkWin32OpenGLRenderWindow (00000000059F3D40): GL version 2.1 with the gpu_shader4 extension is not supported by your graphics driver but is required for the new OpenGL rendering backend. Please update your OpenGL driver. If you are using Mesa please make sure you have version 10.6.5 or later and make sure your driver in Mesa supports OpenGL 3.2
If I understand right, this happens because Intel HD Graphics installed comes with vulkan, the new generation of OpenGL. But I don't know how can I solve this to work with VTK.Could anyone point out a solution?  Thanks.