We Keep Coding

Keras create dataset from CSV without TensorFlow

On every question and tutorial I have found, tf.data.Dataset is used for CSV files, but I am not using tensorflow, I am using PlaidML because my AMD GPU is not supported in ROCm. I have tried using the same code by doing
os.environ["KERAS_BACKEND"] = "plaidml.keras.backend"
from tensorflow import keras
but that still does not use the plaidml backend. How do I load this dataset: https://www.kaggle.com/keplersmachines/kepler-labelled-time-series-data into keras without tensorflow? Thank you. I check if the plaidml backend is used by looking at the output. If it says "Using plaidml.keras.backend backend", then it is using plaidml. Also, only plaidml recognizes my GPU, so tensorflow will use my CPU.

Tensorflow-Keras: Loading weights from checkpoint increases graph size significantly

I am working with a rather large network (98 million parameters), I am using the Keras ModelCheckPoint callback to save my weights as follows, when I reload my saved weights using keras, I can see that the loading operation adds approximately 10 operations per layer in my graph. This results in a huge memory increase of my total network. Is this expected behavior? And if so, are there any known work arounds?
Details:
I am using: tf.keras.callbacks.ModelCheckpoint with "save_weights_only=True" as argument to save the weights
The code for loading it is:
model.load_weights(path_to_existing_weights)
where model is a custom keras model.
I am using Tensorflow 1.14 and Keras 2.3.0
Anyone that has any ideas?

This seems to me to be unexpected behavior but I can't see anything obvious that you are doing wrong. Are you sure there were no changes to your model between the time you saved the weights and the time you reloaded the weights? All I can suggest is try to do the same thing except this time in the callback change it to save the entire model. Then reload the model then check the graph. I also ran across this, doubt it is the problem but I would check it out
In order to save your Keras models as HDF5 files, e.g. via keras.callbacks.ModelCheckpoint,
Keras uses the h5py Python package. It is a dependency of Keras and should be installed by default.
If you are unsure if h5py is installed you can open a Python shell and load the module via
import h5py If it imports without error it is installed, otherwise you can find detailed installation
instructions here: http://docs.h5py.org/en/latest/build.html
Perhaps you might try reinstalling it.

Training RNN on GPU - which tf.keras layer should I use?

I am training RNNs, which I built using tf.keras.layers.GRU layers. They are taking a long time to train (>2 hours), so I am going to deploy them to the GPU for training. I am wondering a few things about training on GPU:
What is the difference between tf.keras.layers.CuDNNGRU and tf.keras.layers.GRU (and also tf.keras.layers.LSTM vs. tf.keras.layers.CuDNNLSTM)? I understand from this post that CuDNNGRU layers train faster than GRU layers, but
Do the 2 layers converge to different results with the same seed?
Do the 2 layers perform the same during inference?
Do CuDNN layers require a GPU during inference?
Can GRU layers run inference on a GPU?
Are CuDNN layers easily deployable? I am currently using coremlconverter to convert my keras model to CoreML for deployment.
Is there an equivalent CuDNN layer for tf.keras.layers.SimpleRNN (i.e. tf.keras.layers.CuDNNSimpleRNN)? I am not committed to a specific architecture yet, and so I believe I would need the tf.keras.layers.CuDNNSimpleRNN layer if I decide on SimpleRNNs and the CuDNN layer has some functionality that I need.
With CuDNN layers, do I need to have tensorflow-gpu installed? Or do they still get deployed to the GPU as long as I have the relevant drivers installed?

if you are using a cuda compatible gpu, it makes absolutely sense to use CuDNN layers. They have a different implementation that tries to overcome computation parallelization issues inherent in the RNN architecture. They usually perform a bit worst though but are 3x-6x faster https://twitter.com/fchollet/status/918170264608817152?lang=en
Do the 2 layers converge to different results with the same seed?
yes
Do the 2 layers perform the same during inference?
You should have a comparable performance but not exactly the same
Do CuDNN layers require a GPU during inference?
Yes but you can convert to a CuDNN compatible GRU/LSTM
Can GRU layers run inference on a GPU?
Yes
With CuDNN layers, do I need to have tensorflow-gpu installed? Or do they still get deployed to the GPU as long as I have the relevant drivers installed?
Yes and you need a cuda compatible gpu

How to change the processing unit during run time (from GPU to CPU)?

In the context of deep neural networks training, the training works faster when it uses the GPU as the processing unit.
This is done by configuring CudNN optimizations and changing the processing unit in the environment variables with the following line (Python 2.7 and Keras on Windows):
os.environ["THEANO_FLAGS"] = "floatX=float32,device=gpu,optimizer_including=cudnn,gpuarray.preallocate=0.8,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic,dnn.include_path=e:/toolkits.win/cuda-8.0.61/include,dnn.library_path=e:/toolkits.win/cuda-8.0.61/lib/x64"
The output is then:
Using gpu device 0: TITAN Xp (CNMeM is disabled, cuDNN 5110)
The problem is that the GPU memory is limited compared to the RAM (12GB and 128GB respectively), and the training is only one phase of the whole flow. Therefore I want to change back to CPU once the training is completed.
I've tried the following line, but it has no effect:
os.environ["THEANO_FLAGS"] = "floatX=float32,device=cpu"
My questions are:
Is it possible to change from GPU to CPU and vice-versa during runtime? (technically)
If yes, how can I do it programmatically in Python? (2.7, Windows, and Keras with Theano backend).

Yes this is possible at least for the tensorflow backend. You just have to also import tensorflow and put your code into the following with:
with tf.device('/cpu:0'):
your code
with tf.device('/gpu:0'):
your code
I am unsure if this also works for theano backend. However, switching from one backend to the other one is just setting a flag beforehand so this should not provide too much trouble.

Loading Torch7 trained models (.t7) in PyTorch

I am using Torch7 library for implementing neural networks. Mostly, I rely on pre-trained models. In Lua I use torch.load function to load a model saved as torch .t7 file. I am curious about switching to PyTorch( http://pytorch.org) and I read the documents. I couldn't find any information regarding the mechanisms to load a pre-trained model. The only relevant information I was able to find is this page:http://pytorch.org/docs/torch.html
But the function torch.load described in the page seems to load a file saved with pickle. If someone has additional information on loading .t7 models in PyTorch, please share it here.

The correct function is load_lua:
from torch.utils.serialization import load_lua
x = load_lua('x.t7')

As of PyTorch 1.0 torch.utils.serialization is completely removed. Hence no one can import models from Lua Torch into PyTorch anymore. Instead, I would suggest installing PyTorch 0.4.1 through pip in a conda environment (so that you can remove it after this) and use this repo to convert your Lua Torch model to PyTorch model, not just the torch.nn.legacy model that you cannot use for training. Then use PyTorch 1.xx to do whatever with it. You can also train your converted Lua Torch models in PyTorch this way :)