Is there a difference (in code) between keras tensorflow-cpu backend and tensorflow-gpu backend? If I want to change tensorflow from cpu to gpu, what code do I need to add or what environmental variables do I need to set?
From keras link I know that I can use tf.devices - something like the code below. But what if I want the whole code, not just some part to run on GPU?
with tf.device('/gpu:0'):
x = tf.placeholder(tf.float32, shape=(None, 20, 64))
y = LSTM(32)(x) # all ops in the LSTM layer will live on GPU:0
with tf.device('/cpu:0'):
x = tf.placeholder(tf.float32, shape=(None, 20, 64))
y = LSTM(32)(x) # all ops in the LSTM layer will live on CPU:0
Just uninstall tensorflow-cpu (pip uninstall tensorflow) and install tensorflow-gpu (pip install tensorflow-gpu). Now tensorflow will always use your gpu(s).
If you only want to use cpu in tensorflow-gpu set the environmental variable CUDA_VISIBLE_DEVICES so that the gpus are invisible. Before loading tensorflow do this in your script:
import os
os.environ["CUDA_VISIBLE_DEVICES"]=""
import tensorflow
Related
I am following this tutorial for SetFit: https://www.philschmid.de/getting-started-setfit
When the training is running, it is using my CPU instead of my GPU. Is there a way I can enable it?
Here is the main part of the code:
from setfit import SetFitModel, SetFitTrainer
from sentence_transformers.losses import CosineSimilarityLoss
# Load a SetFit model from Hub
model_id = "sentence-transformers/all-mpnet-base-v2"
model = SetFitModel.from_pretrained(model_id)
# Create trainer
trainer = SetFitTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=test_dataset,
loss_class=CosineSimilarityLoss,
metric="accuracy",
batch_size=64,
num_iterations=20, # The number of text pairs to generate for contrastive learning
num_epochs=1, # The number of epochs to use for constrastive learning
)
# Train and evaluate
trainer.train()
metrics = trainer.evaluate()
If your training is running on CPU rather than GPU, it is because:
Either you installed the CPU version of the PyTorch.
Either the version of CUDA/CUDNN and PyTorch are not compatible, and the training falls back to CPU instead of GPU.
In essence it has nothing to do with the SetFit model.
A working example for me in recent projects is:
(1) pip/pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
(2) pip install transformers==4.22.0
Note that you may have to uninstall pytorch first before reinstalling it: pip uninstall pytorch.
In order to make sure your GPU is visible, a short print would suffice:
training_device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
I want to train a sequential tensorflow (version 2.3.0) model on a single NVIDIA graphic card (RTX 2080 super). I am using the following code snippet to build and train the model. However, everytime I am running this code I do not see any GPU utilization. Any suggestion how to modify my code so I can run it on 1 GPU?
strategy = tf.distribute.OneDeviceStrategy(device="/GPU:0")
with strategy.scope():
num_classes=len(pd.unique(cats.No))
model = BuildModel((image_height, image_width, 3), num_classes)
model.summary()
model=train_model(model,valid_generator,train_generator,EPOCHS,BATCH_SIZE)
run the code below to see if tensorflow detects your GPU.
import tensorflow as tf
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
print(tf.__version__)
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
tf.test.is_gpu_available()
!python --version
On every question and tutorial I have found, tf.data.Dataset is used for CSV files, but I am not using tensorflow, I am using PlaidML because my AMD GPU is not supported in ROCm. I have tried using the same code by doing
os.environ["KERAS_BACKEND"] = "plaidml.keras.backend"
from tensorflow import keras
but that still does not use the plaidml backend. How do I load this dataset: https://www.kaggle.com/keplersmachines/kepler-labelled-time-series-data into keras without tensorflow? Thank you. I check if the plaidml backend is used by looking at the output. If it says "Using plaidml.keras.backend backend", then it is using plaidml. Also, only plaidml recognizes my GPU, so tensorflow will use my CPU.
The following code generates the warning in tensorflow r1.12 python API:
#!/usr/bin/python3
import tensorflow as tf
M = tf.keras.models.Sequential();
M.add(tf.keras.layers.Dense(2));
The complete warning text is this:
WARNING: Logging before flag parsing goes to stderr.
W0213 15:50:07.239809 140701996246848 deprecation.py:506] From /home/matias/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py:1253: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
I have tried different approaches like initializing and calling a kernel initializer before adding Dense layer and passing it to Dense constructor, but it seems to not change anything. Is this warning inevitable? A 'yes' as an answer would be enough for me.
The warning may be caused upstream by abseil-py, a dependency of tensorflow.
See the details here.
An easy fix may be to update abseil-py by running:
pip install --upgrade absl-py
(In my case, the conflicting version was 0.7.1 and the problem was fixed in the updated version, 0.8.1)
You are running tensor flow 2.0 and it looks like VarianceScaling.init is deprecated. It might mean that Sequential will need to be more explicitly initialized in the future.
for example:
model = tf.keras.Sequential([
# Adds a densely-connected layer with 64 units to the model:
layers.Dense(64, activation='relu', input_shape=(32,)),
# Add another:
layers.Dense(64, activation='relu'),
# Add a softmax layer with 10 output units:
layers.Dense(10, activation='softmax')])
This is just a warning based on the changes in Tensorflow 2.0.
If you don't want to see these warnings, upgrade to TensorFlow 2.0. You can install the beta version via pip:
pip install tensorflow==2.0.0-beta1
I'm quite new in deep learning and, in order to improve my knowledge, I've been reading some books and following a video course on line.
In this videocourse I have to do an exercise with convolution neaural network.
I've builded a CNN with 10.000 images with dimension 64x64 pixels. (to recognize cats and dogs images)
from keras.models import Sequential
from keras.layers import Convolution2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
# Initialising the CNN
classifier = Sequential()
# Step 1 - Convolution
classifier.add(Convolution2D(32,3,3,input_shape=(64,64,3),activation='relu'))
# Step 2 - Pooling
classifier.add(MaxPooling2D(pool_size = (2,2)))
classifier.add(Convolution2D(32,3,3,activation='relu'))
classifier.add(MaxPooling2D(pool_size = (2,2)))
# Step 3 - Flattening
classifier.add(Flatten())
#step 4 - Full Connection CNN
classifier.add(Dense(output_dim = 128 ,activation='relu'))
classifier.add(Dense(output_dim = 1 ,activation='sigmoid'))
# Compiling the CNN
classifier.compile(optimizer = 'adam' , loss = 'binary_crossentropy', metrics = ['accuracy'])
# Fitting the CNN to the images
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
traininig_set = train_datagen.flow_from_directory(
'dataset/training_set',
target_size=(64, 64),
batch_size=32,
class_mode='binary')
test_set = test_datagen.flow_from_directory(
'dataset/test_set',
target_size=(64, 64),
batch_size=32,
class_mode='binary')
classifier.fit_generator(traininig_set,
steps_per_epoch=8000,
epochs=25,
validation_data=test_set,
validation_steps=2000)
The first time I installed Anaconda I didn't install the GPU module and when I started fitting my CNN
I had to wait 1190 seconds per epoch with the CPU working at 70%.
For your information my computer is quite fast. It's an i7 6800k overclocked to 4.2ghz an MSI GTX1080 video cards and 32gb 3333Mhz.
I've tought that with this computer installing the tensorflow gpu module was almost compulsory.
I watched in some posts how to check if the tensorflow is correctly configured to use GPU
and launching:
In [1]: from tensorflow.python.client import device_lib
In [2]: print(device_lib.list_local_devices())
I have this result:
2017-10-16 10:41:25.780983: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-10-16 10:41:25.781067: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-16 10:41:26.635590: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.8225
pciBusID 0000:03:00.0
Total memory: 8.00GiB
Free memory: 6.61GiB
2017-10-16 10:41:26.635807: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-10-16 10:41:26.636324: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0: Y
2017-10-16 10:41:26.637179: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:03:00.0)
[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 16495731140373557390
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 6740156088
locality {
bus_id: 1
}
incarnation: 6266244792178813148
physical_device_desc: "device: 0, name: GeForce GTX 1080, pci bus id: 0000:03:00.0"
]
With gpu:0, I read in the documentation that TensorFlow automatically will use GPU for computation.
Launching the fit method with this configuration I have to wait 950 sec per epoch, well better than 1190 seconds. The cpu never gets over 10% and, strangely, the GPU never gets over 10-13%.
I assume there is something wrong with my configuration because, the teacher in the course, with a MacBook notebook (I don't know the exact configuration actually) without tensorflow GPU module takes approximately 90 seconds per epoch.
I'm not a python or tensorflow expert, but it really seems there is something wrong or something else to understand.
Could someone give some advice, something to read, some tests to do to understand better where is the bottleneck?
Thank you
I don't have a GPU on windows, but I got a really good deal installing the Intel Distribution of Python with Anaconda: https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda.
For tensorflow, the best seems to be a python 3.5 environment (in the previous link, use python=3.5)
I then installed tensorflow with pip inside this environment made with anaconda. Follow installing with anaconda.
Then Keras with conda install keras. (But make sure it won't replace previous numpy and other installations, find proper installation commands not to replace these optimal packages). Maybe pip install keras could be better in case the conda version doesn't work. (Again, use the proper options not to replace your existing packages) - Don't let this keras installation replace your numpy packages or your tensorflow packages!
This gave me all processors absolutely 100% (according to windows resource monitor)
If this doesn't solve your problem, you can also try getting the numpy and scipy packages from here. Unfortunately I had no success at all with the keras and tensorflow packages from this source, but numpy is quality stuff.
With GPU, your problem may be the lack of a proper CUDA driver and the CUDNN library?
Follow this and this.
Unfortunatelly these things vary a lot from computer to computer. I followed strictly the instructions in these sites, and in tensorflow site, for a linux machine, and the results were astonishing.
On top of Daniel's answer (check CUDA & cuDNN) - it is never a good idea to have both tensorflow and tensorflow-gpu packages installed side by side; most probably, you are using the tensorflow (i.e. the CPU) one.
To avoid this, you should uninstall both packages, and then re-install tensorflow-gpu, i.e.:
pip uninstall tensorflow tensorflow-gpu
pip install tensorflow-gpu
See also accepted answer (and comment) here, on a similar issue.