I have installed tensorflow for GPU using native pip using native pip to install tensorflow
While I use import tensorflow as tf with python3,
ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory
I don't know how to solve this. Please help. I have crawling through google and stack overflow! I am using ubuntu17.04!
You are missing nvidia cuda or some parts of it.
This error says that it can't find a lib called libcublas.so.8.0 wich is part of nvidias cuda.
I don't know which Linux you have but if it is debian/ubuntu related you can try to install cuda like this:
sudo apt-get install nvidia-cuda-dev nvidia-cuda-toolkit
You also probably need this:
sudo apt-get install libcupti-dev
Note that you can also build tensorflow without GPU support see: https://www.tensorflow.org/install/install_sources
Related
I am working on a project on Reinforcement Learning - and completely new at this. I installed keras-rl as
pip install keras-rl, however it caused an error as many has mentioned:
TypeError: Keras symbolic inputs/outputs do not implement `__len__`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model. This error will also get raised if you try asserting a symbolic input/output directly.
And the solution for this is to used keras-rl2. Howver I cannot install keras-rl2 on my Mac M1 as the output is following:
ERROR: Cannot install keras-rl2==1.0.0, keras-rl2==1.0.1, keras-rl2==1.0.2, keras-rl2==1.0.3, keras-rl2==1.0.4 and keras-rl2==1.0.5 because these package versions have conflicting dependencies.
When I tried a specific version, i.e
pip install keras-rl2==1.0.5, the output is:
ERROR: Could not find a version that satisfies the requirement tensorflow (from keras-rl2) (from versions: none)
ERROR: No matching distribution found for tensorflow
I have tensorflow 2.6.0 installed by Apple Developer guide. Any idea how I can install keras-rl2?
1. To install Tensorflow on M1 Macs
https://developer.apple.com/metal/tensorflow-plugin/
2. To Install Keras-rl2
Open a terminal window and run these commands from:
https://github.com/keras-rl/keras-rl (last line changed)
git clone https://github.com/wau/keras-rl2.git
cd keras-rl
python3 setup.py install
The setup.py script will still tell you that it can't find a TensorFlow version but you will still be able to import it using:
import rl
I tried to run matterport/MaskRCNN code but faced the following error
----> 6 from mrcnn.model import MaskRCNN
/usr/local/lib/python3.7/dist-packages/mrcnn/model.py in <module>()
253
254
--> 255 class ProposalLayer(KE.Layer):
256 """Receives anchor scores and selects a subset to pass as proposals
257 to the second stage. Filtering is done based on anchor scores and
AttributeError: module 'keras.engine' has no attribute 'Layer'
I found this in the github issue discussion and it worked for me.
You need to uninstall those :
pip uninstall keras -y
pip uninstall keras-nightly -y
pip uninstall keras-Preprocessing -y
pip uninstall keras-vis -y
pip uninstall tensorflow -y
pip uninstall h5py -y
and impose those versions :
pip install tensorflow==1.13.1
pip install keras==2.0.8
pip install h5py==2.10.0
For lines where you are using Layers like ProposalLayer(KE.Layer)
Instead of using KE.Layer do
import keras.layers as KL
and replace all instances of KE by KL
I encountered this problem when I was running the project.
https://github.com/matterport/Mask_RCNN
In the file model.py,
there was a line
import keras.engine as KE
I changed it to
import keras.engine.topology as KE
and the problem disappeared.
Installing tensorflow with version as following
pip uninstall tensorflow -y
pip uninstall keras -y
pip install tensorflow==2.4.3
pip install keras==2.4.0
After above, some errors will arise. You could solve them by following steps.
#Error: [module 'tensorflow' has no attribute XXXXXXXX]
In the model.py or your code, resolving some api with tf.compat.v1, e.g. tf.compat.v1.Session or import tensorflow.compat.v1 as tf
#Error: [ValueError: Tried to convert 'shape' to a tensor and failed. Error: None values not supported.]
mrcnn_bbox = KL.Reshape((-1, num_classes, 4), name="mrcnn_bbox")(x)
replace with this this if-else code block:
if s[1]==None:
mrcnn_bbox = KL.Reshape((-1, num_classes, 4), name="mrcnn_bbox")(x)
else:
mrcnn_bbox = KL.Reshape((s[1], num_classes, 4), name="mrcnn_bbox")(x)
#Error: [ValueError: None values not supported.]
indices = tf.stack([tf.range(probs.shape[0]), class_ids], axis=1)
replace with
indices = tf.stack([tf.range(tf.shape(probs)[0]), class_ids], axis = 1)
#Error: [AttributeError: module 'keras.engine.saving' has no attribute 'load_weights_from_hdf5_group_by_name']
from keras import saving
replace with
from tensorflow.python.keras.saving import hdf5_format
and
saving.load_weights_from_hdf5_group(f, layers)
saving.load_weights_from_hdf5_group_by_name(f, layers)
replace with
hdf5_format.load_weights_from_hdf5_group(f, layers)
hdf5_format.load_weights_from_hdf5_group_by_name(f, layers)
Reference:
Tried to convert 'shape' to a tensor and failed. Error: None values not supported
module 'keras.engine.saving' has no attribute 'load_weights_from_hdf5_group_by_name'
This isn’t strictly a duplicate, but a similar question is found here: AttributeError: module 'keras.engine' has no attribute 'input_layer'
In essence, many of the import and attribute errors from keras come from the fact that keras changes its imports depending on whether you are using a CPU or using a GPU or ASIC. Some of the engine classes don’t get imported in every case.
Instead, use from keras.layers import Layer and use that layer class in place of the one from the engine.
You should write keras.layers
instead keras.engine at import section in model.py file
When running the https://github.com/matterport/Mask_RCNN repository, I faced also all aforementioned issues. After some days, I finally found a way how to run this repository which I would like to share with you:
First, I installed WSL2 + Ubuntu 20.04 GUI (https://medium.com/#japheth.yates/the-complete-wsl2-gui-setup-2582828f4577) and then created the following environment:
conda create tf1_maskrcnn python=3.6 -y
conda activate tf1_maskrcnn
pip install -r requirements.txt
python setup.py install
It should be noted that I have adjusted requirements.txt:
numpy==1.19.5
scipy==1.5.4
Pillow==8.4.0
cython==0.29.28
matplotlib==3.3.4
scikit-image==0.17.2
tensorflow==1.3.0
keras==2.0.8
opencv-python==4.5.5.64
h5py==2.10.0
imgaug==0.4.0
ipykernel
pycocotools
Even though the https://github.com/akTwelve/Mask_RCNN repository which is based upon TensorFlow 2 is available, the pre-trained weights - which are automatically downloaded or can be retrieved from https://github.com/matterport/Mask_RCNN/releases - lead to unsatisfactory results. However, if this repository is used to train the model from scratch it should definitely be preferred over the tf1 version. Nonetheless, if the intention is to see how well this model works on a different dataset, which requires proper pre-trained weights, the tf1 version is the repository to go with.
Peronal option: As most Github repositories concerning deep learning computer vision tasks are tested on Ubuntu, implementing those models on Windows often lead to a multitude of errors which can be avoided by using a virtual machine. The main advantage by using WSL + Ubuntu 20.04 GUI is that it is much faster than using virtual machines. Even though some time needs to be invested at the beginning it is worth investigating this option.
Tried and Tested on 23-10-2022
GoTo matterpot's Mask_RCNN/mrcnn/model.py file
Search for KE. and replace with KL.
the code runs fine and able to download coco weights.
I struggled for a bit with both the outdated Keras and Tensorflow versions on this Mask-RCNN repo.
If you want to use the current versions, instead of replacing lines I recommend you cloning the following repo:
https://github.com/akTwelve/Mask_RCNN
It has been updated to run on tensorflow v2+ and keras v2+.
Where there is KE put KL in the mrcnn/model.py file it worked for me
I am new to tensorflow. In fact using it because the server code I am writing calls that.
I am using conda to setup the various packages. I did conda install -c anaconda tensorflow-mkl. (Note: I dont have a GPU - using a CPU)
I always get this error:
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
The specific line of code where this happens:
tensorflow.contrib.predictor.from_saved_model(path)
On further research, I figured out that this is because the tensorflow package I have does not support this above instruction and needs to have support for the same.
Some questions:
1. How do we ensure the tensorflow package I have does support the above function? Any source from which I can download?
If it is not important, is there a way to suppress this instruction or any errors from it?
Thanks in advance!
You can use conda or pip installations to download the tensorflow that supports cpu. You can use the following commands from your terminal
conda install tensorflow -c anaconda
or
pip install tensorflow==1.13.1
You can use this link if you havent installed pip yet
How to install pip3 on Windows?
Hope this helps..
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX
This is just a warning.Not an error.
To suppress this warning please add the following lines before your actual code:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
import tensorflow as tf
As per the tensorflow official documentation,starting with TensorFlow 1.6, binaries use AVX instructions which may not run on older CPUs.
You can refer the below url for more details on Intel Optimized tensorflow installation:
https://software.intel.com/en-us/articles/intel-optimization-for-tensorflow-installation-guide
Hope this answer your query.Thank you.
I have set up a Ubuntu 18.04 and tried to make Tensorflow 2.2 GPU work (I have an Nvidia/CUDA graphic card) with Python.
Even after reading the documentation https://www.tensorflow.org/install/gpu#linux_setup, it failed (see below for details about how it failed).
Question: would you have a canonical "todo" list (starting point: freshly installed Ubuntu server) on how to install tensorflow-gpu and make it work, with a few steps?
Notes:
I have read many similar forum posts, and I think that having a canonical "todo" (from a fresh Ubuntu install to having tensorflow-gpu working) would be interesting, with a few steps/bash commands
the documentation I used involved
export LD_LIBRARY_PATH...
# Add NVIDIA package repository
sudo apt-key adv --fetch-keys http://developer.download...
...
# Install CUDA and tools. Include optional NCCL 2.x
sudo apt install cuda9.0 cuda...
Even after a lot of trial and errors (I don't copy/paste all the different errors here, would be too long), then at the end:
import tensorflow
always failed. Some reasons included `ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory. I have already read the relevant question here, or this very long (!) Github issue.
After some trial and error, import tensorflow works, but it doesn't use the GPU (see also Tensorflow not running on GPU).
Well, I was facing the same problem. The first thing to do is to look up, which Tensorflow version is required. In your case Tensorflow 2.2. requires CUDA 10.1. The correct cuDNN version is also important. In your case it would be cuDNN 7.4. An additional point is the installed python version. I would recommend Python 3.5-3.8. If one those mismatch, a fully compatibility is almost impossible.
So if you want a check list, here you go:
Install CUDA 10.1 by installing nvidia-cuda-toolkit.
Install the cuDNN version compatible with CUDA 10.1.
Export CUDA environment variables.
If Bazel is not installed, you will be asked on that.
Install TensorFlow 2.2 using pip. I would highly recommend the usage of a virtual environment.
You can find the compatibility check list of Tensorflow and CUDA here
You can find the CUDA Toolkit here
Finally get cuDNN in the correct version here
That's all.
I faced the problem as well when using the Google Cloud Platform for two projects involving deep learning. They provide servers with nothing but a freshly installed Ubuntu OS. Regarding my experience, I recommend doing the following steps:
Look up the cuda and cuDNN version supported by the current Tensorflow release on the Tensorflow page.
Install the targeted cuda version from the deb package retrieved from Nvidias cuda page and be careful that more recent cuda versions might not work! This will automatically install the corresponding Nvidia drivers.
Install the targeted cuDNN version from this page and again be careful that a more recent cuDNN version might not work.
Install tensorflow-gpu using pip.
This should work. Your problem is probably that you are using a more recent cuda version than targeted by the current Tensorflow release.
To install tensorflow-gpu, the guidelines which are provided on official website are very tedious for beginers, instead we can do these simple steps:
Note : NVIDIA driver must be installed before this(you can verify this using command nvidia-smi).
Install Anaconda https://www.anaconda.com/distribution/?
Create an virtual environment using command "conda create -n envname"
Then activate env using command "conda activate envname"
Finally install tensorflow using command "conda install tensorflow-gpu"
With the given code
import tensorflow as tf
if tf.test.gpu_device_name():
print('Default GPU Device{}'.format(tf.test.gpu_device_name()))
else:
print("not using gpu")
You can find the tutorial on link given below
https://www.pugetsystems.com/labs/hpc/Install-TensorFlow-with-GPU-Support-the-Easy-Way-on-Ubuntu-18-04-without-installing-CUDA-1170/?
I would suggest to first check the availability of GPU using nvidia-smi command.
I had faced the same issue, i was able to resolve it by using docker container, you can install docker using Install Docker Engine on Ubuntu or use the Digital Ocean guide (i used this one) How To Install and Use Docker on Ubuntu 18.04
After that it is simple just run the following command based on the requirements
NV_GPU='0' nvidia-docker run --runtime=nvidia -it -v /path/to/folder:/path/to/folder/for/docker/container nvcr.io/nvidia/tensorflow:17.11
NV_GPU='0' nvidia-docker run --runtime=nvidia -it -v /storage/research/:/storage/research/ nvcr.io/nvidia/tensorflow:20.12-tf2-py3
Here '0' represents the GPU number, if you want to use more than one GPU just use '0,1,2' and so on ....
Hope this solves the issue.
I know there are a lot of tutorials of installing keras on win. But I am faced with a weird problem and I cannot find a solution on Google.
The version of python I used is 2.7(anaconda 64bit).
When I typed 'pip install keras' in cmd, an error occurred.
It is said that no matching distribution found for tensorflow(from keras).
I googled it and one saying goes that this problem occurred when python is 32bit on windows. However, my python is 64-bit, so this saying does not hold.
I want to know how to handle this, given that I want to use theano as keras backend instead of tensorflow.
I think the problem is that TensorFlow on Windows only support Python 3.5.X.
You should try installing anaconda3.
Source: https://www.tensorflow.org/install/install_windows
using commad pip install -v -v -v keras may offer you more information about this
and I find this commad may download Keras tar file in site https://pypi.python.org/simple/keras/
and after download file, it will check the requires.txt in Keras.egg-info
then i find in Keras 2.0.0 it require tensorflow instead of theano in this version, otherwise you may fail to install
so the solution is, download the other keras in site which requires theano not tensorflow, and unpack it, then cd its directory, and use python setup.py install to install this version of keras