How to install tensorflow on ubuntu (python is installed with linuxbrew)? - python

I got the following error when I try to install tensorflow in linux.
I'd like to install tensorflow on linux. I prefer not to use virtualenv.
https://www.tensorflow.org/install/install_linux
Does anybody know what the correct sequence of commends is to install tensorflow? Thanks.
$ pip install --upgrade tensorflow
Requirement already up-to-date: tensorflow in /home/py/.linuxbrew/lib/python2.7/site-packages
Requirement already up-to-date: mock>=2.0.0 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Requirement already up-to-date: tensorflow-tensorboard<0.5.0,>=0.4.0rc1 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Requirement already up-to-date: backports.weakref>=1.0rc1 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Requirement already up-to-date: wheel in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Requirement already up-to-date: numpy>=1.12.1 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Collecting protobuf>=3.3.0 (from tensorflow)
Using cached protobuf-3.5.1-py2.py3-none-any.whl
Requirement already up-to-date: six>=1.10.0 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Requirement already up-to-date: enum34>=1.1.6 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow)
Requirement already up-to-date: funcsigs>=1; python_version < "3.3" in /home/py/.linuxbrew/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow)
Requirement already up-to-date: pbr>=0.11 in /home/py/.linuxbrew/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow)
Requirement already up-to-date: bleach==1.5.0 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already up-to-date: futures>=3.1.1; python_version < "3.2" in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Collecting markdown>=2.6.8 (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already up-to-date: html5lib==0.9999999 in /home/py/.linuxbrew/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Collecting werkzeug>=0.11.10 (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Using cached Werkzeug-0.13-py2.py3-none-any.whl
Collecting setuptools (from protobuf>=3.3.0->tensorflow)
Using cached setuptools-38.2.5-py2.py3-none-any.whl
Installing collected packages: setuptools, protobuf, markdown, werkzeug
Found existing installation: setuptools 38.2.4
Uninstalling setuptools-38.2.4:
Successfully uninstalled setuptools-38.2.4
Found existing installation: protobuf 3.5.0.post1
Uninstalling protobuf-3.5.0.post1:
Successfully uninstalled protobuf-3.5.0.post1
Found existing installation: Markdown 2.6.9
Uninstalling Markdown-2.6.9:
Successfully uninstalled Markdown-2.6.9
Found existing installation: Werkzeug 0.12.2
Uninstalling Werkzeug-0.12.2:
Successfully uninstalled Werkzeug-0.12.2
Successfully installed markdown-2.6.10 protobuf-3.5.1 setuptools-38.2.5 werkzeug-0.13
$ python
Python 2.7.12 (default, Feb 3 2017, 09:33:51)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import *
File "/home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 72, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: /home/py/.linuxbrew/Cellar/python/2.7.12_3/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: invalid ELF header
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/install_sources#common_installation_problems
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
>>>

Related

ThunderGBM: libcusparse.so.10.0: cannot open shared object file: No such file or directory

I am trying to work with ThunderGBM on Ubuntu. Installation seems to be successful:
$ pip install thundergbm
Requirement already satisfied: thundergbm in ./venv/lib/python3.8/site-packages (0.3.16)
Requirement already satisfied: numpy in ./venv/lib/python3.8/site-packages (from thundergbm) (1.21.4)
Requirement already satisfied: scipy in ./venv/lib/python3.8/site-packages (from thundergbm) (1.7.2)
Requirement already satisfied: scikit-learn in ./venv/lib/python3.8/site-packages (from thundergbm) (1.0.1)
Requirement already satisfied: joblib>=0.11 in ./venv/lib/python3.8/site-packages (from scikit-learn->thundergbm) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in ./venv/lib/python3.8/site-packages (from scikit-learn->thundergbm) (3.0.0)
But them working with it generates error:
>>> from thundergbm import TGBMClassifier
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "~/Desktop/gpuimlearn/src/CPU-imLearn/venv/lib/python3.8/site-packages/thundergbm/__init__.py", line 10, in <module>
from .thundergbm import *
File "~/Desktop/gpuimlearn/src/CPU-imLearn/venv/lib/python3.8/site-packages/thundergbm/thundergbm.py", line 32, in <module>
thundergbm = CDLL(lib_path)
File "~/anaconda3/lib/python3.8/ctypes/__init__.py", line 373, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libcusparse.so.10.0: cannot open shared object file: No such file or directory
Ubuntu distribution:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Your current ThunderGBM install might not be compatible with the CUDA version that you have. You can check your version via: echo $LD_LIBRARY_PATH. You could downgrade your CUDA version to match, but I think the simple solution is to just rebuild the Python wheel.
This open issue on their Github has more information about your problem.

I can't use matplotlib

I can't use matplotlib. I installed it with pip and it shows up in the pip list, but when I try to import it it says the module is not there. Others work, like pygame and I didn't do anything different when installing.
C:\Users\Torben>pip install matplotlib
Collecting matplotlib
Using cached matplotlib-3.4.2-cp39-cp39-win_amd64.whl (7.1 MB)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from matplotlib) (1.3.1)
Requirement already satisfied: numpy>=1.16 in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from matplotlib) (1.21.0)
Requirement already satisfied: cycler>=0.10 in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: pillow>=6.2.0 in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from matplotlib) (8.3.0)
Requirement already satisfied: python-dateutil>=2.7 in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from matplotlib) (2.8.1)
Requirement already satisfied: pyparsing>=2.2.1 in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: six in c:\users\torben\appdata\local\programs\python\python39\lib\site-packages (from cycler>=0.10->matplotlib) (1.16.0)
Installing collected packages: matplotlib
Successfully installed matplotlib-3.4.2
C:\Users\Torben>pip list
Package Version
cycler 0.10.0
kiwisolver 1.3.1
matplotlib 3.4.2
numpy 1.21.0
Pillow 8.3.0
pip 21.1.3
pygame 2.0.1
pyparsing 2.4.7
python-dateutil 2.8.1
setuptools 56.0.0
six 1.16.0
C:\Users\Torben>python
Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import matplotlib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Torben\AppData\Local\Programs\Python\Python39\lib\site-packages\matplotlib\__init__.py", line 107, in <module>
from . import _api, cbook, docstring, rcsetup
File "C:\Users\Torben\AppData\Local\Programs\Python\Python39\lib\site-packages\matplotlib\rcsetup.py", line 24, in <module>
from matplotlib import _api, animation, cbook
File "C:\Users\Torben\AppData\Local\Programs\Python\Python39\lib\site-packages\matplotlib\animation.py", line 34, in <module>
from PIL import Image
File "C:\Users\Torben\AppData\Local\Programs\Python\Python39\lib\site-packages\PIL\Image.py", line 114, in <module>
from . import _imaging as core
ImportError: DLL load failed while importing _imaging: Das angegebene Modul wurde nicht gefunden.`][1]
(last line -> DLL load failed while importing _imaging: The specified module was not found.)

Probably the C++ runtime library is not installed in your machine or environment. Try python -m pip install msvc-runtime

ImportError: No module named requests - even though it is installed

I'm running Mac OS X and Python (version 2.7).
Apparently I'm missing requests, but it should be installed.
$ python setup.py
Traceback (most recent call last):
File "setup.py", line 1, in <module>
import requests
ImportError: No module named requests
$ pip install requests
Requirement already satisfied: requests in /usr/local/lib/python2.7/site-packages
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/site-packages (from requests)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/site-packages (from requests)
Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/site-packages (from requests)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/site-packages (from requests)

alias python=/usr/local/bin/python3 fixed the issue..

KeyError: "Couldn't find field google.protobuf.DescriptorProto.ExtensionRange.options"

How should I fix this error?
Mon Nov 13 00:37:49 :~$ /Users/mona/anaconda/bin/pip install --upgrade tensorflow tensorflow-tensorboard
Collecting tensorflow
Downloading tensorflow-1.4.0-cp36-cp36m-macosx_10_11_x86_64.whl (39.3MB)
100% |████████████████████████████████| 39.3MB 28kB/s
Collecting tensorflow-tensorboard
Downloading tensorflow_tensorboard-0.1.8-py3-none-any.whl (1.6MB)
100% |████████████████████████████████| 1.6MB 560kB/s
Collecting wheel>=0.26 (from tensorflow)
Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB)
100% |████████████████████████████████| 51kB 604kB/s
Collecting protobuf>=3.3.0 (from tensorflow)
Downloading protobuf-3.4.0-py2.py3-none-any.whl (375kB)
100% |████████████████████████████████| 378kB 57kB/s
Collecting numpy>=1.12.1 (from tensorflow)
Downloading numpy-1.13.3-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.5MB)
100% |████████████████████████████████| 4.6MB 200kB/s
Collecting enum34>=1.1.6 (from tensorflow)
Downloading enum34-1.1.6-py3-none-any.whl
Collecting six>=1.10.0 (from tensorflow)
Using cached six-1.11.0-py2.py3-none-any.whl
Requirement already up-to-date: werkzeug>=0.11.10 in ./anaconda/lib/python3.6/site-packages (from tensorflow-tensorboard)
Requirement already up-to-date: bleach==1.5.0 in ./anaconda/lib/python3.6/site-packages (from tensorflow-tensorboard)
Requirement already up-to-date: markdown>=2.6.8 in ./anaconda/lib/python3.6/site-packages (from tensorflow-tensorboard)
Requirement already up-to-date: html5lib==0.9999999 in ./anaconda/lib/python3.6/site-packages (from tensorflow-tensorboard)
Collecting setuptools (from protobuf>=3.3.0->tensorflow)
Downloading setuptools-36.7.1-py2.py3-none-any.whl (482kB)
100% |████████████████████████████████| 491kB 883kB/s
Installing collected packages: wheel, six, setuptools, protobuf, numpy, tensorflow-tensorboard, enum34, tensorflow
Found existing installation: wheel 0.29.0
Uninstalling wheel-0.29.0:
Successfully uninstalled wheel-0.29.0
Found existing installation: six 1.10.0
Uninstalling six-1.10.0:
Successfully uninstalled six-1.10.0
Found existing installation: setuptools 36.5.0.post20170921
Uninstalling setuptools-36.5.0.post20170921:
Successfully uninstalled setuptools-36.5.0.post20170921
Found existing installation: protobuf 3.3.2
Uninstalling protobuf-3.3.2:
Successfully uninstalled protobuf-3.3.2
Found existing installation: numpy 1.12.1
Uninstalling numpy-1.12.1:
Successfully uninstalled numpy-1.12.1
Found existing installation: tensorflow 1.1.0
Uninstalling tensorflow-1.1.0:
Successfully uninstalled tensorflow-1.1.0
Successfully installed enum34-1.1.6 numpy-1.13.3 protobuf-3.4.0 setuptools-36.7.1 six-1.11.0 tensorflow-1.4.0 tensorflow-tensorboard-0.1.8 wheel-0.30.0
Mon Nov 13 00:40:10 :~$ tensorboard --logdir=/var/folders/bk/2b2d879912bgnl5s768s7crh0000gn/T/tmpzy0cj6xc
Traceback (most recent call last):
File "/Users/mona/anaconda/bin/tensorboard", line 7, in <module>
from tensorboard.main import main
File "/Users/mona/anaconda/lib/python3.6/site-packages/tensorboard/main.py", line 29, in <module>
import tensorflow as tf
File "/Users/mona/anaconda/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import *
File "/Users/mona/anaconda/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 52, in <module>
from tensorflow.core.framework.graph_pb2 import *
File "/Users/mona/anaconda/lib/python3.6/site-packages/tensorflow/core/framework/graph_pb2.py", line 10, in <module>
from google.protobuf import descriptor_pb2
File "/Users/mona/anaconda/lib/python3.6/site-packages/google/protobuf/descriptor_pb2.py", line 409, in <module>
options=None),
File "/Users/mona/anaconda/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 501, in __new__
return _message.default_pool.FindFieldByName(full_name)
KeyError: "Couldn't find field google.protobuf.DescriptorProto.ExtensionRange.options"
Mon Nov 13 00:40:22 :~$

A related bug report on Google protobuf can be found here, you may want to check it out. There's a bug report on GitHub tensorflow here. The solution may be to run
pip install tensorflow tensorflow-tensorboard --upgrade --force-reinstall
Can you try that?

You can try this recipe:
pip uninstall tensorflow
pip install tensorflow
pip uninstall protobuf
pip install protobuf==3.3.0.
It works for me.

airflow + why airflow return unclear erros

I am using the airflow application on my Linux server
note - Airflow is a platform to programmatically author, schedule and monitor workflows.
from some unclear reason we get the following issues ( when we restart the airflow or initialize the database , )
what should I check regarding the following errors?
airflow webserver
Traceback (most recent call last):
File "/usr/bin/airflow", line 4, in <module>
from airflow import configuration
File "/usr/lib/python2.7/site-packages/airflow/__init__.py", line 29, in <module>
from airflow import configuration as conf
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 657, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 519, in read
self._validate()
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 484, in get
option = self._get_cmd_option(section, key)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 465, in _get_cmd_option
return run_command(command)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 70, in run_command
.format(command, process.returncode, output, stderr)
airflow.configuration.AirflowConfigException: Cannot execute python /home/airflow/.sec/security.py get_connection_string pg. Error code is: 1. Output: , Stderr: Traceback (most recent call last):
File "/home/airflow/.sec/security.py", line 4, in <module>
from cryptography.fernet import Fernet
ImportError: No module named cryptography.fernet
# airflow initdb
Traceback (most recent call last):
File "/usr/bin/airflow", line 4, in <module>
from airflow import configuration
File "/usr/lib/python2.7/site-packages/airflow/__init__.py", line 29, in <module>
from airflow import configuration as conf
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 657, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 519, in read
self._validate()
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 484, in get
option = self._get_cmd_option(section, key)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 465, in _get_cmd_option
return run_command(command)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 70, in run_command
.format(command, process.returncode, output, stderr)
airflow.configuration.AirflowConfigException: Cannot execute python /home/airflow/.sec/security.py get_connection_string pg. Error code is: 1. Output: , Stderr: Traceback (most recent call last):
File "/home/airflow/.sec/security.py", line 4, in <module>
from cryptography.fernet import Fernet
ImportError: No module named cryptography.fernet
]# airflow list_dags
Traceback (most recent call last):
File "/usr/bin/airflow", line 4, in <module>
from airflow import configuration
File "/usr/lib/python2.7/site-packages/airflow/__init__.py", line 29, in <module>
from airflow import configuration as conf
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 657, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 519, in read
self._validate()
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 484, in get
option = self._get_cmd_option(section, key)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 465, in _get_cmd_option
return run_command(command)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 70, in run_command
.format(command, process.returncode, output, stderr)
airflow.configuration.AirflowConfigException: Cannot execute python /home/airflow/.sec/security.py get_connection_string pg. Error code is: 1. Output: , Stderr: Traceback (most recent call last):
File "/home/airflow/.sec/security.py", line 4, in <module>
from cryptography.fernet import Fernet
ImportError: No module named cryptography.fernet
we try also
pip install "apache-airflow[crypto]"
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting apache-airflow[crypto]
Using cached https://files.pythonhosted.org/packages/74/cb/3a4bcfcdb75897607c0c4d9ec17f3c3d6eb4973930d2fa59bd0b47c20a67/apache_airflow-1.10.6-py2.py3-none-any.whl
Requirement already satisfied: setproctitle<2,>=1.1.8 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.1.10)
Requirement already satisfied: termcolor==1.1.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.1.0)
Requirement already satisfied: colorlog==4.0.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (4.0.2)
Requirement already satisfied: dill<0.4,>=0.2.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.2.5)
Requirement already satisfied: typing; python_version < "3.5" in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (3.7.4.1)
Collecting flask-admin==1.5.3
Using cached https://files.pythonhosted.org/packages/c3/bc/4cb71a88d8b0d478ed7f46e7bba632e988daae5ae07ce5413a31acbe8166/Flask-Admin-1.5.3.tar.gz
Collecting flask-login<0.5,>=0.3
Using cached https://files.pythonhosted.org/packages/c1/ff/bd9a4d2d81bf0c07d9e53e8cd3d675c56553719bbefd372df69bf1b3c1e4/Flask-Login-0.4.1.tar.gz
Requirement already satisfied: markdown<3.0,>=2.5.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (2.6.7)
Requirement already satisfied: unicodecsv>=0.14.1 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.14.1)
Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.3.3)
Requirement already satisfied: enum34~=1.1.6; python_version < "3.4" in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.1.6)
Requirement already satisfied: text-unidecode==1.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.2)
Collecting flask-wtf<0.15,>=0.14.2
Using cached https://files.pythonhosted.org/packages/60/3a/58c629472d10539ae5167dc7c1fecfa95dd7d0b7864623931e3776438a24/Flask_WTF-0.14.2-py2.py3-none-any.whl
Requirement already satisfied: cached-property~=1.5 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.5.1)
Requirement already satisfied: graphviz>=0.12 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.13.2)
Collecting gunicorn<20.0,>=19.5.0
Using cached https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl
Requirement already satisfied: tabulate<0.9,>=0.7.5 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.8.5)
Requirement already satisfied: python-dateutil<3,>=2.3 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (2.6.0)
Requirement already satisfied: tenacity==4.12.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (4.12.0)
Requirement already satisfied: lazy-object-proxy~=1.3 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.4.3)
Requirement already satisfied: flask-swagger==0.2.13 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.2.13)
Collecting flask<2.0,>=1.1.0
Using cached https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl
Requirement already satisfied: argcomplete~=1.10 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.10.0)
Requirement already satisfied: marshmallow-sqlalchemy<0.19.0,>=0.16.1 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.18.0)
Requirement already satisfied: croniter<0.4,>=0.3.17 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.3.30)
Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (3.5.3)
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (4.4.0)
Requirement already satisfied: sqlalchemy~=1.3 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.3.10)
Collecting flask-appbuilder<2.0.0,>=1.12.5
Using cached https://files.pythonhosted.org/packages/90/9f/d8c4e693eb208042cc6dffcfa7b407db8e36cc89ec790a53d7cea5fa4bb8/Flask-AppBuilder-1.13.1.tar.gz
Processing /root/.cache/pip/wheels/2f/4c/cd/f0cabf03ccb7edd9002875db9001108b33ebfc60b4ff9a61ae/pendulum-1.4.4-cp27-cp27mu-linux_x86_64.whl
Collecting psutil<6.0.0,>=4.2.0
Using cached https://files.pythonhosted.org/packages/03/9a/95c4b3d0424426e5fd94b5302ff74cea44d5d4f53466e1228ac8e73e14b4/psutil-5.6.5.tar.gz
Collecting funcsigs==1.0.0
Using cached https://files.pythonhosted.org/packages/09/8d/17528625d12ca90651dd1f7958fd0d32b23b15f2197023372669fd683321/funcsigs-1.0.0-py2.py3-none-any.whl
Collecting future<0.17,>=0.16.0
Using cached https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz
Collecting requests<3,>=2.20.0
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Requirement already satisfied: thrift>=0.9.2 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (0.9.3)
Collecting alembic<2.0,>=1.0
Using cached https://files.pythonhosted.org/packages/70/3d/d5ed7a71fe84f9ed0a69e91232a40b0b148b151524dc5bb1c8e4211eb117/alembic-1.3.0.tar.gz
Collecting json-merge-patch==0.2
Using cached https://files.pythonhosted.org/packages/39/62/3b783faabac9a099877397d8f7a7cc862a03fbf9fb1b90d414ea7c6bb096/json-merge-patch-0.2.tar.gz
Collecting tzlocal<2.0.0,>=1.4
Using cached https://files.pythonhosted.org/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz
Requirement already satisfied: jinja2<2.11.0,>=2.10.1 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (2.10.3)
Requirement already satisfied: pygments<3.0,>=2.0.1 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (2.1.3)
Collecting iso8601>=0.1.12
Using cached https://files.pythonhosted.org/packages/ef/57/7162609dab394d38bbc7077b7ba0a6f10fb09d8b7701ea56fa1edc0c4345/iso8601-0.1.12-py2.py3-none-any.whl
Requirement already satisfied: pandas<1.0.0,>=0.17.1 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (0.19.2)
Requirement already satisfied: python-daemon<2.2,>=2.1.1 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (2.1.2)
Requirement already satisfied: cryptography>=0.9.3; extra == "crypto" in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.7.1)
Requirement already satisfied: wtforms in /usr/lib/python2.7/site-packages (from flask-admin==1.5.3->apache-airflow[crypto]) (2.1)
Requirement already satisfied: Werkzeug>=0.12 in /usr/lib/python2.7/site-packages (from flask-caching<1.4.0,>=1.3.3->apache-airflow[crypto]) (0.12.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil<3,>=2.3->apache-airflow[crypto]) (1.9.0)
Requirement already satisfied: monotonic>=0.6; python_version == "2.7" in /usr/lib/python2.7/site-packages (from tenacity==4.12.0->apache-airflow[crypto]) (1.5)
Requirement already satisfied: futures>=3.0; python_version == "2.7" in /usr/lib/python2.7/site-packages (from tenacity==4.12.0->apache-airflow[crypto]) (3.3.0)
Requirement already satisfied: PyYAML>=3.0 in /usr/lib64/python2.7/site-packages (from flask-swagger==0.2.13->apache-airflow[crypto]) (3.12)
Requirement already satisfied: click>=5.1 in /usr/lib64/python2.7/site-packages (from flask<2.0,>=1.1.0->apache-airflow[crypto]) (7.0)
Requirement already satisfied: itsdangerous>=0.24 in /usr/lib/python2.7/site-packages (from flask<2.0,>=1.1.0->apache-airflow[crypto]) (0.24)
Requirement already satisfied: marshmallow>=2.0.0 in /usr/lib/python2.7/site-packages (from marshmallow-sqlalchemy<0.19.0,>=0.16.1->apache-airflow[crypto]) (2.20.5)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from zope.deprecation<5.0,>=4.0->apache-airflow[crypto]) (32.3.1)
Requirement already satisfied: colorama<1,>=0.3.9 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (0.4.1)
Requirement already satisfied: apispec[yaml]>=1.1.1<2 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (2.0.2)
Requirement already satisfied: Flask-Babel<1,>=0.11.1 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (0.12.2)
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (1.2.5)
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.3 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (2.4.1)
Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (3.24.1)
Requirement already satisfied: marshmallow-enum<2,>=1.4.1 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (1.5.1)
Collecting prison==0.1.0
Using cached https://files.pythonhosted.org/packages/a1/c4/076e1e2677ef928765b0fb537be6476fb8a96624be4669fb5a0618b32455/prison-0.1.0-py2.py3-none-any.whl
Collecting jsonschema>=3.0.1<4
Using cached https://files.pythonhosted.org/packages/ce/6c/888d7c3c1fce3974c88a01a6bc553528c99d3586e098eee23e8383dd11c3/jsonschema-3.1.1-py2.py3-none-any.whl
Requirement already satisfied: PyJWT>=1.7.1 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (1.7.1)
Collecting pytzdata>=2018.3.0.0
Using cached https://files.pythonhosted.org/packages/7f/f9/cdd39831b0465285c02ed90cfbf0db25bb951cb2a35ded0e69222375bea3/pytzdata-2019.3-py2.py3-none-any.whl
Collecting ordereddict
Using cached https://files.pythonhosted.org/packages/53/25/ef88e8e45db141faa9598fbf7ad0062df8f50f881a36ed6a0073e1572126/ordereddict-1.1.tar.gz
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (1.25.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (2019.9.11)
Requirement already satisfied: Mako in /usr/lib/python2.7/site-packages (from alembic<2.0,>=1.0->apache-airflow[crypto]) (1.0.6)
Requirement already satisfied: python-editor>=0.3 in /usr/lib/python2.7/site-packages (from alembic<2.0,>=1.0->apache-airflow[crypto]) (1.0.3)
Requirement already satisfied: pytz in /usr/lib/python2.7/site-packages (from tzlocal<2.0.0,>=1.4->apache-airflow[crypto]) (2016.10)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib64/python2.7/site-packages (from jinja2<2.11.0,>=2.10.1->apache-airflow[crypto]) (1.0)
Requirement already satisfied: numpy>=1.7.0 in /usr/lib64/python2.7/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow[crypto]) (1.11.3)
Requirement already satisfied: docutils in /usr/lib/python2.7/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow[crypto]) (0.13.1)
Requirement already satisfied: lockfile>=0.10 in /usr/lib/python2.7/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow[crypto]) (0.12.2)
Requirement already satisfied: pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (0.1.9)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/site-packages (from cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (1.0.17)
Requirement already satisfied: cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (1.9.1)
Collecting Babel>=2.3
Using cached https://files.pythonhosted.org/packages/2c/60/f2af68eb046c5de5b1fe6dd4743bf42c074f7141fe7b2737d3061533b093/Babel-2.7.0-py2.py3-none-any.whl
Requirement already satisfied: python-openid>=2.0 in /usr/lib/python2.7/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (2.2.5)
Collecting pyrsistent>=0.14.0
Using cached https://files.pythonhosted.org/packages/30/86/53a88c0a57698fa228db29a4000c28f4124823010388cb7042fe6e2be8dd/pyrsistent-0.15.5.tar.gz
Collecting attrs>=17.4.0
Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting importlib-metadata
Using cached https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl
Collecting functools32; python_version < "3"
Using cached https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz
Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (2.17)
Collecting contextlib2; python_version < "3"
Using cached https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Collecting pathlib2; python_version == "3.4.*" or python_version < "3"
Using cached https://files.pythonhosted.org/packages/e9/45/9c82d3666af4ef9f221cbb954e1d77ddbb513faf552aea6df5f37f1a4859/pathlib2-2.3.5-py2.py3-none-any.whl
Collecting zipp>=0.5
Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
Collecting scandir; python_version < "3.5"
Using cached https://files.pythonhosted.org/packages/df/f5/9c052db7bd54d0cbf1bc0bb6554362bba1012d03e5888950a4f5c5dadc4e/scandir-1.10.0.tar.gz
Collecting more-itertools
Using cached https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl
ERROR: flask 1.1.1 has requirement Werkzeug>=0.15, but you'll have werkzeug 0.12.1 which is incompatible.
ERROR: flask-appbuilder 1.13.1 has requirement marshmallow<2.20,>=2.18.0, but you'll have marshmallow 2.20.5 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement alembic<0.9,>=0.8.3, but you'll have alembic 1.3.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement babel<2.0,>=1.3, but you'll have babel 2.7.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement flask<0.11,>=0.10.1, but you'll have flask 1.1.1 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement flask-admin==1.4.0, but you'll have flask-admin 1.5.3 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement flask-login==0.2.11, but you'll have flask-login 0.4.1 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement Flask-WTF==0.12, but you'll have flask-wtf 0.14.2 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement funcsigs<1,>=0.4, but you'll have funcsigs 1.0.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement future<0.16,>=0.15.0, but you'll have future 0.16.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement gunicorn<19.4.0,>=19.3.0, but you'll have gunicorn 19.9.0 which is incompatible.
ERROR: jsonschema 3.1.1 has requirement six>=1.11.0, but you'll have six 1.9.0 which is incompatible.
ERROR: flask-jwt-extended 3.24.1 has requirement Werkzeug>=0.14, but you'll have werkzeug 0.12.1 which is incompatible.
Installing collected packages: flask, flask-admin, flask-login, flask-wtf, gunicorn, requests, prison, pyrsistent, attrs, contextlib2, scandir, pathlib2, more-itertools, zipp, importlib-metadata, functools32, jsonschema, flask-appbuilder, pytzdata, tzlocal, pendulum, psutil, ordereddict, funcsigs, future, alembic, json-merge-patch, iso8601, apache-airflow, Babel
Found existing installation: Flask 0.10.1
Uninstalling Flask-0.10.1:
Successfully uninstalled Flask-0.10.1
Found existing installation: Flask-Admin 1.4.0
Uninstalling Flask-Admin-1.4.0:
Successfully uninstalled Flask-Admin-1.4.0
Running setup.py install for flask-admin ... done
Found existing installation: Flask-Login 0.2.11
Uninstalling Flask-Login-0.2.11:
Successfully uninstalled Flask-Login-0.2.11
Running setup.py install for flask-login ... done
Found existing installation: Flask-WTF 0.12
Uninstalling Flask-WTF-0.12:
Successfully uninstalled Flask-WTF-0.12
Found existing installation: gunicorn 19.3.0
Uninstalling gunicorn-19.3.0:
Successfully uninstalled gunicorn-19.3.0
Found existing installation: requests 2.6.0
ERROR: Cannot uninstall 'requests'. It is a distutils installed project and thus we cannot accurately deter
the airflow script
!/usr/bin/python2
# PYTHON_ARGCOMPLETE_OK
# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import os
import argcomplete
from airflow.configuration import conf
from airflow.bin.cli import CLIFactory
if __name__ == '__main__':
if conf.get("core", "security") == 'kerberos':
os.environ['KRB5CCNAME'] = conf.get('kerberos', 'ccache')
os.environ['KRB5_KTNAME'] = conf.get('kerberos', 'keytab')
parser = CLIFactory.get_parser()
argcomplete.autocomplete(parser)
args = parser.parse_args()
args.func(args)
seems its not load the module as:
airflow
Traceback (most recent call last):
File "/usr/bin/airflow", line 25, in <module>
from airflow.configuration import conf

Can you install the following:
pip install "apache-airflow[crypto]"
This will encrypt the passwords field (in Airflow connections) in the Airlfow MetaDB

Did you add fernet key on airflow.cfg? there should be a line where you can put the fernet
# Secret key to save connection passwords in the db
fernet_key = {insert here}

Categories