I am using the airflow application on my Linux server
note - Airflow is a platform to programmatically author, schedule and monitor workflows.
from some unclear reason we get the following issues ( when we restart the airflow or initialize the database , )
what should I check regarding the following errors?
airflow webserver
Traceback (most recent call last):
File "/usr/bin/airflow", line 4, in <module>
from airflow import configuration
File "/usr/lib/python2.7/site-packages/airflow/__init__.py", line 29, in <module>
from airflow import configuration as conf
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 657, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 519, in read
self._validate()
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 484, in get
option = self._get_cmd_option(section, key)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 465, in _get_cmd_option
return run_command(command)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 70, in run_command
.format(command, process.returncode, output, stderr)
airflow.configuration.AirflowConfigException: Cannot execute python /home/airflow/.sec/security.py get_connection_string pg. Error code is: 1. Output: , Stderr: Traceback (most recent call last):
File "/home/airflow/.sec/security.py", line 4, in <module>
from cryptography.fernet import Fernet
ImportError: No module named cryptography.fernet
# airflow initdb
Traceback (most recent call last):
File "/usr/bin/airflow", line 4, in <module>
from airflow import configuration
File "/usr/lib/python2.7/site-packages/airflow/__init__.py", line 29, in <module>
from airflow import configuration as conf
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 657, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 519, in read
self._validate()
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 484, in get
option = self._get_cmd_option(section, key)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 465, in _get_cmd_option
return run_command(command)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 70, in run_command
.format(command, process.returncode, output, stderr)
airflow.configuration.AirflowConfigException: Cannot execute python /home/airflow/.sec/security.py get_connection_string pg. Error code is: 1. Output: , Stderr: Traceback (most recent call last):
File "/home/airflow/.sec/security.py", line 4, in <module>
from cryptography.fernet import Fernet
ImportError: No module named cryptography.fernet
]# airflow list_dags
Traceback (most recent call last):
File "/usr/bin/airflow", line 4, in <module>
from airflow import configuration
File "/usr/lib/python2.7/site-packages/airflow/__init__.py", line 29, in <module>
from airflow import configuration as conf
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 657, in <module>
conf.read(AIRFLOW_CONFIG)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 519, in read
self._validate()
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 484, in get
option = self._get_cmd_option(section, key)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 465, in _get_cmd_option
return run_command(command)
File "/usr/lib/python2.7/site-packages/airflow/configuration.py", line 70, in run_command
.format(command, process.returncode, output, stderr)
airflow.configuration.AirflowConfigException: Cannot execute python /home/airflow/.sec/security.py get_connection_string pg. Error code is: 1. Output: , Stderr: Traceback (most recent call last):
File "/home/airflow/.sec/security.py", line 4, in <module>
from cryptography.fernet import Fernet
ImportError: No module named cryptography.fernet
we try also
pip install "apache-airflow[crypto]"
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting apache-airflow[crypto]
Using cached https://files.pythonhosted.org/packages/74/cb/3a4bcfcdb75897607c0c4d9ec17f3c3d6eb4973930d2fa59bd0b47c20a67/apache_airflow-1.10.6-py2.py3-none-any.whl
Requirement already satisfied: setproctitle<2,>=1.1.8 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.1.10)
Requirement already satisfied: termcolor==1.1.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.1.0)
Requirement already satisfied: colorlog==4.0.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (4.0.2)
Requirement already satisfied: dill<0.4,>=0.2.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.2.5)
Requirement already satisfied: typing; python_version < "3.5" in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (3.7.4.1)
Collecting flask-admin==1.5.3
Using cached https://files.pythonhosted.org/packages/c3/bc/4cb71a88d8b0d478ed7f46e7bba632e988daae5ae07ce5413a31acbe8166/Flask-Admin-1.5.3.tar.gz
Collecting flask-login<0.5,>=0.3
Using cached https://files.pythonhosted.org/packages/c1/ff/bd9a4d2d81bf0c07d9e53e8cd3d675c56553719bbefd372df69bf1b3c1e4/Flask-Login-0.4.1.tar.gz
Requirement already satisfied: markdown<3.0,>=2.5.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (2.6.7)
Requirement already satisfied: unicodecsv>=0.14.1 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.14.1)
Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.3.3)
Requirement already satisfied: enum34~=1.1.6; python_version < "3.4" in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.1.6)
Requirement already satisfied: text-unidecode==1.2 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.2)
Collecting flask-wtf<0.15,>=0.14.2
Using cached https://files.pythonhosted.org/packages/60/3a/58c629472d10539ae5167dc7c1fecfa95dd7d0b7864623931e3776438a24/Flask_WTF-0.14.2-py2.py3-none-any.whl
Requirement already satisfied: cached-property~=1.5 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.5.1)
Requirement already satisfied: graphviz>=0.12 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.13.2)
Collecting gunicorn<20.0,>=19.5.0
Using cached https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl
Requirement already satisfied: tabulate<0.9,>=0.7.5 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.8.5)
Requirement already satisfied: python-dateutil<3,>=2.3 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (2.6.0)
Requirement already satisfied: tenacity==4.12.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (4.12.0)
Requirement already satisfied: lazy-object-proxy~=1.3 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.4.3)
Requirement already satisfied: flask-swagger==0.2.13 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.2.13)
Collecting flask<2.0,>=1.1.0
Using cached https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl
Requirement already satisfied: argcomplete~=1.10 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (1.10.0)
Requirement already satisfied: marshmallow-sqlalchemy<0.19.0,>=0.16.1 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.18.0)
Requirement already satisfied: croniter<0.4,>=0.3.17 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (0.3.30)
Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (3.5.3)
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (4.4.0)
Requirement already satisfied: sqlalchemy~=1.3 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.3.10)
Collecting flask-appbuilder<2.0.0,>=1.12.5
Using cached https://files.pythonhosted.org/packages/90/9f/d8c4e693eb208042cc6dffcfa7b407db8e36cc89ec790a53d7cea5fa4bb8/Flask-AppBuilder-1.13.1.tar.gz
Processing /root/.cache/pip/wheels/2f/4c/cd/f0cabf03ccb7edd9002875db9001108b33ebfc60b4ff9a61ae/pendulum-1.4.4-cp27-cp27mu-linux_x86_64.whl
Collecting psutil<6.0.0,>=4.2.0
Using cached https://files.pythonhosted.org/packages/03/9a/95c4b3d0424426e5fd94b5302ff74cea44d5d4f53466e1228ac8e73e14b4/psutil-5.6.5.tar.gz
Collecting funcsigs==1.0.0
Using cached https://files.pythonhosted.org/packages/09/8d/17528625d12ca90651dd1f7958fd0d32b23b15f2197023372669fd683321/funcsigs-1.0.0-py2.py3-none-any.whl
Collecting future<0.17,>=0.16.0
Using cached https://files.pythonhosted.org/packages/00/2b/8d082ddfed935f3608cc61140df6dcbf0edea1bc3ab52fb6c29ae3e81e85/future-0.16.0.tar.gz
Collecting requests<3,>=2.20.0
Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Requirement already satisfied: thrift>=0.9.2 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (0.9.3)
Collecting alembic<2.0,>=1.0
Using cached https://files.pythonhosted.org/packages/70/3d/d5ed7a71fe84f9ed0a69e91232a40b0b148b151524dc5bb1c8e4211eb117/alembic-1.3.0.tar.gz
Collecting json-merge-patch==0.2
Using cached https://files.pythonhosted.org/packages/39/62/3b783faabac9a099877397d8f7a7cc862a03fbf9fb1b90d414ea7c6bb096/json-merge-patch-0.2.tar.gz
Collecting tzlocal<2.0.0,>=1.4
Using cached https://files.pythonhosted.org/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz
Requirement already satisfied: jinja2<2.11.0,>=2.10.1 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (2.10.3)
Requirement already satisfied: pygments<3.0,>=2.0.1 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (2.1.3)
Collecting iso8601>=0.1.12
Using cached https://files.pythonhosted.org/packages/ef/57/7162609dab394d38bbc7077b7ba0a6f10fb09d8b7701ea56fa1edc0c4345/iso8601-0.1.12-py2.py3-none-any.whl
Requirement already satisfied: pandas<1.0.0,>=0.17.1 in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (0.19.2)
Requirement already satisfied: python-daemon<2.2,>=2.1.1 in /usr/lib/python2.7/site-packages (from apache-airflow[crypto]) (2.1.2)
Requirement already satisfied: cryptography>=0.9.3; extra == "crypto" in /usr/lib64/python2.7/site-packages (from apache-airflow[crypto]) (1.7.1)
Requirement already satisfied: wtforms in /usr/lib/python2.7/site-packages (from flask-admin==1.5.3->apache-airflow[crypto]) (2.1)
Requirement already satisfied: Werkzeug>=0.12 in /usr/lib/python2.7/site-packages (from flask-caching<1.4.0,>=1.3.3->apache-airflow[crypto]) (0.12.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil<3,>=2.3->apache-airflow[crypto]) (1.9.0)
Requirement already satisfied: monotonic>=0.6; python_version == "2.7" in /usr/lib/python2.7/site-packages (from tenacity==4.12.0->apache-airflow[crypto]) (1.5)
Requirement already satisfied: futures>=3.0; python_version == "2.7" in /usr/lib/python2.7/site-packages (from tenacity==4.12.0->apache-airflow[crypto]) (3.3.0)
Requirement already satisfied: PyYAML>=3.0 in /usr/lib64/python2.7/site-packages (from flask-swagger==0.2.13->apache-airflow[crypto]) (3.12)
Requirement already satisfied: click>=5.1 in /usr/lib64/python2.7/site-packages (from flask<2.0,>=1.1.0->apache-airflow[crypto]) (7.0)
Requirement already satisfied: itsdangerous>=0.24 in /usr/lib/python2.7/site-packages (from flask<2.0,>=1.1.0->apache-airflow[crypto]) (0.24)
Requirement already satisfied: marshmallow>=2.0.0 in /usr/lib/python2.7/site-packages (from marshmallow-sqlalchemy<0.19.0,>=0.16.1->apache-airflow[crypto]) (2.20.5)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from zope.deprecation<5.0,>=4.0->apache-airflow[crypto]) (32.3.1)
Requirement already satisfied: colorama<1,>=0.3.9 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (0.4.1)
Requirement already satisfied: apispec[yaml]>=1.1.1<2 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (2.0.2)
Requirement already satisfied: Flask-Babel<1,>=0.11.1 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (0.12.2)
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (1.2.5)
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.3 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (2.4.1)
Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (3.24.1)
Requirement already satisfied: marshmallow-enum<2,>=1.4.1 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (1.5.1)
Collecting prison==0.1.0
Using cached https://files.pythonhosted.org/packages/a1/c4/076e1e2677ef928765b0fb537be6476fb8a96624be4669fb5a0618b32455/prison-0.1.0-py2.py3-none-any.whl
Collecting jsonschema>=3.0.1<4
Using cached https://files.pythonhosted.org/packages/ce/6c/888d7c3c1fce3974c88a01a6bc553528c99d3586e098eee23e8383dd11c3/jsonschema-3.1.1-py2.py3-none-any.whl
Requirement already satisfied: PyJWT>=1.7.1 in /usr/lib/python2.7/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (1.7.1)
Collecting pytzdata>=2018.3.0.0
Using cached https://files.pythonhosted.org/packages/7f/f9/cdd39831b0465285c02ed90cfbf0db25bb951cb2a35ded0e69222375bea3/pytzdata-2019.3-py2.py3-none-any.whl
Collecting ordereddict
Using cached https://files.pythonhosted.org/packages/53/25/ef88e8e45db141faa9598fbf7ad0062df8f50f881a36ed6a0073e1572126/ordereddict-1.1.tar.gz
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (1.25.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python2.7/site-packages (from requests<3,>=2.20.0->apache-airflow[crypto]) (2019.9.11)
Requirement already satisfied: Mako in /usr/lib/python2.7/site-packages (from alembic<2.0,>=1.0->apache-airflow[crypto]) (1.0.6)
Requirement already satisfied: python-editor>=0.3 in /usr/lib/python2.7/site-packages (from alembic<2.0,>=1.0->apache-airflow[crypto]) (1.0.3)
Requirement already satisfied: pytz in /usr/lib/python2.7/site-packages (from tzlocal<2.0.0,>=1.4->apache-airflow[crypto]) (2016.10)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib64/python2.7/site-packages (from jinja2<2.11.0,>=2.10.1->apache-airflow[crypto]) (1.0)
Requirement already satisfied: numpy>=1.7.0 in /usr/lib64/python2.7/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow[crypto]) (1.11.3)
Requirement already satisfied: docutils in /usr/lib/python2.7/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow[crypto]) (0.13.1)
Requirement already satisfied: lockfile>=0.10 in /usr/lib/python2.7/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow[crypto]) (0.12.2)
Requirement already satisfied: pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (0.1.9)
Requirement already satisfied: ipaddress in /usr/lib/python2.7/site-packages (from cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (1.0.17)
Requirement already satisfied: cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (1.9.1)
Collecting Babel>=2.3
Using cached https://files.pythonhosted.org/packages/2c/60/f2af68eb046c5de5b1fe6dd4743bf42c074f7141fe7b2737d3061533b093/Babel-2.7.0-py2.py3-none-any.whl
Requirement already satisfied: python-openid>=2.0 in /usr/lib/python2.7/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow[crypto]) (2.2.5)
Collecting pyrsistent>=0.14.0
Using cached https://files.pythonhosted.org/packages/30/86/53a88c0a57698fa228db29a4000c28f4124823010388cb7042fe6e2be8dd/pyrsistent-0.15.5.tar.gz
Collecting attrs>=17.4.0
Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting importlib-metadata
Using cached https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl
Collecting functools32; python_version < "3"
Using cached https://files.pythonhosted.org/packages/c5/60/6ac26ad05857c601308d8fb9e87fa36d0ebf889423f47c3502ef034365db/functools32-3.2.3-2.tar.gz
Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=0.9.3; extra == "crypto"->apache-airflow[crypto]) (2.17)
Collecting contextlib2; python_version < "3"
Using cached https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Collecting pathlib2; python_version == "3.4.*" or python_version < "3"
Using cached https://files.pythonhosted.org/packages/e9/45/9c82d3666af4ef9f221cbb954e1d77ddbb513faf552aea6df5f37f1a4859/pathlib2-2.3.5-py2.py3-none-any.whl
Collecting zipp>=0.5
Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
Collecting scandir; python_version < "3.5"
Using cached https://files.pythonhosted.org/packages/df/f5/9c052db7bd54d0cbf1bc0bb6554362bba1012d03e5888950a4f5c5dadc4e/scandir-1.10.0.tar.gz
Collecting more-itertools
Using cached https://files.pythonhosted.org/packages/2f/9d/dcfe59e213093695f108508af1214cf9cd95cc5489e46877ec5cb56369e5/more_itertools-5.0.0-py2-none-any.whl
ERROR: flask 1.1.1 has requirement Werkzeug>=0.15, but you'll have werkzeug 0.12.1 which is incompatible.
ERROR: flask-appbuilder 1.13.1 has requirement marshmallow<2.20,>=2.18.0, but you'll have marshmallow 2.20.5 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement alembic<0.9,>=0.8.3, but you'll have alembic 1.3.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement babel<2.0,>=1.3, but you'll have babel 2.7.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement flask<0.11,>=0.10.1, but you'll have flask 1.1.1 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement flask-admin==1.4.0, but you'll have flask-admin 1.5.3 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement flask-login==0.2.11, but you'll have flask-login 0.4.1 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement Flask-WTF==0.12, but you'll have flask-wtf 0.14.2 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement funcsigs<1,>=0.4, but you'll have funcsigs 1.0.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement future<0.16,>=0.15.0, but you'll have future 0.16.0 which is incompatible.
ERROR: airflow 1.7.1.3 has requirement gunicorn<19.4.0,>=19.3.0, but you'll have gunicorn 19.9.0 which is incompatible.
ERROR: jsonschema 3.1.1 has requirement six>=1.11.0, but you'll have six 1.9.0 which is incompatible.
ERROR: flask-jwt-extended 3.24.1 has requirement Werkzeug>=0.14, but you'll have werkzeug 0.12.1 which is incompatible.
Installing collected packages: flask, flask-admin, flask-login, flask-wtf, gunicorn, requests, prison, pyrsistent, attrs, contextlib2, scandir, pathlib2, more-itertools, zipp, importlib-metadata, functools32, jsonschema, flask-appbuilder, pytzdata, tzlocal, pendulum, psutil, ordereddict, funcsigs, future, alembic, json-merge-patch, iso8601, apache-airflow, Babel
Found existing installation: Flask 0.10.1
Uninstalling Flask-0.10.1:
Successfully uninstalled Flask-0.10.1
Found existing installation: Flask-Admin 1.4.0
Uninstalling Flask-Admin-1.4.0:
Successfully uninstalled Flask-Admin-1.4.0
Running setup.py install for flask-admin ... done
Found existing installation: Flask-Login 0.2.11
Uninstalling Flask-Login-0.2.11:
Successfully uninstalled Flask-Login-0.2.11
Running setup.py install for flask-login ... done
Found existing installation: Flask-WTF 0.12
Uninstalling Flask-WTF-0.12:
Successfully uninstalled Flask-WTF-0.12
Found existing installation: gunicorn 19.3.0
Uninstalling gunicorn-19.3.0:
Successfully uninstalled gunicorn-19.3.0
Found existing installation: requests 2.6.0
ERROR: Cannot uninstall 'requests'. It is a distutils installed project and thus we cannot accurately deter
the airflow script
!/usr/bin/python2
# PYTHON_ARGCOMPLETE_OK
# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import os
import argcomplete
from airflow.configuration import conf
from airflow.bin.cli import CLIFactory
if __name__ == '__main__':
if conf.get("core", "security") == 'kerberos':
os.environ['KRB5CCNAME'] = conf.get('kerberos', 'ccache')
os.environ['KRB5_KTNAME'] = conf.get('kerberos', 'keytab')
parser = CLIFactory.get_parser()
argcomplete.autocomplete(parser)
args = parser.parse_args()
args.func(args)
seems its not load the module as:
airflow
Traceback (most recent call last):
File "/usr/bin/airflow", line 25, in <module>
from airflow.configuration import conf
Can you install the following:
pip install "apache-airflow[crypto]"
This will encrypt the passwords field (in Airflow connections) in the Airlfow MetaDB
Did you add fernet key on airflow.cfg? there should be a line where you can put the fernet
# Secret key to save connection passwords in the db
fernet_key = {insert here}
Related
I am working on a project and have shifted my environment from local windows to a linux server (via SSH). I only have limited access as the host server is from my college, I've installed many packages without issues (both with and without virtualenv). I'm working on Python 3.6.9.
I was able to install spacy and import it but I need to use the en_core_web_sm package which has to be installed additionally using the command python3 -m spacy download en_core_web_sm. However, I consistently face a PermissionError as seen in the logs below.
Why am I facing this error? Is it because I don't have administrator access on the /usr level (refer to last line of logs)? If yes, how come only this package in particular requires a higher level access? If no, are there any workaround for me to install the package, or do I need to contact the server administrator?
jiayi#cdas1:~/adverse_drug_event_extraction/app/preprocessing$ python3 -m spacy download en_core_web_sm
Collecting en-core-web-sm==3.1.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl#egg=en_core_web_sm==3.1.0
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl (13.6MB)
100% |████████████████████████████████| 13.6MB 117kB/s
Requirement already satisfied: spacy<3.2.0,>=3.1.0 in /home/jiayi/.local/lib/python3.6/site-packages (from en-core-web-sm==3.1.0)
Requirement already satisfied: pathy>=0.3.5 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: blis<0.8.0,>=0.4.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: typer<0.4.0,>=0.3.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: jinja2 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.7 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/lib/python3/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: setuptools in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: catalogue<2.1.0,>=2.0.4 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: packaging>=20.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4; python_version < "3.8" in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: thinc<8.1.0,>=8.0.8 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: numpy>=1.15.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /home/jiayi/.local/lib/python3.6/site-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: dataclasses<1.0,>=0.6; python_version < "3.7" in /home/jiayi/.local/lib/python3.6/site-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: click<7.2.0,>=7.1.1 in /home/jiayi/.local/lib/python3.6/site-packages (from typer<0.4.0,>=0.3.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/jiayi/.local/lib/python3.6/site-packages (from jinja2->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: zipp>=0.5; python_version < "3.8" in /home/jiayi/.local/lib/python3.6/site-packages (from catalogue<2.1.0,>=2.0.4->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/jiayi/.local/lib/python3.6/site-packages (from packaging>=20.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: contextvars<3,>=2.4; python_version < "3.7" in /home/jiayi/.local/lib/python3.6/site-packages (from thinc<8.1.0,>=8.0.8->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: immutables>=0.9 in /home/jiayi/.local/lib/python3.6/site-packages (from contextvars<3,>=2.4; python_version < "3.7"->thinc<8.1.0,>=8.0.8->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Installing collected packages: en-core-web-sm
Exception:
Traceback (most recent call last):
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
**kwargs
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files
clobber(source, lib_dir, True)
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/wheel.py", line 316, in clobber
ensure_dir(destdir)
File "/home/jiayi/.local/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
os.makedirs(path)
File "/usr/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/en_core_web_sm-3.1.0.dist-info'
As you suspect, this is happening because you don't have permission to install to /usr. Normally pip would install there, but it looks like your pip is basically running with --user, which will install to your user home directory instead.
The spaCy models can't be installed directly via pip because they are large data files, so they can't be hosted on PyPI like ordinary code. An unfortunate side effect of this is that some of the options around pip configuration are ignored.
You can pass extra arguments to the pip install command by appending them to your command, so in your case you can do this:
python3 -m spacy download en_core_web_sm --user
And everything should work.
That said, I would strongly recommend you learn how to use virtual environments with the virtualenv tool, which will make working with Python projects easier and allow to you avoid this problem, as anything you install will just go in the local virtualenv rather than your global pip install.
It seems all other packages are installed under your user /home/jiayi/.local/python3.6/lib and this one tries to install itself globally in /usr/local/lib/python3.6/, not sure why. I guess you can give it installation folder or something.
Look here Where does spacy language model download?
I am facing a classic "Module not found error". However, I cannot solved my problem with the questions already posted.
I am trying to use the google.cloud API in order to download some data from a public Google repository to a local linux server.
Below are the instructions I used in a Jupyter server:
# Upgrading pip
!/usr/bin/python3.8 -m pip install --upgrade pip
# Installing the Google cloud storage API
!pip3 install google
!pip3 install google-cloud-storage
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google in /home/antoine/.local/lib/python3.8/site-packages (3.0.0)
Requirement already satisfied: beautifulsoup4 in /home/antoine/.local/lib/python3.8/site-packages (from google) (4.9.1)
Requirement already satisfied: soupsieve>1.2 in /home/antoine/.local/lib/python3.8/site-packages (from beautifulsoup4->google) (2.0.1)
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: google-cloud-storage in /home/antoine/.local/lib/python3.8/site-packages (1.30.0)
Requirement already satisfied: google-cloud-core<2.0dev,>=1.2.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-cloud-storage) (1.4.1)
Requirement already satisfied: google-resumable-media<2.0dev,>=0.6.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-cloud-storage) (0.7.1)
Requirement already satisfied: google-auth<2.0dev,>=1.11.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-cloud-storage) (1.20.1)
Requirement already satisfied: google-api-core<2.0.0dev,>=1.19.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage) (1.22.1)
Requirement already satisfied: google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5" in /home/antoine/.local/lib/python3.8/site-packages (from google-resumable-media<2.0dev,>=0.6.0->google-cloud-storage) (0.1.0)
Requirement already satisfied: six in /home/antoine/.local/lib/python3.8/site-packages (from google-resumable-media<2.0dev,>=0.6.0->google-cloud-storage) (1.15.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage) (4.1.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/lib/python3/dist-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage) (0.2.1)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.5" in /home/antoine/.local/lib/python3.8/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage) (4.6)
Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage) (49.3.0)
Requirement already satisfied: pytz in /usr/lib/python3/dist-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage) (2018.3)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /usr/lib/python3/dist-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage) (2.18.4)
Requirement already satisfied: protobuf>=3.12.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage) (3.13.0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-api-core<2.0.0dev,>=1.19.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage) (1.52.0)
Requirement already satisfied: cffi>=1.0.0 in /home/antoine/.local/lib/python3.8/site-packages (from google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"->google-resumable-media<2.0dev,>=0.6.0->google-cloud-storage) (1.14.1)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<5,>=3.1.4; python_version >= "3.5"->google-auth<2.0dev,>=1.11.0->google-cloud-storage) (0.4.2)
Requirement already satisfied: pycparser in /home/antoine/.local/lib/python3.8/site-packages (from cffi>=1.0.0->google-crc32c<0.2dev,>=0.1.0; python_version >= "3.5"->google-resumable-media<2.0dev,>=0.6.0->google-cloud-storage) (2.20)
from google.cloud import storage
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-48-d626caf5a463> in <module>
----> 1 from google.cloud import storage
2
3
4 def download_blob(bucket_name, source_blob_name, destination_file_name):
5 """Downloads a blob from the bucket."""
ModuleNotFoundError: No module named 'google'
Thank you for your advice!
Sometimes, when you have multiple versions of Python installed (with multiple Python 3.X versions), using pip3 is not enough, as it does not necessarily install a package for the latest version of Python installed on your machine.
You can check what version of python your pip3 is using by using the pip3 --version command.
If you plan to run a script with Python 3.8 and need to install a package for this specific version of Python, you can use:
pip3.8 install PACKAGE or
python3.8 -m pip install PACKAGE
Libraries are installed - not sure why this is throwing an error.
It works on one of my rigs with the same platform - ubuntu 19.10 - python3.8 and the associated libraries in a virtualenv.
I'm going to show you what I see:
Here's im trying to validate spacy -- which fails.
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ python -m spacy validate
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.8/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.8/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/home/gordon/p3env/lib/python3.8/site-packages/spacy/__init__.py", line 10, in <module>
from thinc.neural.util import prefer_gpu, require_gpu
File "/home/gordon/p3env/lib/python3.8/site-packages/thinc/neural/__init__.py", line 4, in <module>
from ._classes.model import Model # noqa: F401
File "/home/gordon/p3env/lib/python3.8/site-packages/thinc/neural/_classes/model.py", line 7, in <module>
import srsly
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/__init__.py", line 7, in <module>
from ._pickle_api import pickle_dumps, pickle_loads
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/_pickle_api.py", line 4, in <module>
from . import cloudpickle
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/cloudpickle/__init__.py", line 1, in <module>
from .cloudpickle import *
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/cloudpickle/cloudpickle.py", line 167, in <module>
_cell_set_template_code = _make_cell_set_template_code()
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/cloudpickle/cloudpickle.py", line 148, in _make_cell_set_template_code
return types.CodeType(
TypeError: an integer is required (got type bytes)
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ ^C
So I just try to install some of those libraries which turn out to be present. - I know everything I need is installed --
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ pip install thinc
Requirement already satisfied: thinc in /home/gordon/p3env/lib/python3.8/site-packages (7.1.1)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (0.9.6)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (1.0.2)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (2.0.2)
Requirement already satisfied: numpy>=1.7.0 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (1.17.3)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (0.4.1)
Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (0.1.0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (4.36.1)
Requirement already satisfied: wasabi<1.1.0,>=0.0.9 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (0.3.0)
Requirement already satisfied: preshed<3.1.0,>=1.0.1 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc) (3.0.2)
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ pip install cloudpickle
Requirement already satisfied: cloudpickle in /home/gordon/p3env/lib/python3.8/site-packages (1.2.2)
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ pip install srsly
Requirement already satisfied: srsly in /home/gordon/p3env/lib/python3.8/site-packages (0.1.0)
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ pip install spacy
Requirement already satisfied: spacy in /home/gordon/p3env/lib/python3.8/site-packages (2.2.1)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (3.0.2)
Requirement already satisfied: numpy>=1.15.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (1.17.3)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (2.22.0)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.4.1)
Requirement already satisfied: srsly<1.1.0,>=0.1.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.1.0)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (1.0.2)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (2.0.2)
Requirement already satisfied: thinc<7.2.0,>=7.1.1 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (7.1.1)
Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.3.0)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.9.6)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.25.6)
Requirement already satisfied: idna<2.9,>=2.5 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2019.9.11)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc<7.2.0,>=7.1.1->spacy) (4.36.1)
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ pip uninstall spacy
Uninstalling spacy-2.2.1:
Would remove:
/home/gordon/p3env/bin/spacy
/home/gordon/p3env/lib/python3.8/site-packages/bin/*
/home/gordon/p3env/lib/python3.8/site-packages/spacy-2.2.1.dist-info/*
/home/gordon/p3env/lib/python3.8/site-packages/spacy/*
Proceed (y/n)? y
Successfully uninstalled spacy-2.2.1
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ pip install spacy
Processing /home/gordon/snap/code/common/.cache/pip/wheels/1d/22/2b/2d5d8fe5e5fe728a766158a89d017051c19005bb7aabc1d79a/spacy-2.2.1-cp38-cp38-linux_x86_64.whl
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (1.0.2)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.4.1)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (3.0.2)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (2.22.0)
Requirement already satisfied: srsly<1.1.0,>=0.1.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.1.0)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (2.0.2)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.9.6)
Requirement already satisfied: thinc<7.2.0,>=7.1.1 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (7.1.1)
Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (0.3.0)
Requirement already satisfied: numpy>=1.15.0 in /home/gordon/p3env/lib/python3.8/site-packages (from spacy) (1.17.3)
Requirement already satisfied: certifi>=2017.4.17 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2019.9.11)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.25.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /home/gordon/p3env/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.8)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/gordon/p3env/lib/python3.8/site-packages (from thinc<7.2.0,>=7.1.1->spacy) (4.36.1)
Installing collected packages: spacy
Successfully installed spacy-2.2.1
RIGHT HERE -- I just try importing spacy and this is what I'm getting on one of the machines... while the other works -- odd I'll need to look into this -
Has anyone run into this ?
(p3env) gordon#citadel:~/p3env/alice/alice/spiders$ python
Python 3.8.0 (default, Oct 15 2019, 09:10:20)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import spacy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/gordon/p3env/lib/python3.8/site-packages/spacy/__init__.py", line 10, in <module>
from thinc.neural.util import prefer_gpu, require_gpu
File "/home/gordon/p3env/lib/python3.8/site-packages/thinc/neural/__init__.py", line 4, in <module>
from ._classes.model import Model # noqa: F401
File "/home/gordon/p3env/lib/python3.8/site-packages/thinc/neural/_classes/model.py", line 7, in <module>
import srsly
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/__init__.py", line 7, in <module>
from ._pickle_api import pickle_dumps, pickle_loads
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/_pickle_api.py", line 4, in <module>
from . import cloudpickle
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/cloudpickle/__init__.py", line 1, in <module>
from .cloudpickle import *
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/cloudpickle/cloudpickle.py", line 167, in <module>
_cell_set_template_code = _make_cell_set_template_code()
File "/home/gordon/p3env/lib/python3.8/site-packages/srsly/cloudpickle/cloudpickle.py", line 148, in _make_cell_set_template_code
return types.CodeType(
TypeError: an integer is required (got type bytes
I don't believe this happens outside of the virtualenv.
Let me know if you have seen this one, when I fix it I'll update the post.
$ python3.8 -m pip install srsly --upgrade
Defaulting to user installation because normal site-packages is not writeable
Collecting srsly
Downloading srsly-2.0.0-cp38-cp38-manylinux1_x86_64.whl (311 kB)
|████████████████████████████████| 311 kB 1.3 MB/s
ERROR: thinc 7.3.1 has requirement srsly<1.1.0,>=0.0.6, but you'll have srsly 2.0.0 which is incompatible.
ERROR: spacy 2.2.3 has requirement srsly<1.1.0,>=0.1.0, but you'll have srsly 2.0.0 which is incompatible.
Installing collected packages: srsly
Attempting uninstall: srsly
Found existing installation: srsly 0.1.0
Uninstalling srsly-0.1.0:
Successfully uninstalled srsly-0.1.0
Successfully installed srsly-2.0.0
You might want to use python3.8 -m pip install srsly==1.0.1.
I've been trying to install the openquake.engine package via Anaconda. At first I got multiple errors one after another but [I thought] I fixed everything (pls see the P.S. for errors and solutions to them). Now, after going through all that, I get the following error:
openquake-engine 3.3.2 has requirement numpy<1.15,>=1.14, but you'll have numpy 1.16.2 which is incompatible.
osmnx 0.9 has requirement networkx>=2.2, but you'll have networkx 2.1 which is incompatible.
osmnx 0.9 has requirement requests>=2.21, but you'll have requests 2.20.1 which is incompatible.
First, it does not make sense as in PyPi the last released date of the package is Jan 22, 2019! Then here is the main issue: when I use conda list, openquake.engine is not listed!!
But when I runpip install openquake.engine, I get the following:
Requirement already satisfied: openquake.engine in c:\users\ameri19\appdata\roaming\python\python36\site-packages (3.3.2)
Requirement already satisfied: numpy<1.15,>=1.14 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (1.14.6)
Requirement already satisfied: nose<1.4,>=1.3 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (1.3.7)
Requirement already satisfied: requests<2.21,>=2.20 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (2.20.1)
Requirement already satisfied: scipy<1.2,>=1.0.1 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (1.1.0)
Requirement already satisfied: docutils<0.15,>=0.11 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (0.14)
Requirement already satisfied: mock<2.1,>=1.0 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (2.0.0)
Requirement already satisfied: rtree==0.8.3 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (0.8.3)
Requirement already satisfied: decorator>=4.3 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (4.3.2)
Requirement already satisfied: shapely<1.7,>=1.3 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (1.6.4.post2)
Requirement already satisfied: matplotlib<2.3,>=1.5 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (2.2.4)
Requirement already satisfied: pyshp==1.2.3 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (1.2.3)
Requirement already satisfied: setuptools in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (40.8.0)
Requirement already satisfied: h5py<2.9,>=2.8 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from openquake.engine) (2.8.0)
Requirement already satisfied: pyzmq<18.0 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (17.1.3)
Requirement already satisfied: django<2.1,>=1.10 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (2.0.13)
Requirement already satisfied: psutil<5.5,>=2.0 in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (5.4.3)
Requirement already satisfied: PyYAML in c:\programdata\anaconda3\lib\site-packages (from openquake.engine) (3.12)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in c:\programdata\anaconda3\lib\site-packages (from requests<2.21,>=2.20->openquake.engine) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in c:\programdata\anaconda3\lib\site-packages (from requests<2.21,>=2.20->openquake.engine) (2018.11.29)
Requirement already satisfied: idna<2.8,>=2.5 in c:\programdata\anaconda3\lib\site-packages (from requests<2.21,>=2.20->openquake.engine) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\programdata\anaconda3\lib\site-packages (from requests<2.21,>=2.20->openquake.engine) (3.0.4)
Requirement already satisfied: pbr>=0.11 in c:\users\ameri19\appdata\roaming\python\python36\site-packages (from mock<2.1,>=1.0->openquake.engine) (5.1.3)
Requirement already satisfied: six>=1.9 in c:\programdata\anaconda3\lib\site-packages (from mock<2.1,>=1.0->openquake.engine) (1.11.0)
Requirement already satisfied: python-dateutil>=2.1 in c:\programdata\anaconda3\lib\site-packages (from matplotlib<2.3,>=1.5->openquake.engine) (2.6.1)
Requirement already satisfied: pytz in c:\programdata\anaconda3\lib\site-packages (from matplotlib<2.3,>=1.5->openquake.engine) (2017.3)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\programdata\anaconda3\lib\site-packages (from matplotlib<2.3,>=1.5->openquake.engine) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\programdata\anaconda3\lib\site-packages (from matplotlib<2.3,>=1.5->openquake.engine) (2.2.0)
Requirement already satisfied: cycler>=0.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib<2.3,>=1.5->openquake.engine) (0.10.0)
Then, when I use pip uninstall openquake.engine, I get the following error:
Exception:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\commands\uninstall.py", line 75, in run
auto_confirm=options.yes, verbose=self.verbosity > 0,
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\req\req_install.py", line 816, in uninstall
uninstalled_pathset = UninstallPathSet.from_dist(dist)
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\req\req_uninstall.py", line 496, in from_dist
for path in uninstallation_paths(dist):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\req\req_uninstall.py", line 50, in unique
for item in fn(*args, **kw):
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\req\req_uninstall.py", line 67, in uninstallation_paths
r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1414, in get_metadata_lines
return yield_lines(self.get_metadata(name))
File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1411, in get_metadata
return value.decode('utf-8') if six.PY3 else value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 152595: invalid continuation byte
Also, after I run pip install openquake.engine, it seems it automatically uninstalls the latest version of numpy and installs a former version that is compatible to openquake.engine, but then when I run something like from openquake.hazardlib import const I get the following error:
could not find or load spatialindex_c.dll
which is error #3 in the P.S. section below! And after I run conda install -c conda-forge osmnx to solve the issue, I start getting this error:
Something is wrong with the numpy installation. While importing we detected an older version of numpy in ['C:\\ProgramData\\Anaconda3\\lib\\site-packages\\numpy']. One method of fixing this is to repeatedly uninstall numpy until none is found, then reinstall this version.
So it seems like a deadlock! Dose anyone know how I could fix this?
P.S. Errors and Solutions:
error #1: (occurred while using 'pip install openquake.engine' in Anaconda)
Command "python setup.py egg_info" failed with error code 1 in C:\temp\pip-install-mco6__1j\shapely\
Solution to error #1:
conda install -c conda-forge shapely
error #2: (occurred while using 'pip install openquake.engine' in Anaconda, EVEN THOUGH I was running the prompt as administrator!!)
PermissionError: [WinError 5] Access is denied
Solution to error #2:
pip install openquake.engine --user
Here, I got multiple warnings, but they did not seem to be that important! I tried import epenquake.engine in the Spyder and as it worked like a charm I thought it is all good (but it was not; see error #3)!
error #3: (occured while running from openquake.hazardlib import const)
could not find or load spatialindex_c.dll
Solution to error #3:
conda install -c conda-forge osmnx
C:\Users\Administrator>pip3 install scrapy
Collecting scrapy
Using cached Scrapy-1.4.0-py2.py3-none-any.whl
Requirement already satisfied: pyOpenSSL in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Requirement already satisfied: service-identity in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Requirement already satisfied: lxml in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Requirement already satisfied: six>=1.5.2 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Requirement already satisfied: w3lib>=1.17.0 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Collecting Twisted>=13.1.0 (from scrapy)
Using cached Twisted-17.5.0.tar.bz2
Requirement already satisfied: cssselect>=0.9 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Requirement already satisfied: queuelib in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Requirement already satisfied: parsel>=1.1 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from scrapy)
Collecting PyDispatcher>=2.0.5 (from scrapy)
Requirement already satisfied: cryptography>=1.9 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from pyOpenSSL->scrapy)
Requirement already satisfied: attrs in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from service-identity->scrapy)
Requirement already satisfied: pyasn1 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from service-identity->scrapy)
Requirement already satisfied: pyasn1-modules in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from service-identity->scrapy)
Requirement already satisfied: zope.interface>=4.0.2 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied: constantly>=15.1 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied: incremental>=16.10.1 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied: Automat>=0.3.0 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied: hyperlink>=17.1.1 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied: idna>=2.1 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from cryptography>=1.9->pyOpenSSL->scrapy)
Requirement already satisfied: cffi>=1.7 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from cryptography>=1.9->pyOpenSSL->scrapy)
Requirement already satisfied: asn1crypto>=0.21.0 in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from cryptography>=1.9->pyOpenSSL->scrapy)
Requirement already satisfied: setuptools in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->scrapy)
Requirement already satisfied: pycparser in c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages (from cffi>=1.7->cryptography>=1.9->pyOpenSSL->scrapy)
Building wheels for collected packages: Twisted
Running setup.py bdist_wheel for Twisted ... error
Failed building wheel for Twisted
Running setup.py clean for Twisted
Failed to build Twisted
Installing collected packages: Twisted, PyDispatcher, scrapy
Running setup.py install for Twisted ... error
Exception:
Traceback (most recent call last):
File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages\pip\compat\__init__.py",
line 73, in console_to_str
return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 2: invalid continuation byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages\pip\commands\install.py",
line 342, in run
prefix=options.prefix_path,
File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages\pip\req\req_set.py",
line 784, in install
**kwargs
File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages\pip\req\req_install.py",
line 878, in install
spinner=spinner,
File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages\pip\utils\__init__.py",
line 676, in call_subprocess
line = console_to_str(proc.stdout.readline())
File "c:\users\administrator\appdata\local\programs\python\python36-32\lib\site-packages\pip\compat\__init__.py",
line 75, in console_to_str
return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 2: invalid continuation byte
I ran into the same problem while installing scrapy. The solution for me was a little bit different, not the file pip_vendor\pkg_resources__init__.py, but the \pip\compat__init__.py, I've changed line 75 to mbcs, and then pip installed scrapy successfully!
After that, I changed that line back to utf-8.