We have some issues deploying our GCP Dataflow pipeline. After some analysis, found that the latest version of apache-beam has some issues while installing. To replicate the issue I created a virtualenv and ran the below
pip install apache-beam==2.32.0
Below errors started to pop while installing the 'orjson' dependency,
Using cached orjson-3.6.3.tar.gz (548 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... error
ERROR: Command errored out with exit status 1:
command: 'c:\temp\virtu\scripts\python.exe' 'c:\temp\virtu\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\prasasaw\AppData\Local\Temp\tmpmekx1jjj'
cwd: C:\Users\prasasaw\AppData\Local\Temp\pip-install-0yofoe55\orjson_62f1ca2674934a7f8c45b08e87e05a4b
Complete output (6 lines):
Cargo, the Rust package manager, is not installed or is not on PATH.
This package requires Rust and Cargo to compile extensions. Install it through
the system's package manager or via https://rustup.rs/
Note that the previous versions of Apache beam like 2.30.0 does not have the dependency on 'orjson' and it works just fine when you do
pip install apache-beam==2.30.0
I tried to install RUST but it failed for some 'pysam' dependency. So would like to know what is the correct way to install the 'orjson' dependency
I saw this GitHub issue for orjson installation but could not find much from it.
https://github.com/readthedocs/readthedocs.org/issues/7687
//Prasad.
I'm a bit late to the party, but I ran into this issue as well today. I solved it by switching to a 64-bit Python environment (I accidently installed 32-bit).
The orjson dependency was introduced in https://github.com/apache/beam/pull/14690/files. According to the comment: orjson, only available on Python 3.6 and above. You may want to check your python version.
According to the orjson docs, you need to upgrade pip to be above 20.3:
pip install --upgrade "pip>=20.3" # manylinux_x_y, universal2 wheel support
pip install --upgrade orjson
(docs)
Related
I was installing firebase-admin using pip on my raspberry pi 3b+ using the following command:
pip3 install firebase-admin
However it always ends with an error saying "Error building wheel for cryptography"
Here is the full error message:
/tmp/pip-build-env-k7qo7p54/overlay/lib/python3.7/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'cryptography.hazmat.bindings._rust' as data is deprecated, please list it in packages.
!!
############################
# Package would be ignored #
############################
Python recognizes 'cryptography.hazmat.bindings._rust' as an importable package,
but it is not listed in the `packages` configuration of setuptools.
'cryptography.hazmat.bindings._rust' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).
Please make sure that 'cryptography.hazmat.bindings._rust' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" and "data files" on setuptools
documentation page.
!!
check.warn(importable)
=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation/ for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq/
4) Ensure you have a recent Rust toolchain installed:
https://cryptography.io/en/latest/installation/#rust
Python: 3.7.3
platform: Linux-5.10.103-v7+-armv7l-with-debian-10.13
pip: n/a
setuptools: 67.0.0
setuptools_rust: 1.5.2
rustc: n/a
=============================DEBUG ASSISTANCE=============================
error: can't find Rust compiler
If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
To update pip, run:
pip install --upgrade pip
and then retry package installation.
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
This package requires Rust >=1.48.0.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects
I have tried different commands like:
pip3 install firebase-admin
or
python -m pip install firebase-admin
I have upgraded pip and tried again, but still it didn't work
I faced a similar problem long time ago and solved by installing openssl. Tried it this time, it didn't work
Also tried installing rust compiler and using it but it didn't work
I am using Raspbian OS Buster on my Raspberry Pi 3
I have tried fresh installs of the OS
Still the same problem occurs
I have 2 of my projects that install perfectly fine on ubuntu, both require python3.8, both are installable using python setup.py install and I occasionally uninstall / install them from / to virtual environments as well as globally for testing purposes. Recently, all kinds of troubles and error messages started to occur not following any special event, I just woke up one day, and tried to install one of them, I got an error, then I tried the other and got another error. I will include both setup.py scripts and show the steps I follow on macOS Big Sur 11.0 to reproduce the results, given that both install perfectly fine on Ubuntu with the same list of respective dependencies.
Versions:
python: 3.8.6
pip: 20.3.1
macOS: 11.0.1
Project 1:
requirements.txt
imagesize==1.2.0
numpy==1.18.5
pandas==1.1.4
seaborn==0.11.0
tensorflow==2.3.1
matplotlib==3.3.3
lxml==4.6.2
imgaug==0.4.0
tensorflow-addons==0.11.2
opencv-python-headless==4.4.0.46
imagecorruptions==1.1.2
configparser~=5.0.1
scipy==1.5.4
PyQt5==5.15.2
tabulate==0.8.7
ipykernel==5.3.4
setup.py
from setuptools import setup, find_packages
install_requires = [dep.strip() for dep in open('requirements.txt')]
setup(
name='project1',
version='1.0',
packages=find_packages(),
url='url',
license='MIT',
author='author',
author_email='email#domain',
description='description goes here',
install_requires=install_requires,
python_requires='>=3.8',
entry_points={
'console_scripts': [
'some_entry_point',
],
},
)
Notes:
I will truncate logs for character limits.
The attempts below are done within a virtualenv
All kinds of problems do not occur if I just use pip install -r requirements
Installation works perfectly fine on Ubuntu and used to work perfectly fine on macOS even after I upgraded to Big Sur, it just suddenly decided to never work again.
python is installed using brew install python#3.8
Attempt 1:
virtualenv proj1
source proj1/bin/activate
python setup.py install
Result:
Searching for PyQt5==5.15.2
Reading https://pypi.org/simple/PyQt5/
Downloading https://files.pythonhosted.org/packages/28/6c/640e3f5c734c296a7193079a86842a789edb7988dca39eab44579088a1d1/PyQt5-5.15.2.tar.gz#sha256=372b08dc9321d1201e4690182697c5e7ffb2e0770e6b4a45519025134b12e4fc
Best match: PyQt5 5.15.2
Processing PyQt5-5.15.2.tar.gz
error: Couldn't find a setup script in /var/folders/hr/61r_7jcx2r3cnklwrr2zwbqw0000gn/T/easy_install-sl9y34mj/PyQt5-5.15.2.tar.gz
Attempt 2: (I install pyqt using pip)
pip install pyqt5==5.15.2 # success
python setup.py install # fail
Result:
No module named numpy.
Attempt 3: (add setup_requires=['numpy==1.18.5'] to setup())
python setup.py install
Result:
numpy.distutils.system_info.NotFoundError: No lapack/blas resources found. Note: Accelerate is no longer supported.
During handling of the above exception, another exception occurred:
# Traceback goes here
RuntimeError: implement_array_function method already has a docstring
Attempt 4:
I checked this issue and there is nothing helpful yet so I manually install numpy and scipy:
pip install numpy==1.18.5 scipy # success
python setup.py install # fail
Result:
ModuleNotFoundError: No module named 'skbuild'
Attempt 5:
pip install scikit-build==0.11.1 # success
python setup.py install # fail
Result:
error: Setup script exited with Problem with the CMake installation, aborting build. CMake executable is cmake
Attempt 6:
I checked the issue here and accordingly:
pip install cmake # success
python setup.py install
Opencv is being built, I'm not sure why or what triggered the build so I aborted and manually installed opencv using pip:
pip install opencv-python-headless==4.4.0.46 # success
python setup.py install # fail
Result:
Could not find suitable distribution for Requirement.parse('tensorflow-addons==0.11.2')
Attempt 7:
pip install tensorflow-addons==0.11.2 # success
python setup.py install
Again lxml is being built, I don't know how or why, so I manually install it and:
pip install lxml==4.6.2 # success
python setup.py install # fail
Result:
Running matplotlib-3.3.3/setup.py -q bdist_egg --dist-dir /var/folders/hr/61r_7jcx2r3cnklwrr2zwbqw0000gn/T/easy_install-q32mufo3/matplotlib-3.3.3/egg-dist-tmp-_js7sem9
UPDATING build/lib.macosx-11.0-x86_64-3.8/matplotlib/_version.py
set build/lib.macosx-11.0-x86_64-3.8/matplotlib/_version.py to '3.3.3'
error: Setup script exited with error: Failed to download FreeType. Please download one of ['https://downloads.sourceforge.net/project/freetype/freetype2/2.6.1/freetype-2.6.1.tar.gz', 'https://download.savannah.gnu.org/releases/freetype/freetype-2.6.1.tar.gz'] and extract it into build/freetype-2.6.1 at the top-level of the source repository.
Attempt 8:
pip install matplotlib==3.3.3 # success
python setup.py install # fail
Result:
Could not find suitable distribution for Requirement.parse('tensorflow==2.3.1')
Attempt 9:(already more than 75% of the requirements were already installed manually using pip in the previous attempts)
pip install tensorflow==2.3.1 # success
python setup.py install # fail
Result:(pandas fails to install)
RuntimeError: Cannot cythonize without Cython installed.
Attempt 10: (I manually install cython)
pip install cython # success
python setup.py install
Again pandas for some reason is being built, so I manually install it and the list goes on ...
I think after this simple demonstration, I don't even need to replicate the steps of the second project however here are its requirements and I pretty much get a variation of the same problems shown above.
requirements.txt for project2
oauth2client==4.1.3
gcloud==0.18.3
pyarrow==2.0.0
requests==2.24.0
pandas==1.1.4
httplib2==0.15.0
TA-Lib==0.4.19
matplotlib==3.3.2
matplotlib fails first, followed by ta-lib (given that I run brew install ta-lib before then) and the list goes on ...
Instead of python setup.py install do python -m pip install ..
Instead of python setup.py develop do python -m pip install --editable ..
I was asked to fix an old installation of a Django App
I was going through the package dependencies and saw that during a server change they probably lost some python modules:
Installed is:
Python 2.7.5 and Django (1.8)
pip 8.1.2
edit: django-mptt 0.9.1
I need to install django-mptt-admin
but I get this error when doing pip install django-mptt-admin:
...
Collecting Django>=1.11 (from django-mptt->django-mptt-admin==0.5.8)
Using cached https://files.pythonhosted.org/packages/7e/ae/29c28f6afddae0e305326078f31372f03d7f2e6d6210c9963843196ce67e/Django-2.1.7.tar.gz
Complete output from command python setup.py egg_info:
==========================
Unsupported Python version
==========================
This version of Django requires Python 3.5, but you're trying to
install it on Python 2.7.
This may be because you are using a version of pip that doesn't
understand the python_requires classifier. Make sure you
have pip >= 9.0 and setuptools >= 24.2, then try again:
the question is now:
How can I find the right version of django-mptt-admin and save myself the problem of upragding all python, django and their dependencies?
Its a non documented app which probably has some more problems in it
In tried
pip install django-mptt-admin==0.2.1 (from 0.5.8 down to 0.2.1) but always with the same error.
I think you install django-mptt as dependency for django-mptt-admin. And there in dependencies is Django>=1.11. You must fix version of django-mptt-admin and django-mptt for both.
Why don't you check the release history?
https://pypi.org/project/django-mptt-admin/#history
After installing Python 3.5.1 on Windows 10 x64, cpplint installed from pip produces the error: failed to create process. There seems to be a possibly related issue with pip related to having a space in the path, which exists in my use case. It seems to be related to how the exe is created. How can this error be remedied? Reinstalling Python and cpplint does not solve the problem.
C:\Users>python -V
Python 3.5.1
C:\Users>pip list
pip (7.1.2)
setuptools (18.2)
C:\Users>pip -V
pip 7.1.2 from c:\users\john hagen\appdata\local\programs\python\python35\lib\site-packages (python 3.5)
C:\Users>pip install cpplint
Collecting cpplint
Using cached cpplint-0.0.6.tar.gz
Installing collected packages: cpplint
Running setup.py install for cpplint
Successfully installed cpplint-0.0.6
C:\Users>where.exe cpplint
C:\Users\John Hagen\AppData\Local\Programs\Python\Python35\Scripts\cpplint.exe
C:\Users>cpplint
failed to create process.
Yep, this is a pip bug.
The workaround is to add quotes around the path in the first line of generated python scripts in the \Scripts\ directory. In the case of cpplint this is 'cpplint-script.py'. In your case the generated first line should something like:
#!c:\users\john hagen\appdata\local\programs\python\python35\python.exe
and should be edited to:
#!"c:\users\john hagen\appdata\local\programs\python\python35\python.exe"
I am just starting with virtualenv, but I am trying to install gevent within a virtualenv environment (I am running Windows). When I use PIP from virtualenv, I get this error:
MyEnv>pip install gevent
Downloading/unpacking gevent
Running setup.py egg_info for package gevent
Please provide path to libevent source with --libevent DIR
The package index has MSIs and EXEs for installing on Windows (http://pypi.python.org/pypi/gevent/0.13.7), but I don't know how to install those into a virtualenv environment (or if that is even possible). When I try pip install gevent-0.13.7.win32-py2.7.exe from the virtualenv promp, I get an error as well:
ValueError: ('Expected version spec in', 'D:\\Downloads\\gevent-0.13.7.win32-py2.7.exe', 'at', ':\\Downloads\\gevent-0.13.7.win32-py2.7.exe')
Does someone know how to do this?
Pip doesn't support installing binary packages, yet. If you want to install from binary package you have to use easy_install - easy_install gevent-0.13.7.win32-py2.7.exe
Microsoft Windows XP [Wersja 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
Z:\>virtualenv z:\venv\gevent-install
New python executable in z:\venv\gevent-install\Scripts\python.exe
Installing distribute..................................................................................................
............................................................................................done.
Installing pip.................done.
Z:\>venv\gevent-install\Scripts\activate
(gevent-install) Z:\>easy_install c:\python\packages\gevent-0.13.7.win32-py2.7.exe
Processing gevent-0.13.7.win32-py2.7.exe
creating 'c:\docume~1\pdobro~1\ustawi~1\temp\easy_install-b5nj3i\gevent-0.13.7-py2.7-win32.egg' and adding 'c:\docume~1
pdobro~1\ustawi~1\temp\easy_install-b5nj3i\gevent-0.13.7-py2.7-win32.egg.tmp' to it
creating z:\venv\gevent-install\lib\site-packages\gevent-0.13.7-py2.7-win32.egg
Extracting gevent-0.13.7-py2.7-win32.egg to z:\venv\gevent-install\lib\site-packages
Adding gevent 0.13.7 to easy-install.pth file
Installed z:\venv\gevent-install\lib\site-packages\gevent-0.13.7-py2.7-win32.egg
Processing dependencies for gevent==0.13.7
Searching for greenlet
Reading http://pypi.python.org/simple/greenlet/
Reading http://bitbucket.org/ambroff/greenlet
Reading https://github.com/python-greenlet/greenlet
Best match: greenlet 0.3.4
Downloading http://pypi.python.org/packages/2.7/g/greenlet/greenlet-0.3.4-py2.7-win32.egg#md5=9941aa246358c586bb274812e
130629
Processing greenlet-0.3.4-py2.7-win32.egg
creating z:\venv\gevent-install\lib\site-packages\greenlet-0.3.4-py2.7-win32.egg
Extracting greenlet-0.3.4-py2.7-win32.egg to z:\venv\gevent-install\lib\site-packages
Adding greenlet 0.3.4 to easy-install.pth file
Installed z:\venv\gevent-install\lib\site-packages\greenlet-0.3.4-py2.7-win32.egg
Finished processing dependencies for gevent==0.13.7
(gevent-install) Z:\>
See Can I install Python windows packages into virtualenvs? Another option is to install from source and you can do this with pip but this requires setting up compiler and environment which is much harder than the simple command above.
From the error message, it would appear you need libevent source code. I would imagine you need to go a step further and compile/install libevent system-wide so pip can find it.
I would start by downloading the latest stable source from http://libevent.org/.
Compile and install it using instructions in the README: https://github.com/libevent/libevent#readme
To compile it on Windows, you'll need to use GNU-style build utilities like make and autoconf. I recommend http://www.mingw.org/.
Once you've installed libevent system-wide, I imagine pip will find it and proceed with gevent installation.
In the msi for gevent-0.13.7 there's an option to select an alternate installation point. point it to the root dir of your particular virtual environment (just above where /Lib and /Scripts are located). That should install it correctly.
You also need to make sure greenlets are installed. For that you can use Piotr's suggested method with easy_install on the .exe.