I've been getting the slug size too large warning (Compiled slug size: 789.8M is too large (max is 500M)) from Heroku and I can't figure out why, as my model size (cnn.pth below) is fairly small and my file directory is only 1.1mb in total: screenshot of directory.
It seems like the size increase is caused by running pipenv install torch, as the slug size was 89.1mb before installing torch and 798.8mb after.
My Pipfile currently has these packages installed:
[packages]
flask = "*"
flask-sqlalchemy = "*"
psycopg2 = "*"
psycopg2-binary = "*"
requests = "*"
numpy = "*"
gunicorn = "*"
pillow = "*"
torch = "*"
Is there any workaround for this?
Edit: I'm running Mac OSX 10.10.5, using Flask and pipenv.
The pytorch package that you're installing comes with both cpu and gpu support, thus has a large size. It seems you're using the free version of heroku, and only require the cpu support. The solution is to install the pytorch package for cpu only i.e.
In requirements.txt, write the wheel file path corresponding to the version of pytorch (cpu) that you're interested in. You can find the list of wheel files, which can be installed with pip. For example, for PyTorch 1.3.1, torchvision 0.4.2, Python 3.7, Linux, you can write the following for pytorch and torchvision respectively:
https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cpu/torchvision-0.4.2%2Bcpu-cp37-cp37m-linux_x86_64.whl
The above will download torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl (107MB) torchvision-0.4.2+cpu-cp37-cp37m-linux_x86_64.whl (13MB) respectively.
Go to PyTorch Get Started page. Choose Stable version, Linux, pip, python, cpu and you can see:
pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
Copy versions to your requirements.txt.
torch==1.8.1+cpu
torchvision==0.9.1+cpu
And add to the beginning of the requirements.txt this:
-f https://download.pytorch.org/whl/torch_stable.html
After that, all packages will be installed on Heroku.
For reference:
torch==1.8.1+cpu (169.1 MB);
torchvision==0.9.1+cpu (13.3 MB)
By today, March 28, 2022, these are the working versions:
torch==1.11.0+cpu
torchvision==0.12.0+cpu
(Aug, 2, 2022) the only solution I found was leaving the requirements.txt like this:
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.11.0+cpu
--find-links https://download.pytorch.org/whl/torch_stable.html
torchvision==0.12.0+cpu
Related
I'm intermediate in python and I started learning about ML. I use qpython3 app on my android phone and as I need to use some packages such as numpy, I have to install them.
when I run the command
pip install numpy
I get an error that says that my
python version needs to be updated.
I tried many other compilers that run python 3.9 but still can't install libraries.
In Pydroid app, it requires to install an additional plugin app but I can never install it and many others have the same problem as me.
Collecting scipy
Downloading scipy-1.9.3.tar.gz (42.1 MB)
━━━━━━━━━━━━━━━━━━━━━ 42.1/42.1 MB 1.8 MB/s eta 0:00:00ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/data/data/ru.iiec.pydroid3/cache/pip-unpack-ddqhn5z9/scipy-1.9.3.tar.gz'
that's the error message
I think if I find a way to install python 3.9 and run it on qpython3 it'll work
We have some issues deploying our GCP Dataflow pipeline. After some analysis, found that the latest version of apache-beam has some issues while installing. To replicate the issue I created a virtualenv and ran the below
pip install apache-beam==2.32.0
Below errors started to pop while installing the 'orjson' dependency,
Using cached orjson-3.6.3.tar.gz (548 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... error
ERROR: Command errored out with exit status 1:
command: 'c:\temp\virtu\scripts\python.exe' 'c:\temp\virtu\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' prepare_metadata_for_build_wheel 'C:\Users\prasasaw\AppData\Local\Temp\tmpmekx1jjj'
cwd: C:\Users\prasasaw\AppData\Local\Temp\pip-install-0yofoe55\orjson_62f1ca2674934a7f8c45b08e87e05a4b
Complete output (6 lines):
Cargo, the Rust package manager, is not installed or is not on PATH.
This package requires Rust and Cargo to compile extensions. Install it through
the system's package manager or via https://rustup.rs/
Note that the previous versions of Apache beam like 2.30.0 does not have the dependency on 'orjson' and it works just fine when you do
pip install apache-beam==2.30.0
I tried to install RUST but it failed for some 'pysam' dependency. So would like to know what is the correct way to install the 'orjson' dependency
I saw this GitHub issue for orjson installation but could not find much from it.
https://github.com/readthedocs/readthedocs.org/issues/7687
//Prasad.
I'm a bit late to the party, but I ran into this issue as well today. I solved it by switching to a 64-bit Python environment (I accidently installed 32-bit).
The orjson dependency was introduced in https://github.com/apache/beam/pull/14690/files. According to the comment: orjson, only available on Python 3.6 and above. You may want to check your python version.
According to the orjson docs, you need to upgrade pip to be above 20.3:
pip install --upgrade "pip>=20.3" # manylinux_x_y, universal2 wheel support
pip install --upgrade orjson
(docs)
I succeeded in pytorch installation thanks to answers here
Poetry and PyTorch.
But, I'm still failed to install torchvision via poetry.
> poetry add torchvision==0.8.2
Updating dependencies
Resolving dependencies...
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
• Installing torchvision (0.8.2)
RuntimeError
Unable to find installation candidates for torchvision (0.8.2)
at ~\.poetry\lib\poetry\installation\chooser.py:72 in choose_for
68│
69│ links.append(link)
70│
71│ if not links:
→ 72│ raise RuntimeError(
73│ "Unable to find installation candidates for {}".format(package)
74│ )
75│
76│ # Get the best link
Failed to add packages, reverting the pyproject.toml file to its original content.
I googled it and found some answers that say 'just pip install torchvision'.
But I'm suspicious that it works because according to PyPi(https://pypi.org/project/torchvision/#files), there is no wheel file for windows. And I tried it and it failed as I expected.
Is there any way to install latest torchvisin which is compatible with latest torch(1.7.1) in windows? + via poetry?
Look in https://download.pytorch.org/whl/torch_stable.html for the version you want to install (torchvision version, python version, CUDA version, OS etc.)
Add a URL dependency to your pyproject.toml file. For example I have torchvision 1.8.0 working with the following in my dependencies
[tool.poetry.dependencies]
python = "^3.8"
torch = {url = "https://download.pytorch.org/whl/cu102/torch-1.8.0-cp38-cp38-win_amd64.whl"}
torchvision = {url = "https://download.pytorch.org/whl/cu102/torchvision-0.9.0-cp38-cp38-win_amd64.whl"}
From your activated virtual environment do poetry update torchvision and you should be good to go.
I got the same issue, I just reopen terminal and all worked good. I guess I was in some wrong environment.
I want to install Tensorflow v1.12.0 on a Docker-container which is running on a Raspberry Pi 3 B Plus using a Dockerfile from python:3.6-slim.
I am aware that i need to take into account that im using an armv7l architecture.
So far I tried installing with pip/pip3 with and without information about version and with and without the use of my requirements.txt.
This way all other needed packages can be installed with pip.
But I am getting the Error:
Step 5/13 : RUN pip install tensorflow==1.12.0
---> Running in d0ecb01c7fbc
Collecting tensorflow==1.12.0
Could not find a version that satisfies the requirement tensorflow==1.12.0 (from versions: )
No matching distribution found for tensorflow==1.12.0
The command '/bin/bash -c pip install tensorflow==1.12.0' returned a non-zero code: 1
Then I tried installing with the armv7l miniconda. But it seems the Tensorflow package is not supported with that version of miniconda.
Lastly I tried installing from a precompiled wheel for arm from this repo: https://github.com/lhelontra/tensorflow-on-arm/releases.
Like so: RUN pip install https://github.com/lhelontra/tensorflow-on-arm/releases/download/v1.12.0/tensorflow-1.12.0-cp35-none-linux_armv7l.whl
Here I changed to python:3.5-slim which starts out fine but seems to stop somewhere and results in a broken pipe (I am doing this remote).
I dont know what I could try next and didn't find anything else online.
I am just starting with virtualenv, but I am trying to install gevent within a virtualenv environment (I am running Windows). When I use PIP from virtualenv, I get this error:
MyEnv>pip install gevent
Downloading/unpacking gevent
Running setup.py egg_info for package gevent
Please provide path to libevent source with --libevent DIR
The package index has MSIs and EXEs for installing on Windows (http://pypi.python.org/pypi/gevent/0.13.7), but I don't know how to install those into a virtualenv environment (or if that is even possible). When I try pip install gevent-0.13.7.win32-py2.7.exe from the virtualenv promp, I get an error as well:
ValueError: ('Expected version spec in', 'D:\\Downloads\\gevent-0.13.7.win32-py2.7.exe', 'at', ':\\Downloads\\gevent-0.13.7.win32-py2.7.exe')
Does someone know how to do this?
Pip doesn't support installing binary packages, yet. If you want to install from binary package you have to use easy_install - easy_install gevent-0.13.7.win32-py2.7.exe
Microsoft Windows XP [Wersja 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
Z:\>virtualenv z:\venv\gevent-install
New python executable in z:\venv\gevent-install\Scripts\python.exe
Installing distribute..................................................................................................
............................................................................................done.
Installing pip.................done.
Z:\>venv\gevent-install\Scripts\activate
(gevent-install) Z:\>easy_install c:\python\packages\gevent-0.13.7.win32-py2.7.exe
Processing gevent-0.13.7.win32-py2.7.exe
creating 'c:\docume~1\pdobro~1\ustawi~1\temp\easy_install-b5nj3i\gevent-0.13.7-py2.7-win32.egg' and adding 'c:\docume~1
pdobro~1\ustawi~1\temp\easy_install-b5nj3i\gevent-0.13.7-py2.7-win32.egg.tmp' to it
creating z:\venv\gevent-install\lib\site-packages\gevent-0.13.7-py2.7-win32.egg
Extracting gevent-0.13.7-py2.7-win32.egg to z:\venv\gevent-install\lib\site-packages
Adding gevent 0.13.7 to easy-install.pth file
Installed z:\venv\gevent-install\lib\site-packages\gevent-0.13.7-py2.7-win32.egg
Processing dependencies for gevent==0.13.7
Searching for greenlet
Reading http://pypi.python.org/simple/greenlet/
Reading http://bitbucket.org/ambroff/greenlet
Reading https://github.com/python-greenlet/greenlet
Best match: greenlet 0.3.4
Downloading http://pypi.python.org/packages/2.7/g/greenlet/greenlet-0.3.4-py2.7-win32.egg#md5=9941aa246358c586bb274812e
130629
Processing greenlet-0.3.4-py2.7-win32.egg
creating z:\venv\gevent-install\lib\site-packages\greenlet-0.3.4-py2.7-win32.egg
Extracting greenlet-0.3.4-py2.7-win32.egg to z:\venv\gevent-install\lib\site-packages
Adding greenlet 0.3.4 to easy-install.pth file
Installed z:\venv\gevent-install\lib\site-packages\greenlet-0.3.4-py2.7-win32.egg
Finished processing dependencies for gevent==0.13.7
(gevent-install) Z:\>
See Can I install Python windows packages into virtualenvs? Another option is to install from source and you can do this with pip but this requires setting up compiler and environment which is much harder than the simple command above.
From the error message, it would appear you need libevent source code. I would imagine you need to go a step further and compile/install libevent system-wide so pip can find it.
I would start by downloading the latest stable source from http://libevent.org/.
Compile and install it using instructions in the README: https://github.com/libevent/libevent#readme
To compile it on Windows, you'll need to use GNU-style build utilities like make and autoconf. I recommend http://www.mingw.org/.
Once you've installed libevent system-wide, I imagine pip will find it and proceed with gevent installation.
In the msi for gevent-0.13.7 there's an option to select an alternate installation point. point it to the root dir of your particular virtual environment (just above where /Lib and /Scripts are located). That should install it correctly.
You also need to make sure greenlets are installed. For that you can use Piotr's suggested method with easy_install on the .exe.