I am trying to create an environment from an environment.yml file by running conda env create -f environment.yml in an administrator command prompt. The environment.yml file is as the following:
name: pytorch0.4
channels:
- pytorch
- defaults
dependencies:
- python=3.6.5
- pytorch=0.4.1
- torchvision
- numpy
- nltk
- ipython
- docopt
- pip
- pip:
- tqdm
After running the command, this error appeared:
Collecting package metadata (repodata.json): done
Solving environment: \
Warning: 2 possible package resolutions (only showing differing packages):
- defaults/noarch::parso-0.8.1-pyhd3eb1b0_0, defaults/win-64::jedi-0.17.0-py36_0
- defaults/noarch::parso-0.7.0-py_0, defaults/win-64::jedi-0.17.2-py36haa95532done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installing pip dependencies: - Ran pip subprocess with arguments:
['C:\\Users\\PC\\anaconda3\\envs\\pytorch0.4\\python.exe', '-m', 'pip', 'install', '-U', '-r', 'C:\\Users\\Assignment\\A2\\a2\\condaenv.1tcu2wl4.requirements.txt']
Pip subprocess output:
Requirement already satisfied: tqdm in c:\users\pc\anaconda3\envs\pytorch0.4\lib\site-packages (from -r C:\Users\Assignment\A2\a2\condaenv.1tcu2wl4.requirements.txt (line 1)) (4.56.0)
Collecting tqdm
Using cached tqdm-4.58.0-py2.py3-none-any.whl (73 kB)
Installing collected packages: tqdm
Attempting uninstall: tqdm
Found existing installation: tqdm 4.56.0
Uninstalling tqdm-4.56.0:
Successfully uninstalled tqdm-4.56.0
Pip subprocess error:
ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-uninstall-yeq37y9s\\tqdm.exe'
Consider using the `--user` option or check the permissions.
failed
CondaEnvException: Pip failed
I don't think my environment was created correctly, since I cannot use the dependencies in the environment nor activate it. I also tried to install tqdm by running pip install tqdm --user, it says that Requirement already satisfied: tqdm in c:\users\pc\appdata\roaming\python\python38\site-packages (4.58.0). How do I initiate that environment then?
Conda should be able to solve this without resorting to pip since tdqm is available on the defaults channel. Moreover, nltk has tdqm as dependency, so the YAML could be simplified to
File: pytorch0.4.yaml
name: pytorch0.4
channels:
- pytorch
- defaults
dependencies:
- python=3.6.5
- pytorch=0.4.1
- torchvision
- numpy
- nltk
- ipython
- docopt
- pip
Also, while using pip in Conda environments is valid, it comes with some caveats. One of these is to never use the --user flag, since this ends up placing packages outside the environment's site-packages and can lead to violating the isolation of environments that Conda is so keen on establishing.
Go to "pyvenv.cfg" file in your Python project and change include-system-site-packages from false to true.
Related
I am working with Ubuntu in WSL and tried to install the required packages for a repo with:
$ conda install --file requirements.txt
I got a PackageNotFoundError for a bunch of different packages. I search on anaconda.org for the required channels and added them. But it doesn't matter which channels I add I always get a PackageNotFoundError for the last two remaining packages:
$ conda install --file requirements.txt
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- openssl==1.1.1=h7b6447c_0
- intel-openmp==2019.5=281
Current channels:
- https://conda.anaconda.org/fastchan/linux-64
- https://conda.anaconda.org/fastchan/noarch
- https://conda.anaconda.org/cctbx202208/linux-64
- https://conda.anaconda.org/cctbx202208/noarch
- https://conda.anaconda.org/pytorch/linux-64
- https://conda.anaconda.org/pytorch/noarch
- https://conda.anaconda.org/conda-forge/linux-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Anaconda.org says conda-forge, fastchan, cctbx202208 for openssl but even though I added all of them it's still not found.
The next thing I tried was to install it with pip:
$ pip install openssl==1.1.1
ERROR: Could not find a version that satisfies the requirement openssl==1.1.1 (from versions: none)
ERROR: No matching distribution found for openssl==1.1.1
But pip detects none versions of this package. Same with intel-openmp, but pip does find packages but not the one I want 2019.5:
$ pip install intel-openmp==2019.5
ERROR: Could not find a version that satisfies the requirement intel-openmp==2019.5 (from versions: 2018.0.0, 2018.0.3, 2019.0, 2020.0.133, 2021.1.1, 2021.1.2, 2021.2.0, 2021.3.0, 2021.4.0, 2022.0.1, 2022.0.2, 2022.1.0, 2022.2.0, 2022.2.1)
ERROR: No matching distribution found for intel-openmp==2019.5
So my question is, is there another way to install the two packages or do they not exists anymore? Because the repo from which I got the code has its last commit from 3 years ago...
Edit:
I tried this command:
conda install -c anaconda openssl
and it installs openssl, but the latest version and than the code still says openssl is missing.
I also tried:
conda install -c anaconda openssl=1.1.1
but I get the same error as in the beginning (PackageNotFoundError in the channels).
Edit2:
TrackNPred is the repo I cloned and want to get working.
As for the required channel, I just searched for the package name on anaconda.org and add the channels i see to my anaconda config with:
conda config --add channels new_channel
I'm not sure if I need the exact version of a package as it's listed in the requirements.txt or if the code also works with another version of the two missing packages.
Edit3:
I changed in the requirements.txt:
openssl=1.1.1*
intel-openmp=2019.5
and that worked.
If you don't require exact package build replication, then edit the file to loosen up the specifications.
I see the intel channel has intel-openmp=2019.5=intel_218 and any openssl=1.1.1 should do. So, find the respective entries in the requirements.txt file and edit them to have:
openssl=1.1.1*
intel-openmp=2019.5
then recreate the environment, with intel channel added.
conda config --set channel_priority flexible
conda create -n foo -c defaults -c intel --file requirements.txt
I was trying to test the pip install -e . on my library ultimate-utils after changing laptops but I keep getting errors of this type:
(uutils_env) brandomiranda~/ultimate-utils ❯ pip install -e .
Obtaining file:///Users/brandomiranda/ultimate-utils
Preparing metadata (setup.py) ... done
ERROR: Could not find a version that satisfies the requirement torchvision==0.10.1 (from ultimate-utils) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
ERROR: No matching distribution found for torchvision==0.10.1
or
(uutils_env) brandomiranda~/ultimate-utils ❯ pip install -e .
Obtaining file:///Users/brandomiranda/ultimate-utils
Preparing metadata (setup.py) ... done
ERROR: Could not find a version that satisfies the requirement torch==1.9.1 (from ultimate-utils) (from versions: none)
ERROR: No matching distribution found for torch==1.9.1
where it seems it stopped working to install anything related to pytorch for some reason.
I tried updating pip and conda but it did not work. I did:
pip install --upgrade pip
conda update conda
conda update conda-build
conda update -n base -c defaults conda
conda update --name base conda
conda update --all
conda install anaconda
but none seem to work.
my setup.py looks as follows:
"""
conda create -n uutils_env python=3.9
conda activate uutils_env
conda remove --all --name uutils_env
rm -rf /Users/brando/anaconda3/envs/uutils_env
pip install -e ~/ultimate-utils/ultimate-utils-proj-src/
pip install ultimate-utils
To test it:
python -c "import uutils; uutils.hello()"
python -c "import uutils; uutils.torch_uu.hello()"
python -c "import uutils; uutils.torch_uu.gpu_test_torch_any_device()"
python -c "import uutils; uutils.torch_uu.gpu_test()"
PyTorch:
basing the torch install from the pytorch website as of this writing: https://pytorch.org/get-started/locally/
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
refs:
- setup tools: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#using-find-or-find-packages
"""
from setuptools import setup
from setuptools import find_packages
import os
# import pathlib
here = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
setup(
name='ultimate-utils', # project name
version='0.5.3',
description="Brando's ultimate utils for science, machine learning and AI",
long_description=long_description,
long_description_content_type="text/markdown",
url='https://github.com/brando90/ultimate-utils',
author='Brando Miranda',
author_email='brandojazz#gmail.com',
python_requires='>=3.9.0',
license='MIT',
package_dir={'': 'ultimate-utils-proj-src'},
packages=find_packages('ultimate-utils-proj-src'), # imports all modules/folders with __init__.py & python files
# for pytorch see doc string at the top of file
install_requires=[
# 'torch==1.9.1',
'torchvision==0.10.1',
'torchaudio==0.9.1',
'dill',
'networkx>=2.5',
'scipy',
'scikit-learn',
'lark-parser',
'torchtext==0.10.1',
'tensorboard',
'pandas',
'progressbar2',
'transformers',
'requests',
'aiohttp',
'numpy',
'plotly',
'wandb',
'matplotlib',
# 'seaborn'
# 'pygraphviz' # removing because it requires user to install graphviz and gives other issues
]
)
the dir structure is simple:
How do I fix this and what is not working?
I am trying to avoid having to run pip commands outside like:
conda install pytorch torchvision torchaudio -c pytorch
at least for cpu in my local laptop...if I can install gpu ones automatically with a flag it could be nice but for future work...
Btw, I prefer no requirements.txt solutions if possible, but post those too in case nothing else works.
This is for an intel mac 2013, but I will have an m1 max in 2 months, so those answers are also welcomed!
still fails:
(uutils_env) brandomiranda~/ultimate-utils ❯ pip install -e .
Obtaining file:///Users/brandomiranda/ultimate-utils
Preparing metadata (setup.py) ... done
ERROR: Could not find a version that satisfies the requirement torch<1.10.0,>=1.4.0 (from ultimate-utils) (from versions: none)
ERROR: No matching distribution found for torch<1.10.0,>=1.4.0
Related resources I tried:
https://docs.conda.io/projects/conda-build/en/latest/install-conda-build.html
PackageNotInstalledError: Package is not installed in prefix
Installing PyTorch with CUDA in setup.py
https://github.com/brando90/ultimate-utils
https://pytorch.org/get-started/locally/
https://discuss.pytorch.org/t/how-does-one-install-pytorch-and-related-tools-from-within-the-setup-py-install-requires-list/138976
How does one install PyTorch and related tools from within the setup.py install_requires list?
https://www.reddit.com/r/pytorch/comments/rcprlk/how_does_one_install_pytorch_and_related_tools/
Could not find a version that satisfies the requirement torch>=1.0.0?
See this Why is python using 3.8.1 and 3.9, then fail to install packages (ERROR: Package pkg requires a different Python: 3.8.1 not in '>=3.9.0')? for a more complete answer but the summary is that pytorch seems to not be working right now reliable with python 3.10 so I created an env with 3.9 and then it installed what I needed.
When running poetry add package, poetry decides to update some out of date dependencies without my asking:
$ poetry add -D ipython
Using version ^7.26.0 for ipython
Updating dependencies
Resolving dependencies... (6.8s)
Writing lock file
Package operations: 13 installs, 2 updates, 0 removals
• Installing ipython-genutils (0.2.0)
• Installing parso (0.8.2)
• Installing ptyprocess (0.7.0)
• Installing traitlets (5.0.5)
• Installing wcwidth (0.2.5)
• Installing appnope (0.1.2)
• Installing backcall (0.2.0)
• Updating connexion (2.8.0 -> 2.9.0) <---------- "I never asked for this"
• Installing jedi (0.18.0)
• Installing matplotlib-inline (0.1.2)
• Installing pexpect (4.8.0)
• Installing pickleshare (0.7.5)
• Installing prompt-toolkit (3.0.19)
• Installing ipython (7.26.0)
• Updating ramda (0.6.0 -> 0.6.1) <---------- "I never asked for this"
And yet when getting the help for the command, there doesn't seem to be anything that suggest that it's possible to disable this behaviour:
$ poetry add --help
USAGE
poetry add [-D] [-E <...>] [--optional] [--python <...>]
[--platform <...>] [--source <...>] [--allow-prereleases]
[--dry-run] [--lock] <name1> ... [<nameN>]
ARGUMENTS
<name> The packages to add.
OPTIONS
-D (--dev) Add as a development dependency.
-E (--extras) Extras to activate for the dependency.
(multiple values allowed)
--optional Add as an optional dependency.
--python Python version for which the dependency must be
installed.
--platform Platforms for which the dependency must be
installed.
--source Name of the source to use to install the
package.
--allow-prereleases Accept prereleases.
--dry-run Output the operations but do not execute
anything (implicitly enables --verbose).
--lock Do not perform operations (only update the
lockfile).
GLOBAL OPTIONS
-h (--help) Display this help message
-q (--quiet) Do not output any message
-v (--verbose) Increase the verbosity of messages: "-v" for
normal output, "-vv" for more verbose output and
"-vvv" for debug
-V (--version) Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n (--no-interaction) Do not ask any interactive question
...
Any way to install only the package of interest?
I prefer not to update deps, even if the bump is a minor bump - dependabot exists for this purpose already.
I am trying to install python packages from the local file system with pip as discussed in this question.
I have used pip2pi as suggested in the accepted answer in the above question.
dmanna#ubuntu:~$ mkdir -p pyt/pkg
dmanna#ubuntu:~$ pip2tgz pyt/pkg/ patroni[zookeeper]
The downloaded packages
dmanna#ubuntu:~$ ls /home/dmanna/pyt/pkg/
cdiff-1.0.tar.gz
certifi-2018.4.16-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
click-6.7-py2.py3-none-any.whl
idna-2.7-py2.py3-none-any.whl
kazoo-2.5.0-py2.py3-none-any.whl
patroni-1.4.4.tar.gz
prettytable-0.7.2.tar.bz2
psutil-5.4.6.tar.gz
psycopg2-2.7.4.tar.gz
python_dateutil-2.7.3-py2.py3-none-any.whl
pytz-2018.4-py2.py3-none-any.whl
PyYAML-3.12.tar.gz
requests-2.19.1-py2.py3-none-any.whl
six-1.11.0-py2.py3-none-any.whl
tzlocal-1.5.1.tar.gz
urllib3-1.23-py2.py3-none-any.whl
Then when I am trying to install the required package from the above local directory. It is giving me the below error
dmanna#ubuntu:~$ sudo pip install --no-index --find-links=pyt/pkg/patroni-1.4.4.tar.gz patroni[zookeeper]
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking patroni[zookeeper]
Running setup.py (path:/tmp/pip_build_root/patroni/setup.py) egg_info for package patroni
Installing extra requirements: 'zookeeper'
Downloading/unpacking urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Could not find any downloads that satisfy the requirement urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Cleaning up...
No distributions at all found for urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Storing debug log for failure in /home/dmanna/.pip/pip.log
Can someone let me know what I am doing wrong?
Python - 2.7.6
PIP - 1.5.4
Ubuntu 14.04
--find-links=pyt/pkg
Find links in the directory, not in the patroni-1.4.4.tar.gz archive.
So I'm building a package with conda on Win7:
conda skeleton pypi $pkgname
conda build $pkgname
binstar upload ./conda-bld/win-64/$pkgname-py27_0.tar.bz2
conda install -c https://conda.binstar.org/myname $pkgname
With python2k, this works like a charm. However, using python3k this fails because the patch package is not available for python3k. Error (example for docopt):
$ conda skeleton pypi docopt
Using url https://pypi.python.org/packages/source/d/docopt/docopt-0.6.2.tar.gz (25 KB) for docopt.
Downloading docopt (use --no-download to skip this step)
Using cached download
Unpacking docopt...
done
working in C:\cygwin64\tmp\tmpy7erx8vyconda_skeleton_docopt-0.6.2.tar.gz
Fetching package metadata: ......
Solving package specifications: .
Applying patch: 'C:\\cygwin64\\tmp\\tmpy7erx8vyconda_skeleton_docopt-0.6.2.tar.gz\\pypi-distutils.patch'
Error:
Did not find 'patch' in: C:\Python\Anaconda3\envs\_build\Scripts;C:\Python\Anaconda3\Scripts;C:\cygwin\bin;C:\cygwin64\usr\local\bin;C:\cygwin64\bin;C:\Python\Python3\Lib\site-packages\PyQt4;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files\MiKTeX 2.9\miktex\bin\x64;C:\cygwin64\bin;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Python\Anaconda3;C:\Python\Anaconda3\Scripts;C:\cygwin64\lib\lapack
You can install 'patch' using apt-get, yum (Linux), Xcode (MacOSX), or conda, cygwin (Windows)
When trying to install patch:
$conda search ^patch$
Fetching package metadata: ....
patch 1.12.11 py27_0 defaults
No patch package for python3k - does this mean no package building for me?
You can reuse the same recipe that you built with Python 2. Or you can create an environment with patch (conda create -n patch patch) and put it on your PATH. Or you can install patch from http://gnuwin32.sourceforge.net/packages/patch.htm.