On a fresh python3.7 virtual environment I try to install a downloaded wheel but the installation step on Windows takes ~20 seconds (on linux is a bit faster but still ~20seconds)
Executed steps
Here I create venv, update pip, setuptools & wheel, download scons, nothing really interesting here.
PS C:\s\tmp> C:\Python37\python.exe -m venv venv37
PS C:\s\tmp> .\venv37\Scripts\activate
(venv37) PS C:\s\tmp> pip install -U pip setuptools wheel
Requirement already up-to-date: pip in c:\s\tmp\venv37\lib\site-packages (20.1.1)
Collecting setuptools
Using cached setuptools-49.1.0-py3-none-any.whl (789 kB)
Collecting wheel
Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Installing collected packages: setuptools, wheel
Attempting uninstall: setuptools
Found existing installation: setuptools 47.1.0
Uninstalling setuptools-47.1.0:
Successfully uninstalled setuptools-47.1.0
Successfully installed setuptools-49.1.0 wheel-0.34.2
(venv37) PS C:\s\tmp> Measure-Command { pip download scons | Out-Default }
Collecting scons
Using cached SCons-4.0.0-py3-none-any.whl (4.0 MB)
Saved c:\s\tmp\scons-4.0.0-py3-none-any.whl
Collecting pywin32>=1.0; platform_system == "Windows"
Using cached pywin32-228-cp37-cp37m-win_amd64.whl (9.1 MB)
Saved c:\s\tmp\pywin32-228-cp37-cp37m-win_amd64.whl
Collecting setuptools
Using cached setuptools-49.1.0-py3-none-any.whl (789 kB)
Saved c:\s\tmp\setuptools-49.1.0-py3-none-any.whl
Successfully downloaded scons pywin32 setuptools
Seconds : 2
(venv37) PS C:\s\tmp> ls
Directory: C:\s\tmp
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/10/2020 2:38 PM venv37
-a---- 7/10/2020 2:40 PM 9100091 pywin32-228-cp37-cp37m-win_amd64.whl
-a---- 7/10/2020 2:40 PM 3970271 SCons-4.0.0-py3-none-any.whl
-a---- 7/10/2020 2:40 PM 789761 setuptools-49.1.0-py3-none-any.whl
Here I try to install the downloaded wheel, which takes up to 17 seconds.
(venv37) PS C:\s\tmp> Measure-Command { python -m pip install -vvv -U --no-index --find-links . scons | Out-Default }
Non-user install because user site-packages disabled
Ignoring indexes: https://pypi.org/simple
Created temporary directory: C:\Users\FRANCE~1\AppData\Local\Temp\pip-ephem-wheel-cache-56rg4dkw
Created temporary directory: C:\Users\FRANCE~1\AppData\Local\Temp\pip-req-tracker-awspyjg9
Initialized build tracking at C:\Users\FRANCE~1\AppData\Local\Temp\pip-req-tracker-awspyjg9
Created build tracker: C:\Users\FRANCE~1\AppData\Local\Temp\pip-req-tracker-awspyjg9
Entered build tracker: C:\Users\FRANCE~1\AppData\Local\Temp\pip-req-tracker-awspyjg9
Created temporary directory: C:\Users\FRANCE~1\AppData\Local\Temp\pip-install-1_k389zs
Looking in links: .
0 location(s) to search for versions of scons:
Skipping link: not a file: . (from -f)
Skipping link: wrong project name (not scons): file:///C:/s/tmp/pywin32-228-cp37-cp37m-win_amd64.whl
Found link file:///C:/s/tmp/SCons-4.0.0-py3-none-any.whl, version: 4.0.0
Skipping link: wrong project name (not scons): file:///C:/s/tmp/setuptools-49.1.0-py3-none-any.whl
Skipping link: not a file: file:///C:/s/tmp/venv37
Local files found: C:\s\tmp\SCons-4.0.0-py3-none-any.whl
Given no hashes to check 1 links for project 'scons': discarding no candidates
Using version 4.0.0 (newest of versions: 4.0.0)
Processing c:\s\tmp\scons-4.0.0-py3-none-any.whl
Added scons from file:///C:/s/tmp/SCons-4.0.0-py3-none-any.whl to build tracker 'C:\\Users\\FRANCE~1\\AppData\\Local\\Temp\\pip-req-tracker-awspyjg9'
Removed scons from file:///C:/s/tmp/SCons-4.0.0-py3-none-any.whl from build tracker 'C:\\Users\\FRANCE~1\\AppData\\Local\\Temp\\pip-req-tracker-awspyjg9'
Requirement already satisfied, skipping upgrade: setuptools in c:\s\tmp\venv37\lib\site-packages (from scons) (49.1.0)
0 location(s) to search for versions of pywin32:
Found link file:///C:/s/tmp/pywin32-228-cp37-cp37m-win_amd64.whl, version: 228
Skipping link: wrong project name (not pywin32): file:///C:/s/tmp/SCons-4.0.0-py3-none-any.whl
Local files found: C:\s\tmp\pywin32-228-cp37-cp37m-win_amd64.whl
Given no hashes to check 1 links for project 'pywin32': discarding no candidates
Using version 228 (newest of versions: 228)
Processing c:\s\tmp\pywin32-228-cp37-cp37m-win_amd64.whl
Added pywin32>=1.0; platform_system == "Windows" from file:///C:/s/tmp/pywin32-228-cp37-cp37m-win_amd64.whl (from scons) to build tracker 'C:\\Users\\FRANCE~1\\AppData\\Local\\Temp\\pip-req-tracker-awspyjg9'
Removed pywin32>=1.0; platform_system == "Windows" from file:///C:/s/tmp/pywin32-228-cp37-cp37m-win_amd64.whl (from scons) from build tracker 'C:\\Users\\FRANCE~1\\AppData\\Local\\Temp\\pip-req-tracker-awspyjg9'
Installing collected packages: pywin32, scons
Created temporary directory: C:\Users\FRANCE~1\AppData\Local\Temp\pip-unpacked-wheel-vwh0ebm3
Created temporary directory: C:\Users\FRANCE~1\AppData\Local\Temp\pip-unpacked-wheel-dxib5q_u
Successfully installed pywin32-228 scons-4.0.0
Removed build tracker: 'C:\\Users\\FRANCE~1\\AppData\\Local\\Temp\\pip-req-tracker-awspyjg9'
Seconds : 17
Is there anything I can do to speed up the installation step?
Update (7.13.2020):
After bdbaddog's comment I shortly had a look at the repo, as he suggested dobook artifacts seem to be the ones to blame for this slow installation.
(venv37) PS C:\src\scons\SCons> dust
397K ┌── Node │ █ │ 2%
522K │ ┌── RELEASE-NOTES.xml │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█ │ 3%
612K │ ├── manpages │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█ │ 3%
666K │ ├── RELEASE-NOTES.html │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█ │ 3%
696K │ ├── webhelp │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█ │ 3%
417K │ │ ┌── template-pages.xml │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 2%
860K │ ├─┴ roundtrip │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 4%
876K │ ├── slides │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 4%
919K │ ├── params │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 4%
1.1M │ │ ┌── reference.xml.included│ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 6%
1.2M │ ├─┴ docsrc │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 6%
1.3M │ ├── xhtml-1_1 │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 7%
1.3M │ ├── xhtml │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓██ │ 7%
1.6M │ ├── html │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓███ │ 8%
447K │ │ ┌── param.xml │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓███ │ 2%
1.8M │ ├─┴ fo │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓███ │ 9%
3.1M │ ├── common │ ░░▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█████ │ 15%
16M │ ┌─┴ docbook-xsl-1.76.1 │ ░░████████████████████████ │ 83%
16M │ ┌─┴ docbook │ ░░████████████████████████ │ 83%
17M ├─┴ Tool │ ██████████████████████████ │ 89%
20M ┌─┴ . │████████████████████████████ │ 100%
Related
my project structure is the following
py/
├─ projects/
│ ├─ project1/
│ │ ├─ ...
│ │ ├─ pyproject.toml
│ │ ├─ tox.ini
├─ libs/
│ ├─ lib1/
│ │ ├─ ...
│ │ ├─ pyproject.toml
py/projects/project1/pyproject.toml is something like this
[tool.poetry.dependencies]
...
lib1 = {path = "../../libs/lib1" }
py/projects/project1/tox.ini is something like this
[tox]
isolated_build = True
envlist = py3
minversion = 3.24.4
requires = setuptools >= 60.1.0
[testenv]
whitelist_externals = poetry
commands =
poetry install -v
poetry run pytest -vvv
Everything is working fine during development and run, but I cannot make the testing work. When running the command poetry run tox I got the following error
.package recreate: ...
.package installdeps: poetry-core>=1.0.0
py3 recreate: ...
py3 inst: ...
ERROR: invocation failed (exit code 1), logfile: ....
============================================================================================================== log start ===============================================================================================================
Processing ./.tox/.tmp/package/1/project1-1.0.0.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [16 lines of output]
Traceback (most recent call last):
...
self.create_dependency(name, constraint, root_dir=package.root_dir)
File "/tmp/pip-build-env-nmn7y8qz/overlay/lib/python3.10/site-packages/poetry/core/factory.py", line 244, in create_dependency
dependency = DirectoryDependency(
File "/tmp/pip-build-env-nmn7y8qz/overlay/lib/python3.10/site-packages/poetry/core/packages/directory_dependency.py", line 41, in __init__
raise ValueError("Directory {} does not exist".format(self._path))
ValueError: Directory ../../libs/lib1 does not exist
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the ...
Does anyone know how to force tox to install local dependencies? Thank you very much!
tox tries to create a sdist package and install it into the environment. This fails, because the creation of the of sdist takes place in a temporary directory and virtual environment, where your relative path dependency does not point to the correct location anymore.
Your [testenv] should look like this instead:
[testenv]
skip_install = true
whitelist_externals = poetry
commands =
poetry install -v
poetry run pytest -vvv
Setting skip_install to true will tell tox not to build and install the package. This is unnecessary because you use poetry install.
I have a local python project (A) that depends on another local python project (B).
I have added the project B as a git submodule to a dependencies folder.
When I pull the repo with all the submodules, I would like to have a setup.py in A such that wenn I run pip install -e . in A, it also installs the setup.py in B.
The repo looks like this:
A
├── A
│ └── foo.py
│
├── dependencies
│ └── B
│ ├── B
│ │ └── foo.py
│ │
│ └── setup.py
│
└── setup.py
I tried from setuptools import setup, find_packages
setup(
name='A',
packages=find_packages(),
)
But that only finds package A, which makes sense.
I also tried adding a preinstall command
class PreInstallCommand(install):
"""Pre-installation for installation mode."""
def run(self):
check_call("pip install -e dependencies/B".split())
install.run(self)
setup(
...
cmdclass={
'install': PreInstallCommand,
},
...
)
But that one is not triggered when I install it in editable mode (pip install -e).
Thanks in advance for your advice! :)
Cheers!
Hi guys i have a problem with creating python package out of my project. I read some tutorials but every one leads to uploading files to pypi which i dont want to do. I just want to pip install it to my machine locally using tar.gz file.
Here's a structure of my project folder:
root
├── src
│ ├── __init__.py
│ ├── config.py
│ └── sth_a
│ │ ├── __init__.py
│ │ └── a.py
│ └── sth_b
│ ├── __init__.py
│ └── b.py
└── setup.py
Here is how my setup.py file looks like:
from setuptools import setup, find_packages
setup(name="mypkg",
version='0.0.1',
author="whatever",
packages=find_packages()
)
First i run command:
python setup.py sdist bdist_wheel
then it creatates dist dictionary with tar.gz file and wheel file, so i just run
pip3 install dist/mypkg-0.0.1.tar.gz
After this first problem emerges. To import these files somewhere else i need to write
from src.sth_a.a import *
but i want to do this like this
from mypgk.src.sth_a.a import *
or even if i just want to 'publish' for example functions from file a.py
from mypck.a import *
Also i was having another issues bit this answer helped me a bit but it is not still what i want pip install . creates only the dist-info not the package
I'm trying to see all the dependencies that are required for an specific package(in this case I'm using pipdeptree) but it turns out that it only shows me the dependency tree for an installed package. Let's say that I'm using graphene:2.1.0, so e.g:
$ pipdeptree -p graphene
Warning!!! Possibly conflicting dependencies found:
* graphql-relay==0.5.0
- graphql-core [required: >=0.5.0,<2, installed: 2.3.1]
* social-auth-core==3.2.0
- requests [required: >=2.9.1, installed: 2.8.1]
------------------------------------------------------------------------
Now when I try $ pipdeptree -p graphene==40.0.2 (which does not exist).
Warning!!! Possibly conflicting dependencies found:
* graphql-relay==0.5.0
- graphql-core [required: >=0.5.0,<2, installed: 2.3.1]
* social-auth-core==3.2.0
- requests [required: >=2.9.1, installed: 2.8.1]
------------------------------------------------------------------------
It seems that it only takes into account a stable version, what I want is a dependency tree related to the specific version that I give by console, I hope I'm being clear with this.
I believe johnnydep can help with that:
$ johnnydep --verbose 0 'graphene==2.1.0'
name summary
------------------------------ ---------------------------------------
graphene==2.1.0 GraphQL Framework for Python
├── aniso8601<4,>=3 A library for parsing ISO 8601 strings.
├── graphql-core<3,>=2.0 GraphQL implementation for Python
│ ├── promise<3,>=2.3 Promises/A+ implementation for Python
│ │ └── six Python 2 and 3 compatibility utilities
│ ├── rx<2,>=1.6 Reactive Extensions (Rx) for Python
│ └── six>=1.10.0 Python 2 and 3 compatibility utilities
├── graphql-relay<1,>=0.4.5 Relay implementation for Python
│ ├── graphql-core<2,>=0.5.0 GraphQL implementation for Python
│ │ ├── promise>=2.0 Promises/A+ implementation for Python
│ │ │ └── six Python 2 and 3 compatibility utilities
│ │ └── six>=1.10.0 Python 2 and 3 compatibility utilities
│ ├── promise>=0.4.0 Promises/A+ implementation for Python
│ │ └── six Python 2 and 3 compatibility utilities
│ └── six>=1.10.0 Python 2 and 3 compatibility utilities
├── promise<3,>=2.1 Promises/A+ implementation for Python
│ └── six Python 2 and 3 compatibility utilities
└── six<2,>=1.10.0 Python 2 and 3 compatibility utilities
How can one command find out all dependencies of the package and links to dependent packages in the repository pip?
You can use johnnydep
pip install johnnydep
Example: johnnydep ipython
ipython IPython: Productive Interactive Computing
├── appnope Disable App Nap on OS X 10.9
├── backcall Specifications for callback functions passed in to an API
├── decorator Better living through Python with decorators
├── jedi>=0.10 An autocompletion tool for Python that can be used for text editors.
│ └── parso>=0.3.0 A Python Parser
├── pexpect Pexpect allows easy control of interactive console applications.
│ └── ptyprocess>=0.5 Run a subprocess in a pseudo terminal
├── pickleshare Tiny 'shelve'-like database with concurrency support
├── prompt-toolkit<2.1.0,>=2.0.0 Library for building powerful interactive command lines in Python
│ ├── six>=1.9.0 Python 2 and 3 compatibility utilities
│ └── wcwidth Measures number of Terminal column cells of wide-character codes
├── pygments Pygments is a syntax highlighting package written in Python.
├── setuptools>=18.5 Easily download, build, install, upgrade, and uninstall Python packages
├── simplegeneric>0.8 Simple generic functions (similar to Python's own len(), pickle.dump(), etc.)
└── traitlets>=4.2 Traitlets Python config system
├── decorator Better living through Python with decorators
├── ipython-genutils Vestigial utilities from IPython
└── six Python 2 and 3 compatibility utilities
You can also display download links to dependant packages by adding the --fields download_link option:
johnnydep ipython --fields download_link
.
name download_link
-------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
ipython https://files.pythonhosted.org/packages/a0/27/29d66ed395a5c2c3a912332d446a54e2bc3277c36b0bbd22bc71623e0193/ipython-7.0.1-py3-none-any.whl
├── appnope https://files.pythonhosted.org/packages/87/a9/7985e6a53402f294c8f0e8eff3151a83f1fb901fa92909bb3ff29b4d22af/appnope-0.1.0-py2.py3-none-any.whl
├── backcall https://files.pythonhosted.org/packages/84/71/c8ca4f5bb1e08401b916c68003acf0a0655df935d74d93bf3f3364b310e0/backcall-0.1.0.tar.gz
├── decorator https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
├── jedi>=0.10 https://files.pythonhosted.org/packages/7a/1a/9bd24a185873b998611c2d8d4fb15cd5e8a879ead36355df7ee53e9111bf/jedi-0.13.1-py2.py3-none-any.whl
│ └── parso>=0.3.0 https://files.pythonhosted.org/packages/09/51/9c48a46334be50c13d25a3afe55fa05c445699304c5ad32619de953a2305/parso-0.3.1-py2.py3-none-any.whl
├── pexpect https://files.pythonhosted.org/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl
│ └── ptyprocess>=0.5 https://files.pythonhosted.org/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
├── pickleshare https://files.pythonhosted.org/packages/9a/41/220f49aaea88bc6fa6cba8d05ecf24676326156c23b991e80b3f2fc24c77/pickleshare-0.7.5-py2.py3-none-any.whl
├── prompt-toolkit<2.1.0,>=2.0.0 https://files.pythonhosted.org/packages/e5/c5/f1ee6698bdcf615f171a77e81ca70293b16a6d82285f1760b388b4348263/prompt_toolkit-2.0.5-py3-none-any.whl
│ ├── six>=1.9.0 https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
│ └── wcwidth https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
├── pygments https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl
├── setuptools>=18.5 https://files.pythonhosted.org/packages/96/06/c8ee69628191285ddddffb277bd5abdf769166e7a14b867c2a172f0175b1/setuptools-40.4.3-py2.py3-none-any.whl
├── simplegeneric>0.8 https://files.pythonhosted.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip
└── traitlets>=4.2 https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl
├── decorator https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
├── ipython-genutils https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
└── six https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl