We'd like to use pip with github to install private packages to our production servers. This question concerns what needs to be in the github repo in order for the install to be successful.
Assuming the following command line (which authenticates just fine and tries to install):
pip install git+ssh://git#github.com/BlahCo/search/tree/prod_release_branch/ProductName
What needs to reside in the ProductName? Is it the contents of what would normally be in the tar file after running setup.py with the sdist option, or is the actual tar.gz file, or something else?
I'm asking here because I've tried several variations and can't make it work. Any help appreciated.
You need the whole python package, with a setup.py file in it.
A package named foo would be:
foo # the installable package
├── foo
│ ├── __init__.py
│ └── bar.py
└── setup.py
And install from github like:
$ pip install git+ssh://git#github.com/myuser/foo.git
or
$ pip install git+https://github.com/myuser/foo.git#v123
or
$ pip install git+https://github.com/myuser/foo.git#newbranch
More info at https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support
I had similar issue when I had to install from github repo, but did not want to install git , etc.
The simple way to do it is using zip archive of the package. Add /zipball/master to the repo URL:
$ pip install https://github.com/hmarr/django-debug-toolbar-mongo/zipball/master
Downloading/unpacking https://github.com/hmarr/django-debug-toolbar-mongo/zipball/master
Downloading master
Running setup.py egg_info for package from https://github.com/hmarr/django-debug-toolbar-mongo/zipball/master
Installing collected packages: django-debug-toolbar-mongo
Running setup.py install for django-debug-toolbar-mongo
Successfully installed django-debug-toolbar-mongo
Cleaning up...
This way you will make pip work with github source repositories.
If you want to use requirements.txt file, you will need git and something like the entry below to anonymously fetch the master branch in your requirements.txt.
For regular install:
git+git://github.com/celery/django-celery.git
For "editable" install:
-e git://github.com/celery/django-celery.git#egg=django-celery
Editable mode downloads the project's source code into ./src in the current directory. It allows pip freeze to output the correct github location of the package.
Clone target repository same way like you cloning any other project:
git clone git#github.com:myuser/foo.git
Then install it in develop mode:
cd foo
pip install -e .
You can change anything you wan't and every code using foo package will use modified code.
There 2 benefits ot this solution:
You can install package in your home projects directory.
Package includes .git dir, so it's regular Git repository. You can push to your fork right away.
Here is the simple solution
With git
pip install git+https://github.com/jkbr/httpie.git
Without git
pip install https://github.com/jkbr/httpie/tarball/master
or
pip install https://github.com/jkbr/httpie/zipball/master
or
pip install https://github.com/jkbr/httpie/archive/master.zip
Note: You need a python package with the setup.py file in it.
Below format could be use to install python libraries via pip from GitHub.
pip install <LibName>#git+ssh://git#github.com/<username>/<LibName>#egg<LibName>
you can try this way in Colab
!git clone https://github.com/UKPLab/sentence-transformers.git
!pip install -e /content/sentence-transformers
import sentence_transformers
Tested Optimized Ubuntu Solution using the terminal command:
Step 1:
In a selected directory clone the git repo.
Example:
$ git clone https://github.com/httpie/httpie.git
Step 2:
select/change path to the directory, to the cloned folder
$ cd ClonedFolderName
Step 3:
Enter following command to install that package
ColnedFolderName(directory Name) $ pip install ./
pip install ./ is command to enter in cloned directory name
Note: Make sure setup.py is inside cloned repo. (which is by default in it)
Related
I know about requirements.txt, but that only includes a list of dependencies.
But what about the other meta information like the package name, author, main function etc. ?
Also i know about setup.py but since i want to programmatically access values inside, i need a configuaration file standard like yaml/json and not python code.
Did the python community come out with something truly comparable to package.json ?
1. Without 3rd Party Packages
pip freeze > requirements.txt
In the local machine. And in the server,
pip install -r requirements.txt
This installs all the dependencies
2. With a 3rd Party Package
pipenv
I would use pipenv instead of pip. pipenv automatically generate Pipfile and Pipfile.lock that is far superior to requirements.txt
Install pipenv and setting it for your project
pip install --user pipenv
cd yourproject
pipenv install package1 package2 ...
to install packages from Pipfile is as simple as
pipenv install
Read more: https://pipenv.pypa.io/en/latest/
poetry
I have recently moved from pipenv to poetry because poetry has everything pipenv offers and much more. It is end-to-end, as it includes building and publishing of your project to pypi.
installing poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
and set .poetry/bin in your path.
poetry new yourproject
cd yourproject
poetry add packagename
Like pipenv this generate pyproject.toml file that context all your requirements. Like Pipenv, to install your depence
poetry install
See more: https://poetry.eustace.io/docs/
See Python packaging war: Pipenv vs. Poetry for short review of these awesome packages
Like the comments on my question answered, it's pyproject.toml together with the poetry package management tool
I'm working with the following directory structure:
package1/
package_content1
package2/
package_content2
setup.py
Up until now I usually installed with pip install -e . commenting the part of the package1 to install package2 of the setup.py and vice versa. Is there a way to create 2 different setup.py and declare the package I want to install without changing the whole structure of the project?
Something like pip install -e 'package1' or pip install -e package1_setup.py
UPDATE:
I have been testing other methods and I've come to the solution of installing it via python3 package1_setup.py develop. Although I see that it is not the preferred method, so if there is a way to do the same with pip install -e let me know.
try having something like this
package1/
package_content1
setup.py
package2/
package_content2
setup.py
and you can do pip install -e ./package1 or pip install -e ./package2 from top level
I trying to download requirements.txt from my repo in GitHub
I use pip install -e git+https://github.com/GabrielCoutz/Problema-Chiado#egg=requirements.txt and this is returning:
ERROR: File "setup.py" not found for legacy project requirements.txt from git+https://github.com/GabrielCoutz/Problema-Chiado#egg=requirements.txt.
How i can create setup.py and what i have to put in?
pip install -r https://raw.githubusercontent.com/GabrielCoutz/Problema-Chiado/main/requirements.txt
-e is for local installs for developing, for example. Check this:
What is the use case for `pip install -e`?
For example, we have project Foo with dependency Bar (that in private Git repo) and we want install Bar into Foo directory via pip from requirements.txt.
We can manually install Bar with console command:
pip install --target=. git+ssh://git.repo/some_pkg.git#egg=SomePackage
But how to install Bar into current directory from requirements.txt?
The best way to do this would be to clone the repository, or just donwload the requirements.txt file, and then run pip install -r requirements.txt to install all the modules dependencies.
You can use the "editable" syntax in requirements.txt to install a package from a VCS (eg git)
From the docs:
pip install -e git+https://git.repo/some_pkg.git#egg=SomeProject
I have a script that creates a virtualenv, installs distribute and pip in it and then optionally clones a git repo.
Now I have the project I will be working on, installed. But its dependencies are not installed. How can I make pip install all the dependencies as if I have issued a pip install MyApp?
EDIT: Appareantly my question is a duplicate of this one.
Not exactly sure but pip install -e . seems to do what I want without too many extra stuff lying around. I'd prefer if my code wasn't linked from site-packages though.
If your dependencies are defined in the setup.py file, you can first dump them to an external file using:
python setup.py egg_info
This will list all your dependencies in YOUR_PROJECT.egg-info/requires.txt file. Then you can install them using pip:
pip install -r *.egg-info/requires.txt
to delete what you just created:
rm -rf *.egg-info/
to save some time copy pasting:
python setup.py egg_info
pip install -r *.egg-info/requires.txt
rm -rf *.egg-info/
In my package root issuing pip install -e . installs dependencies.
To install your project's dependencies (i.e. install_requires + extra_requires) you have to extract your dependencies using setuptools egg-info and then install the filtered list of the combined dependencies:
python setup.py egg_info
pip install `grep -v '^\[' *.egg-info/requires.txt`
You should use the pip requirements file.
Essentially, place all your requirements, one in each line in a file and pass that to pip using the command
pip install -r requirements.txt
What more, if you have a standard environment, pip can actually dump such a file from existing installs using the command:
pip freeze
You can put the file thus generated directly into the pip requirements, and call the previous command from your deployment script.
Pretty cool, isnt it? :)
You can use pip-tools to create a requirements.txt that only contains the dependencies of your package:
$ pip-compile -o requirements.txt setup.py
Note that the command above only works if you do not already have a requirements.txt file. If you happen to have one already, just delete it.
Using the generated requirements.txt you can then run pip to install the dependencies:
$ pip install -r requirements.txt
Bonus 1:
The requirements.txt will include comments that indicate where the regarding dependency originates from.
Bonus 2:
If you have have an extras_require section for optional dependencies in your setup.py that looks e.g. like this:
...
extras_require={
"development": [
"wheel",
"debugpy",
"pytest",
],
},
...
You can create the requirements.txt including the optional dependencies by using:
$ pip-compile -o requirements.txt --extra development setup.py