We Keep Coding

Pip install from private Git repo, with Personal access token in Git URL

I am trying to install a package from a private repository on Git.
I am using Personal Access Token in my Git URL in order to bypass the manual authentication step. (You can read about Personal Access Tokens here)
If I add this git URL in requirements file and then use the requirements file in pip to install build it works.
requirements.txt
<package name> # git+https://<Personal Access Token>#<git server address>/<username>/<repository name>.git#<branch name>#egg=<package name>
But, if I use the same URL directly it asks for password, how do I avoid this password prompt (as mentioned below):
pip install git+https://<Personal Access Token>#<git server address>/<username>/<repository name>.git#<branch name>#egg=<package name>
This issue is not observed on all machines that i tested on. It worked on Win 10 x64 and Win 10 x86. But it didn't work on Ubuntu x64. I made sure all the 3 systems has same Python version (3.8.0) and same Pip version (19.3.1).

Use environment variables with the syntax ${VARIABLE} (POSIX format, upper case and underscores allowed) so you're not hard-coding your secrets.
Pip will replace when installing from requirements.txt.
So you can refer to a token to clone the private repo, for example:
in requirements.txt
Github
git+https://${GITHUB_TOKEN}#github.com/user/project.git#{version}
Gitlab
git+https://${GITLAB_TOKEN_USER}:${GITLAB_TOKEN}#gitlab.com/user/project.git#{version}
Bitbucket
git+https://${BITBUCKET_USER}:${BITBUCKET_APP_PASSWORD}#bitbucket.org/user/project.git#{version}
More info here:
https://docs.readthedocs.io/en/stable/guides/private-python-packages.html

Go to GitLab profile settings and generate an read access token:
Select access tokens
give it a name (you can leave expiration date empty)
give it access to read all repositories you have access
generate it
Now edit your requirement file:
pandas==1.0.5
git+https://yourgitlabuser:<generated_token>#gitlab/group/repo#hash#egg=piplib
requests==2.24.0

I just had the same issue. In the end, I could install the package as follows.
from the command line:
pip install mypackagename --no-deps --index-url https://gitlab+deploy-token-mytokenname:tokenpassword#gitlab.mycompany.com/api/v4/projects/123456789/packages/pypi/simple
by specifying it in the requirements.txt file:
(Note that the flask and flask-cors package requirements in the example below are just an example, because it may seem really weird to a reader that the other lines in the example are really content that can be written in a requirements.txt.)
flask==1.1.1
flask-cors==3.0.8
--index-url https://pypi.org/simple --extra-index-url https://gitlab+deploy-token-mytokenname:tokenpassword#gitlab.mycompany.com/api/v4/projects/123456789/packages/pypi/simple
mypackagename
Then of course run pip install -r requirements.txt.
Note that both fragments above show how to provide your password, as you asked.

How to make a continuous delivery of a python function app deployed in Azure?

For the first time I deployed a Python function app to Azure using a deployment pipeline:
https://learn.microsoft.com/bs-latn-ba/azure/azure-functions/functions-how-to-azure-devops
The package is deployed to Azure using Kudu Zip deploy.
My http triggered function runs wonderfully locally (on Windows), but I have a 500 internal errors on Azure because it does not find the module requests.
Exception: ModuleNotFoundError: No module named 'requests'
imports of __init__.py:
import logging, requests, os
import azure.functions as func
If I remove the 'requests' dependency the function works on Azure (status 200).
The requests library is imported by the requirement.txt and copied to the .venv36/lib/site-packages/requests by the build pipeline.
So I am wondering if the virtual environment .venv36 that is built in the package is used by the function deployed in Azure. There is no indication about how to activate virtual environments in Azure.

If you name your virtual env worker_venv as named in the documentation you linked, it should work (assuming you are using a Linux environment for your pipeline).
However, the Python Azure Functions documentation is to be updated very soon, and the recommended way would be to not deploy the entire virtual environment from your deployment pipeline.
Instead, you'd want to install your packages in .python_packages/lib/site-packages.
You could do --
pip3.6 install --target .python_packages/lib/site-packages -r requirements.txt
Instead of --
python3.6 -m venv worker_venv
source worker_venv/bin/activate
pip3.6 install setuptools
pip3.6 install -r requirements.txt
And it should work fine.

We are also having the same issue using the newest version of the YAML pipeline template:
- task: UsePythonVersion#0
displayName: 'Use Python 3.6'
inputs:
versionSpec: 3.6 # Functions V2 supports Python 3.6 as of today
- bash: |
python -m venv worker_venv
source worker_venv/bin/activate
pip install -r requirements.txt
workingDirectory: $(workingDirectory)
displayName: 'Install application dependencies'
Removing the virtual environment step, the Function App deployed and run without any issues. This does not seem to be Python best practices; however, it was the only thing we could do to get this deployed correctly on Azure DevOps Pipelines.
Separately, before making this change, we were able to deploy using the Visual Studio code plugin, which indicated to us that this was an environment issue.
Updated docs from Microsoft (1/12/2020)
https://learn.microsoft.com/en-us/azure/azure-functions/functions-how-to-azure-devops?tabs=python
azure-pipelines.yml (our working version on Azure DevOps Pipelines)
- master
variables:
# Azure Resource Manager connection created during pipeline creation
azureSubscription: '<subscription-id>'
# Function app name
functionAppName: '<built-function-app-name>'
# Agent VM image name
vmImageName: 'ubuntu-latest'
# Working Directory
workingDirectory: '$(System.DefaultWorkingDirectory)/__app__'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- bash: |
if [ -f extensions.csproj ]
then
dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
fi
workingDirectory: $(workingDirectory)
displayName: 'Build extensions'
- task: UsePythonVersion#0
displayName: 'Use Python 3.7'
inputs:
versionSpec: 3.7 # Functions V2 supports Python 3.6 as of today
- bash: |
pip install --upgrade pip
pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
workingDirectory: $(workingDirectory)
displayName: 'Install application dependencies'
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(workingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'production'
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureFunctionApp#1
displayName: 'Azure functions app deploy'
inputs:
azureSubscription: '$(azureSubscription)'
appType: functionAppLinux
appName: $(functionAppName)
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'

It definitely needs to be more clearly pointed out that the proper directory for Python packages when deploying Azure Functions is .python_packages/lib/site-packages. I had to go digging through the Azure Function Core Tools source code to see where they put Python packages.
Also had to dig around in the Function debug console to see where Oryx grabs packages from.
I guess there is a pointer in the Version 3.7 YAML file here, but no callout of the directory's importance and does it apply to Python 3.8 Functions?
If I'm not mistaken, this is a requirement to use DevOps to deploy Python Functions (unless you want to install Function Core Tools as part of your build pipeline!).

You need to handle those 2 imports separately,
import azure.functions as func
import requests

Hopefully I am understanding your problem correctly.
When you are installing on your local machine, libs are installed where python is (or at least somewhere other than where your actual code is). This means, when you package your code, you aren't actually keeping the libs together.
To get around this, you can use a virtual env. Python provide a venv tool (there is also a a standard linux virtual env tool) which you can run via:
python -m venv /path/to/my/dir
source /path/to/my/dir/bin/activate
cd /path/to/my/dir/bin/activate
pip install -r requirements.txt
deactivate
I know you mentioned windows, so I would suggest using WSL and the ubuntu image (generally a nice tool to have anyway). There probably is a way to get that working in windows otherwise though I don't know it.
EDIT: Fixed format

Although its old but:
*pip(python version) install --target .python_packages/lib/site-packages -r requirements.txt
For ex. if you are using 3.7 then
pip3.7 install --target .python_packages/lib/site-packages -r requirements.txt
Works like a charm

How to force pip to get a wheel package (even for package dependencies)?

I'm trying to build a multistage docker image with some python packages. For some reason, pip wheel command still downloads source files .tar.gz for few packages even though .whl files exist in Pypi. For example: it does it for pandas, numpy.
Here is my requirements.txt:
# REST client
requests
# ETL
pandas
# SFTP
pysftp
paramiko
# LDAP
ldap3
# SMB
pysmb
First stage of the Dockerfile:
ARG IMAGE_TAG=3.7-alpine
FROM python:${IMAGE_TAG} as python-base
COPY ./requirements.txt /requirements.txt
RUN mkdir /wheels && \
apk add build-base openssl-dev pkgconfig libffi-dev
RUN pip wheel --wheel-dir=/wheels --requirement /requirements.txt
ENTRYPOINT tail -f /dev/null
Output below shows that it is downloading source package for Pandas but it got a wheel for Requests package. Also, surprisingly it takes a lot of time (I really mean a lot of time) to download and build these packages !!
Step 5/11 : RUN pip wheel --wheel-dir=/wheels --requirement /requirements.txt
---> Running in d7bd8b3bd471
Collecting requests (from -r /requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
Saved /wheels/requests-2.22.0-py2.py3-none-any.whl
Collecting pandas (from -r /requirements.txt (line 7))
Downloading https://files.pythonhosted.org/packages/0b/1f/8fca0e1b66a632b62cc1ae38e197befe48c5cee78f895edf4bf8d340454d/pandas-0.25.0.tar.gz (12.6MB)
I would like to know how I can force it get a wheel file for all the required packages and also for the dependencies listed in these packages. I observed that some dependencies get a wheel file but others get the source packages.
NOTE: code above is a combination of multiple online sources.
Any help to make this build process easier is greatly appreciated.
Thanks in Advance.

You are using Alpine Linux. This one is somewhat unique as it uses musl as the underlying libc implementation, as opposed to the most other Linux distros which use glibc.
If a Python project implements C extensions (this is what e.g. numpy or pandas do), it has two options: either
offer a source dist (.tar.gz, .tar.bz2 or .zip) so that the C extensions are compiled using the C compiler/library found on the target system, or
offer a wheel that contains compiled C extensions. If the extensions are compiled against glibc, they will be unusable on systems using musl, and AFAIK vice versa too.
Now, Python defines the manylinux1 platform tag which is specified in PEP 513 and updated in PEP 571. Basically, the name says it all - wheels with compiled C extensions should be built against glibc and thus will work on many distros (that use glibc), but not on some (Alpine being one of them).
For you, it means that you have two possibilities: either build packages from source dists (this is what pip already does), or install the prebuilt packages via Alpine's package manager. E.g. for py3-pandas it would mean doing:
# echo "#edge http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
# apk update
# apk add py3-pandas#edge
However, I don't see a big issue with building packages from source. When done right, you capture it in a separate layer placed as high as possible in the image, so it is cached and not rebuilt each time.
You might ask, why there's no platform tag analogous to manylinux1, but for musl-based distros? Because no one has written a PEP similar to PEP 513 that defines a musllinux platform tag yet. If you are interested in the current state of it, take a look at the issue #37.
Update
PEP 656 That defines a musllinux platform tag is now accepted, so it (hopefully) won't last long until prebuilt wheels for Alpine start to ship. You can track the current implementation state in auditwheel#305.

For Python 3, your packages will be installed from wheels with ordinary pip call:
pip install pandas numpy
From the docs:
Pip prefers Wheels where they are available. To disable this, use the --no-binary flag for pip install.
If no satisfactory wheels are found, pip will default to finding source archives.

pip3 setup.py install_requires PEP 508 git URL for private repo

I am trying to run:
pip3 install -e .
in my Python project where I have the following setup.py:
from setuptools import setup
setup(
name='mypackage',
install_requires=[
"anotherpackage#git+git#bitbucket.org:myorg/anotherpackage.git"
]
)
but it fails with:
error in mypackage setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid URL given
I guess it is correct about the format of my URL as PEP 508 doesn't allow specifying git user name for ssh clone URLs.
What is the correct syntax for PEP 508 URLs with git+ssh protocol for install_requires dependency for private git repositories (in this case hosted on BitBucket)? What is the syntax for specifying a specific branch, tag or sha?
More context to avoid XY problem
I have an internal Python project that depends on multiple internally developed Python packages. I would like to avoid the necessity for hosting my own PIP repository in the organisation and thus I am trying to use git URLs directly. I need to use ssh protocol for git URLs as all the users have their ssh keys configured and it would be cumbersome to ask all the users to configure their app passwords in BitBuckets (I have 2FA required and the regular user password doesn't work).
I have already tried to use:
dependency_links
setup(
name='mypackage',
install_requires=[
"anotherpackage==0.0.1"
],
dependency_links=[
"git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage-0.0.1"
]
)
But they are deprecated and they are ignored by pip3 install -e .. According to documentation I've found, PEP 508 URLs should be used instead.
requirements.txt file with entries duplicated from install_requires entries
I have a requirements.txt file with:
-e git+git#bitbucket.org:myorg/anotherpackage.git#0.0.1#egg=anotherpackage
and I use pip3 install -r requirements.txt instead of pip3 install -e .. It works but is suboptimal as I have to keep both setyp.py and requirements.txt in sync.
If there is any other recommended solution for my problem I would like to learn about it :)

After checking pip source code I found the correct syntax for private BitBucket repositories.
The general form for the packages with URLs is <package name>#<URI> and the URI must start with a <scheme>://.
So I fixed it to:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git
and then I was getting a different error - this time git command (invoked by pip) was complaining about repository URL ssh://git#bitbucket.org:myorg/anotherpackage.git.
I checked the git documentation for the ssh:// URLs format and found out that hostname and organisation parts must be separated with / instead of ::
ssh://git#bitbucket.org/myorg/anotherpackage.git
This URL works fine. I also learned from the pip source code that the actual revision/branch/tag can be specified by appending #<rev-spec> so I can specify for example the tag 0.0.1 with the following in install_requires:
anotherpackage#git+ssh://git#bitbucket.org:myorg/anotherpackage.git#0.0.1
The only issue that I still have is that when I change the revision and run pip3 install -e . again it doesn't detect the change (even when run with --upgrade). I have to manually uninstall the package (pip3 uninstall anotherpackage) and run pip3 install -e . again.

How to install dependencies from requirements.txt in a Yocto recipe for a local Python project

What I should have:
I want my Yocto Project to build a package for my Python project with all dependencies inside. The project has to run out of box on the resulting read-only sdcard image.
It simply should install all requirements in the required version to the package.
What I tried without luck:
Calling pip in do_install():
"pip/pip3 is not found", even it's in RDEPENDS.
Anyway, I really prefer this way.
With inherit pypi:
When trying with inherit pypi, it tries to get also my local sources (my pyton project) from pypi. And I have always to copy the requirements to the recipe. This is not my preferred way.
Calling pip in pkg_postinst():
It tries to install the modules on first start and fails, because the system has no internet connection and it's a read-only system. It must run out of the box without installation on first boot time. Does its stuff to late.
Where I'll get around:
There should be no need to change anything in the recipes when something changes in requirements.txt.
Background information
I'm working with Yocto Rocko in a Linux environment.
In the Hostsystem, there is no pip installed. I want to run this one installed from RDEPENDS in the target system.
Building the Package (only this recipe) with:
bitbake myproject
Building the whole sdcard image:
bitbake myProject-image-base
The recipe:
myproject.bb (relevant lines):
RDEPENDS_${PN} = "python3 python3-pip"
APP_SOURCES_DIR := "${#os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../../../../app-sources')}"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = " \
file://${APP_SOURCES_DIR}/myProject \
...
"
inherit allarch # tried also with pypi and setuptools3 for the pypi way.
do_install() { # Line 116
install -d -m 0755 ${D}/myProject
cp -R --no-dereference --preserve=mode,links -v ${APP_SOURCES_DIR}/myProject/* ${D}/myProject/
pip3 install -r ${APP_SOURCES_DIR}/myProject/requirements.txt
# Tried also python ${APP_SOURCES_DIR}/myProject/setup.py install
}
# Tried also this, but it's no option because the data MUST be included in the Package:
# pkg_postinst_${PN}() {
# #!/bin/sh -e
# pip3 install -r /myProject/requirements.txt
# }
FILES_${PN} = "/myProject/*"
Resulting Errors:
Expected to install the listed modules from requirements.txt into the myProject package, so that the python app will run directly on the resulting readonly sdcard image.
With pip, I get:
| /*/tmp/work/*/myProject/0.1.0-r0/temp/run.do_install: 116: pip3: not found
| WARNING: exit code 127 from a shell command.
| ERROR: Function failed: do_install ...
When using pypi:
404 Not Found
ERROR: myProject-0.1.0-r0 do_fetch: Fetcher failure for URL: 'https://files.pythonhosted.org/packages/source/m/myproject/myproject-0.1.0.tar.gz'. Unable to fetch URL from any source.
=> But it should not fetch myProject, since it is already local and nowhere remote.
Any ideas? What would be the best way to reach to a ready to use sdcard image without the need to change recipes when requirements.txt changes?

You should use RDEPENDS_${PN} to take care of your dependencies for your app in the recipe.
For example, assuming your python app needs aws-iot-device-sdk-python module, you should add it to RDEPENDS in the recipe. In your case, it would be like this:
RDEPENDS_${PN} = "python3 \
python3-pip \
python3-aws-iot-device-sdk-python \
"
Here's the link showing the Python modules supported by OpenEmbedded Layer.
https://layers.openembedded.org/layerindex/branch/master/layer/meta-python/
If the modules you need are not there, you will likely need to create recipes for the modules.

My newest findings:
Yocto/bitbake seems to suppress interpreting the requirements, because this breaks automatic dependency resolving what could lead to conflicts.
Reason: The required modules from setup.py would not be stored as independent packages, but as part of my package. So, bitbake does not know about this modules what could conflict with other packages that probably requires same modules in different versions.
What was in my recipe:
MY_INSTALL_ARGS = "--root=${D} \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
do_install() {
PYTHONPATH=${PYTHON_SITEPACKAGES_DIR} \
${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${MY_INSTALL_ARGS}
}
If I execute this outside of bitbake as python3 setup.py install ${MY_INSTALL_ARGS}, all will be installed correctly, but in the recipe, no requirements are installed.
There is a parameter --no-deps, but I didn't find where it is set.
I think there could be one possibility to exploit the requirements out of setup.py:
Find out where to disable --no-deps in the openembedded/poky layer for easy_install.
Creating a separate PYTHON_SITEPACKAGES_DIR
Install this separate PYTHON_SITEPACKAGES_DIR in eg the home directory as private python modules dir.
This way, no python module would trigger a conflict.
Since I do not have the time to experiment with this, I'll define now one recipe per requirement.

You try installing pip?
Debian
apt-get install python-pip
apt-get install python3-pip
Centos
yum install python-pip