We Keep Coding

Calling pip package files from within pip package? (`No module named 'src.snnalgorithms'`) [duplicate]

This question already has answers here:
What is the use case for `pip install -e`?
(3 answers)
Closed 4 months ago.
Whilst in some_other_package, I am importing files from the snnalgorithms pip package. I received the error:
No module named 'src.snnalgorithms'. This is a valid error because that src.snnalgorithms file does not exist in the some_other_package project from which I was calling the pip package.
As a solution, I can make all the imports in the snn.algorithms pip package, relative to itself. Instead of:
from src.snnalgorithms.population.DUMMY import DUMMY
One could write:
from snnalgorithms.population.DUMMY import DUMMY
However, that implies that each time I want to run the code to briefly verify a minor change, or run the tests after a change, I will have to:
Upload the changes into the pip package.
Re-install the pip package locally to reflect the changes.
This significantly slows down development. Hence I was wondering are there more efficient solutions for this?

You can use editable mode for development mode
pip install -e . # Install package locally
From pip documentation:
Editable installs allow you to install your project without copying any files. Instead, the files in the development directory are added to Python’s import path. This approach is well suited for development and is also known as a “development installation”.
With an editable install, you only need to perform a re-installation if you change the project metadata (eg: version, what scripts need to be generated etc). You will still need to run build commands when you need to perform a compilation for non-Python code in the project (eg: C extensions).
Example
If you have project: some_other_package from which you call pip package snnalgorithms you can:
cd snnalgorithms
pip install -e .
cd ..
cd some_other_package
python -m src.some_other_package
Assuming you use the same conda environment for both packages, both packages will then be able to use your newest changes that have not even been published to pypi.org yet.

Unable to find installation candidates for nvidia-cudnn-cu11 when trying to install kornia

I'm trying to install kornia using poetry.
System details
Python 3.8.5 (I have to use this version)
poetry 1.1.13 (upgrading to v1.2 is not an option)
I've previously installed kornia (poetry add kornia) but for some unknown reason I'm now getting the following traceback:
RuntimeError
Unable to find installation candidates for nvidia-cudnn-cu11
(8.5.0.96)
at ~.poetry\lib\poetry\installation\chooser.py:72 in choose_for
68│
69│ links.append(link)
70│
71│ if not links:
→ 72│ raise RuntimeError(
73│ "Unable to find installation candidates for {}".format(package)
74│ )
Since this wasn't a very helpful traceback I thought I'd try to pip install to see if I could get kornia installed somehow, then repeat the process using poetry.
First I ran poetry run pip install nvidia-pyindex, then poetry run pip install kornia. kornia was installed, giving me confidence that I might be able to do the same using poetry.
So I ran poetry add nvidia-pyindex which installed, but when I ran poetry add kornia I got the same RuntimeError.
I wondered if having poetry add without pip uninstall might have caused issues. So I started from scratch - I ran:
rm -rf ~/.cache/pypoetry/cache/repositories/pypi/
poetry rm {name of venv}
git checkout master pyproject.toml poetry.lock
poetry add nvidia-pyindex
which created the venv (by the same name as previously - not sure if this is an issue in itself, since the name of the venv was first created in PyCharm and I wonder if there are some files lingering in places I don't know that is the root of my problems?), then poetry add kornia but got the same error.
I also tried to clear poetry's cache as suggested here but that didn't help. I didn't find a solution here. I don't believe this applies since poetry was installed properly and is working fine otherwise. I read through this but didn't see any working solution. I tried a fix mentioned here, which was to add
[[tool.poetry.source]]
name = "nvidia"
url = "https://pypi.ngc.nvidia.com"
secondary = true
to pyproject.toml, hoping that it would avoid the error when poetry tries to install nvidia-cudnn-cu11, but it didn't help either.
I tried poetry add nvidia-cudnn-cu11 and poetry add nvidia-cudnn-cu116 hoping that having added the ndvidia repo as a secondary source would get kornia's dependencies installed before trying to install kornia itself.
{I don't know if secondary sources are only used when poetry adding a package (i.e. kornia) and "ignored" for installation of dependencies (nvidia-cudnn-cu11) of the package to be installed (kornia).}
In any case, that didn't work either.
Running poetry source add nvidia https://pypi.ngc.nvidia.com didn't work - presumably because poetry source add was added to poetry v1.2 and was not part of v1.1.13.
I also tried poetry add --source nvidia https://pypi.ngc.nvidia.com but I got
PermissionError
[Errno 13] Permission denied:
'C:\Users\...\AppData\Local\Temp\tmp2t6am7ei'
at ~.poetry\lib\poetry\utils\helpers.py:101 in download_file
97│
98│ with get(url, stream=True) as response:
99│ response.raise_for_status()
100│
→ 101│ with open(dest, "wb") as f:
102│ for chunk in response.iter_content(chunk_size=chunk_size):
103│ if chunk:
104│ f.write(chunk)
I'm out of ideas and hoped someone might help.

could this be related to this problem?
https://github.com/python-poetry/poetry/issues/6939
https://github.com/pytorch/pytorch/issues/88049
I had a similar issue and was able to fix it by making the torch dependency explicit and install torch version < 1.13:
[tool.poetry.dependencies]
...
torch = "~1.12.1"

How to install dependencies from requirements.txt in a Yocto recipe for a local Python project

What I should have:
I want my Yocto Project to build a package for my Python project with all dependencies inside. The project has to run out of box on the resulting read-only sdcard image.
It simply should install all requirements in the required version to the package.
What I tried without luck:
Calling pip in do_install():
"pip/pip3 is not found", even it's in RDEPENDS.
Anyway, I really prefer this way.
With inherit pypi:
When trying with inherit pypi, it tries to get also my local sources (my pyton project) from pypi. And I have always to copy the requirements to the recipe. This is not my preferred way.
Calling pip in pkg_postinst():
It tries to install the modules on first start and fails, because the system has no internet connection and it's a read-only system. It must run out of the box without installation on first boot time. Does its stuff to late.
Where I'll get around:
There should be no need to change anything in the recipes when something changes in requirements.txt.
Background information
I'm working with Yocto Rocko in a Linux environment.
In the Hostsystem, there is no pip installed. I want to run this one installed from RDEPENDS in the target system.
Building the Package (only this recipe) with:
bitbake myproject
Building the whole sdcard image:
bitbake myProject-image-base
The recipe:
myproject.bb (relevant lines):
RDEPENDS_${PN} = "python3 python3-pip"
APP_SOURCES_DIR := "${#os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../../../../app-sources')}"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = " \
file://${APP_SOURCES_DIR}/myProject \
...
"
inherit allarch # tried also with pypi and setuptools3 for the pypi way.
do_install() { # Line 116
install -d -m 0755 ${D}/myProject
cp -R --no-dereference --preserve=mode,links -v ${APP_SOURCES_DIR}/myProject/* ${D}/myProject/
pip3 install -r ${APP_SOURCES_DIR}/myProject/requirements.txt
# Tried also python ${APP_SOURCES_DIR}/myProject/setup.py install
}
# Tried also this, but it's no option because the data MUST be included in the Package:
# pkg_postinst_${PN}() {
# #!/bin/sh -e
# pip3 install -r /myProject/requirements.txt
# }
FILES_${PN} = "/myProject/*"
Resulting Errors:
Expected to install the listed modules from requirements.txt into the myProject package, so that the python app will run directly on the resulting readonly sdcard image.
With pip, I get:
| /*/tmp/work/*/myProject/0.1.0-r0/temp/run.do_install: 116: pip3: not found
| WARNING: exit code 127 from a shell command.
| ERROR: Function failed: do_install ...
When using pypi:
404 Not Found
ERROR: myProject-0.1.0-r0 do_fetch: Fetcher failure for URL: 'https://files.pythonhosted.org/packages/source/m/myproject/myproject-0.1.0.tar.gz'. Unable to fetch URL from any source.
=> But it should not fetch myProject, since it is already local and nowhere remote.
Any ideas? What would be the best way to reach to a ready to use sdcard image without the need to change recipes when requirements.txt changes?

You should use RDEPENDS_${PN} to take care of your dependencies for your app in the recipe.
For example, assuming your python app needs aws-iot-device-sdk-python module, you should add it to RDEPENDS in the recipe. In your case, it would be like this:
RDEPENDS_${PN} = "python3 \
python3-pip \
python3-aws-iot-device-sdk-python \
"
Here's the link showing the Python modules supported by OpenEmbedded Layer.
https://layers.openembedded.org/layerindex/branch/master/layer/meta-python/
If the modules you need are not there, you will likely need to create recipes for the modules.

My newest findings:
Yocto/bitbake seems to suppress interpreting the requirements, because this breaks automatic dependency resolving what could lead to conflicts.
Reason: The required modules from setup.py would not be stored as independent packages, but as part of my package. So, bitbake does not know about this modules what could conflict with other packages that probably requires same modules in different versions.
What was in my recipe:
MY_INSTALL_ARGS = "--root=${D} \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
do_install() {
PYTHONPATH=${PYTHON_SITEPACKAGES_DIR} \
${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${MY_INSTALL_ARGS}
}
If I execute this outside of bitbake as python3 setup.py install ${MY_INSTALL_ARGS}, all will be installed correctly, but in the recipe, no requirements are installed.
There is a parameter --no-deps, but I didn't find where it is set.
I think there could be one possibility to exploit the requirements out of setup.py:
Find out where to disable --no-deps in the openembedded/poky layer for easy_install.
Creating a separate PYTHON_SITEPACKAGES_DIR
Install this separate PYTHON_SITEPACKAGES_DIR in eg the home directory as private python modules dir.
This way, no python module would trigger a conflict.
Since I do not have the time to experiment with this, I'll define now one recipe per requirement.

You try installing pip?
Debian
apt-get install python-pip
apt-get install python3-pip
Centos
yum install python-pip

ImportError: cannot import name _imaging

I installed Pillow, and after I want to do:
from PIL import Image
I get the following error:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 61, in <module>
ImportError: cannot import name _imaging
However, if I import these separately, everything is fine, ie:
import _imaging
import Image
Do you know what the problem might be?

I had the same problem and I solved that by upgrading this package using the command below:
pip install -U Pillow

This also happens if you built Pillow in one OS and then copied the contents of site-packages to another one. For example, if you are creating AWS Lambda deployment package, that's the error you will face when running the Lambda function. If that's the case, then Pillow needs to be installed in a Amazon Linux instance and you have to use the resulting site-packages in your deployment package. See instructions and details here:
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html

I ran into this problem as well. It can happen if you have PIL installed, then install Pillow on top of it.
Go to /usr/local/lib/python2.7/dist-packages/ and delete anything with "PIL" in the name (including directories). If the Pillow .egg file is there you might as well delete that too.
Then re-install Pillow.
substitute "python2.7" for the version of python you're using.

What is your version of pillow?
Pillow >= 2.1.0 no longer supports import _imaging. Please use from PIL.Image import core as _imaging instead. Here's the official documentation.

I have got the same error with Python 3.6. Upgrading Pillow did the job for me.
sudo python3.6 -m pip install Pillow --upgrade
Probably for other python versions use your version instead of 3.6.

This can happen if you're trying to run Pillow installed on a Mac in a Linux environment (for example, e.g. building an AWS Lambda on a Mac then deploying it to a Linux runtime).
To make sure you're installing it for the right platform do the following:
pip3 install --platform manylinux1_x86_64 --only-binary=:all:
The --only-binary=:all: is required when specifying --platform and the platform itself can be found by looking at https://pypi.org/project/Pillow/7.2.0/#files (for example) - the platform is the last part of the filename e.g. win32, manylinux1_x86_64, manylinux1_i686 etc.
This avoids the need to be running Linux to install the Linux build of Pillow.

This may be a niche solution but I was able to fix this problem on Pycharm by going to file->settings->python interpreter and clicking the upgrade symbol next to the pillow package.

For pillow to work PIL must be in /usr/local/lib/python2.7 or3/dist-packages/PIL.py.
In dist-packages PIL.py should have a folder.
sudo apt-get update
pip install Pillow
PIL != PiL

I had the same problem when it tried to deploy a lambda package, the thing is that you have to precompile the package emulating the lambda architecture/runtime that you are going to use, otherwise you'll get cannot import name _imaging. 2 ways of solving this:
1 - spin up an EC2 Amazon Linux instance.( i will only cover this part)
2 - Use dockers.
Short solution
Install Python 3 in Amazon Linux 2 intance. (Must be python3.X you plan to use in lambda)
Install a virtual environment under the ec2-user home directory.
Activate the environment, and then install Boto 3.
Install Pillow
Create a ZIP archive with the contents of the library(PIL and Pillow.libs)
Add your function code to the archive.
Update your the lambda.(AWS CLI)
Long solution
If Python 3 isn't already installed, then install the package using the yum package manager.
`$ sudo yum install python3 -y`
Create a virtual environment under the ec2-user home directory
The following command creates the app directory with the virtual environment inside of it. You can change my_app to another name. If you change my_app, make sure that you reference the new name in the remaining resolution steps.
`$ python3 -m venv my_app/env`
Activate the virtual environment and install Boto 3
Attach an AWS Identity and Access Management (IAM) role to your EC2 instance with the proper permissions policies so that Boto 3 can interact with the AWS APIs. For other authentication methods....For a quick use you can set your credential using $ aws confifure see documentation ( you will need this in step 7)
3.1 Activate the environment by sourcing the activate file in the bin directory under your project directory.
`$ source ~/my_app/env/bin/activate`
3.2. Make sure that you have the latest pip module installed within your environment.
$ pip install pip --upgrade
3.3 Use the pip command to install the Boto 3 library within our virtual environment.
`pip install boto3`
Install libraries with pip.
$ pip install Pillow
4.1 Deactivate the virtual environment.
`$ deactivate`
Create a ZIP archive with the contents of the library.
change directory to where pip is installes. it should be something like /my_app/env/lib/python3.x/site-packages.
IMPORTANT: the key here is to zip the file inside site-packages into
your lambda.(i only used PIL and Pillow.libs to save space but you can
zip everything)
5.1 ZIP everything thats inside the PIL folder.
`zip -r9 PIL.zip ./PIL/`
add the Pillow.libs to your ZIP
`zip -gr PIL.zip Pillow.libs`
Add your function code to the archive.
you can do this in the console if it just on file of code, but i recomend doing it in this step.If you don't have your code,just create a file using vi or nano and save it with the name that your lambda handler will use (in this case will use lambda_function.py).
`zip -g PIL.zip lambda_function.py`
Update your the lambda.(AWS CLI)
if you haven't create a lambda function,do it now before updating the function from the aws cli, make sure that you have the right permission to update lambda from the aws cli.
change LAMBDAFUNCTIONNAME for your function name
aws lambda update-function-code --function-name LAMBDAFUNCTIONNAME P --zip-file fileb://PIL.zip
Getting out of the first loop of hell
go to your lambda console and test your code, make sure you use the same runtime/python version you used in the EC2 instance

Quick solution - import PyQt5 as well,
you will not get that error message.
import PyQt5
from PIL import ImageGrab

As some other answers have alluded to, this can happen when you build Pillow on MacOS and try to import PIL in another OS like some Amazon Linux flavor.
My exact use-case was to package imagehash as a Lambda layer which includes pillow as a dependency. The following guideline has worked great for me for all python packages.
Install the SAM CLI SAM Installation
Create your python script with the lambda handler defined
Create your template.yml file with your Lambda function defined. Your CodeUri should be the relative path to your python script.
Add the package you are trying to create a layer for to your requirements.txt.
Run the following SAM command sam build -t path_to_template
You will now have the following directory .aws-sam/build/{Logical ID Of Lambda Function}. Inside you will see that your python packages and their dependencies have been installed just as if you ran pip download package and unzipped the wheel files.
Now, the python files have been prepped by SAM specifically for Lambda and you can continue with creating your Lambda Layer as desired. Configuring Lambda Layers
Since I use AWS SAM CLI already for running Lambda functions locally, this has been the easiest method for me to create my layers.

Just uninstall pillow:
pip uninstall pillow
then install pillow again:
pip install pillow
works great

I'm using Flask with Google App Engine. I have the module Pillow installed via this command:
pip install -t lib pillow
I fixed this error by defined PIL in my app.yaml file:
libraries:
- name: PIL
version: latest

Solution
pip uninstall PIL
pip uninstall Pillow
pip install Pillow

How to state in requirements.txt a direct github source

I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?

Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.

“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils

requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support

First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt

Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.

None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.

Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.

It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.

I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname

For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.