I have a local package set up like this:
D:.
| .gitignore
| LICENSE
| pyproject.toml
| README.md
| requirements.txt
+---.venv
| | ...
+---mypackage
| | __init__.py
| +---moduleA
| | | module_a_src.py
| | | module_a_helpers.py
| +---tools
| | tools.py
\---tests
The __init__.py file is empty. The tools.py file contains the following:
def working(string):
print(string)
print("here i am")
I installed the package in edit mode to my local venv using pip install -e .
I don't have/want entry points yet. I can run the following from the shell and it works as expected:
$ py -c "from mypackage.tools import tools; tools.working('foo')"
here i am
foo
(.venv)
However, I want to be able to run py -c "import mypackage; tools.working('foo')". I tried adding the following to the __init__.py file:
from tools import tools
# other things that didn't work and return the same error:
# from .tools import tools
# import tools.tools
# from . import tools
# from mypackage.tools import tools
But I get this:
$ py -c "import mypackage; tools.working('foo')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'tools' is not defined
here i am
(.venv)
I tried adding an empty __init__.py to the tools folder, no luck.
The pyproject.toml contains this, if that matters:
[tool.setuptools]
include-package-data = true
[tool.setuptools.packages.find]
where = [".", "mypackage"]
Related
Consider the following minimal Python project with setuptools packaging and a "pyproject.toml" file (see setuptools Build System Support):
> tree myproject
myproject
|-- myproject
| `-- __init__.py
|-- pyproject.toml
|-- setup.cfg
`-- setup.py
"setup.py" is only a minimal dummy file to enable support for editable installs, as described here:
from setuptools import setup
if __name__ == '__main__':
setup()
When performing an editable install (pip install -e) to a virtualenv, everything works as expected:
> ls venv/lib/python3.9/site-packages | grep myproject
myproject.egg-link
> cat venv/lib/python3.9/site-packages/easy-install.pth
/myproject
> python3
>>> import myproject
Hello world!
The same is true for a non-editable system-wide install:
> ls /usr/local/lib/python3.9/dist-packages | grep myproject
myproject
myproject-1.0.dist-info
> python3
>>> import myproject
Hello world!
For an editable system-wide install, however, pip succeeds but does not result in a usable module:
> ls /usr/local/lib/python3.9/dist-packages | grep myproject
(No output)
> python3
>>> import myproject
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'myproject'
I'm aware that there were some issues in the past with "pyproject.toml" and editable installs. However, these appear to be fixed since 2019.
I have a relatively recent Debian Bullseye system with pip 20.3.4 and setuptools 52.0.0.
There also is PEP 660, which has not been implemented by setuptools yet. However, the dummy "setup.py" file (see above) should work around that limitation.
This is a Debian-specific problem.
Having a "pyproject.toml" file (i.e. a PEP 518 package), enables build isolation by default. On Debian, that results in editable installs ending up in "/usr/lib/python3.9/site-packages" instead of "/usr/local/lib/python3.9/dist-packages":
> ls /usr/lib/python3.9/site-packages
easy-install.pth myproject.egg-link
> cat /usr/lib/python3.9/site-packages/easy-install.pth
/myproject
However, "/usr/lib/python3.9/site-packages" is not part of the default Debian Python module paths.
I reported this behavior as Debian bug #1004149. Until it has been fixed, one can work around it by using pip install --no-build-isolation -e.
I recently came across a Python package that had setup.py located not inside the package root.
package
|
|--cpp
| |
| |--<cpp stuff>
|
|--python
| |
| |--setup.py
| |--<python stuff>
I cannot simply point dependency_links in setup.py to the python folder, because the repo can only be downloaded as a whole (it comes in a tarball).
How would one handle a dependency like this? Possibly a download and then a local reference in setup like
install_requires=['package # file://localhost/{}'.format(
os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'package', 'python'))]
To em-answer my comment:
You could just build a wheel (.whl) of the Python bits with
python setup.py bdist_wheel
and use the generated wheel as your dependency.
I am building a flask api and want to create a docker image of it. However, when I do docker-compose run (after build), it cannot find the module.
Error:
api_1 | Traceback (most recent call last):
api_1 | File "app.py", line 6, in <module>
api_1 | from api.classify.classify import get_prediction
api_1 | ModuleNotFoundError: No module named 'api'
My folder structure looks like this:
- api
-- classify
--- classify.py
-- app.py
-- Dockerfile
-- requirements.txt
-- setup.py
The setup.py looks like this:
from setuptools import setup, find_packages
setup(
name='image_api',
keywords='',
version='0.1',
packages=find_packages()
)
And the Dockerfile looks like this:
FROM python:3
WORKDIR /user/src/app
ENV PYTHONPATH=/api
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN python setup.py install
CMD ["python", "app.py"]
How do I fix this and what is best practice to install custom packages when building a Docker image?
You should never over-write PYTHONPATH like that, nstead append your path, or the system will not find the installed Python packages.
You can do either of the following to make it work:
RUN export PYTHONPATH="$PYTHONPATH:/api"
ENV PYTHONPATH="$PYTHONPATH:/api"
Also your Dockerfile should be on api level, it wont be able to look for it in the present directory structure.
I'm trying to compile and install the following python package, system-wide:
https://github.com/mathurinm/BlitzL1/
(note that the init.py of the module is inside a folder named python)
So I run, at the root of the repo,
pip install -e .
I get:
zongo#zongo-HP-EliteBook-840-G3:~/workspace/BlitzL1$ pip install -e .
Obtaining file:///home/zongo/workspace/BlitzL1
Installing collected packages: blitzl1
Running setup.py develop for blitzl1
Successfully installed blitzl1
zongo#zongo-HP-EliteBook-840-G3:~/workspace/BlitzL1$ ipython
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import blitzl1
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-8bb5a22c28e9> in <module>
----> 1 import blitzl1
ModuleNotFoundError: No module named 'blitzl1'
after trial and error, I found that renaming the python folder to blitzl1 and replacing, in setup.py:
package_dir = {"blitzl1": "python"},
by
package_dir = {"blitzl1": "blitzl1"},
makes it possible to import the package. Why is the first one not working?
By the way:
zongo#zongo-HP-EliteBook-840-G3:~/workspace/BlitzL1$ which pip
/home/zongo/anaconda3/bin/pip
This is due to a long lasting issue in pip with installing a package in develop mode when the package directory is not in the same folder as the setup.py. See here for more info.
To be clearer, if the package name is my_package and the structure of the source is:
|- setup.py
|- src
|- __init__.py
|- ...
with package_dir={'my_package':'src'}, installing the package with either pip install -e . or python setup.py develop will raise the error reported by the OP.
A way to mitigate this is to change to package_dir={'':'src'} and change the structure of the repo to
|- setup.py
|- src
|- mypackage
|- __init__.py
|- ...
I'd like to use "scikits.samplerate", but installation fails.
I'm using Windows10 (64 Bits) for Python 3.51 with Anaconda.
Firstly, I followed this instruction:
https://scikits.appspot.com/samplerate
>pip install scikits.samplerate Collecting scikits.samplerate Using cached scikits.samplerate-0.3.3.tar.gz
Complete output from command python setup.py egg_info:
SamplerateInfo:
libraries samplerate not found in c:\users\username\anaconda3\lib
libraries samplerate not found in C:\
libraries samplerate not found in c:\users\username\anaconda3\libs
Traceback (most recent call last):
File "scikits\samplerate\setup.py", line 15, in configuration
sf_config = sf_info.get_info(2)
File "c:\users\username\anaconda3\lib\site-packages\numpy\distutils\system_info.py", line 568, in get_info
raise self.notfounderror(self.notfounderror.__doc__)
numpy.distutils.system_info.NotFoundError: Some third-party program or library is not found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\username\AppData\Local\Temp\pip-build-9sjnkaf5\scikits.samplerate\setup.py", line 74, in <module>
classifiers = CLASSIFIERS,
File "c:\users\username\anaconda3\lib\site-packages\numpy\distutils\core.py", line 135, in setup
config = configuration()
File "C:\Users\username\AppData\Local\Temp\pip-build-9sjnkaf5\scikits.samplerate\setup.py", line 59, in configuration
config.add_subpackage(DISTNAME)
File "c:\users\username\anaconda3\lib\site-packages\numpy\distutils\misc_util.py", line 1002, in add_subpackage
caller_level = 2)
File "c:\users\username\anaconda3\lib\site-packages\numpy\distutils\misc_util.py", line 971, in get_subpackage
caller_level = caller_level + 1)
File "c:\users\username\anaconda3\lib\site-packages\numpy\distutils\misc_util.py", line 908, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "scikits\samplerate\setup.py", line 20, in configuration
[samplerate].""")
numpy.distutils.system_info.NotFoundError: SRC (http://www.mega-nerd.com/SRC/) library not found. Directories to search
for the libraries can be specified in the site.cfg file, in section
[samplerate].
---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in C:\Users\username\AppData\Local\Temp\pip-build-9sjnkaf5\scikits.samplerate\
... Next, I followed this instruction:
https://anaconda.org/hcc/scikits.samplerate
>conda install -c hcc scikits.samplerate=0.3.3 Using Anaconda Cloud api site https://api.anaconda.org Fetching package metadata: ...... Solving package specifications: . Error: Package missing in current win-64 channels:
- scikits.samplerate 0.3.3*
You can search for this package on anaconda.org with
anaconda search -t conda scikits.samplerate 0.3.3*
... so, I serached:
[Anaconda3] C:\Users\username>anaconda search -t conda scikits↲ Using Anaconda Cloud api site https://api.anaconda.org↲ Run 'anaconda show <USER/PACKAGE>' to get more details:↲ Packages:↲
Name | Version | Package Types | Platforms↲
------------------------- | ------ | --------------- | ---------------↲
HCC/scikits.samplerate | 0.3.3 | conda | linux-64↲
: A python module for high quality audio resampling↲
anaconda/scikits-image | 0.7.1 | conda | linux-64, win-32, win-64, linux-32, osx-64↲
davidbgonzalez/scikits.talkbox | 0.2.5 | conda | linux-64↲
desilinguist/scikits-bootstrap | 0.3.1 | conda | linux-64, osx-64↲
krisvanneste/scikits.timeseries | 0.91.3 | conda | win-64↲
lukepfister/scikits.cuda | master_2016.2 | conda | linux-64↲
: Python interface to GPU-powered libraries↲
menpo/scikits.sparse | 0.2 | conda | linux-64, osx-64↲
miguelalexanderdiaz/scikits.cuda | 0.5.0b1 | conda | linux-64↲
: Python interface to GPU-powered libraries↲
poppy-project/scikits.samplerate | 0.3.3 | conda | linux-armv7l↲
: Simple Hamming Marker Detection using OpenCV↲
rgrout/scikits.bootstrap | 0.3.2 | conda | linux-64, osx-64↲
: Bootstrap confidence interval estimation routines for SciPy.↲ Found 10 packages↲ ↲ [Anaconda3] C:\Users\username>anaconda show poppy-project/scikits.samplerate↲ Using Anaconda Cloud api site https://api.anaconda.org↲ Name: scikits.samplerate↲ Summary: Simple Hamming Marker Detection using OpenCV↲ Access: public↲ Package Types: conda↲ Versions:↲ + 0.3.3↲ ↲ To install this package with conda run:↲
conda install --channel https://conda.anaconda.org/poppy-project scikits.samplerate↲ ↲ [Anaconda3] C:\Users\username>conda install
--channel https://conda.anaconda.org/poppy-project scikits.samplerate↲ Using Anaconda Cloud api site https://api.anaconda.org↲ Fetching package metadata: ......↲ Solving package specifications: .↲ Error: Package missing in current win-64 channels:↲
- scikits.samplerate↲ ↲ You can search for this package on anaconda.org with↲ ↲
anaconda search -t conda scikits.samplerate↲
... I have done what I was told, but still it fails.
Does anyone have a solution?
Is this really installable?
I am not sure if this would work, but glad if it does. Have you tried to edit the site.cfg file and try the installation again.This is what line 20 error in your question says as well.
The user here has done it on Ubuntu, maybe a similar approach works for Windows as well.
http://msnoise.org/doc/installation.html
You first need to install the SRC library:
sudo apt-get install libsamplerate0 libsamplerate0-dev
This python package will probably be the most tricky to install. If you are lucky, you can just
pip install scikits.samplerate
On my Ubuntu 12.04, this results in an error because the SRC library path is not found. The reason is that the setup searches SRC in /usr/lib and not in /usr/lib/x86_64-linux-gnu where the library is actually present. To install, you need to download the archive from pypi and edit some configuration file:
wget https://pypi.python.org/packages/source/s/scikits.samplerate/scikits.samplerate-0.3.3.tar.gz#md5=96c8d8ba3aa95a9db15994f78792efb4
tar -xvf scikits.samplerate-0.3.3.tar.gz
cd scikits.samplerate-0.3.3
then edit the site.cfg example file and insert the following lines:
[samplerate]
library_dirs=/usr/lib/x86_64-linux-gnu
include_dirs=/usr/include
To know where the SRC library is on you machine:
sudo dpkg -L libsamplerate0
sudo dpkg -L libsamplerate0-dev
then, build and install:
python setup.py build
python setup.py install
Firstly, notice I used UNIX system, not Windows.
I had same/similar error:
...numpy.distutils.system_info.NotFoundError: SRC
(http://www.mega-nerd.com/SRC/) library not found. Directories to
search
for the libraries can be specified in the site.cfg file, in section...
I followed link given in it, found download site:
http://www.mega-nerd.com/SRC/download.html
downloaded sources,
compiled and installed them
then just installed scikits.samplerate using pip
In Windows it might be more difficult (i haven't tried it!), but on the site there is a link to instructions for Windows: http://www.mega-nerd.com/SRC/win32.html
For Centos7 when I had that error I did:
yum install libsamplerate-devel libsamplerate
For Windows I am using Miniconda2 and in order to get librosa to work installed ffmpeg using this command:
conda install -c conda-forge ffmpeg