We Keep Coding

Python cannot find newly installed module

I've created a module using the following setup.py
# -*- coding: utf-8 -*-
# Learn more: https://github.com/kennethreitz/setup.py
from setuptools import setup, find_packages
with open('README.md') as f:
readme = f.read()
with open('LICENSE') as f:
license = f.read()
setup(
name='mymod',
version='1.0a1',
description='test',
long_description=readme,
long_description_content_type="text/markdown",
author='Ray Salemi',
author_email='ray#raysalemi.com',
url='https://rayboston#bitbucket.org/rayboston/mymod',
license=license,
packages=find_packages(exclude=('tests', 'docs', 'examples'))
)
But when I try to install it using
% python setup.py install
I see that it gets installed in my site packages:
Processing mymod-1.0a1-py3.8.egg
Copying mymod-1.0a1-py3.8.egg to /Users/raysalemi/PycharmProjects/testenv/lib/python3.8/site-packages
Adding mymod 1.0a1 to easy-install.pth file
Installed /Users/raysalemi/PycharmProjects/testenv/lib/python3.8/site-packages/mymod-1.0a1-py3.8.egg
Processing dependencies for mymod==1.0a1
Finished processing dependencies for mymod==1.0a1
(testenv) (base) raysalemi#WriteNow mymod % cd ../testenv
(testenv) (base) raysalemi#WriteNow testenv % python
Python 3.8.3 (default, Jul 2 2020, 11:26:31)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mymod
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mymod'
How do I debug this? I can't see an error.
I'm running Big Sur 11.0.1 and
Python 3.8.3 from Anaconda
Pip shows the module is there
Package Version
---------- -------
pip 20.3.1
mymod 1.0a1
setuptools 41.2.0
The problem is that the package is being misnamed:
(testenv) (base) raysalemi#WriteNow site-packages % ls
__pycache__ mymod-1.0a0-py3.8.egg
easy-install.pth mymod-1.0a0.dist-info
easy_install.py setuptools
pip setuptools-41.2.0.dist-info
pip-20.3.1.dist-info src
pkg_resources
It is mymod-1.0a0-py3.8.egg instead of mymod

To debug you can run the setup:
python setup.py sdist --formats=gztar
and unzip the resulting .tar.gz file and check if all your source code files are in it.
(or use --formats=zip instead of gztar to get a simpler file to extract)
The resulting package is always of the form package_name-package_version, so the name you received is not incorrect. (In case you are wondering, you can find the valid package_version formatting rules here.)
You can later use this package by adding it to the requirements.txt file of the project you want to be dependent on it. E.g.
my-package>=1.2.0,<2.0.0
In your case, since the version is a pre-release (mymod-1.0a0-py3.8.egg ==> version is 1.0a0-py3.8.egg which means version 1.0 pre-relase version alpha0-py3.8).
The version 1.0a0-py3.8.egg < than version 1.0 (pre-release always < release with same number), so you will need something like >0,<2.0.
Personally, I put the source code in the repo under src/ and then select these files in setup.py using:
packages=find_namespace_packages(where="src")
There are other parameters I recommend using e.g. make sure environment has a new enough setuptools to recognize find_namespace_packages, take list of dependencies from requirements.txt files etc.:
from setuptools import setup, find_namespace_packages
with open('requirements.txt') as f:
required = f.read().splitlines()
setup(
name='your_project_name',
version='1.0.0',
description='your project description',
url='your repo url',
author='your username',
author_email='your email',
license='your license type',
package_dir={'': 'src'},
setup_requires='setuptools>=45.2.0',
packages=find_namespace_packages(where="src"),
install_requires=required,
data_files=['requirements.txt'],
include_package_data=True
)
See the full list of options and what they are for in the documentation.

I found my problem.
My source directory was named src not mymod. So there was a src directory in site-packages instead of a mymod directory. This is a surprise since the package is named in setup.py.

mognoengine and bson package not work together

I have one project which is depend on mongoengine and I am using one library which was using bson library.
It structure like below.
bsNotify/
├── setup.cfg
├── setup.py
└── src
└── bsnotify
├── __init__.py
└── resources.py
setup.py
$ cat bsNotify/setup.py
"""Base module setup."""
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True
)
setup.cfg
$ cat bsNotify/setup.cfg
[metadata]
name = bsNotify
classifiers =
License :: N/A :: N/A
Programming Language :: Python :: 3.7
[options]
zip_safe = False
include_package_data = True
python_requires = >= 3.7
install_requires =
mongoengine
bson
package_dir=
=src
packages=find:
[options.packages.find]
where=src
[tool:wheel]
universal = 1
[flake8]
exclude =
venv,
.tox,
.git,
__pycache__,
*.pyc,
*.egg-info,
.cache,
.eggs,
max-line-length = 80
[tox]
envlist = py37,unittest,lint
[testenv]
basepython=python3.7
deps =
ipython
pylint
pytest
pytest-cov
pytest-xdist
flake8
flake8-docstrings
[testenv:unittest]
commands=
pytest -v -s -n auto -l --cov=bsnotify --cov-report term-missing --cov-report xml --no-cov-on-fail tests/unit
[testenv:lint]
commands=
flake8 src/bsnotify
pylint src/bsnotify
src/bsnotify/__init__.py
$ cat bsNotify/src/bsnotify/__init__.py
src/bsnotify/resources.py
$ cat bsNotify/src/bsnotify/resources.py
Files in src are empty.
When I run the tox command, it create a virtualenv for testing.
$ tox -v -e unittest --notest
using tox.ini: /Users/myuser/bsNotify/setup.cfg (pid 30954)
using tox-3.14.6 from /Users/myuser/.pyenv/versions/3.7.4/lib/python3.7/site-packages/tox/__init__.py (pid 30954)
GLOB sdist-make: /Users/myuser/bsNotify/setup.py
[30956] /Users/myuser/bsNotify$ /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 setup.py sdist --formats=zip --dist-dir /Users/myuser/bsNotify/.tox/dist >.tox/log/GLOB-0.log
package .tmp/package/1/bsNotify-0.1.2.dev2.zip links to dist/bsNotify-0.1.2.dev2.zip (/Users/myuser/bsNotify/.tox)
unittest cannot reuse: no previous config /Users/myuser/bsNotify/.tox/unittest/.tox-config1
unittest create: /Users/myuser/bsNotify/.tox/unittest
[30992] /Users/myuser/bsNotify/.tox$ /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 -m virtualenv --no-download --python /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 unittest >unittest/log/unittest-0.log
unittest installdeps: ipython, pylint, pytest, pytest-cov, pytest-xdist, flake8, flake8-docstrings
[30993] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip install ipython pylint pytest pytest-cov pytest-xdist flake8 flake8-docstrings >.tox/unittest/log/unittest-1.log
unittest inst: /Users/myuser/bsNotify/.tox/.tmp/package/1/bsNotify-0.1.2.dev2.zip
write config to /Users/myuser/bsNotify/.tox/unittest/.tox-config1 as '7186e9f46c94b6d9f7dde810ce83f8fe46740d9d42f2863cfd063c4c6f4e4a88 /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7\n3.14.6 0 0 0\n00000000000000000000000000000000 ipython\n00000000000000000000000000000000 pylint\n00000000000000000000000000000000 pytest\n00000000000000000000000000000000 pytest-cov\n00000000000000000000000000000000 pytest-xdist\n00000000000000000000000000000000 flake8\n00000000000000000000000000000000 flake8-docstrings'
[31000] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip install --exists-action w .tox/.tmp/package/1/bsNotify-0.1.2.dev2.zip >.tox/unittest/log/unittest-2.log
[31030] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip freeze >.tox/unittest/log/unittest-3.log
unittest installed: apipkg==1.5,appnope==0.1.0,astroid==2.4.2,attrs==19.3.0,backcall==0.1.0,bsNotify==0.1.2.dev2,bson==0.5.10,coverage==5.1,decorator==4.4.2,execnet==1.7.1,flake8==3.8.3,flake8-docstrings==1.5.0,importlib-metadata==1.6.1,ipython==7.15.0,ipython-genutils==0.2.0,isort==4.3.21,jedi==0.17.0,lazy-object-proxy==1.4.3,mccabe==0.6.1,mongoengine==0.20.0,more-itertools==8.3.0,packaging==20.4,parso==0.7.0,pexpect==4.8.0,pickleshare==0.7.5,pluggy==0.13.1,prompt-toolkit==3.0.5,ptyprocess==0.6.0,py==1.8.1,pycodestyle==2.6.0,pydocstyle==5.0.2,pyflakes==2.2.0,Pygments==2.6.1,pylint==2.5.3,pymongo==3.10.1,pyparsing==2.4.7,pytest==5.4.3,pytest-cov==2.9.0,pytest-forked==1.1.3,pytest-xdist==1.32.0,python-dateutil==2.8.1,six==1.15.0,snowballstemmer==2.0.0,toml==0.10.1,traitlets==4.3.3,typed-ast==1.4.1,wcwidth==0.2.4,wrapt==1.12.1,zipp==3.1.0
_______________________________________________________ summary _______________________________________________________
unittest: skipped tests
congratulations :)
Then I try to access mongoengine in newly created virtualenv, it gives error.
$ .tox/unittest/bin/python -c "from mongoengine import connection"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/mongoengine/__init__.py", line 2, in <module>
from mongoengine import connection
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/mongoengine/connection.py", line 1, in <module>
from pymongo import MongoClient, ReadPreference, uri_parser
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/pymongo/__init__.py", line 77, in <module>
from pymongo.collection import ReturnDocument
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/pymongo/collection.py", line 20, in <module>
from bson.code import Code
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/bson/code.py", line 18, in <module>
from bson.py3compat import abc, string_type, PY3, text_type
ImportError: cannot import name 'abc' from 'bson.py3compat' (/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/bson/py3compat.py)
When I was using mongoengine alone with bson, it works fine.
$ python -m venv .venv
$ .venv/bin/pip install mongoengine
Collecting mongoengine
Using cached https://files.pythonhosted.org/packages/7d/bd/9a7239b0032157f948c69febdf71dd82cb54fcd2499077300496a3f076c9/mongoengine-0.20.0-py3-none-any.whl
Collecting pymongo<4.0,>=3.4 (from mongoengine)
Using cached https://files.pythonhosted.org/packages/23/cd/27fbc08f0bd835b4735504a758756e979b42c5bc9ebaac5ed3c2cbffd83f/pymongo-3.10.1-cp37-cp37m-macosx_10_9_x86_64.whl
Installing collected packages: pymongo, mongoengine
Successfully installed mongoengine-0.20.0 pymongo-3.10.1
$ .venv/bin/python -c "from mongoengine import connection"
$ .venv/bin/pip install bson
Collecting bson
Using cached https://files.pythonhosted.org/packages/4d/53/7c534a38850f2252275d7f949aed2219095e90df1e2d180a9c8ed139e499/bson-0.5.10.tar.gz
Collecting python-dateutil>=2.4.0 (from bson)
Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting six>=1.9.0 (from bson)
Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, bson
Running setup.py install for bson ... done
Successfully installed bson-0.5.10 python-dateutil-2.8.1 six-1.15.0
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ .venv/bin/python -c "from mongoengine import connection"
It check the pip freeze between both the virtualenv. Only difference I found is in .tox env has bsNotify==0.1.2.dev2 installed which is not exists in .venv env.
What make bson change the file py3compat.py?

change this
install_requires =
mongoengine
bson
to this
install_requires =
mongoengine
pymongo
or remove altogether (mongoengine already requires pymongo
install_requires =
mongoengine
requirements=bson is the same things as
pip install bson
which installs this 3rd party package which does not include all the goodies found in MongoDB's package
https://pypi.org/project/bson/
pymongo (official MongoDB driver) contains a bson package. I'll drop both a pymongo folder and a bson folder into your site-packages.
bson (third party bson implementation) also wants to drop a bson folder into your site packages.
Installing pymongo and installing bson will conflict folders inside your site-packages. PyMongo only knows how to use it's own implementation, which is why you're seeing pymongo looking for py3compat which is not part of the 3rd party bson package.
Based on your install order, it would have installed mongo engine (and pymongo as a dependency) then installed bson (overwriting MongoDB's bson implementation packaged with pymongo). Pymongo called py3compat expecting it to be there, but 3td party did not implement that (as well as several other things).
pip install pymongo
https://pypi.org/project/pymongo/
As noted on the pymongo pypi page
Do not install the “bson” package from pypi. PyMongo comes with its
own bson package; doing “easy_install bson” installs a third-party
package that is incompatible with PyMongo.

single-sourcing package version for setup.cfg Python projects

For traditional Python projects with a setup.py, there are various ways of ensuring that the version string does not have to be repeated throughout the code base. See PyPA's guide on "Single-sourcing the package version" for a list of recommendations.
Many are trying to move away from setup.py to setup.cfg (probably under the influence of PEP517 and PEP518; setup.py was mostly used declaratively anyway, and when there was logic in setup.py, it was probably for the worse.) This means that most the suggestions won't work anymore since setup.cfg cannot contain "code".
How can I single-source the package version for Python projects that use setup.cfg?

There are a couple of ways to do this (see below for the project structure used in these examples):
1.
setup.cfg
[metadata]
version = 1.2.3.dev4
src/my_top_level_package/__init__.py
import importlib.metadata
__version__ = importlib.metadata.version('MyProject')
2.
setup.cfg
[metadata]
version = file: VERSION.txt
VERSION.txt
1.2.3.dev4
src/my_top_level_package/__init__.py
import importlib.metadata
__version__ = importlib.metadata.version('MyProject')
3.
setup.cfg
[metadata]
version = attr: my_top_level_package.__version__
src/my_top_level_package/__init__.py
__version__ = '1.2.3.dev4'
And more...
There are probably other ways to do this, by playing with different combinatons.
References:
https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
https://docs.python.org/3/library/importlib.metadata.html
Structure assumed in the previous examples is as follows...
MyProject
├── setup.cfg
├── setup.py
└── src
└── my_top_level_package
└── __init__.py
setup.py
#!/usr/bin/env python3
import setuptools
if __name__ == '__main__':
setuptools.setup(
# see 'setup.cfg'
)
setup.cfg
[metadata]
name = MyProject
# See above for the value of 'version = ...'
[options]
package_dir =
= src
packages = find:
[options.packages.find]
where = src
$ cd path/to/MyProject
$ python3 setup.py --version
1.2.3.dev4
$ python3 -m pip install .
# ...
$ python3 -c 'import my_top_level_package; print(my_top_level_package.__version__)'
1.2.3.dev4
$ python3 -V
Python 3.6.9
$ python3 -m pip list
Package Version
------------- ----------
MyProject 1.2.3.dev4
pip 20.0.2
pkg-resources 0.0.0
setuptools 45.2.0
wheel 0.34.2
zipp 3.0.0

python ModuleNotFoundError after running pip install - e .

I'm trying to compile and install the following python package, system-wide:
https://github.com/mathurinm/BlitzL1/
(note that the init.py of the module is inside a folder named python)
So I run, at the root of the repo,
pip install -e .
I get:
zongo#zongo-HP-EliteBook-840-G3:~/workspace/BlitzL1$ pip install -e .
Obtaining file:///home/zongo/workspace/BlitzL1
Installing collected packages: blitzl1
Running setup.py develop for blitzl1
Successfully installed blitzl1
zongo#zongo-HP-EliteBook-840-G3:~/workspace/BlitzL1$ ipython
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import blitzl1
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-8bb5a22c28e9> in <module>
----> 1 import blitzl1
ModuleNotFoundError: No module named 'blitzl1'
after trial and error, I found that renaming the python folder to blitzl1 and replacing, in setup.py:
package_dir = {"blitzl1": "python"},
by
package_dir = {"blitzl1": "blitzl1"},
makes it possible to import the package. Why is the first one not working?
By the way:
zongo#zongo-HP-EliteBook-840-G3:~/workspace/BlitzL1$ which pip
/home/zongo/anaconda3/bin/pip

This is due to a long lasting issue in pip with installing a package in develop mode when the package directory is not in the same folder as the setup.py. See here for more info.
To be clearer, if the package name is my_package and the structure of the source is:
|- setup.py
|- src
|- __init__.py
|- ...
with package_dir={'my_package':'src'}, installing the package with either pip install -e . or python setup.py develop will raise the error reported by the OP.
A way to mitigate this is to change to package_dir={'':'src'} and change the structure of the repo to
|- setup.py
|- src
|- mypackage
|- __init__.py
|- ...

Install a package using pip does not create all my files

Hello Denizens of the Exchange of Stackness,
I have a library that I'm trying to distribute. I have created a setup.py and run
python setup.py sdist
I see that it creates a .tar.gz file under the dist/ directory, which has all my files and folders that I want in it. However, when I install it on a Windows 8 box (running Python 3.6.5rc1), I don't get any files- only a kivydnd-0.5.dist-info directory. When I install it on Linux (running Fedora 26, Python 2.7), I do see the package's files but I don't see the examples directory.
Can you tell me what I'm doing wrong?
The setup.py is here:
from setuptools import setup, find_packages
setup(
name='kivydnd',
version='0.5',
description='Kivy Drag-n-Drop for Widgets',
url='https://github.com/GreyGnome/KivyDnD',
author='GreyGnome',
author_email='myemail#example.com',
license='Apache License 2.0',
#packages=find_packages('kivydnd'),
packages=['kivydnd'],
zip_safe=False,
scripts=[
'examples/dndexample1.py',
'examples/dndexample2.py',
'examples/dndexample3.py',
'examples/dndexample_copy_draggable.py',
'examples/dndexample_drop_groups.py',
'examples/dndexample_relative_layout.py',
'examples/example_base_classes.py',
'examples/example_base_classes.pyc',
]
)
In my development directory, I perform:
python setup.py sdist
The resulting .tar.gz looks like this; this will also reflect the structure of the directory where I'm doing my development:
kivydnd-0.5/
kivydnd-0.5/setup.py
kivydnd-0.5/PKG-INFO
kivydnd-0.5/examples/
kivydnd-0.5/examples/example_base_classes.pyc
kivydnd-0.5/examples/dndexample1.py
kivydnd-0.5/examples/dndexample_copy_draggable.py
kivydnd-0.5/examples/dndexample3.py
kivydnd-0.5/examples/dndexample_relative_layout.py
kivydnd-0.5/examples/dndexample_drop_groups.py
kivydnd-0.5/examples/dndexample2.py
kivydnd-0.5/examples/example_base_classes.py
kivydnd-0.5/README.md
kivydnd-0.5/RELEASE_NOTES.md
kivydnd-0.5/LICENSE
kivydnd-0.5/kivydnd.egg-info/
kivydnd-0.5/kivydnd.egg-info/top_level.txt
kivydnd-0.5/kivydnd.egg-info/PKG-INFO
kivydnd-0.5/kivydnd.egg-info/not-zip-safe
kivydnd-0.5/kivydnd.egg-info/SOURCES.txt
kivydnd-0.5/kivydnd.egg-info/dependency_links.txt
kivydnd-0.5/setup.cfg
kivydnd-0.5/MANIFEST.in
kivydnd-0.5/kivydnd/
kivydnd-0.5/kivydnd/dnd_storage_singletons.py
kivydnd-0.5/kivydnd/debug_print.py
kivydnd-0.5/kivydnd/__init__.py
kivydnd-0.5/kivydnd/dropdestination.py
kivydnd-0.5/kivydnd/dragndropwidget.py
Here is what happens on Windows 8:
F:\>pip install kivydnd-0.5.tar.gz
Processing f:\kivydnd-0.5.tar.gz
Building wheels for collected packages: kivydnd
Running setup.py bdist_wheel for kivydnd ... done
Stored in directory: C:\Users\schwager\AppData\Local\pip\Cache\wheels\9a\11\cd
\68bfb0d34c7b73ec7e25c6f9c40c5926377747b5951ac2e6ab
Successfully built kivydnd
Installing collected packages: kivydnd
Successfully installed kivydnd-0.5
` c:\users\schwager\python\Lib\site-packages\kivydnd-0.5.dist-info\` I have:
DESCRIPTION.rst
INSTALLER
METADATA
metadata.json
RECORD
top_level.txt
WHEEL
Here is what happens on Linux:
pip install --target=/home/schwager/lib/python kivydnd-0.5.tar.gz
Processing ./kivydnd-0.5.tar.gz
Installing collected packages: kivydnd
Running setup.py install for kivydnd ... done
Successfully installed kivydnd-0.5
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
 $ ls /home/schwager/lib/python
kivydnd kivydnd-0.5-py2.7.egg-info
 $ ls -R /home/schwager/lib/python
/home/schwager/lib/python:
kivydnd kivydnd-0.5-py2.7.egg-info
/home/schwager/lib/python/kivydnd:
debug_print.py dnd_storage_singletons.py dragndropwidget.py dropdestination.py __init__.py
debug_print.pyc dnd_storage_singletons.pyc dragndropwidget.pyc dropdestination.pyc __init__.pyc
/home/schwager/lib/python/kivydnd-0.5-py2.7.egg-info:
dependency_links.txt installed-files.txt not-zip-safe PKG-INFO SOURCES.txt top_level.txt

I appears that my setup.py should look like this. The package will get installed under Python's site-packages directory, the examples under <path-to-share>/kivydnd-examples.
from setuptools import setup, find_packages
from codecs import open
from os import path
with open(path.join('.', 'README.md'), encoding='utf-8') as f:
long_description = f.read()
setup(
name='kivydnd',
version='0.5.0',
description='Kivy Drag-n-Drop for Widgets',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://github.com/GreyGnome/KivyDnD',
author='GreyGnome',
author_email='myemail#example.com',
license='Apache License 2.0',
keywords='kivy drag-n-drop',
packages=find_packages(exclude=[]),
data_files=[('share/kivydnd-examples',
[
'examples/dndexample1.py',
'examples/dndexample2.py',
'examples/dndexample3.py',
'examples/dndexample_copy_draggable.py',
'examples/dndexample_drop_groups.py',
'examples/dndexample_relative_layout.py',
'examples/example_base_classes.py',
'examples/example_base_classes.pyc',
]
)],
)