We Keep Coding

pip install -e . vs setup.py

I have been locally editing (inside a conda env) the package GSTools cloned from the github repo https://github.com/GeoStat-Framework/GSTools, to adapt it to my own purposes. The package is c++ wrapped in python (cython).
I've thus far used pip install -e . in the main package dir for my local changes. But I want to now use their OpenMP support by setting the env variable export GSTOOLS_BUILD_PARALLEL=1 . Then doing pip install -e . I get among other things in the terminal ...
Installing collected packages: gstools
Running setup.py develop for gstools
Successfully installed gstools-1.3.6.dev37
The issue is nothing actually changed because, setup.py (shown below) is supposed to print "OpenMP=True" if the env variable is set to GSTOOLS_BUILD_PARALLEL=1 in the linux terminal , and print something else if its not set to 1.
here is setup.py.
# -*- coding: utf-8 -*-
"""GSTools: A geostatistical toolbox."""
import os
​
import numpy as np
from Cython.Build import cythonize
from extension_helpers import add_openmp_flags_if_available
from setuptools import Extension, setup
​
# cython extensions
CY_MODULES = [
Extension(
name=f"gstools.{ext}",
sources=[os.path.join("src", "gstools", *ext.split(".")) + ".pyx"],
include_dirs=[np.get_include()],
define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
)
for ext in ["field.summator", "variogram.estimator", "krige.krigesum"]
]
# you can set GSTOOLS_BUILD_PARALLEL=0 or GSTOOLS_BUILD_PARALLEL=1
if int(os.getenv("GSTOOLS_BUILD_PARALLEL", "0")):
added = [add_openmp_flags_if_available(mod) for mod in CY_MODULES]
print(f"## GSTools setup: OpenMP used: {any(added)}")
else:
print("## GSTools setup: OpenMP not wanted by the user.")
​
# setup - do not include package data to ignore .pyx files in wheels
setup(ext_modules=cythonize(CY_MODULES), include_package_data=False)
I tried instead just python setup.py install but that gives
UNKNOWN 0.0.0 is already the active version in easy-install.pth
Installed /global/u1/b/benabou/.conda/envs/healpy_conda_gstools_dev/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8-linux-x86_64.egg
Processing dependencies for UNKNOWN==0.0.0
Finished processing dependencies for UNKNOWN==0.0.0
and import gstools
no longer works correctly.
So how can I install my edited version of the package with OpenMP support?

developer of GSTools here.
I guess you don't see the printed message, because pip is suppressing output for the setup. So you could try making pip verbose with:
GSTOOLS_BUILD_PARALLEL=1 pip install -v -e .
BTW, we are always interested in enhancements. So maybe you are willing the share your edits on GSTools? :-)
Cheers,
Sebastian

In tox testenv "ModuleNotFoundError: No module named 'pandas" error comes whenever pandas is added to deps

After I implemented my test, which is using pandas, my build is failing with "ModuleNotFoundError: No module named 'pandas'" error, however, I added pandas to the testenv deps and in the log file I also see that it is installed. I got the same error in case of boto3 but after I added it to the deps, it solved the problem but in case of pandas it doesn't work.
tox.ini
[tox]
envlist=flake8
py36_tests
[testenv]
deps=pytest
flake8
[testenv:py36_tests]
basepython=python3.6
deps=boto3
pandas
commands=py.test -s -v tests --junitxml=report.xml
[testenv:flake8]
commands=flake8 --exclude=.git,__pycache__,__init__.py data_collector/
flake8 tests/
flake8 setup.py
flake8 setup-cy.py
flake8 Docker/startup_scripts/
flake8 bin/data_collector
Log:
$ tox -e py36_tests
GLOB sdist-make: /builds/<path>/<my_package>/setup.py
py36_tests create: /builds/<path>/<my_package>/.tox/py36_tests
py36_tests installdeps: boto3, pandas
py36_tests inst: /builds/<path>/.tox/.tmp/package/1/<my_package>-0.0.0.zip
py36_tests installed: <many packages>, pandas==1.1.1, <many packages>
y36_tests run-test-pre: PYTHONHASHSEED='3745093701'
py36_tests run-test: commands[0] | py.test -s -v tests --junitxml=report.xml
WARNING: test command found but not installed in testenv
cmd: /usr/local/bin/py.test
env: /builds/<path>/<my_package>/.tox/py36_tests
Maybe you forgot to specify a dependency? See also the allowlist_externals envconfig setting.
DEPRECATION WARNING: this will be an error in tox 4 and above!
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.6
cachedir: .tox/py36_tests/.pytest_cache
rootdir: /builds/<path>/<my_package>
collecting ... collected 10 items / 1 error / 9 selected
==================================== ERRORS ====================================
________________ ERROR collecting tests/ingestion/test_task.py _________________
ImportError while importing test module '/builds/<path>/tests/ingestion/test_task.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.6/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/ingestion/test_task.py:3: in <module>
import pandas as pd
E ModuleNotFoundError: No module named 'pandas'
- generated xml file: /builds/<path>/<my_package>/report.xml -
=========================== short test summary info ============================
ERROR tests/ingestion/test_task.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.24s ===============================
ERROR: InvocationError for command /usr/local/bin/py.test -s -v tests --junitxml=report.xml (exited with code 2)
___________________________________ summary ____________________________________
ERROR: py36_tests: commands failed
While I hadn't got any import in my test it worked well. What is the problem?

See this
WARNING: test command found but not installed in testenv
cmd: /usr/local/bin/py.test
You have not installed py.test into the virtual environment so tox found a global one. And of course the global one runs with a global Python and doesn't know anything about your virtual environments in tox.
To fix it install pytest into the virtual environment:
[testenv:py36_tests]
deps=boto3
pandas
pytest

I used a virtual env after the tox call, the solution was to move the tox call when the virtual env was already activated.

install local package in tox unencumbered by previous install from GitHub

Question
How to use tox to install a the local package without it refering to a random time when I installed the pacakge from GitHub?
Context
In this PR, I'm trying to build and test my package, dbt-synapse, before publishing a new minor version to PyPI. At some point (rookie move), I think I tried installing the package directly from a specific GitHub commit with the below line.
pip install -e git+https://github.com/dbt-msft/dbt-synapse.git#345d7cafcb08bac25d23867a2e22c0e9b741f603#egg=dbt_synapse
Here's a gist of the requirements.txt, tox.ini, setup.py and some other files
When I call tox -e integration-synapse to kick off the tests, having listing the actual package in the deps as -e. compared to just ., causes the package to be installed from GitHub (see the integration-synapse installed: line below, shortened for clarity)!
when deps has just ., however it lists this as having been installed (which also doesn't seem right): dbt-synapse # file:///Users/anders.swanson/repos/dbt-synapse.
Unfortunately, neither are giving me what I know should be happening.
integration-synapse recreate: /Users/anders.swanson/repos/dbt-synapse/.tox/integration-synapse
integration-synapse installdeps: -rrequirements.txt, -e.
integration-synapse installed: [...],-e git+https://github.com/dbt-msft/dbt-synapse.git#345d7cafcb08bac25d23867a2e22c0e9b741f603#egg=dbt_synapse,[...]
integration-synapse runtests: PYTHONHASHSEED='1932234374'
integration-synapse runtests: commands[0] | /bin/bash -c '/Users/anders.swanson/repos/dbt-synapse/.tox/integration-synapse/bin/python -m pytest -v test/integration/synapse.dbtspec'
======================================================================================================================================== test session starts =========================================================================================================================================
platform darwin -- Python 3.6.12, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/anders.swanson/repos/dbt-synapse/.tox/integration-synapse/bin/python
cachedir: .pytest_cache
rootdir: /Users/anders.swanson/repos/dbt-synapse
plugins: dbt-adapter-0.4.0
collected 1 item
tox.ini
[tox]
skipsdist = True
envlist = integration-synapse
[testenv:integration-synapse]
basepython = python3
commands = /bin/bash -c '{envpython} -m pytest -v test/integration/synapse.dbtspec'
passenv = DBT_SYNAPSE_DB DBT_SYNAPSE_PORT DBT_SYNAPSE_PWD DBT_SYNAPSE_SERVER DBT_SYNAPSE_UID
deps =
-rrequirements.txt
-e.

If you want to start from a clean state, you can you can delete the hidden .tox folder and start from scratch.
Also, you can run tox -r which will recreate the environemnts.
In order to test the code from your local development environment, you do not have to include your package in the deps list (-e.). tox will build the package from your setup.py.
Well.. it would. But in your tox.ini you specified skipsdist = True which afaik means tox should not build the package.
There are some more issues in your tox.ini, e.g. unit and flake8 envs do nothing.
Why do you use the /bin/bash -c construct?
Unfortunately, I am missing some header files, so I cannot run your tox.ini file, but I would create a tox.ini file like this...
[tox]
envlist = py36,py37,py38,py39,integration-synapse,flake8
[testenv]
deps = whatever needed for your unit tests
commands = python -m pytest ...unit tests...
[testenv:integration-synapse]
deps =
pytest
pytest-dbt-adapter
commands =
python -m pytest -v test/integration/synapse.dbtspec
passenv = DBT_SYNAPSE_DB DBT_SYNAPSE_PORT DBT_SYNAPSE_PWD DBT_SYNAPSE_SERVER DBT_SYNAPSE_UID
[testenv:flake8]
skipsdist = True
deps = flake8
commands = flake8 setup.py ... (and more)

mognoengine and bson package not work together

I have one project which is depend on mongoengine and I am using one library which was using bson library.
It structure like below.
bsNotify/
├── setup.cfg
├── setup.py
└── src
└── bsnotify
├── __init__.py
└── resources.py
setup.py
$ cat bsNotify/setup.py
"""Base module setup."""
from setuptools import setup
setup(
setup_requires=['pbr'],
pbr=True
)
setup.cfg
$ cat bsNotify/setup.cfg
[metadata]
name = bsNotify
classifiers =
License :: N/A :: N/A
Programming Language :: Python :: 3.7
[options]
zip_safe = False
include_package_data = True
python_requires = >= 3.7
install_requires =
mongoengine
bson
package_dir=
=src
packages=find:
[options.packages.find]
where=src
[tool:wheel]
universal = 1
[flake8]
exclude =
venv,
.tox,
.git,
__pycache__,
*.pyc,
*.egg-info,
.cache,
.eggs,
max-line-length = 80
[tox]
envlist = py37,unittest,lint
[testenv]
basepython=python3.7
deps =
ipython
pylint
pytest
pytest-cov
pytest-xdist
flake8
flake8-docstrings
[testenv:unittest]
commands=
pytest -v -s -n auto -l --cov=bsnotify --cov-report term-missing --cov-report xml --no-cov-on-fail tests/unit
[testenv:lint]
commands=
flake8 src/bsnotify
pylint src/bsnotify
src/bsnotify/__init__.py
$ cat bsNotify/src/bsnotify/__init__.py
src/bsnotify/resources.py
$ cat bsNotify/src/bsnotify/resources.py
Files in src are empty.
When I run the tox command, it create a virtualenv for testing.
$ tox -v -e unittest --notest
using tox.ini: /Users/myuser/bsNotify/setup.cfg (pid 30954)
using tox-3.14.6 from /Users/myuser/.pyenv/versions/3.7.4/lib/python3.7/site-packages/tox/__init__.py (pid 30954)
GLOB sdist-make: /Users/myuser/bsNotify/setup.py
[30956] /Users/myuser/bsNotify$ /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 setup.py sdist --formats=zip --dist-dir /Users/myuser/bsNotify/.tox/dist >.tox/log/GLOB-0.log
package .tmp/package/1/bsNotify-0.1.2.dev2.zip links to dist/bsNotify-0.1.2.dev2.zip (/Users/myuser/bsNotify/.tox)
unittest cannot reuse: no previous config /Users/myuser/bsNotify/.tox/unittest/.tox-config1
unittest create: /Users/myuser/bsNotify/.tox/unittest
[30992] /Users/myuser/bsNotify/.tox$ /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 -m virtualenv --no-download --python /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 unittest >unittest/log/unittest-0.log
unittest installdeps: ipython, pylint, pytest, pytest-cov, pytest-xdist, flake8, flake8-docstrings
[30993] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip install ipython pylint pytest pytest-cov pytest-xdist flake8 flake8-docstrings >.tox/unittest/log/unittest-1.log
unittest inst: /Users/myuser/bsNotify/.tox/.tmp/package/1/bsNotify-0.1.2.dev2.zip
write config to /Users/myuser/bsNotify/.tox/unittest/.tox-config1 as '7186e9f46c94b6d9f7dde810ce83f8fe46740d9d42f2863cfd063c4c6f4e4a88 /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7\n3.14.6 0 0 0\n00000000000000000000000000000000 ipython\n00000000000000000000000000000000 pylint\n00000000000000000000000000000000 pytest\n00000000000000000000000000000000 pytest-cov\n00000000000000000000000000000000 pytest-xdist\n00000000000000000000000000000000 flake8\n00000000000000000000000000000000 flake8-docstrings'
[31000] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip install --exists-action w .tox/.tmp/package/1/bsNotify-0.1.2.dev2.zip >.tox/unittest/log/unittest-2.log
[31030] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip freeze >.tox/unittest/log/unittest-3.log
unittest installed: apipkg==1.5,appnope==0.1.0,astroid==2.4.2,attrs==19.3.0,backcall==0.1.0,bsNotify==0.1.2.dev2,bson==0.5.10,coverage==5.1,decorator==4.4.2,execnet==1.7.1,flake8==3.8.3,flake8-docstrings==1.5.0,importlib-metadata==1.6.1,ipython==7.15.0,ipython-genutils==0.2.0,isort==4.3.21,jedi==0.17.0,lazy-object-proxy==1.4.3,mccabe==0.6.1,mongoengine==0.20.0,more-itertools==8.3.0,packaging==20.4,parso==0.7.0,pexpect==4.8.0,pickleshare==0.7.5,pluggy==0.13.1,prompt-toolkit==3.0.5,ptyprocess==0.6.0,py==1.8.1,pycodestyle==2.6.0,pydocstyle==5.0.2,pyflakes==2.2.0,Pygments==2.6.1,pylint==2.5.3,pymongo==3.10.1,pyparsing==2.4.7,pytest==5.4.3,pytest-cov==2.9.0,pytest-forked==1.1.3,pytest-xdist==1.32.0,python-dateutil==2.8.1,six==1.15.0,snowballstemmer==2.0.0,toml==0.10.1,traitlets==4.3.3,typed-ast==1.4.1,wcwidth==0.2.4,wrapt==1.12.1,zipp==3.1.0
_______________________________________________________ summary _______________________________________________________
unittest: skipped tests
congratulations :)
Then I try to access mongoengine in newly created virtualenv, it gives error.
$ .tox/unittest/bin/python -c "from mongoengine import connection"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/mongoengine/__init__.py", line 2, in <module>
from mongoengine import connection
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/mongoengine/connection.py", line 1, in <module>
from pymongo import MongoClient, ReadPreference, uri_parser
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/pymongo/__init__.py", line 77, in <module>
from pymongo.collection import ReturnDocument
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/pymongo/collection.py", line 20, in <module>
from bson.code import Code
File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/bson/code.py", line 18, in <module>
from bson.py3compat import abc, string_type, PY3, text_type
ImportError: cannot import name 'abc' from 'bson.py3compat' (/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/bson/py3compat.py)
When I was using mongoengine alone with bson, it works fine.
$ python -m venv .venv
$ .venv/bin/pip install mongoengine
Collecting mongoengine
Using cached https://files.pythonhosted.org/packages/7d/bd/9a7239b0032157f948c69febdf71dd82cb54fcd2499077300496a3f076c9/mongoengine-0.20.0-py3-none-any.whl
Collecting pymongo<4.0,>=3.4 (from mongoengine)
Using cached https://files.pythonhosted.org/packages/23/cd/27fbc08f0bd835b4735504a758756e979b42c5bc9ebaac5ed3c2cbffd83f/pymongo-3.10.1-cp37-cp37m-macosx_10_9_x86_64.whl
Installing collected packages: pymongo, mongoengine
Successfully installed mongoengine-0.20.0 pymongo-3.10.1
$ .venv/bin/python -c "from mongoengine import connection"
$ .venv/bin/pip install bson
Collecting bson
Using cached https://files.pythonhosted.org/packages/4d/53/7c534a38850f2252275d7f949aed2219095e90df1e2d180a9c8ed139e499/bson-0.5.10.tar.gz
Collecting python-dateutil>=2.4.0 (from bson)
Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting six>=1.9.0 (from bson)
Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, bson
Running setup.py install for bson ... done
Successfully installed bson-0.5.10 python-dateutil-2.8.1 six-1.15.0
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ .venv/bin/python -c "from mongoengine import connection"
It check the pip freeze between both the virtualenv. Only difference I found is in .tox env has bsNotify==0.1.2.dev2 installed which is not exists in .venv env.
What make bson change the file py3compat.py?

change this
install_requires =
mongoengine
bson
to this
install_requires =
mongoengine
pymongo
or remove altogether (mongoengine already requires pymongo
install_requires =
mongoengine
requirements=bson is the same things as
pip install bson
which installs this 3rd party package which does not include all the goodies found in MongoDB's package
https://pypi.org/project/bson/
pymongo (official MongoDB driver) contains a bson package. I'll drop both a pymongo folder and a bson folder into your site-packages.
bson (third party bson implementation) also wants to drop a bson folder into your site packages.
Installing pymongo and installing bson will conflict folders inside your site-packages. PyMongo only knows how to use it's own implementation, which is why you're seeing pymongo looking for py3compat which is not part of the 3rd party bson package.
Based on your install order, it would have installed mongo engine (and pymongo as a dependency) then installed bson (overwriting MongoDB's bson implementation packaged with pymongo). Pymongo called py3compat expecting it to be there, but 3td party did not implement that (as well as several other things).
pip install pymongo
https://pypi.org/project/pymongo/
As noted on the pymongo pypi page
Do not install the “bson” package from pypi. PyMongo comes with its
own bson package; doing “easy_install bson” installs a third-party
package that is incompatible with PyMongo.

pytest before 4.6 was able to import from a local package with the same name as an installed namespace package, and now it isn't. Is that a bug?

I clone this toy repository that demonstrates how namespace packages work:
C:\workspace>git clone https://github.com/pypa/sample-namespace-packages.git
Specifically, I'll use its pkg_resources directory, which has the structure
pkg_a/
setup.py
example_pkg/
__init__.py
a/
__init__.py
pkg_b/
setup.py
example_pkg/
__init__.py
b/
__init__.py
The example_pkg package is a pkg_resources-style namespace package (explained here).
I set up my python environment:
C:\workspace>\Python35\python.exe -m venv localpython
C:\workspace>localpython\Scripts\activate.bat
(localpython) C:\workspace>python -m pip install --upgrade pip setuptools
I install pkg_a from the toy repository:
(localpython) C:\workspace>python -m pip install c:\workspace\sample-namespace-packages\pkg_resources\pkg_a
I put pkg_b from the toy repository on my PYTHONPATH:
(localpython) C:\workspace>set PYTHONPATH=c:\workspace\sample-namespace-packages\pkg_resources\pkg_b
I write a test suite for pkg_b, consisting of one line:
(localpython) C:\workspace>echo import example_pkg.b > test_b.py
Now, if I run that test suite in pytest 4.5 or less, it succeeds:
(localpython) C:\workspace>python -m pip install pytest==4.5.0
Collecting pytest==4.5.0
...
(localpython) C:\workspace>pytest test_b.py
================================================= test session starts =================================================
platform win32 -- Python 3.5.2, pytest-4.5.0, py-1.8.0, pluggy-0.12.0
rootdir: C:\workspace
collected 0 items
============================================ no tests ran in 0.02 seconds =============================================
But if I run it in pytest 4.6 or greater, it errors:
(localpython) C:\workspace>python -m pip install pytest==4.6.0
Collecting pytest==4.6.0
...
(localpython) C:\workspace>pytest test_b.py
================================================= test session starts =================================================
platform win32 -- Python 3.5.2, pytest-4.6.0, py-1.8.0, pluggy-0.12.0
rootdir: C:\workspace
collected 0 items / 1 errors
======================================================= ERRORS ========================================================
_____________________________________________ ERROR collecting test_b.py ______________________________________________
ImportError while importing test module 'C:\workspace\test_b.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_b.py:1: in <module>
import example_pkg.b
E ImportError: No module named 'example_pkg.b'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================== 1 error in 0.08 seconds ===============================================
The import error is understandable: It's presumably looking in the example_pkg in the site-packages and therefore not finding the b package, which is under the example_pkg on the PYTHONPATH.
Yet pytest 4.5 or earlier manages to find example_pkg.b.
Edit: According to a comment, the relevant difference is that pytest 4.5 and earlier imported pkg_resources. Indeed, if I add the line
import pkg_resources
at the top of my test file, test_b.py, then the test succeeds even in pytest 4.6 or greater. Furthermore, if I have multiple test files that try to import example_pkg.b, then it's necessary and sufficient to import pkg_resources in whichever of them pytest happens to run first.
That means that some side effect of importing pkg_resources makes example_pkg.b importable. What exactly does pkg_resources do that achieves that effect, and can I do it directly instead of getting it as a side effect of a seemingly unused import? And does my needing that side effect mean the setup is invalid?

Import your packages through sys as written in this project (using unittest not pytest)
import sys
sys.path.insert(0,"pkg_a")
sys.path.insert(0,"pkg_b")
#Test goes here
My project load test/testSpecie.py which tests files in src/...