We Keep Coding

tox does not find numpy

I try to run tox in order to automatically test my project. My tox file is very simple:
[tox]
envlist = py3
[testenv]
deps = pytest
commands =
pytest --doctest-modules
I also have a requirements.txt file that defines the required modules:
cmake
osqp
numpy
cvxpy
networkx
matplotlib
and a setup.py file:
from setuptools import setup, find_packages
...
requirements = ["numpy","cvxpy","networkx","matplotlib"]
...
setup(
name='fairpy',
version='1.0',
description=...,
packages=find_packages(),
install_requires=requirements,
...
)
But when I run tox from the terminal, I get an error saying that the module numpy is not found:
GLOB sdist-make: /mnt/d/Dropbox/ariel/fairpy/setup.py
py3 inst-nodeps: /mnt/d/Dropbox/ariel/fairpy/.tox/.tmp/package/1/fairpy-1.0.zip
py3 installed: attrs==19.3.0,fairpy==1.0,more-itertools==8.1.0,packaging==20.1,pluggy==0.13.1,py==1.8.1,pyparsing==2.4.6,pytest==5.3.4,six==1.14.0,wcwidth==0.1.8
py3 run-test-pre: PYTHONHASHSEED='188600482'
py3 run-test: commands[0] | pytest --doctest-modules
========================================================== test session starts ==========================================================
platform linux -- Python 3.8.1, pytest-5.3.4, py-1.8.1, pluggy-0.13.1
cachedir: .tox/py3/.pytest_cache
rootdir: /mnt/d/Dropbox/ariel/fairpy
collected 0 items / 20 errors
================================================================ ERRORS =================================================================
__________________________________________________ ERROR collecting Deng_Qi_Saberi.py ___________________________________________________
Deng_Qi_Saberi.py:13: in <module>
from agents import *
agents.py:10: in <module>
import numpy as np
E ModuleNotFoundError: No module named 'numpy'
...
I have numpy installed - in my python 3.8.1 terminal, I can import numpy with no problem.
What am I doing wrong with tox?

First,
I have numpy installed - in my python 3.8.1 terminal, I can import numpy with no problem.
tox creates a virtual environment when you run it so it doesn't matter if you have numpy installed on your computer original interpreter.
Second,
If you want tox to install the requirements file you must add
deps = -rrequirements.txt to tox.ini.
And you can always add numpy manually as a dependency there as well.
Third,
tox has some dependencies tracking issues in some cases. Try run tox -r to force tox to recreate its environment and make sure that numpy mentioned in the "install_requires" section in the setup.py.

Module gets imported under a syntactically false name space?

I am following along with the O'Riley Head First Python (2nd Edition) Course.
At one point you will create a webapp and deploy it to pythonanywhere (chapter5).
The webapp uses two functions, imported from a module, created earlier.
The module is called vsearch.py. I also created a readme.txt and a setup.py and used setuptools to create a source distribution file using :
python3 setup.py sdist
The code of the setup.py read as follows:
from setuptools import setup
setup(
name = "vsearch",
version = "1.0",
description = "The Head First Python Seach Tools",
author = "HF Python 2e",
author_email = "hfpy2e#gmail.com",
url = "headfirstlabs.com",
py_modules = ["vsearch"],
)
The source distribution file gets created without errors and creates a file called vsearch-1.0.tar.gz
The file then gets uploaded to pythonanywhere and installed via console using:
python3 -m pip install vsearch-1.0.tar.gz --user
Console outputs:
15:36 ~/mysite $ python3 -m pip install vsearch-1.0.tar.gz --user
Looking in links: /usr/share/pip-wheels
Processing ./vsearch-1.0.tar.gz
Building wheels for collected packages: vsearch
Running setup.py bdist_wheel for vsearch ... done
Stored in directory: /home/Mohr/.cache/pip/wheels/85/fd/4e/5302d6f3b92e4057d341443ed5ef0402eb04994663282c12f7
Successfully built vsearch
Installing collected packages: vsearch
Found existing installation: vsearch 1.0
Uninstalling vsearch-1.0:
Successfully uninstalled vsearch-1.0
Successfully installed vsearch-1.0
Now when I try to run my webapp I get the following error:
2020-03-24 16:18:14,592: Error running WSGI application
2020-03-24 16:18:14,592: ModuleNotFoundError: No module named 'vsearch'
2020-03-24 16:18:14,593: File "/var/www/mohr_eu_pythonanywhere_com_wsgi.py", line 16, in <module>
2020-03-24 16:18:14,593: from vsearch4web import app as application # noqa
2020-03-24 16:18:14,593:
2020-03-24 16:18:14,593: File "/home/Mohr/mysite/vsearch4web.py", line 3, in <module>
2020-03-24 16:18:14,593: from vsearch import search4letters
Judging from this error I assume that "vsearch" can not be found because it was installed as "vsearch-1.0". However when I try to change this line to:
from vsearch-1.0 import search4letters
I rightfully get a synthax error since I can not adress modules this way. So what can I do about this? When creating the module in the beginning I added a version number to the setup.py file because according to the lecture it is good practice. Setuptools then automatically creates the source distribution file with the "-1.0" at the end. Also when importing it using the command shown above i automatically gets importet as "vsearch-1.0" which in turn I am unable to reference in my python code because of bad synthax.
Am I doing something wrong? Is there a way to import this under another namespace? Is there a way to reference "vsearch-1.0" in my python code without getting a synthax error?

There are different python3 versions installed on PythonAnywhere. When you install something using python3 -m pip or pip3 you use default python3 that is probably not matching python version setting of your web app. Use python3.7 and pip3.7 or python3.6 and pip3.6 etc. for --user installations to be sure.

pip install --user (with emphasized --user) installed the package into your user directory: /home/Mohr/.local/lib/pythonX.Y/site-packages/.
To run your WSGI application you probably use a virtual environment in which the user-installed modules are not available. To use modules in the venv you have to install everything in the venv. So activate the venv in a terminal and install the module with the venv's pip:
pip install vsearch-1.0.tar.gz

how to require a specific package version in tox?

In my tox.ini file, the dependencies are installed via the requirements.txt file which is also used by setup.py, as follows:
The requirements.txt file contains the acceptable range of django packages, depending on the python version installed, as follows:
Django>=1.11,<2 ; python_version == '2.7'
Django>=1.11,<3 ; python_version > '3'
For python3, I want to make sure the tests run on django 2.0 as well as the latest django 2.1+ that will be installed by default, obeying the version constraints specified in the requirements.txt file. To achieve that, I force the installation of the desired django version with commands, as follows:
[tox]
envlist = {py27,py3}-django111,py3-django{20,21}
[testenv]
deps =
-r{toxinidir}/requirements.txt
commands =
django111: pip install 'Django>=1.11,<1.12'
py3-django20: pip install 'Django>=2.0,<2.1'
py3-django21: pip install 'Django>=2.1'
pytest
Ideally I could just add to the deps variable like so:
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
But pip does not support double requirements and will throw an error even though there is no conflict in how the version constraints are specified.
The drawback of using commands to override the installation is that it needs to remove the django package version installed via requirements.txt to install the desired one. Is there a way to avoid that extra step?

One trick is to move the requirement from requirements.txt into setup.py - where it's loosely pinned so that all your django versions are possible. For example
# setup.py
from setuptools import setup, find_packages
setup(
...
install_requires=[
"Django>=1.11,<2.1",
]
)
and then use your second suggestion in tox.ini
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
... so long as the Django requirement isn't listed in requirements.txt.
This works because the pip install is split in two parts, the first from tox:deps where you specify the hard requirement, and the second from the equivalent of pip install -e . where the setup.py has the looser requirement.

Python build CFFI in API mode with Setuptools

I'm trying to learn about creating a CFFI modules, and packaging them with setuptools. When I run the build script build_foo.py I get an API mode library, but when I try to pip install . I get an ABI mode library.
Creates API mode
$> python build_foo.py
$> ls
build_foo.py _one_cffi.cpython-36m-x86_64-linux-gnu.so
_one_cffi.c _one_cffi.o
Creates ABI mode
$> pip install .
$> ls env/bin/site-packages
cffi pkg_resources
cffi-1.11.5.dist-info pkg_resources-0.0.0.dist-info
_cffi_backend.cpython-36m-x86_64-linux-gnu.so __pycache__
easy_install.py pycparser
Foo-0.1.dist-info pycparser-2.19.dist-info
foopkg setuptools
_one_cffi.abi3.so setuptools-40.6.2.dist-info
pip wheel
pip-18.1.dist-info wheel-0.32.3.dist-info
Files
build_foo.py
#!/usr/bin/env python3
import cffi
ffi = cffi.FFI()
ffi.cdef("int get_one();")
ffi.set_source("_one_cffi",
"""
int get_one() {
return 1;
}
"""
)
if __name__ == '__main__':
ffi.compile(verbose=True)
setup.py
from setuptools import setup
setup(
name = 'Foo',
version = '0.1',
packages = ['foopkg'],
cffi_modules=["foopkg/build_foo.py:ffi"],
install_requires = ['cffi']
)

I think that the filename ending with abi3.so has to do with Python's PEP 425 abi tag, not the ABI-mode option for how to use CFFI generated bindings. As far as I can tell the cffi_module install is simply failing to add any platform specific information to the shared object file, but I think that Python code that calls foopkg can still access get_one() in API-mode.

setuptools very simple (one source file module) configuration

I want to use setuptools to create a package consisting of two files: foo.py (script) and foo.conf.
Then I want to publish the package on my devpi-server and then install the package using pip.
Suppose I that initially I have my current working directory clean
$ ls -l
total 0
Then I issue pip install (or download?) command
$ pip install -i http://mydevpi.server foo
And get a dir with my two files created
$ tree
.
|
foo
|
|\_ foo.py
|
\_ foo.conf
So questions are:
what setuptools configuration should I use?
what exact pip command should I use to install the package the way I want? Will pip install -i http://mydevpi.server --target=. do the trick?

First write somethings as setup.py in foo directory like:
import setuptools
setuptools.setup(
name='foo_pip',
version='1',
packages=[''],
url='1',
license='1',
author='1',
author_email='1',
description='1'
)
(You can use distutils or setuptools)
Then python setup.py bdist_wheel -d TARGET and there will be a whl file in target directory, copy the path.
You can now install using pip install the_wheel_file_path --prefix="the_path_to_install"
Something like this
Processing .../TARGET/foo_pip-1-py2-none-any.whl
Installing collected packages: foo-pip
Successfully installed foo-pip-1
Then use it by import foo