In my Python project, I'm reading environment variables from a .env file. I am actually using pydantic to read/verify the env vars.
When using tox, the .env file will be completely ignored. I am wondering how to make tox acknowledging the existence of .env?
Here's my tox.ini
[tox]
envlist = py39
[testenv]
deps = -r requirements-dev.txt
commands = pytest {posargs}
My .env file:
ENV_STATE="prod" # dev or prod
At first, I thought maybe pydantic loads the content of the .env file as environment variables, that is why I wrote this as my first answer:
original answer
tox does some isolation work, so your builds / tests are more reproducible.
This means that e.g. environment variables are filtered out, except you whitelist them.
You probably need to set
passenv = YOUR_ENVIRONMENT_VARIABLE
Also see in the tox documentation.
updated answer
This does not seem to be a tox issue at all.
I just created a simple project with pydantic and dotenv, and it works like a charm with tox.
tox.ini
[tox]
envlist = py39
skipsdist = True
[testenv]
deps = pydantic[dotenv]
commands = pytest {posargs}
.env
ENVIRONMENT="production"
main.py
from pydantic import BaseSettings
class Settings(BaseSettings):
environment: str
class Config:
env_file = ".env"
env_file_encoding = "utf-8"
test_main.py
from main import Settings
def test_settings():
settings = Settings(_env_file=".env")
assert settings.environment == "production"
Related
This is my tox.ini file:
# Tox (https://tox.readthedocs.io/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
#
# See also https://tox.readthedocs.io/en/latest/config.html for more
# configuration options.
[tox]
# Choose your Python versions. They have to be available
# on the system the tests are run on.
# skipsdist=True
ignore_basepython_conflict=false
[testenv:{setup,lint,codestyle,docstyle,tests,doc-linux,doc-darwin,doc-win32}]
basepython=python3.9
envdir = {toxworkdir}/py39
setenv =
PROJECT_NAME = project_name
passenv =
WINDIR
install_command=
pip install \
--find-links=pkg \
--trusted-host=pypi.python.org \
--trusted-host=pypi.org \
--trusted-host=files.pythonhosted.org \
{opts} {packages}
platform = doc-linux: linux
doc-darwin: darwin
doc-win32: win32
deps =
-r{toxinidir}/requirements-dev.txt
-r{toxinidir}/requirements.txt
commands =
setup: python -c "print('All SetUp')"
# Mind the gap, use a backslash :)
lint: pylint -f parseable -r n --disable duplicate-code \
lint: --extension-pkg-whitelist=PyQt5,numpy,torch,cv2,boto3 \
lint: --ignored-modules=PyQt5,numpy,torch,cv2,boto3 \
lint: --ignored-classes=PyQt5,numpy,torch,cv2,boto3 \
lint: project_name \
lint: {toxinidir}/script
lint: pylint -f parseable -r n --disable duplicate-code \
lint: demo/demo_file.py
codestyle: pycodestyle --max-line-length=100 \
codestyle: --exclude=project_name/third_party/* \
codestyle: project_name demo script
docstyle: pydocstyle \
docstyle: --match-dir='^((?!(third_party|deprecated)).)*' \
docstyle: project_name demo script
doc-linux: make -C {toxinidir}/doc html
doc-darwin: make -C {toxinidir}/doc html
doc-win32: {toxinidir}/doc/make.bat html
tests: python -m pytest -v -s --cov-report xml --durations=10 \
tests: --cov=project_name --cov=script \
tests: {toxinidir}/test
tests: coverage report -m --fail-under 100
On tox<4.0 it was very convinient to run tox -e lint to fix linting stuff or tox -e codestyle tox fix codestyle stuff, etc. But now, with version tox>4.0 each time I run one of these commands I get this message (for instance):
codestyle: recreate env because env type changed from {'name': 'lint', 'type': 'VirtualEnvRunner'} to {'name': 'codestyle', 'type': 'VirtualEnvRunner'}
codestyle: remove tox env folder .tox/py39
And it takes forever to run these commands since the evironments are recreated each time ...
I also use these structure for running tests on jenkins so I can map each of these commands to a jenkins stage.
How can I reuse the environment? I have read that it is possible to do it using plugins, but no idea how this can be done, or how to install/use plugins.
I have tried this:
tox multiple tests, re-using tox environment
But it does not work in my case.
I spect to reuse the environment for each of the environments defined in the tox file.
As an addition to N1ngu's excellent answer...
You could re-structure your tox.ini as following:
[tox]
...
[testenv]
<here goes all the common configuration>
[testenv:lint]
<here goes the lint specific configuration>
[testenv:codestyle]
...
And so on. This is a common setup.
While still the environments need to be created at least once, they won't get recreated on each invocation.
This all said, you could also have a look at https://pre-commit.com/ to run your linters, which is very common in the Python community.
Then you would have a tox.ini like the following...
[tox]
...
[testenv]
<here goes all the common configuration>
[testenv:lint]
deps = pre-commit
commands = pre-commit run --all-files
There is now a definite answer about re-use of environments in the faq:
https://tox.wiki/en/latest/upgrading.html#re-use-of-environments
I fear the generative names + factor-specific commands solution you linked relied on tox-3 not auto-recreating the environments by default, which is among the new features in tox 4. Now, environment recreation is something that can be forced (--recreate) but can't be opted-out.
Official answer on this https://github.com/tox-dev/tox/issues/425 boils down to
Officially we don't allow sharing tox environments at the moment [...] As of today, each tox environment has to have it's own virtualenv even if the Python version and dependencies are identical [...] We'll not plan to support this. However, tox 4 allows one to do this via a plugin, so we'd encourage people [...] to try it [...]. Once the project is stable and widely used we can revisit accepting it in core.
So that's it, write a plugin. No idea on how to do that either, so my apologies if this turns out as "not an answer"
I'm developing a simple FastAPI app and I'm using Pydantic for storing app settings.
Some settings are populated from the environment variables set by Ansible deployment tools but some other settings are needed to be set explicitly from a separate env file.
So I have this in config.py
class Settings(BaseSettings):
# Project wide settings
PROJECT_MODE: str = getenv("PROJECT_MODE", "sandbox")
VERSION: str
class Config:
env_file = "config.txt"
And I have this config.txt
VERSION="0.0.1"
So project_mode env var is being set by deployment script and version is being set from the env file. The reason for that is that we'd like to keep deployment script similar across all projects, so any custom vars are populated from the project specific env files.
But the problem is that when I run the app, it fails with:
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
VERSION
field required (type=value_error.missing)
So how can I populate Pydantic settings model from the local ENV file?
If the environment file isn't being picked up, most of the time it's because it isn't placed in the current working directory. In your application in needs to be in the directory where the application is run from (or if the application manages the CWD itself, to where it expects to find it).
In particular when running tests this can be a bit confusing, and you might have to configure your IDE to run tests with the CWD set to the project root if you run the tests from your IDE.
The path of env_file is relative to the current working directory, which confused me as well. In order to always use a path relative to the config module I set it up like this:
env_file: f"{pathlib.Path(__file__).resolve().parent}/config.txt"
This is the path to the project
D:\QA\test-framework\python-client
This is a test frame work implemented by python
This is the python file that contains tests
This is the path to the test case that I need to run
D:\QA\test-framework\python-client\test_data\tests\curve.json
This is the beginning of the curve.json file.
{
"Sklearn - Sklearn - Regression - Curve M2" : [
{
"dataImport": {
"
"
"
]
}
This is the tox.ini file
[tox]
envlist = py38
[testenv]
deps =
pytest
pytest-html
pytest-sugar
pytest-logger
allure-pytest
pytest-xdist
pytest_steps
datetime
oauth2client
gspread
aiclub
commands =
pytest -s -v -k _workflow --html=test_report.html --alluredir=allure-
results/ -n auto --dist=loadfile
allure serve allure-results
pytest {posargs}
I need to run only this curve.json using tox command
I want to run my test with different values of environment variables. I have this tox.ini, which doesn't do what I want:
# tox.ini
[tox]
envlist = py37-{foo,bar}
[testenv]
description = Tests common
setenv =
MY_VAR=COMMON
commands =
env
[testenv:foo]
description = Tests foo
setenv =
MY_VAR=FOO
[testenv:bar]
description = Tests bar
setenv =
MY_VAR=BAR
Above ini produced the following output:
$ tox
GLOB sdist-make:
***
py37-foo run-test: commands[0] | env
***
MY_VAR=COMMON <<<--- MY_VAR=foo is expected
***
py37-bar run-test: commands[0] | env
***
MY_VAR=COMMON <<<--- MY_VAR=bar is expected
Whats wrong?
I use:
Win 10.0.18363 Build 18363
Python 3.7.4
tox: 3.14.0
tox doesn't combine environments. Every environment is derived from [testenv] but other environments are not considered for inclusion. You have to combine them yourself in tox.ini. This should work:
[tox]
envlist = py37-{foo,bar}
[testenv]
description = Tests common
setenv =
MY_VAR=COMMON
commands =
env
[foo]
description = Tests foo
setenv =
MY_VAR=FOO
[bar]
description = Tests bar
setenv =
MY_VAR=BAR
[testenv:py37-foo]
description = Tests py37 foo
setenv = {[foo]setenv}
[testenv:py37-bar]
description = Tests py37 bar
setenv = {[bar]setenv}
The key is the Compressing dependency matrix. This technique results compact, and non-redundant solution:
[tox]
envlist = py37-{foo,bar,baz}
[testenv]
setenv =
MY_VAR=COMMON
foo: MY_VAR=FOO
bar: MY_VAR=BAR
commands =
env
e.g. Let's say I have following in tox.ini
[tox]
envlist = py27, py35, testenv2
[testenv]
# settings related to "default" testenv - includes deps, commands
[testenv:testenv2]
# settings related to testenv2 - includes deps, commands
Now when I run tox command, it invokes testenv commands with python 2.7 and 3.5 interpreter but testenv2 commands only with base python installed on the machine (in my case 2.7). How to get tox to also test a "named" (non-default) test environment like testenv2 to be tested with multiple python versions?
The first answer describes two viable ways. For completeness: you can also generate environments and use conditional settings - e.g.:
[tox]
skipsdist = True
envlist = py{27,35}-{test,lint}
[testenv]
skip_install = True
deps =
test: pytest
test: pytest-xprocess
lint: flake8
lint: black
commands =
test: pytest -v
lint: flake8
lint: black .
would generate (tox -a):
py27-test
py27-lint
py35-test
py35-lint
You can use any factor (e.g. py27 or test) to conditionally add commands, deps, etc.
See also the docs.
BTW: to see which settings each testenv exactly has you can run tox --showconfig.
List all environments explicitly:
[tox]
envlist = py27, py35, py27-testenv2, py35-testenv2
If that's not enough refactor tox.ini:
[testenv]
# settings related to "default" testenv - includes deps, commands
[testenv2]
# settings related to testenv2 - includes deps, commands
[testenv:py27-testenv2]
deps = {[testenv2]deps}
commands = {[testenv2]commands}
[testenv:py35-testenv2]
deps = {[testenv2]deps}
commands = {[testenv2]commands}
I could get the "named" test environment to be tested with multiple python versions by making multiple "named" test environments, one each for the different python versions I wanted it to be tested with and using the basepython option to specify the python version for the test environment to be tested with. Below is the example that works:
[tox]
envlist = py27, py35, testenv2_py27, testenv2_py35
[testenv]
# settings related to "default" testenv - includes deps, commands
[testenv:testenv2_py27]
basepython = python2.7
# settings related to testenv2 - includes deps, commands
[testenv:testenv2_py35]
basepython = python3.5
# settings related to testenv2 - includes deps, commands
Just in case someone still needs this, I figured out an even more concise solution. It combines all the good suggestions from the other answers:
[tox]
envlist = py{27,35}, testenv2-py{27,35}
[testenv]
# settings related to "default" testenv - includes deps, commands
[testenv:testenv2-py{27,35}]
# settings related to testenv2 - includes deps, commands