We Keep Coding

How to load environment variables from file using python-dotenv before loading pytest scritps?

I have tests which on inside docker container with https://github.com/pytest-docker-compose/pytest-docker-compose, but it does take too long for the container to start up/shutdown. Then, I would like to let docker-compose run tests only CI machine or when need.
For this, I used this way of defining tests on simple_test_runner.py:
import os
THIS_FOLDER = os.path.dirname(os.path.realpath(__file__))
RUN_TESTS_LOCALLY = os.environ.get('RUN_TESTS_LOCALLY')
if RUN_TESTS_LOCALLY:
def test_simple(run_process, load_env):
output, returncode = run_process("python3 " + THIS_FOLDER + "/simple_test.py")
assert returncode == 0
else:
def test_simple(function_scoped_container_getter):
container = function_scoped_container_getter.get("container_name")
exec = container.create_exec("python3 /simple_test.py")
...
This works file if I export RUN_TESTS_LOCALLY=1 before calling pytest -vs ., but if I try to use https://github.com/theskumar/python-dotenv in my conftest.py:
from dotenv import load_dotenv
#pytest.fixture(scope='session', autouse=True)
def load_env():
print("Loading .env file")
load_dotenv()
The environment variables are only loaded after python test loaded the simple_test_runner.py. Then, my tests are always running inside docker instead of outside it when RUN_TESTS_LOCALLY=1 is defined.
How can I make pytest to call load_dotenv() before my switch if RUN_TESTS_LOCALLY: is evaluated inside my tests?
Or do you know an alternative to if RUN_TESTS_LOCALLY: which allows me to use load_dotenv() and switch between running my tests inside docker-compose or not?
Related:
https://github.com/quiqua/pytest-dotenv
How to load variables from .env file for pytests
pytest -- how do I use global / session-wide fixtures?

The code I originally posted is working:
#pytest.fixture(scope='session', autouse=True)
def load_env(request):
file = THIS_FOLDER + '/.env'
if request.config.getoption("verbose") > 0:
print("Loading", file, file=sys.stderr)
load_dotenv(dotenv_path=file)
The problem was that when I had tested, I had set my environment variable to RUN_TESTS_LOCALLY= (the empty string). This was causing dotenv to not override my environment variable. Once I unset the variable with bash unset RUN_TESTS_LOCALLY, dotenv was finally loading the environment file correctly.

Import path in pytest

I have Python Flask project with following structure of my project (I removed uneccesary things):
server
src
service
__init__.py
User.py
tests
pytest.ini
service
test_user.py
Where pytest.ini contains:
[pytest]
python_paths = ../src
And test_user.py:
from service.User import UserService
def test_empty_db():
service = UserService()
users = service.get_all() # Get all users from DB.
assert len(users) = 0 # There should be users.
Now, I would like to run this test. When I run pytest or pytest server from root of project, everything is ok. However, when I want to run the specified file pytest server/tests/service/test_user.py error appears:
from services.User import UserService
E ModuleNotFoundError: No module named 'service'
Is there any way to fix it?

You can try to make your imports more absolute inside your test_user.py like: from server.src.service.User import …
Or you can also try adding a __init__.py to your tests directory.

Your error message is showing services.User while you code is showing service.User, could it be a typo?

How to load variables from .env file for pytests

I am writing an API in Flask and in some point I send email to users who register. I store variables concerning this email service in .env file. Now want to test a piece where I use these variables, but I have no idea how to load them from the .env file.
I tried basically all the answers here https://rb.gy/0nro1a, monkey patching setenv as show here https://rb.gy/kd07wa + other tips here and there. Each failed on some point. I also tried using pytest-dotenv. pytest-env, pytest.ini etc..but nothing really worked as expected, and it is all pretty confusing to me.
My pytests fixture looks like this
#pytest.fixture(autouse=True)
def test_client_db():
# set up
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///"
app.config["JWT_SECRET_KEY"] = "testing"
with app.app_context():
db.init_app(app)
db.create_all()
testing_client = app.test_client()
ctx = app.app_context()
ctx.push()
# do testing
yield testing_client
# tear down
with app.app_context():
db.session.remove()
db.drop_all()
ctx.pop()
I am wondering why I cant just simply load the .env file with a line like this load_dotenv(path/to/.env) somewhere in the set up of the fixture and be done?
Can someone explain to me as a newbie how to read the .env variables in a simple straightforward way to work with pytest?
The only way that actually works for me is to pass the environment variables on the command line as I run the tests.
FROM_EMAIL="some#email.com" MAILGUN_DOMAIN="sandbox6420919ab29b4228sdfda9d43ff37f7689072.mailgun.org" MAILGUN_API_KEY="245d6d0asldlasdkjfc380fba7fbskfsj1ad3125649esadbf2-7cd1ac2b-47fb3ac2" pytest tests
But this is a terrible way and I don't want to write all these var into the command line every time I run tests.
I just want to write pytest test, the .env file should be loaded somewhere automatically I believe. But where and how?
Any help appreciated.

If you install python-dotenv, you can use that to load the variables from the .env file. Here is a minimal example:
.env
SQLALCHEMY_DATABASE_URI="sqlite:///"
JWT_SECRET_KEY="testing"
test.py
import os
import pytest
from dotenv import load_dotenv
#pytest.fixture(scope='session', autouse=True)
def load_env():
load_dotenv()
#pytest.fixture(autouse=True)
def test_client_db():
print(f"\nSQLALCHEMY_DATABASE_URI"
f"={os.environ.get('SQLALCHEMY_DATABASE_URI')}")
print(f"JWT_SECRET_KEY={os.environ.get('JWT_SECRET_KEY')}")
def test():
pass
python -m pytest -s test.py gives:
============================================ test session starts ============================================
...
collected 1 item
test.py
SQLALCHEMY_DATABASE_URI=sqlite:///
JWT_SECRET_KEY=testing
.
============================================= 1 passed in 0.27s =============================================
e.g. the enviroment variables are set throughout the test session and can be used to configure your app. Note that I didn't provide a path in load_dotenv(), because I put the .env file in the same directory as the test - in your code, you probably have to add the path (load_dotenv(dotenv_path=your_path)).

You can use the monkeypatch fixture provided by pytest to manipulate env variables:
#pytest.fixture(scope="function")
def configured_env(monkeypatch):
monkeypatch.setenv("SQLALCHEMY_DATABASE_URI", "sqlite:///")
monkeypatch.setenv("JWT_SECRET_KEY", testing)
def test_client(configured_env):
#env variables are set here

How can I get module name automatically from __init__.py or conftest.py?

I am running multiple tests in a tests package, and I want to print each module name in the package, without duplicating code.
So, I wanted to insert some code to __init__.py or conftest.py that will give me the executing module name.
Let's say my test modules are called: checker1, checker2, etc...
My directory structure is like this:
tests_dir/
├── __init__.py
├── conftest.py
├── checker1
├── checker2
└── checker3
So, inside __init__.py I tried inserting:
def module_name():
return os.path.splitext(__file__)[0]
But it still gives me __init__.py from each file when I call it.
I also tried using a fixture inside conftest.py, like:
#pytest.fixture(scope='module')
def module_name(request):
return request.node.name
But it seems as if I still need to define a function inside each module to get module_name as a parameter.
What is the best method of getting this to work?
Edit:
In the end, what I did is explained here:
conftest.py
#pytest.fixture(scope='module', autouse=True)
def module_name(request):
return request.node.name
example for a test file with a test function. The same needs to be added to each file and every function:
checker1.py
from conftest import *
def test_columns(expected_res, actual_res, module_name):
expected_cols = expected_res.columns
actual_cols = actual_res.columns
val = expected_cols.difference(actual_cols) # verify all expected cols are in actual_cols
if not val.empty:
log.error('[{}]: Expected columns are missing: {}'.format(module_name, val.values))
assert val.empty
Notice the module_name fixture I added to the function's parameters.
expected_res and actual_res are pandas Dataframes from excel file.
log is a Logger object from logging package

In each module (checker1, checker2, checker3, conftest.py), in the main function, execute
print(__name__)
When the __init__.py file imports those packages, it should print the module name along with it.
Based on your comment, you can perhaps modify the behaviour in the __init__.py file for local imports.
__init.py__
import sys, os
sys.path.append(os.path.split(__file__)[0])
def my_import(module):
print("Module name is {}".format(module))
exec("import {}".format(module))
testerfn.py
print(__name__)
print("Test")
Directory structure
tests_dir/
├── __init__.py
└── testerfn.py
Command to test
import tests_dir
tests_dir.my_import("testerfn")

Optimal file structure organization of Python module unittests?

Sadly I observed that there are are too many ways to keep your unittest in Python and they are not usually well documented.
I am looking for an "ultimate" structure, one would accomplish most of the below requirements:
be discoverable by test frameworks, including:
pytest
nosetests
tox
the tests should be outside the module files and in another directory than the module itself (maintenance), probably in a tests/ directory at package level.
it should be possible to just execute a test file (the test must be able to know where is the module that is supposed to test)
Please provide a sample test file that does a fake test, specify filename and directory.

Here's the approach I've been using:
Directory structure
# All __init__.py files are empty in this example.
app
package_a
__init__.py
module_a.py
package_b
__init__.py
module_b.py
test
__init__.py
test_app.py
__init__.py
main.py
main.py
# This is the application's front-end.
#
# The import will succeed if Python can find the `app` package, which
# will occur if the parent directory of app/ is in sys.path, either
# because the user is running the script from within that parect directory
# or because the user has included the parent directory in the PYTHONPATH
# environment variable.
from app.package_a.module_a import aaa
print aaa(123, 456)
module_a.py
# We can import a sibling module like this.
from app.package_b.module_b import bbb
def aaa(s, t):
return '{0} {1}'.format(s, bbb(t))
# We can also run module_a.py directly, using Python's -m option, which
# allows you to run a module like a script.
#
# python -m app.package_a.module_a
if __name__ == '__main__':
print aaa(111, 222)
print bbb(333)
module_b.py
def bbb(s):
return s + 1
test_app.py
import unittest
# From the point of view of testing code, our working modules
# are siblings. Imports work accordingly, as seen in module_a.
from app.package_a.module_a import aaa
from app.package_a.module_a import bbb
class TestApp(unittest.TestCase):
def test_aaa(self):
self.assertEqual(aaa(77, 88), '77 89')
def test_bbb(self):
self.assertEqual(bbb(99), 100)
# Simiarly, we can run our test modules directly as scripts using the -m option,
# or using nose.
#
# python -m app.test.test_app
# nosetests app/test/test_app.py
if __name__ == '__main__':
unittest.main()