Mock global variable, which connects to database in Python - python

I am preparing unittests for Airflow DAGs, and I have ran into below problem.
I have a module with common functions, which imports airflow variables into common global variable:
"""
Script for common functions used in airflow DAGs
"""
[...]
from airflow.models import Variable
[...]
config = Variable.get("some_dag_details")
[...]
I have code that import DAG files for test validation.
import unittest
from unittest.mock import patch
from airflow.models import DagBag
from some_package import common_functions
class DAGValidationTest(unittest.TestCase):
DETAILS = {
"some important data"
}
def setUp(self):
with patch.object(common_functions.Variable, 'get', return_value=self.DETAILS) as mock_get_variable:
[...]
Problem is that code for connecting to database is called before I am able to mock class Variable or variable config. Python is showing 4th line (from some_package import common_functions) as the error source. How can I mock this object before calling this script ?

Related

How to override "env_file" during tests?

I'm reading env variables from .prod.env file in my config.py:
from pydantic import BaseSettings
class Settings(BaseSettings):
A: int
class Config:
env_file = ".prod.env"
env_file_encoding = "utf-8"
settings = Settings()
in my main.py I'm creating the app like so:
from fastapi import FastAPI
from app.config import settings
app = FastAPI()
print(settings.A)
I am able to override settings variables like this in my conftest.py:
import pytest
from fastapi.testclient import TestClient
from app.main import app
from app.config import settings
settings.A = 42
#pytest.fixture(scope="module")
def test_clinet():
with TestClient(app) as client:
yield client
This works fine, whenever I use settings.A I get 42.
But is it possible to override the whole env_file from .prod.env to another env file .test.env?
Also I probably want to call settings.A = 42 in conftest.py before I import app, right?
You can override the env file you use by creating a Settings instance with the _env_file keyword argument.
From documentation:
Passing a file path via the _env_file keyword argument on instantiation (method 2) will override the value (if any) set on the Config class. If the above snippets were used in conjunction, prod.env would be loaded while .env would be ignored.
For example, this should work for your test -
import pytest
from fastapi.testclient import TestClient
import app.config as conf
from app.config import Settings
# replace the settings object that you created in the module
conf.settings = Settings(_env_file='.test.env')
from app.main import app
# just to show you that you changed the module-level
# settings
from app.config import settings
#pytest.fixture(scope="module")
def test_client():
with TestClient(app) as client:
yield client
def test_settings():
print(conf.settings)
print(settings)
And you could create a .test.env, set A=10000000, and run with
pytest -rP conftest.py
# stuff
----- Captured stdout call -----
A=10000000
A=10000000
This looks a little messy (though this is probably only used for test purposes), so I'd recommend not creating a settings object that is importable by everything in your code, and instead making it something you create in, say, your __main__ that actually creates and runs the app, but that's a design choice for you to make.
I ran into the same issue today, really annoying. My goal was to set a different postgresql database for the unit tests. The default configuration comes from a .env file. But when you think of it thoroughly it is not that difficult to understand. It all boils down to the order of the imported modules in conftest.py. I based the example below on the answer of #wkl:
import pytest
from typing import Generator
from fastapi.testclient import TestClient
import app.core.config as config
from app.core.config import Settings
# Replace individual attribute in the settings object
config.settings = Settings(
POSTGRES_DB="test_db")
# Or replace the env file in the settings object
config.settings = Settings(_env_file='.test.env')
# All other modules that import settings are imported here
# This ensures that those modules will use the updated settings object
# Don't forget to use "noqa", otherwise a formatter might put it back on top
from app.main import app # noqa
from app.db.session import SessionLocal # noqa
#pytest.fixture(scope="session")
def db() -> Generator:
try:
db = SessionLocal()
yield db
finally:
db.close()
#pytest.fixture(scope="module")
def client() -> Generator:
with TestClient(app) as c:
yield c
One workaround I have found is to remove the env_file from Config completely and replace it's functionality with load_dotenv() from dotenv like this:
config.py:
from pydantic import BaseSettings
from dotenv import load_dotenv
load_dotenv(".prod.env")
class Settings(BaseSettings):
A: int
settings = Settings()
conftest.py:
import pytest
from fastapi.testclient import TestClient
from dotenv import load_dotenv
load_dotenv("test.env")
from app.config import settings
from app.main import app
#pytest.fixture(scope="module")
def test_clinet():
with TestClient(app) as client:
yield client
Please note, that calling load_dotenv("test.env") happens before importing the settings (from app.config import settings)
and also note that load_dotenv() will load environment variables globally for the whole python script.
Loading env variables like this will not override already exported variables, same as using the env_file in pydantic's BaseSettings

Pytest mock: how to patch an import statement inside a function [duplicate]

Is it possible to mock a module in python using unittest.mock? I have a module named config, while running tests I want to mock it by another module test_config. how can I do that ? Thanks.
config.py:
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
test_config.py:
CONF_VAR1 = "test_VAR1"
CONF_VAR2 = "test_VAR2"
All other modules read config variables from the config module. While running tests I want them to read config variables from test_config module instead.
If you're always accessing the variables in config.py like this:
import config
...
config.VAR1
You can replace the config module imported by whatever module you're actually trying to test. So, if you're testing a module called foo, and it imports and uses config, you can say:
from mock import patch
import foo
import config_test
....
with patch('foo.config', new=config_test):
foo.whatever()
But this isn't actually replacing the module globally, it's only replacing it within the foo module's namespace. So you would need to patch it everywhere it's imported. It also wouldn't work if foo does this instead of import config:
from config import VAR1
You can also mess with sys.modules to do this:
import config_test
import sys
sys.modules["config"] = config_test
# import modules that uses "import config" here, and they'll actually get config_test
But generally it's not a good idea to mess with sys.modules, and I don't think this case is any different. I would favor all of the other suggestions made over it.
foo.py:
import config
VAR1 = config.CONF_VAR1
def bar():
return VAR1
test.py:
import unittest
import unittest.mock as mock
import test_config
class Test(unittest.TestCase):
def test_one(self):
with mock.patch.dict('sys.modules', config=test_config):
import foo
self.assertEqual(foo.bar(), 'test_VAR1')
As you can see, the patch works even for code executed during import foo.
If you want to mock an entire module just mock the import where the module is used.
myfile.py
import urllib
test_myfile.py
import mock
import unittest
class MyTest(unittest.TestCase):
#mock.patch('myfile.urllib')
def test_thing(self, urllib):
urllib.whatever.return_value = 4
Consider this following setup
configuration.py:
import os
class Config(object):
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
class TestConfig(object):
CONF_VAR1 = "test_VAR1"
CONF_VAR2 = "test_VAR2"
if os.getenv("TEST"):
config = TestConfig
else:
config = Config
now everywhere else in your code you can use:
from configuration import config
print config.CONF_VAR1, config.CONF_VAR2
And when you want to mock your coniguration file just set the environment variable "TEST".
Extra credit:
If you have lots of configuration variables that are shared between your testing and non-testing code, then you can derive TestConfig from Config and simply overwrite the variables that need changing:
class Config(object):
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
CONF_VAR3 = "VAR3"
class TestConfig(Config):
CONF_VAR2 = "test_VAR2"
# CONF_VAR1, CONF_VAR3 remain unchanged
If your application ("app.py" say) looks like
import config
print config.var1, config.var2
And gives the output:
$ python app.py
VAR1 VAR2
You can use mock.patch to patch the individual config variables:
from mock import patch
with patch('config.var1', 'test_VAR1'):
import app
This results in:
$ python mockimport.py
test_VAR1 VAR2
Though I'm not sure if this is possible at the module level.

Unable to patch logging.Logger.info called inside celery task

I want to test that a specific celery task call logger.info exactly once when the task is invoked with the delay() API.
And I want to do the test by patching logger.info .
I want to test as described here for the Product.order case https://docs.celeryproject.org/en/latest/userguide/testing.html.
The setup is : python 2.7 on ubuntu 16.04 . celery 4.3.0 , pytest 4.0.0, mock 3.0.3.
I have the following file system structure:
poc/
prj/
-celery_app.py
-tests.py
celery_app.py
from __future__ import absolute_import
from celery import Celery
from celery.utils.log import get_task_logger
app = Celery('celery_app')
logger = get_task_logger(__name__)
#app.task(bind=True)
def debug_task(self):
logger.info('Request: {0!r}'.format(self.request))
tests.py
from __future__ import absolute_import
from mock import patch
from prj.celery_app import debug_task
#patch('logging.Logger.info')
def test_log_info_is_called_only_once_when_called_sync(log_info):
debug_task()
log_info.assert_called_once()
#patch('logging.Logger.info')
def test_log_info_is_called_only_once_when_called_async(log_info):
debug_task.delay()
log_info.assert_called_once()
I expect both tests to have success.
Instead the first has success, while the second fails with AssertionError: Expected 'info' to have been called once. Called 0 times.
I expect that the evaluation of the expression logger.info inside debug_task() context evaluates to <MagicMock name='info' id='someinteger'> in both cases, instead it evaluates to <bound method Logger.info of <logging.Logger object at 0x7f894eeb1690>> in the second case, showing no patching.
I know that in the second case the celery worker executes the task inside a thread.
I ask for a way to patch the logger.info call when debug_task.delay() is executed.
Thanks in advance for any answer.

How to unittest a function that use app.logger of Flask

For sure I'm missing something in Flask and unit test integration (or logger configuration maybe)
But when I'm trying to unittest some class methods that have some app.logger I'm having troubles with RuntimeError: working outside of the application context
So a practical example:
utils.py
import boto3
from flask import current_app as app
class CustomError(BaseException):
type = "boto"
class BotoManager:
def upload_to_s3(self, file):
try:
# do something that can trigger a boto3 error
except boto3.exceptions.Boto3Error as e:
app.logger.error(e)
raise CustomError()
test_utils.py
import pytest
from utils.py import CustomError, BotoManager
def test_s3_manager_trigger_error():
boto_manager = BotoManager()
with pytest.raises(CustomError):
boto_manager.upload_to_s3('file.txt') # file doesn't exist so trigger error
So the thing is that when I run it show me the error:
RuntimeError: Working outside of application context.
Becuase the app is not created and I'm not working with the app, so have sense.
So I only see two possible solutions (spoiler I don't like any of them):
Don't log anything with app.logger outside of the views (I think I can use the python logging system, but this is not the desired behaviour)
Don't unittest the parts that use app.logger
Did someone face this problem already? How did you solve it? Any other possible solution?

How do I write a Python test that checks that my code imports a module

I have a Python project that relies on a particular module, receivers.py, being imported.
I want to write a test to make sure it is imported, but I also want to write other tests for the behaviour of the code within the module.
The trouble is, that if I have any tests anywhere in my test suite that import or patch anything from receivers.py then it will automatically import the module, potentially making the test for import pass wrongly.
Any ideas?
(Note: specifically this is a Django project.)
One (somewhat imperfect) way of doing it is to use the following TestCase:
from django.test import TestCase
class ReceiverConnectionTestCase(TestCase):
"""TestCase that allows asserting that a given receiver is connected
to a signal.
Important: this will work correctly providing you:
1. Do not import or patch anything in the module containing the receiver
in any django.test.TestCase.
2. Do not import (except in the context of a method) the module
containing the receiver in any test module.
This is because as soon as you import/patch, the receiver will be connected
by your test and will be connected for the entire test suite run.
If you want to test the behaviour of the receiver, you may do this
providing it is a unittest.TestCase, and there is no import from the
receiver module in that test module.
Usage:
# myapp/receivers.py
from django.dispatch import receiver
from apples.signals import apple_eaten
from apples.models import Apple
#receiver(apple_eaten, sender=Apple)
def my_receiver(sender, **kwargs):
pass
# tests/integration_tests.py
from apples.signals import apple_eaten
from apples.models import Apple
class TestMyReceiverConnection(ReceiverConnectionTestCase):
def test_connection(self):
self.assert_receiver_is_connected(
'myapp.receivers.my_receiver',
signal=apple_eaten, sender=Apple)
"""
def assert_receiver_is_connected(self, receiver_string, signal, sender):
receivers = signal._live_receivers(sender)
receiver_strings = [
"{}.{}".format(r.__module__, r.__name__) for r in receivers]
if receiver_string not in receiver_strings:
raise AssertionError(
'{} is not connected to signal.'.format(receiver_string))
This works because Django runs django.test.TestCases before unittest.TestCases.

Categories