I have a project assigning 2 configs in my main __init__ file since they are used frequently throughout the project.
#__init__.py
from config import Config
config1 = Config('Email')
config2 = Config('Test')
My Config class within config.py has a method called content that I need to mock out on instances config1 and config2. The config does call out to a third party library to do an http request, so I need to return a json dictionary for the response to content.
In a validators function I have the following:
#validation.py
from parser import config1, config2
def validation(msg):
if "email" in config1.keys():
...
I'm not trying to mock out the tests but keep getting errors. I've tried various mock patch paths but none work.
My latest attempt is the following:
from mock import patch
from parser import validation
#patch('parser.Config')
def test_is_valid(mock_config):
mock_config.return_value.content = "Test"
assert validation.is_valid("email") == True
What am I doing wrong that my instances of Config (config1, and config2) are not correctly returning the .content values? Thanks
Related
So I am trying to test an api lookup with my mock data.
I am testing a method within transform.py that imports a module lookup
import lookup
col = lookup.ColFinder()
url = "xyz"
if col.is_present(url):
do this
lookup.py
import secdata
class ColFinder():
def is_present(url):
if url in secdata.CUSTOM_STUFF:
return secdata.CUSTOM_STUFF[url]
secdata.py
secdata.CUSTOM_STUFF=load("some_file")
I want to mock the JSON (file being loaded within secdata.CUSTOM_STUFF)
I have tried mocking using the unittest.mock with some custom config within tests/resources
CUSTOM_CONFIG = secdata.load(os.path.join(os.path.dirname(__file__),'/resources/custom_config.json'))
import mock
#mock.patch("transform.lookup.secdata.CUSTOM_STUFF" , return_value=CUSTOM_CONFIG)
def test_blah_blah(self, *_):
but this doesn't seem to load the file I am trying to reference. Please can someone help me mock this, point out what am I doing wrong.
Thank you in advance.
Say I want to mock the following:
session = boto3.Session(profile_name=profile)
resource = session.resource('iam')
iam_users = resource.users.all()
policies = resource.policies.filter(Scope='AWS', OnlyAttached=True, PolicyUsageFilter='PermissionsPolicy')
How do I go about starting to mock this with in pytest? I could create mocked objects by creating a dummy class and the necessary attributes, but I suspect that's the wrong approach.
Some additional details, here's what I'm trying to test out:
def test_check_aws_profile(self, mocker):
mocked_boto3 = mocker.patch('myapp.services.utils.boto3.Session')
mocker.patch(mocked_boto3.client.get_caller_identity.get, return_value='foo-account-id')
assert 'foo-account-id' == my_func('foo')
#in myapp.services.utils.py
def my_func(profile):
session = boto3.Session(profile_name=profile)
client = session.client('sts')
aws_account_number = client.get_caller_identity().get('Account')
return aws_account_number
But I can't quite seem to be able to get this patched correctly. I'm trying to make it so that I can patch session and the function calls in that method
I tried using moto and got this:
#mock_sts
def test_check_aws_profile(self):
session = boto3.Session(profile_name='foo')
client = session.client('sts')
client.get_caller_identity().get('Account')
But I'm running into
> raise ProfileNotFound(profile=profile_name)
E botocore.exceptions.ProfileNotFound: The config profile (foo) could not be found
So it seems like it's not mocking anything :|
Edit:
Turns out you need to have the mocked credentials in a config and credentials file for this to work.
If you want to use moto, you can use the AWS_SHARED_CREDENTIALS_FILE environment variable, to point it to a dummy credentials file which can be kept in the tests folder.
You can define your profiles there. Example:
Files: test_stuff.py. dummy_aws_credentials
test_stuff.py:
import os
from pathlib import Path
import boto3
import pytest
from moto import mock_sts
#pytest.fixture(scope='module')
def aws_credentials():
"""Mocked AWS Credentials for moto."""
moto_credentials_file_path = Path(__file__).parent.absolute() / 'dummy_aws_credentials'
os.environ['AWS_SHARED_CREDENTIALS_FILE'] = str(moto_credentials_file_path)
#mock_sts
def test_check_aws_profile(aws_credentials):
session = boto3.Session(profile_name='foo')
client = session.client('sts')
client.get_caller_identity().get('Account')
dummy_aws_credentials:
[foo]
aws_access_key_id = mock
aws_secret_access_key = mock
I'm not sure what exactly you want, so I'll give you something to start.
You let unittest.mock to mock everything for you, for example. (Useful reading: https://docs.python.org/3/library/unittest.mock.html)
module.py:
import boto3
def function():
session = boto3.Session(profile_name="foobar")
client = session.resource("sts")
return client.get_caller_identity().get('Account')
test_module.py:
from unittest.mock import patch
import module
#patch("module.boto3") # this creates mock which is passed to test below
def test_function(mocked_boto):
# mocks below are magically created by unittest.mock when they are accessed
mocked_session = mocked_boto.Session()
mocked_client = mocked_session.resource()
mocked_identity = mocked_client.get_caller_identity()
# now mock the return value of .get()
mocked_identity.get.return_value = "foo-bar-baz"
result = module.function()
assert result == "foo-bar-baz"
# we can make sure mocks were called properly, for example
mocked_identity.get.assert_called_once_with("Account")
Results of pytest run:
$ pytest
================================ test session starts ================================
platform darwin -- Python 3.7.6, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: /private/tmp/one
collected 1 item
test_module.py . [100%]
================================= 1 passed in 0.09s =================================
I would also recommend to install pytest-socket and run pytest --disable-socket to make sure your tests do not talk with network by accident.
Although there is nothing wrong with manually patching boto using mock.patch, you could also consider using a higher level testing utility like moto.
I have a question about how to mock a nested method and test what it was called with. I'm having a hard time getting my head around: https://docs.python.org/3/library/unittest.mock-examples.html#mocking-chained-calls.
I'd like to test that the "put" method from the fabric library is called by the deploy_file method in this class, and maybe what values are given to it. This is the module that gathers some information from AWS and provides a method to take action on the data.
import json
import os
from aws.secrets_manager import get_secret
from fabric import Connection
class Deploy:
def __init__(self):
self.secrets = None
self.set_secrets()
def set_secrets(self):
secrets = get_secret()
self.secrets = json.loads(secrets)
def deploy_file(self, source_file):
with Connection(host=os.environ.get('SSH_USERNAME'), user=os.environ.get("SSH_USERNAME")) as conn:
destination_path = self.secrets["app_path"] + '/' + os.path.basename(source_file)
conn.put(source_file, destination_path)
"get_secret" is a method in another module that uses the boto3 library to get the info from AWS.
These are the tests I'm working on:
from unittest.mock import patch
from fabric import Connection
from jobs.deploy import Deploy
def test_set_secrets_dict_from_expected_json_string():
with patch('jobs.deploy.get_secret') as m_get_secret:
m_get_secret.return_value = '{"app_path": "/var/www/html"}'
deployment = Deploy()
assert deployment.secrets['app_path'] == "/var/www/html"
def test_copy_app_file_calls_fabric_put():
with patch('jobs.deploy.get_secret') as m_get_secret:
m_get_secret.return_value = '{"app_path": "/var/www/html"}'
deployment = Deploy()
with patch('jobs.deploy.Connection', spec=Connection) as m_conn:
local_file_path = "/tmp/foo"
deployment.deploy_file(local_file_path)
m_conn.put.assert_called_once()
where the second test results in "AssertionError: Expected 'put' to have been called once. Called 0 times."
the first test mocks the "get_secret" function just fine to test that the constructor for "Deploy" sets "Deploy.secrets" from the fake AWS data.
In the second test, get_secrets is mocked just as before, and I mock "Connection" from the fabric library. If I don't mock Connection, I get an error related to the "host" parameter when the Connection object is created.
I think that when "conn.put" is called its creating a whole new Mock object and I'm not testing that object when the unittest runs. I'm just not sure how to define the test to actually test the call to put.
I'm also a novice at understanding what to test (and how) and what not to test as well as how to use mock and such. I'm fully bought in on the idea though. It's been very helpful to find bugs and regressions as I work on projects.
Can you please help me out to figure what I did wrong? I have the following unit test for a python lambdas
class Tests(unittest.TestCase):
def setUp(self):
//some setup
#mock.patch('functions.tested_class.requests.get')
#mock.patch('functions.helper_class.get_auth_token')
def test_tested_class(self, mock_auth, mock_get):
mock_get.side_effect = [self.mock_response]
mock_auth.return_value = "some id token"
response = get_xml(self.event, None)
self.assertEqual(response['statusCode'], 200)
The problem is that when I run this code, I get the following error for get_auth_token:
Invalid URL '': No schema supplied. Perhaps you meant http://?
I debugged it, and it doesn't look like I patched it correctly. The Authorization helper file is in the same folder "functions" as the tested class.
EDIT:
In the tested_class I was importing get_auth_token like this:
from functions import helper_class
from functions.helper_class import get_auth_token
...
def get_xml(event, context):
...
response_token = get_auth_token()
After changing to this, it started to work fine
import functions.helper_class
...
def get_xml(event, context):
...
response_token = functions.helper_class.get_auth_token()
I still don't fully understand why though
In your first scenario
in tested_class.py, get_auth_token is imported
from functions.helper_class import get_auth_token
The patch should be exactly the get_auth_token at tested_class
#mock.patch('functions.tested_class.get_auth_token')
Second scenario
With the following usage
response_token = functions.helper_class.get_auth_token()
The only way to patch is this
#mock.patch('functions.helper_class.get_auth_token')
alternative
With import like this in tested_class
from functions import helper_class
helper_class.get_auth_token()
patch could be like this:
#mock.patch('functions.tested_class.helper_class.get_auth_token')
patch() works by (temporarily) changing the object that a name points to with another one. There can be many names pointing to any individual object, so for patching to work, you must ensure that you patch the name used by the system under test.
The basic principle is that you patch where an object is looked up, which is not necessarily the same place as where it is defined.
Python documentation has a very good example. where to patch
I have a Sanic application, and want to retrieve app.config from a blueprint as it holds MONGO_URL, and I will pass it to a repository class from the blueprint.
However, I could not find how to get app.config in a blueprint. I have also checked Flask solutions, but they are not applicable to Sanic.
My app.py:
from sanic import Sanic
from routes.authentication import auth_route
from routes.user import user_route
app = Sanic(__name__)
app.blueprint(auth_route, url_prefix="/auth")
app.blueprint(user_route, url_prefix="/user")
app.config.from_envvar('TWEETBOX_CONFIG')
app.run(host='127.0.0.1', port=8000, debug=True)
My auth blueprint:
import jwt
from sanic import Blueprint
from sanic.response import json, redirect
from domain.user import User
from repository.user_repository import UserRepository
...
auth_route = Blueprint('authentication')
mongo_url = ?????
user_repository = UserRepository(mongo_url)
...
#auth_route.route('/signin')
async def redirect_user(request):
...
The Sanic way...
Inside a view method, you can access the app instance from the request object. And, therefore access your configuration.
#auth_route.route('/signin')
async def redirect_user(request):
configuration = request.app.config
2021-10-10 Update
There are two newer ways to get to the configuration values (or, perhaps more accuratlely getting the application instance from which you can get the configuration). The first version might be more on point to answering the question of how to get to the config from the blueprint. However, the second option is probably the preferred method since it is precisely intended for this kind of use.
Alternative #1
Blueprints have access to the Sanic applications they are attached to beginning with v21.3.
Therefore, if you have a blueprint object, you can trace that back to the application instance, and therefore also the config value.
app = Sanic("MyApp")
bp = Blueprint("MyBlueprint")
app.blueprint(bp)
assert bp.apps[0] is app
The Blueprint.apps property is a set because it is possible to attach a single blueprint to multiple applications.
Alternative #2
Sanic has a built-in method for retrieving an application instance from the global scope beginning in v20.12. This means that once an application has been instantiated, you can retrieve it using: Sanic.get_app().
app = Sanic("MyApp")
assert Sanic.get_app() is app
This method will only work if there is a single Sanic instance available. If you have multiple application instances, you will need to use the optional name argument:
app1 = Sanic("MyApp")
app2 = Sanic("MyOtherApp")
assert Sanic.get_app("MyApp") is app1
I would suggest a slightly different approach, based on the 12 Factor App (very interesting read which, among others, provides a nice guideline on how to protect and isolate your sensitive info).
The general idea is to place your sensitive and configuration variables in a file that is going to be gitignored and therefore will only be available locally.
I will try to present the method I tend to use in order to be as close as possible to the 12 Factor guidelines:
Create a .env file with your project variables in it:
MONGO_URL=http://no_peeking_this_is_secret:port/
SENSITIVE_PASSWORD=for_your_eyes_only
CONFIG_OPTION_1=config_this
DEBUG=True
...
(Important) Add .env and .env.* on your .gitignore file, thus protecting your sensitive info from been uploaded to GitHub.
Create an env.example (be careful not to name it with a . in the beginning, because it will get ignored).
In that file, you can put an example of the expected configuration in order to be reproducible by simply copy, paste, rename to .env.
In a file named settings.py, use decouple.config to read your config file into variables:
from decouple import config
MONGO_URL = config('MONGO_URL')
CONFIG_OPTION_1 = config('CONFIG_OPTION_1', default='')
DEBUG = config('DEBUG', cast=bool, default=True)
...
Now you can use these variables wherever is necessary for your implementation:
myblueprint.py:
import settings
...
auth_route = Blueprint('authentication')
mongo_url = settings.MONGO_URL
user_repository = UserRepository(mongo_url)
...
As a finisher, I would like to point out that this method is framework (and even language) agnostic so you can use it on Sanic as well as Flask and everywhere you need it!
I think you can create a config.py to save your configuration, just like
config.py
config = {
'MONGO_URL'：'127.0.0.1:27017'
}
and use it in app.py
from config import config
mongo_url = config['MONGO_URL']
There is a variable named current_app in Flask. You can use current_app.config["MONGO_URL"].
But I am not familiar with Sanic.