We Keep Coding

Mocking os.environ with python unittests

I am trying to test a class that handles for me the working directory based on a given parameter. To do so, we are using a class variable to map them.
When a specific value is passed, the path is retrieved from the environment variables (See baz in the example below). This is the specific case that I'm trying to test.
I'm using Python 3.8.13 and unittest.
I'm trying to avoid:
I don't want to mock the WorkingDirectory.map dictionary because I want to make sure we are fetching from the environ with that particular variable (BAZ_PATH).
Unless is the only solution, I would like to avoid editing the values during the test, i.e I would prefer not to do something like: os.environ["baz"] = DUMMY_BAZ_PATH
What I've tried
I tried mocking up the environ as a dictionary as suggested in other publications, but I can't make it work for some reason.
# working_directory.py
import os
class WorkingDirectory:
map = {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH"),
}
def __init__(self, env: str):
self.env = env
self.path = self.map[self.env]
#property
def data_dir(self):
return os.path.join(self.path, "data")
# Other similar methods...
Test file:
# test.py
import os
import unittest
from unittest import mock
from working_directory import WorkingDirectory
DUMMY_BAZ_PATH = "path/to/baz"
class TestWorkingDirectory(unittest.TestCase):
#mock.patch.dict(os.environ, {"BAZ_PATH": DUMMY_BAZ_PATH})
def test_controlled_baz(self):
wd = WorkingDirectory("baz")
self.assertEqual(wd.path, DUMMY_BAZ_PATH)
Error
As shown in the error, os.environ doesn't seem to be properly patched as it returns Null.
======================================================================
FAIL: test_controlled_baz (test_directory_structure_utils.TestWorkingDirectory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "~/.pyenv/versions/3.8.13/lib/python3.8/unittest/mock.py", line 1756, in _inner
return f(*args, **kw)
File "~/Projects/dummy_project/tests/unit/test_directory_structure_utils.py", line 127, in test_controlled_baz
self.assertEqual(wd.path, DUMMY_BAZ_PATH)
AssertionError: None != 'path/to/baz'
----------------------------------------------------------------------
Ran 136 tests in 0.325s
FAILED (failures=1, skipped=5)
This seems to be because the BAZ_PATH doesn't exist actually. However, I would expect this to be OK since is being patched.
When, in the mapping dictionary, "baz": os.environ.get("BAZ_PATH"), I repalce BAZ_PATH for a variable that actually exist in my environment, i.e HOME, it returns the actual value of HOME instead of the DUMMY_BAZ_PATH, which lead me to think that I'm definetely doing something wrong patching
AssertionError: '/Users/cestla' != 'path/to/baz'
Expected result
Well, obviously, I am expecting the test_controlled_baz passes succesfully.

So the problem is that you added map as a static variable.
Your patch works correctly as you can see here:
patch actually works
The problem is that when it runs it's already too late because the map variable was already calculated (before the patch).
If you want you can move it to the init function and it will function correctly:
class WorkingDirectory:
def __init__(self, env: str):
self.map = {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH")
}
self.env = env
self.path = self.map[self.env]
If for some reason you wish to keep it static, you have to also patch the object itself.
writing something like this will do the trick:
class TestWorkingDirectory(unittest.TestCase):
#mock.patch.dict(os.environ, {"BAZ_PATH": DUMMY_BAZ_PATH})
def test_controlled_baz(self):
with mock.patch.object(WorkingDirectory, "map", {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH")
}):
wd = WorkingDirectory("baz")
self.assertEqual(wd.path, DUMMY_BAZ_PATH)

That's not directly answer to your question but a valid answer either way imo:
Don't try to patch that (it's possible, but harder and cumbersome).
Use config file for your project.
e.g. use pyproject.toml and inside configure the pytest extension:
[tool.pytest.ini_options]
env=[
"SOME_VAR_FOR_TESTS=some_value_for_that_var"
]

How to handle Spyne XSD exceptions gracefully

Whenever my Spyne application receives a request, XSD validation is performed. This is good, but whenever there is an XSD violation a fault is raised and my app returns a Client.SchemaValidationError like so:
<soap11env:Fault>
<faultcode>soap11env:Client.SchemaValidationError</faultcode>
<faultstring>:25:0:ERROR:SCHEMASV:SCHEMAV_CVC_DATATYPE_VALID_1_2_1: Element '{http://services.sp.pas.ng.org}DateTimeStamp': '2018-07-25T13:01' is not a valid value of the atomic type 'xs:dateTime'.</faultstring>
<faultactor></faultactor>
</soap11env:Fault>
I would like to know how to handle the schema validation error gracefully and return the details in the Details field of my service's out_message, rather than just raising a standard Client.SchemaValidationError. I want to store the details of the error as a variable and pass it to my OperationOne function.
Here is my code, I have changed var names for sensitivity.
TNS = "http://services.so.example.org"
class InMessageType(ComplexModel):
__namespace__ = TNS
class Attributes(ComplexModel.Attributes):
declare_order = 'declared'
field_one = Unicode(values=["ONE", "TWO"],
min_occurs=1)
field_two = Unicode(20, min_occurs=1)
field_three = Unicode(20, min_occurs=0)
Confirmation = Unicode(values=["ACCEPTED", "REJECTED"], min_occurs=1)
FileReason = Unicode(200, min_occurs=0)
DateTimeStamp = DateTime(min_occurs=1)
class OperationOneResponse(ComplexModel):
__namespace__ = TNS
class Attributes(ComplexModel.Attributes):
declare_order = 'declared'
ResponseMessage = Unicode(values=["SUCCESS", "FAILURE"], min_occurs=1)
Details = Unicode(min_len=0, max_len=2000)
class ServiceOne(ServiceBase):
#rpc(InMessageType,
_returns=OperationOneResponse,
_out_message_name='OperationOneResponse',
_in_message_name='InMessageType',
_body_style='bare',
)
def OperationOne(ctx, message):
# DO STUFF HERE
# e.g. return {'ResponseMessage': Failure, 'Details': XSDValidationError}
application = Application([ServiceOne],
TNS,
in_protocol=Soap11(validator='lxml'),
out_protocol=Soap11(),
name='ServiceOne',)
wsgi_application = WsgiApplication(application)
if __name__ == '__main__':
pass
I have considered the following approach but I can't quite seem to make it work yet:
create subclass MyApplication with call_wrapper() function overridden.
Instantiate the application with in_protocol=Soap11(validator=None)
Inside the call wrapper set the protocol to Soap11(validator='lxml') and (somehow) call something which will validate the message. Wrap this in a try/except block and in case of error, catch the error and handle it in whatever way necessary.
I just haven't figured out what I can call inside my overridden call_wrapper() function which will actually perform the validation. I have tried protocol.decompose_incoming_envelope() and other such things but no luck yet.

Overriding the call_wrapper would not work as the validation error is raised before it's called.
You should instead use the event subsystem. More specifically, you must register an application-level handler for the method_exception_object event.
Here's an example:
def _on_exception_object(ctx):
if isinstance(ctx.out_error, ValidationError):
ctx.out_error = NicerValidationError(...)
app = Application(...)
app.event_manager.add_listener('method_exception_object', _on_exception_object)
See this test for more info: https://github.com/arskom/spyne/blob/4a74cfdbc7db7552bc89c0e5d5c19ed5d0755bc7/spyne/test/test_service.py#L69
As per your clarification, if you don't want to reply with a nicer error but a regular response, I'm afraid Spyne is not designed to satisfy that use-case. "Converting" an errored-out request processing state to a regular one would needlessly complicate the already heavy request handling logic.
What you can do instead is to HACK the heck out of the response document.
One way to do it is to implement an additional method_exception_document event handler where the <Fault> tag and its contents are either edited to your taste or even swapped out.
Off the top of my head:
class ValidationErrorReport(ComplexModel):
_type_info = [
('foo', Unicode),
('bar', Integer32),
]
def _on_exception_document(ctx):
fault_elt, = ctx.out_document.xpath("//soap11:Fault", namespaces={'soap11': NS_SOAP11_ENV})
explanation_elt = get_object_as_xml(ValidationErrorReport(...))
fault_parent = fault_elt.parent()
fault_parent.remove(fault_elt)
fault_parent.add(explanation_elt)
The above needs to be double-checked with the relevant Spyne and lxml APIs (maybe you can use find() instead of xpath()), but you get the idea.
Hope that helps!

Django TestCase: recreate database in self.subTest(...)

I need to test a function with different parameters, and the most proper way for this seems to be using the with self.subTest(...) context manager.
However, the function writes something to the db, and it ends up in an inconsistent state. I can delete the things I write, but it would be cleaner if I could recreate the whole db completely. Is there a way to do that?

Not sure how to recreate the database in self.subTest() but I have another technique I am currently using which might be of interest to you. You can use fixtures to create a "snapshot" of your database which will basically be copied in a second database used only for testing purposes. I currently use this method to test code on a big project I'm working on at work.
I'll post some example code to give you an idea of what this will look like in practice, but you might have to do some extra research to tailor the code to your needs (I've added links to guide you).
The process is rather straighforward. You would be creating a copy of your database with only the data needed by using fixtures, which will be stored in a .yaml file and accessed only by your test unit.
Here is what the process would look like:
List item you want to copy to your test database to populate it using fixtures. This will only create a db with the needed data instead of stupidly copying the entire db. It will be stored in a .yaml file.
generate.py
django.setup()
stdout = sys.stdout
conf = [
{
'file': 'myfile.yaml',
'models': [
dict(model='your.model', pks='your, primary, keys'),
dict(model='your.model', pks='your, primary, keys')
]
}
]
for fixture in conf:
print('Processing: %s' % fixture['file'])
with open(fixture['file'], 'w') as f:
sys.stdout = FixtureAnonymiser(f)
for model in fixture['models']:
call_command('dumpdata', model.pop('model'), format='yaml',indent=4, **model)
sys.stdout.flush()
sys.stdout = stdout
In your test unit, import your generated .yaml file as a fixture and your test will automatically use this the data from the fixture to carry out the tests, keeping your main database untouched.
test_class.py
from django.test import TestCase
class classTest(TestCase):
fixtures = ('myfile.yaml',)
def setUp(self):
"""setup tests cases"""
# create the object you want to test here, which will use data from the fixtures
def test_function(self):
self.assertEqual(True,True)
# write your test here
You can read up more here:
Django
YAML
If you have any questions because things are unclear just ask, I'd be happy to help you out.

Maybe my solution will help someone
I used transactions to roll back to the database state that I had at the start of the test.
I use Eric Cousineau's decorator function to parametrizing tests
More about database transactions at django documentation page
import functools
from django.db import transaction
from django.test import TransactionTestCase
from django.contrib.auth import get_user_model
User = get_user_model()
def sub_test(param_list):
"""Decorates a test case to run it as a set of subtests."""
def decorator(f):
#functools.wraps(f)
def wrapped(self):
for param in param_list:
with self.subTest(**param):
f(self, **param)
return wrapped
return decorator
class MyTestCase(TransactionTestCase):
#sub_test([
dict(email="new#user.com", password='12345678'),
dict(email="new#user.com", password='password'),
])
def test_passwords(self, email, password):
# open a transaction
with transaction.atomic():
# Creates a new savepoint. Returns the savepoint ID (sid).
sid = transaction.savepoint()
# create user and check, if there only one with this email in DB
user = User.objects.create(email=email, password=password)
self.assertEqual(User.objects.filter(email=user.email).count(), 1)
# Rolls back the transaction to savepoint sid.
transaction.savepoint_rollback(sid)

Best way to make subapps with Traversal

Ok so I have my apps, that takes requests from root / Almost everything is using traversal.
But i'd like to make on top of that site a rest api.
So I'm off with two choices. I either separate the that in two different apps and put that rest application to : rest.site.com, Or I can move it to site.com/rest/*traversal
If I'm doing "/rest/*traversal", I guess I'll have to add a route called rest_traversal where the traversal path will be *traversal with the route /rest/*traversal. I did that once for an admin page.
I was wondering if there was a cleanest way to do that. I tried to use virtual_root, but as I understand virtual_root is actually getting added to the path for traversal.
like having virtual_root = /cms and requesting /fun will create the following path /cms/fun
I on the other hand wish to have /cms/fun turned into /fun

I know this has been answered already, but in case someone arrives here looking for another possible way to make "subapps" and using them in pyramid, I wanted to point out that some interesting things can be done with pyramid.wsgi
"""
example of wsgiapp decorator usage
http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/api/wsgi.html
"""
from pyramid.wsgi import wsgiapp2, wsgiapp
from pyramid.config import Configurator
from webob import Request, Response
import pprint
# define some apps
def wsgi_echo(environ, start_response):
"""pretty print out the environ"""
response = Response(body=pprint.pformat({k: v for k, v in environ.items()
if k not in ["wsgi.errors",
"wsgi.input",
"SCRIPT_NAME"]}))
return response(environ, start_response)
print Request.blank("/someurl").send(wsgi_echo).body
# convert wsgi app to a pyramid view callable
pyramid_echo = wsgiapp(wsgi_echo)
pyramid_echo_2 = wsgiapp2(wsgi_echo)
# wire up a pyramid application
config = Configurator()
config.add_view(pyramid_echo, name="foo") # /foo
config.add_view(pyramid_echo, name="bar") # /bar
config.add_view(pyramid_echo_2, name="foo_2") # /foo
config.add_view(pyramid_echo_2, name="bar_2") # /bar
pyramid_app = config.make_wsgi_app()
#call some urls
foo_body = Request.blank("/foo").send(pyramid_app).body
bar_body = Request.blank("/bar").send(pyramid_app).body
foo_body_2 = Request.blank("/foo_2").send(pyramid_app).body
bar_body_2 = Request.blank("/bar_2").send(pyramid_app).body
# both should be different because we arrived at 2 different urls
assert foo_body != bar_body, "bodies should not be equal"
# should be equal because wsgiapp2 fixes stuff before calling
# application in fact there's an additional SCRIPT_NAME in the
# environment that we are filtering out
assert foo_body_2 == bar_body_2, "bodies should be equal"
# so how to pass the path along? like /foo/fuuuu should come back
# /fuuuu does it
foo_body = Request.blank("/foo_2/fuuuu").send(pyramid_app).body
assert "'/fuuuu'," in foo_body, "path didn't get passed along"
# tldr: a wsgi app that is decorated with wsgiapp2 will recieve data
# as if it was mounted at "/", any url generation it has to do should
# take into account the SCRIPT_NAME variable that may arrive in the
# environ when it is called

If you're using traversal already, why not just use it to return your "rest API root" object when Pyramid traverses to /rest/? From there, everything will work naturally.
class ApplicationRoot(object):
def __getitem__(self, name):
if name == "rest":
return RestAPIRoot(parent=self, name=name)
...
If your "application tree" and "API tree" have the same children and you want to have different views registered for them depending on which branch of the tree the child is located in, you can use containment view predicate to register your API views, so they will only match when the child is inside the "API branch":
containment
This value should be a reference to a Python class or interface that a
parent object in the context resource’s lineage must provide in order
for this view to be found and called. The resources in your resource
tree must be “location-aware” to use this feature.
If containment is not supplied, the interfaces and classes in the
lineage are not considered when deciding whether or not to invoke the
view callable.
Another approach would be not to build a separate "API tree" but to use your "main" application's "URI-space" as RESTful API. The only problem with this is that GET and possibly POST request methods are already "taken" on your resources and mapped to your "normal" views which return HTML or consume HTTP form POSTs. There are numerous ways to work around this:
register the API views with a separate name, so, say GET /users/123 would return HTML and GET /users/123/json would return a JSON object. Similarly, POST /users/123 would expect HTTP form to be posted and POST /users/123/json would expect JSON. A nice thing about this approach is that you can easily add, say, an XML serializer at GET /users/123/xml.
use custom view predicates so GET /users/123 and GET /users/123?format=json are routed to different views. Actually, there's a built-in request_param predicate for that since Pyramid 1.2
use xhr predicate to differentiate requests based on HTTP_X_REQUESTED_WITH header or accept predicate to differentiate on HTTP_ACCEPT header

Get list of Cache Keys in Django

I'm trying to understand how Django is setting keys for my views. I'm wondering if there's a way to just get all the saved keys from Memcached. something like a cache.all() or something. I've been trying to find the key with cache.has_key('test') but still cant figure out how the view keys are being named.
UPDATE: The reason I need this is because I need to manually delete parts of the cache but dont know the key values Django is setting for my cache_view key

For RedisCache you can get all available keys with.
from django.core.cache import cache
cache.keys('*')

As mentioned there is no way to get a list of all cache keys within django. If you're using an external cache (e.g. memcached, or database caching) you can inspect the external cache directly.
But if you want to know how to convert a django key to the one used in the backend system, django's make_key() function will do this.
https://docs.djangoproject.com/en/1.8/topics/cache/#cache-key-transformation
>>> from django.core.cache import caches
>>> caches['default'].make_key('test-key')
u':1:test-key'

For debugging, you can temporarily switch to LocMemCache instead of PyMemcacheCache:
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'unique-snowflake',
}
}
then see this question:
from django.core.cache.backends import locmem
print(locmem._caches)

In my setup with Django 3.2 there is a method to get a "raw" client for Redis which you can get the keys from.
from django.core.cache import cache
cache.get_client(1).keys()

You can use http://www.darkcoding.net/software/memcached-list-all-keys/ as explained in How do I check the content of a Django cache with Python memcached?

The Memcached documentation recommends that instead of listing all the cache keys, you run memcached in verbose mode and see everything that gets changed. You should start memcached like this
memcached -vv
and then it will print the keys as they get created/updated/deleted.

For Redis Backend
I'm going to add this answer because I landed on this SO question searching for exactly the same question but using a different cache backend. Also with REDIS in particular if you are using the same REDIS server for multiple applications you will want to scope your cache keys with the KEY_PREFIX option otherwise you could end up with cache keys from another application.
My answer is for if you have setup KEY_PREFIX in your settings.py and if you are using either redis_cache.RedisCache or django.core.cache.backends.redis.RedisCache
e.g.
CACHES = {
"default": {
"BACKEND": "redis_cache.RedisCache",
"LOCATION": f"redis://localhost:6379",
"KEY_PREFIX": "my_prefix",
},
}
or
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.redis.RedisCache",
"LOCATION": f"redis://localhost:6379",
"KEY_PREFIX": "my_prefix",
},
}
redis_cache.RedisCache
from django.conf import settings
from django.core.cache import cache
cache_keys = cache.get_client(1).keys(
f"*{settings.CACHES['default']['KEY_PREFIX']}*"
)
django.core.cache.backends.redis.RedisCache
Doing some tests shows that using Django's built in RedisCache may already be scoped but in my case I'm doing it to be explicit. calling .keys("*") will also return keys that belong to celery tasks
from django.conf import settings
from django.core.cache import cache
cache_keys = cache._cache.get_client().keys(
f"*{settings.CACHES['default']['KEY_PREFIX']}*"
)
Bonus: Deleting all app keys
If you want to clear the cache for your specific app instead of ALL the keys in REDIS you'll want to using the prior technique and then call cache.delete_many(cache_keys) instead of cache.clear() as the Django Docs warns that using cache.clear() will remove ALL keys in your cache, not just the ones that are created by your app.

You can use memcached_stats from: https://github.com/dlrust/python-memcached-stats. This package makes it possible to view the memcached keys from within the python environment.

If this is not too out of date, I have had similar issue, due I have had to iterate over whole cache. I managed it, when I add something to my cache like in following pseudocode:
#create caches key list if not exists
if not my_cache.get("keys"):
my_cache.set("keys", [])
#add to my cache
my_cache.set(key, value)
#add key to keys
if key not in my_cache.get("keys"):
keys_list = my_cache.get("keys")
keys_list.append(key)
my_cache.set("keys", keys_list)

this helps.
Ref:
https://lzone.de/blog/How-to%20Dump%20Keys%20from%20Memcache
https://github.com/dlrust/python-memcached-stats
import re, telnetlib, sys
key_regex = re.compile(r"ITEM (.*) \[(.*); (.*)\]")
slab_regex = re.compile(r'STAT items:(.*):number')
class MemcachedStats:
def __init__(self, host='localhost', port='11211'):
self._host = host
self._port = port
self._client = None
#property
def client(self):
if self._client is None:
self._client = telnetlib.Telnet(self._host, self._port)
return self._client
def command(self, cmd):
' Write a command to telnet and return the response '
self.client.write("{}\n".format(cmd).encode())
res = self.client.read_until('END'.encode()).decode()
return res
def slab_ids(self):
' Return a list of slab ids in use '
slab_ids = slab_regex.findall(self.command('stats items'))
slab_ids = list(set(slab_ids))
return slab_ids
def get_keys_on_slab(self, slab_id, limit=1000000):
cmd = "stats cachedump {} {}".format(slab_id, limit)
cmd_output = self.command(cmd)
matches = key_regex.findall(cmd_output)
keys = set()
for match_line in matches:
keys.add(match_line[0])
return keys
def get_all_keys(self):
slab_ids = self.slab_ids()
all_keys = set()
for slab_id in slab_ids:
all_keys.update(self.get_keys_on_slab(slab_id))
return list(all_keys)
def main():
m = MemcachedStats()
print(m.get_all_keys())
if __name__ == '__main__':
main()

There are some weird workarounds you can do to get all keys from the command line, but there is no way to do this with memcached inside of Django. See this thread.