We Keep Coding

Mocking os.environ with python unittests

I am trying to test a class that handles for me the working directory based on a given parameter. To do so, we are using a class variable to map them.
When a specific value is passed, the path is retrieved from the environment variables (See baz in the example below). This is the specific case that I'm trying to test.
I'm using Python 3.8.13 and unittest.
I'm trying to avoid:
I don't want to mock the WorkingDirectory.map dictionary because I want to make sure we are fetching from the environ with that particular variable (BAZ_PATH).
Unless is the only solution, I would like to avoid editing the values during the test, i.e I would prefer not to do something like: os.environ["baz"] = DUMMY_BAZ_PATH
What I've tried
I tried mocking up the environ as a dictionary as suggested in other publications, but I can't make it work for some reason.
# working_directory.py
import os
class WorkingDirectory:
map = {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH"),
}
def __init__(self, env: str):
self.env = env
self.path = self.map[self.env]
#property
def data_dir(self):
return os.path.join(self.path, "data")
# Other similar methods...
Test file:
# test.py
import os
import unittest
from unittest import mock
from working_directory import WorkingDirectory
DUMMY_BAZ_PATH = "path/to/baz"
class TestWorkingDirectory(unittest.TestCase):
#mock.patch.dict(os.environ, {"BAZ_PATH": DUMMY_BAZ_PATH})
def test_controlled_baz(self):
wd = WorkingDirectory("baz")
self.assertEqual(wd.path, DUMMY_BAZ_PATH)
Error
As shown in the error, os.environ doesn't seem to be properly patched as it returns Null.
======================================================================
FAIL: test_controlled_baz (test_directory_structure_utils.TestWorkingDirectory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "~/.pyenv/versions/3.8.13/lib/python3.8/unittest/mock.py", line 1756, in _inner
return f(*args, **kw)
File "~/Projects/dummy_project/tests/unit/test_directory_structure_utils.py", line 127, in test_controlled_baz
self.assertEqual(wd.path, DUMMY_BAZ_PATH)
AssertionError: None != 'path/to/baz'
----------------------------------------------------------------------
Ran 136 tests in 0.325s
FAILED (failures=1, skipped=5)
This seems to be because the BAZ_PATH doesn't exist actually. However, I would expect this to be OK since is being patched.
When, in the mapping dictionary, "baz": os.environ.get("BAZ_PATH"), I repalce BAZ_PATH for a variable that actually exist in my environment, i.e HOME, it returns the actual value of HOME instead of the DUMMY_BAZ_PATH, which lead me to think that I'm definetely doing something wrong patching
AssertionError: '/Users/cestla' != 'path/to/baz'
Expected result
Well, obviously, I am expecting the test_controlled_baz passes succesfully.

So the problem is that you added map as a static variable.
Your patch works correctly as you can see here:
patch actually works
The problem is that when it runs it's already too late because the map variable was already calculated (before the patch).
If you want you can move it to the init function and it will function correctly:
class WorkingDirectory:
def __init__(self, env: str):
self.map = {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH")
}
self.env = env
self.path = self.map[self.env]
If for some reason you wish to keep it static, you have to also patch the object itself.
writing something like this will do the trick:
class TestWorkingDirectory(unittest.TestCase):
#mock.patch.dict(os.environ, {"BAZ_PATH": DUMMY_BAZ_PATH})
def test_controlled_baz(self):
with mock.patch.object(WorkingDirectory, "map", {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH")
}):
wd = WorkingDirectory("baz")
self.assertEqual(wd.path, DUMMY_BAZ_PATH)

That's not directly answer to your question but a valid answer either way imo:
Don't try to patch that (it's possible, but harder and cumbersome).
Use config file for your project.
e.g. use pyproject.toml and inside configure the pytest extension:
[tool.pytest.ini_options]
env=[
"SOME_VAR_FOR_TESTS=some_value_for_that_var"
]

How can I access the Kamailio configuration file from the KEMI script?

I need to access defines and variables from the Kamailio configuration file in my Python script. So far, I am able to access the global variables only through self.my_var = int(KSR.pv.get("$sel(cfg_get.my_group.my_var)")) where this variable is defined in the configuration file as my_group.my_var = 664. How can I access the definitions (to know that #!ifdef MY_DEFINE succeeds or not)? Or at least the configuration file to read them myself ?
I found nothing about on the official documentation and even $sel(cfg_get.my_group.my_var) I found it elsewhere.
UPDATE These values are not normally available at runtime (preprocessing), so the native code can use them like this:
#!define WITH_SIPTRACE
#!substdef "!SERVER_ID!654!g"
...
request_route {
#!ifdef WITH_SIPTRACE
xlog("L_NOTICE", "$C(yx)========= server_id=SERVER_ID NEW $rm $pr:$si:$sp$C(xx)\n");
#!endif
...
Can this same behavior be achieved in Python ?
UPDATE 2 Found partial answer (below) in KSR.kx.get_def() and KSR.kx.get_defn().

You should not change defines. There is no mechanism for that.
To make configurable parameters, use database or curl module + in-memory cache via htable module.
Here is htable optimization for auth, for example
AUTH WITH CACHING
# authentication with password caching using htable
modparam("htable", "htable", "auth=>size=10;autoexpire=300;")
modparam("auth_db", "load_credentials", "$avp(password)=password")
route[AUTHCACHE] {
if($sht(auth=>$au::passwd)!=$null) {
if (!pv_auth_check("$fd", "$sht(auth=>$au::passwd)", "0", "1")) {
auth_challenge("$fd", “1”);
exit;
}
} else {
# authenticate requests
if (!auth_check("$fd", "subscriber", "1")) {
auth_challenge("$fd", "0");
exit;
}
$sht(auth=>$au::passwd) = $avp(password);
}
# user authenticated - remove auth header
if(!is_method("REGISTER|PUBLISH"))
consume_credentials();
}
Here is how to get info from db using sql.
https://kamailio.org/docs/modules/5.0.x/modules/avpops.html#avpops.f.avp_db_query
avp_db_query("select password, ha1 from subscriber where username='$tu'",
"$avp(i:678);$avp(i:679)");
As opensource sample of such optimization you can see code of kazoo project.

A solution I dislike because it adds extra variables for each interesting definition.
On the configuration side:
####### Custom Parameters #########
/*
Is there a better method to access these flags (!define)?
For the constants (!substdef) there are KSR.kx.get_def(), KSR.kx.get_defn().
*/
#!ifdef WITH_OPTIONA
my_group.option_a = yes
#!else
my_group.option_a = no
#!endif
#!ifdef WITH_OPTIONB
my_group.option_b = yes
#!else
my_group.option_b = no
#!endif
On the Python side:
def __init__(self):
self.my_domain = KSR.kx.get_def("MY_DOMAIN")
self.server_id = KSR.kx.get_defn("SERVER_ID")
assert self.my_domain and self.server_id
self.flags = None
self.initialized = False
def real_init(self):
'''
Object data is initialized here, dynamically,
because__init__ and child_init are called too early.
'''
def read_flags():
flags = {}
for i in ['option_a', 'option_b']:
value = bool(int(KSR.pv.get("$sel(cfg_get.my_group.{})".format(i))))
if value:
flags[i[len('option_'):]] = True
return flags
self.flags = read_flags()
self.initialized = True
Update Partial solution found: !substdef constants, but not !define ones.
Update 5.5.2 KSR.kx.ifdef() and KSR.kx.ifndef() added on the master branch, but not yet exported in a official release (the latest being 5.5.2)
See kemix: added KSR.kx.ifdef() and KSR.kx.ifndef().

How would I separate my Python File to multiple plugins?

So first thing I want to say: I have been looking into modules and such, I just don't quiet know how I would rewrite it to fit this in.
Project: What I have is a Skype Bot using the Skype4Py module. I have about 11 commands I noticed the one script is getting a little large.
I'm trying to think about how to link one main.py file to a Plugin Folder, which contains each and every bot function in it's own respectable Python file. It sounds simple an all, except when it comes to how the function is called.
Here is just a basic look at my Skype bot, missing some of the larger functions.
import Skype4Py, random
class SkypeBot():
def __init__(self):
self.skype = Skype4Py.Skype()
if self.skype.Client.IsRunning == False:
self.skype.Client.Start()
self.skype.Attach()
self.results = ['Yes', 'No', 'Maybe', 'Never']
def main(self):
print ' Skype Bot currently running on user: %s' % self.skype.CurrentUserHandle
print "\n\nCommands Called:\n"
while True:
self.skype.OnMessageStatus = self.RunFunction
def RunFunction(self, Message, Status):
if Status == 'SENT' or Status == 'RECEIVED':
cmd = Message.Body.split(' ')[0]
if cmd in self.functions.keys():
self.context = Message
self.caller = self.context.FromHandle
self.functions[cmd](self)
def ping(self):
print " %s : Ping" % self.caller
self.context.Chat.SendMessage('Pong')
def say(self):
try:
response = self.context.Body.split(' ', 1)
if response[1] == "-info":
print " %s : say -info" % self.caller
self.context.Chat.SendMessage("Resends the message entered. \n"
"Usage: !say Hello. \n"
"Example: Bot: Hello.")
else:
say = response[1]
print " %s : Say [%s]" % (self.caller, say)
self.context.Chat.SendMessage(say)
except:
self.context.Chat.SendMessage("Please use -info to properly use the !say command")
def eightball(self):
try:
question = self.context.Body.split(' ', 1)
if question[1] == "-info":
print " %s : 8Ball -info" % self.caller
self.context.Chat.SendMessage("Responds with an answer.\n"
"Usage: !8ball 'Do I have swag?'\n"
"Example: !8Ball Response: 'Yes'")
else:
random.shuffle(self.results)
answer = self.results[3]
print " %s : 8Ball [%s]" % (self.caller, question[1])
self.context.Chat.SendMessage("!8Ball Response: %s" % answer)
except:
self.context.Chat.SendMessage("Please use -info to properly use the !8ball command")
#FUNCTIONS LIST
#********************
functions = {
"!ping": ping,
"!say": say,
"!8ball": eightball,
}
if __name__ == "__main__":
snayer = SkypeBot()
snayer.main()
So basically, what I am wondering, how can I change
self.skype.OnMessageStatus = self.RunFunction
so that it'll run functions from another file?

For a program of this size it's not really necessary to put your command functions into separate files, but I guess it is good organization. And good practice for when you write a program that has thousands of lines of code. :)
One way to do this is to create a basic SkypeBot class without any command methods and then import the command methods from your plugins directory and add them to the class. It's easy enough to add new attributes to an existing class, and it doesn't matter if the new attributes are properties or methods, the syntax to add them is identical. (With a tiny bit more work it's even possible to add new attributes to an instance, so you can have multiple instances, each with their own individual set of commands. But I guess that's not necessary here, since a program that uses the SkypeBot class will normally only create a single instance).
So we can break your question into two parts:
How to add methods to an existing class.
How to import those methods
from other source files.
As I said, 1) is easy. 2) is quite easy as well, but I've never done it before, so I had to do a little bit of research and testing, and I can't promise that what I've done is best practice, but it works. :)
I don't know much about Skype, and I don't have that Skype4Py module, and as you said, the code above is not the complete program, so I've written some fairly simple code to illustrate the process of adding plugin methods from separate files to an existing class.
The name of the main program is "plugin_demo.py". To keep things neat, it lives in its own directory, "plugintest/", which you should create somewhere in your Python path (eg where you normally keep your Python programs). This path must be specified in your PYTHONPATH environment variable.
"plugintest/" has the following structure:
plugintest/
__init__.py
plugin_demo.py
plugins/
__init__.py
add.py
multiply.py
The __init__.py files are used by Python's import machinery to let it know that a directory contains a Python package, see 6.4. Packages in the Python docs for further details.
Here are the contents of those files. Firstly, the files that go into "plugintest/" itself:
__init__.py
__all__ = ['plugin_demo', 'plugins']
from plugintest import *
plugin_demo.py
#! /usr/bin/env python
#A simple class that will get methods added later from plugins directory
class Test(object):
def __init__(self, data):
self.data = data
def add_plugins(cls):
import plugins
print "Adding plugin methods to %s class" % cls.__name__
for name in plugins.__all__:
print name
plug = getattr(plugins, name)
print plug
method = getattr(plug, name)
print method
setattr(cls, name, method)
print
print "Done\n"
add_plugins(Test)
def main():
#Now test it!
t = Test([1, 2, 3]); print t.data
t.multiply(10); print t.data
t.add(5); print t.data
if __name__ == '__main__':
main()
And now the contents of the "plugintest/plugins/" directory:
__init__.py
__all__ = ['add', 'multiply']
from plugintest.plugins import *
add.py
#A method for the Test class of plugin_demo.py
def add(self, m):
self.data = [m + i for i in self.data]
multiply.py
#A method for the Test class of plugin_demo.py
def multiply(self, m):
self.data = [m * i for i in self.data]
If you cd to the directory containing the "plugintest/" folder, you should be able to run it with
python plugintest/plugin_demo.py
and if you cd to "plugintest/" itself
python plugin_demo.py
Also, in the interpreter (or another Python program), you should be able to do
import plugintest
and then run the main() function of "plugin_demo.py" with
plugintest.plugin_demo.main()
The other usual variations of from ... import ... etc should also work as expected.
The function in "plugin_demo.py" that performs the magic of adding the imported methods to the Test class is add_plugins(). When it runs it prints out each method name, its module, and its function. This could be handy during development, but you'd probably comment out some of those print statements once the program's working properly.
I hope this helps, and if you have any questions please don't hesitate to ask.

Python Mock not correctly setting return value

I am attempting to build unit tests and have been using mock, However upon using two patch statements, I was not able to set the proper return values.
#patch('pulleffect.lib.google.gcal_helper.validate_and_refresh_creds')
#patch('pulleffect.lib.google.gcal_helper.get_google_creds')
def test_get_calendar_list_for_gcalhelper_without_credentials(self,
mock_get_google_creds,
mock_validate_and_refresh_creds):
mock_validate_and_refresh_creds = "redirect"
mock_get_google_creds = "credentials"
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
however the assert fails and instead of the expected string redirect I instead get
<MagicMock name = "validate_and_refresh_creds() id = 14054613955344>
I was wondering what is necessary to have redirect returned instead. I have not encountered this issue when only patching a single method.

I was able to fix the issue of
<MagicMock name = "foo()" id = number>
incorrectly appearing by replacing my earlier code with:
from mock import MagicMock
def test_get_calendar_list_for_gcalhelper_without_credentials(self):
rtn = { "redirect": "/gcal/authenticate"}
pulleffect.lib.google.gcal_helper.validate_and_refresh_creds = MagicMock(name = "sup", return_value = rtn)
pulleffect.lib.google.gcal_helper.get_google_creds = MagicMock(name = "sup2", return_value = "redirect")
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
this allowed the return values to be properly set.

mock_get_google_creds and mock_validate_and_refresh_creds created with patch decorator are ordinary mock objects (Mock or MagicMock). Direct assignment is not the correct way to set return values. Use return_value attribute:
mock_validate_and_refresh_creds.return_value = "redirect"
Also you can set it during patching:
patch takes arbitrary keyword arguments. These will be passed to the
Mock (or new_callable) on construction.
#patch('pulleffect.lib.google.gcal_helper.get_google_creds', return_value="redirect")

I recommend you to use this solution. You should move your functions to helper class and instead static methods user class methods, because it's possible to mock class in this way.
class GCallHelper(object):
#classmethond
def validate_and_refresh(cls):
...
return result
def test_get_calendar_list_for_gcalhelper_without_credentials(self):
with patch('pulleffect.lib.google.gcal_helper') as mocked_gcal:
mocked_gcal.return_value.validate_and_refresh_creds.return_value = 'redirect'
mocked_gcal.return_value.get_google_creds.return_value = 'credentials'
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
p.s. And you forgot 'return_value' in your example.

python django mock cache

In my settings.py , I have specified my cache as :
CACHES = {
'default': {
......
}
}
In my views.py, I have
import requests
from django.core.cache import cache, get_cache
def aview():
#check cache
if not get_cache('default').get('key'):
#make request and save in cache
result = request.get('some_url')
get_cache('default').set('key', result)
return result
else:
return get_cache('default').get('key')
Now in my tests.py, I have been able to mock requests.get('aurl'), so that makes sure that no external requests are made.
But the test code still hits the cache and gets/sets from it. So if my prod has already set the cache, then test is failing because it gets the data from same cache. Or if I run my tests first, then the test case is setting the cache with test data and I see that same reflected when I run prod website.
How can I mock the calls to
get_cache('default').set('key', result)
and
get_cache('default').get('key')
so that the set call does not sets the real cache ( return None?) and get does not return anything in actual cache.
Please provide me with code sample to how to get this done.
Here is how I have mocked my requests.get
def test_get_aview(self):
with mock.patch('requests.get') as mymock:
mymock.side_effect = (lambda url: MOCKED_DATA[url])
What code can I put after this to make it work? I tried something like
class MockCacheValue(mock.MagicMock):
def get(self, key):
print 'here'
return None
def set(self, key, value):
print 'here 2'
pass
def test_get_aview(self):
with mock.patch('requests.get') as mymock:
mymock.side_effect = (lambda url: MOCKED_DATA[url])
mock.patch('django.core.cache.get_cache', new=MockCacheValue)
but it does not work and putting a print statement inside get/set above does not print anything giving me an idea that its not mocked properly

I think you should use dummy cache while running tests by:
overriding settings in test cases, see docs
checking what cache backend to use while testing right in settings.py:
CACHES = ...
if 'test' in sys.argv:
CACHES['default'] = {'BACKEND': 'django.core.cache.backends.dummy.DummyCache',}
having a separate settings.py for testing
mocking, see good article on how to do it
Hope that helps.