I have a class which makes requests to a remote API. I'd like to be able to reduce the number of calls I'm making. Some of the methods in my class make the same API calls (but for different reasons), so I'ld like the ability for them to 'share' a cached API response.
I'm not entirely sure if it's more Pythonic to use optional parameters or to use multiple methods, as the methods have some required parameters if they are making an API call.
Here are the approches as I see them, which do you think is best?
class A:
def a_method( item_id, cached_item_api_response = None):
""" Seems awkward having to supplied item_id even
if cached_item_api_response is given
"""
api_response = None
if cached_item_api_response:
api_response = cached_item_api_response
else:
api_response = ... # make api call using item_id
... #do stuff
Or this:
class B:
def a_method(item_id = None, cached_api_response = None):
""" Seems awkward as it makes no sense NOT to supply EITHER
item_id or cached_api_response
"""
api_response = None
if cached_item_api_response:
api_response = cached_item_api_response
elif item_id:
api_response = ... # make api call using item_id
else:
#ERROR
... #do stuff
Or is this more appropriate?
class C:
"""Seems even more awkward to have different method calls"""
def a_method(item_id):
api_response = ... # make api call using item_id
api_response_logic(api_response)
def b_method(cached_api_response):
api_response_logic(cached_api_response)
def api_response_logic(api_response):
... # do stuff
Normally when writing method one could argue that a method / object should do one thing and it should do it well. If your method get more and more parameters which require more and more ifs in your code that probably means that your code is doing more then one thing. Especially if those parameters trigger totally different behavior. Instead maybe the same behavior could be produced by having different classes and having them overload methods.
Maybe you could use something like:
class BaseClass(object):
def a_method(self, item_id):
response = lookup_response(item_id)
return response
class CachingClass(BaseClass):
def a_method(self, item_id):
if item_id in cache:
return item_from_cache
return super(CachingClass, self).a_method(item_id)
def uncached_method(self, item_id)
return super(CachingClass, self).a_method(item_id)
That way you can split the logic of how to lookup the response and the caching while also making it flexible for the user of the API to decide if they want the caching capabilities or not.
There is nothing wrong with the method used in your class B. To make it more obvious at a glance that you actually need to include either item_id or cached_api_response, I would put the error checking first:
class B:
def a_method(item_id = None, cached_api_response = None):
"""Requires either item_id or cached_api_response"""
if not ((item_id == None) ^ (cached_api_response == None)):
#error
# or, if you want to allow both,
if (item_id == None) and (cached_api_response == None):
# error
# you don't actually have to do this on one line
# also don't use it if cached_item_api_response can evaluate to 'False'
api_response = cached_item_api_response or # make api call using item_id
... #do stuff
Ultimately this is a judgement that must be made for each situation. I would ask myself, which of these two more closely fits:
Two completely different algorithms or actions, with completely different semantics, even though they may be passed similar information
A single conceptual idea, with consistent semantics, but with nuance based on input
If the first is closest, go with separate methods. If the second is closest, go with optional arguments. You might even implement a single method by testing the type of the argument(s) to avoid passing additional arguments.
This is an OO anti-pattern.
class API_Connection(object):
def do_something_with_api_response(self, response):
...
def do_something_else_with_api_response(self, response):
...
You have two methods on an instance and you're passing state between them explicitly? Why are these methods and not bare functions in a module?
Instead, think about using encapsulation to help you by having the instance of the class own the api response.
For example:
class API_Connection(object):
def __init__(self, api_url):
self._url = api_url
self.cached_response = None
#property
def response(self):
"""Actually use the _url and get the response when needed."""
if self._cached_response is None:
# actually calculate self._cached_response by making our
# remote call, etc
self._cached_response = self._get_api_response(self._url)
return self._cached_response
def _get_api_response(self, api_param1, ...):
"""Make the request and return the api's response"""
def do_something_with_api_response(self):
# just use self.response
do_something(self.response)
def do_something_else_with_api_response(self):
# just use self.response
do_something_else(self.response)
You have caching and any method which needs this response can run in any order without making multiple api requests because the first method that needs self.response will calculate it and every other will use the cached value. Hopefully it's easy to imagine extending this with multiple URLs or RPC calls. If you have a need for a lot of methods that cache their return values like response above then you should look into a memoization decorator for your methods.
The cached response should be saved in the instance, not passed around like a bag of Skittles -- what if you dropped it?
Is item_id unique per instance, or can an instance make queries for more than one? If it can have more than one, I'd go with something like this:
class A(object):
def __init__(self):
self._cache = dict()
def a_method( item_id ):
"""Gets api_reponse from cache (cache may have to get a current response).
"""
api_response = self._get_cached_response( item_id )
... #do stuff
def b_method( item_id ):
"""'nother method (just for show)
"""
api_response = self._get_cached_response( item_id )
... #do other stuff
def _get_cached_response( self, item_id ):
if item_id in self._cache:
return self._cache[ item_id ]
response = self._cache[ item_id ] = api_call( item_id, ... )
return response
def refresh_response( item_id ):
if item_id in self._cache:
del self._cache[ item_id ]
self._get_cached_response( item_id )
And if you may have to get the most current info about item_id, you can have a refresh_response method.
Related
I'm trying to unit test a function with nested for loop using the Mock module in Python:
main.py:
#dataclass
Helper_class():
Attr1: Set[some_object]
Attr2: Set[str]
def fetch_rule_for(api, id):
return some rule_objects based on id
def my_func(list_ids):
api = create some api connection
result_dict = dict()
for id in list_ids:
Attr1_set = set()
Attr2_set = set()
for rule in fetch_rule_for(api, id):
if any(a.name == 'some_action_name' for a in rule.actions):
Attr2_set.add(rule.attr1)
Attr1_set.add(rule)
result_dict[id] = Helper_class(Attr1_set, Attr2_set)
return result_dict
Rule is a class object with method actions, action is another class object with attribute name.
Two questions I'm struggling with:
(1) How do I pacth the return value of function fetch_rule_for(api, id) where the return is a complicated class object?
(2) How do I deal with the for loops in Python using unit test? I have seen mock.call_count mentioned but can someone please get into more details or point me to relevant resources?
I'm new to unit testing in Python so any help is much appreicated!
If you want to patch the return value of a class just instantiate the class and manually set any necessary class variables then it's just a matter of using configure_mock(return_value=class_object)
Here is an example with requests.Response:
resp = Response()
resp.status_code = 200
resp._content = json.dumps({'token': 'xyz123'})
mock_post.configure_mock(return_value=resp)
In the case of the for loop you likely want to mock any returns contained within using side_effect
#mock.patch('your_class.fetch_rule_for', side_effect=iter([1,2,3]))
I'm attempting to create a few unit tests for my class. I want to mock these, so that I don't burn through my API quota running some of these tests. I have multiple test cases that will call the fetch method, and depending on the passed URL I'll get different results back.
My example class looks like this:
import requests
class ExampleAPI(object):
def fetch(self, url, params=None, key=None, token=None, **kwargs):
return requests.get(url).json() # Returns a JSON string
The tutorial I'm looking at shows that I can do something like this:
import unittest
from mock import patch
def fake_fetch_test_one(url):
...
class TestExampleAPI(unittest.TestCase):
#patch('mymodule.ExampleAPI.fetch', fake_fetch_test_one)
def test_fetch(self):
e = ExampleAPI()
self.assertEqual(e.fetch('http://my.api.url.example.com'), """{'result': 'True'}""")
When I do this, though, I get an error that says:
TypeError: fake_fetch_test_one() takes exactly 1 argument (3 given)
What is the proper way to mock a requests.get call that is in a method in my class? I'll need the ability to change the mock'd response per test, because different URLs can provide different response types.
Your fake fetch needs to accept the same arguments as the original:
def fake_fetch(self, url, params=None, key=None, token=None, **kwargs):
Note that it's better to mock just the external interface, which means letting fetch call requests.get (or at least, what it thinks is requests.get):
#patch('mymodule.requests.get')
def test_fetch(self, fake_get):
# It would probably be better to just construct
# a valid fake response object whose `json` method
# would return the right thing, but this is a easier
# for demonstration purposes. I'm assuming nothing else
# is done with the response.
expected = {"result": "True"}
fake_get.return_value.json.return_value = expected
e = ExampleAPI()
self.assertEqual(e.fetch('http://my.api.url.example.com'), expected)
from you test method you can monkeypatch your requests module
import unittest
class Mock:
pass
ExampleAPI.requests = Mock()
def fake_get_test_one(url):
/*returns fake get json */
ExampleAPI.requests.get= Mock()
ExampleAPI.requests.json = fake_get_test_one
class TestExampleAPI(unittest.TestCase):
def test_fetch(self):
e = ExampleAPI()
self.assertEqual(e.fetch('http://my.api.url.example.com'), """{'result': 'True'}""")
you can setup the patch in each setup() and corresponding teardown() methods of your test class if needed
I'm trying to implement a wrapper around a redis database that does some bookkeeping, and I thought about using descriptors. I have an object with a bunch of fields: frames, failures, etc., and I need to be able to get, set, and increment the field as needed. I've tried to implement an Int-Like descriptor:
class IntType(object):
def __get__(self,instance,owner):
# issue a GET database command
return db.get(my_val)
def __set__(self,instance,val):
# issue a SET database command
db.set(instance.name,val)
def increment(self,instance,count):
# issue an INCRBY database command
db.hincrby(instance.name,count)
class Stream:
_prefix = 'stream'
frames = IntType()
failures = IntType()
uuid = StringType()
s = Stream()
s.frames.increment(1) # float' object has no attribute 'increment'
Is seems like I can't access the increment() method in my descriptor. I can't have increment be defined in the object that the __get__ returns. This would require an additional db query if all I want to do is increment! I also don't want increment() on the Stream class, as later on when I want to have additional fields like strings or sets in Stream, then I'd need to type check the heck out of everything.
Does this work?
class Stream:
_prefix = 'stream'
def __init__(self):
self.frames = IntType()
self.failures = IntType()
self.uuid = StringType()
Why not define the magic method iadd as well as get and set. This will allow you to do normal addition with assignment on the class. It will also mean you can treat the increment separately from the get function and thereby minimise the database accesses.
So change:
def increment(self,instance,count):
# issue an INCRBY database command
db.hincrby(instance.name,count)
to:
def __iadd__(self,other):
# your code goes here
Try this:
class IntType(object):
def __get__(self,instance,owner):
class IntValue():
def increment(self,count):
# issue an INCRBY database command
db.hincrby(self.name,count)
def getValue(self):
# issue a GET database command
return db.get(my_val)
return IntValue()
def __set__(self,instance,val):
# issue a SET database command
db.set(instance.name,val)
I have a main page that has a GET and a POST function. The POST function gets data from a search screen and should pass this information, via an ajax call, to the worldMarkers class. This is separate because it will be needed for other aspects of the application.
The goal of this, is to have a user press submit on index and during the POST call, it is able to limit the results retrieved. This logic exists in the worldMarkers class.
class index(object):
def GET(self):
# do things to generate the page
return html
def POST(self):
continents = web.input(search_continents=[])
countries = web.input(search_countries=[])
searchDict = {}
if continents['search_continents']:
searchDict['continents'] = continents['search_continents']
if countries['search_countries']:
searchDict['countries'] = countries['search_countries']
markers = worldMarkers()
# Yes, this just spits out the results, nothing fancy right now
return markers.GET()
#alternatively,
#return markers.GET(searchDict)
class worldMarkers(object):
def __init__(self, **kargs):
self.searchDict = None
if 'searchDict' in kargs:
self.searchDict = kargs['searchDict']
def GET(self):
print "SearchDict: %s" % (self.searchDict)
# No searchDict data exists
The first option, with no parameters to markers.GET() means that none of my search criteria has been passed. If I do markers.GET(searchDict), I receive this error:
<type 'exceptions.TypeError'> at /
GET() takes exactly 1 argument (2 given)
How can I pass my search parameters to the worldMarkers class?
It looks like you should actually create an instance of worldMarkers as follows in order for your searchDict to exist:
markers = worldMarkers(searchDict=searchDict)
Right now, you're creating it without the argument:
markers = worldMarkers()
and in that case, the condition if 'searchDict' in kargs is false and self.searchDict = kargs['searchDict'] is not run.
And, as #TyrantWave points out, your GET is not really prepared to take any arguments since it is only declared as def GET(self). See the last sample code of this section of the docs.
I have the following function signature, and it looks really ugly, what can I do to make it look cleaner ?
def contact(
request, sender=settings.DEFAULT_FROM_EMAIL,
subj_tmpl='contato/subject.txt',msg_tmpl='contato/msg.html',
template='contato/contato.html', success_template='contato/success.html',
success_redir='/',append_message=None,):
if i were you i think i will do it like this:
def contact(request, sender=None, append_message=None, context=None):
if not sender:
sender = settings.DEFAULT_FROM_EMAIL # i hope that you can access settings here
# The context arg is a dictionary where you can put all the others argument and
# you can use it like so :
subj_tmpl = context.get('subj_tmpl', 'contato/subject.txt')
# ....
hope this will help you.
My proposal is to drop parameters. Do you really need to be able to specify all the templates separately? Wouldn't it be sufficient to just specify the template folder, and then mandate that it has subject.txt, msg.html, etc in it?
If you just want to improve readability, reformat it to have one parameter per line:
def contact(
request,
sender=settings.DEFAULT_FROM_EMAIL,
subj_tmpl='contato/subject.txt',
msg_tmpl='contato/msg.html',
template='contato/contato.html',
success_template='contato/success.html',
success_redir='/',
append_message=None,):
This will allow a reader to more quickly grasp what the parameter names are.
You could rewrite it as:
def contact( request, **kargs):
try:
sender = kwargs.pop ('sender')
except KeyError:
sender=settings.DEFAULT_FROM_EMAIL
try:
subj_tmpl = kwargs.pop ('subj_tmpl')
except KeyError:
subj_tmpl='contato/subject.txt'
# ...
# and so on
# ...
def contact(request, **kwargs):
sender = kwargs.get('sender', settings.DEFAULT_FROM_EMAIL)
subj_template = kwargs.get('subj_template', 'contato/subject.txt')
..
With that said, I think your current solution is waaay better than using **kwargs.
this does not seem so ugly to me: you have a function, you have enough parameters to modify the way the function behave, and you have sensible default values so that you don't need to specify all arguments at each function call.
there is the possibility to package the function in a class: in the class constructor, you specify all those values which are part of the parameter list, and you have a special method without arguments to execute the core feature.
something like this:
class ContactForm(object):
def __init__( self,
subj_tmpl='contato/subject.txt',
msg_tmpl='contato/msg.html',
template='contato/contato.html',
success_template='contato/success.html',
success_redir='/',
append_message=None):
self.subj_tmpl = subj_tmpl
self.msg_tmpl = msg_tmpl
self.template = template
self.success_template = success_template
self.success_redir = success_redir
self.append_message = append_message
def __call__( self, request, sender=settings.DEFAULT_FROM_EMAIL ):
# do something
# use case:
contact = ContactForm()
contact( req, sndr )
(i guessed which values is site specific and which is user specific from the name of the parameters. i don't know your specific application, adapt it the way you want)