I am trying to implement a type of custom authentication by using aiohttp something like the example in this link but I also need request body. Here is an example for requests:
class CustomAuth(AuthBase):
def __init__(self, secretkey):
self.secretkey = secretkey
def get_hash(self, request):
if request.body:
data = request.body.decode('utf-8')
else:
data = "{}"
signature = hmac.new(
str.encode(self.secretkey),
msg=str.encode(data),
digestmod=hashlib.sha256
).hexdigest().upper()
return signature
def __call__(self, request):
request.headers["CUSTOM-AUTH"] = self.get_hash(request)
return request
I've looked into tracing and BasicAuth but they are useless in my situation. On on_request_start request body is not ready, on on_request_chunk_sent headers have already been sent. A solution like BasicAuth don't have access the request data at all.
Do you have any idea?
Thanks in advance.
I've got a piece of code that I can't figure out how to unit test! The module pulls content from external XML feeds (twitter, flickr, youtube, etc.) with urllib2. Here's some pseudo-code for it:
params = (url, urlencode(data),) if data else (url,)
req = Request(*params)
response = urlopen(req)
#check headers, content-length, etc...
#parse the response XML with lxml...
My first thought was to pickle the response and load it for testing, but apparently urllib's response object is unserializable (it raises an exception).
Just saving the XML from the response body isn't ideal, because my code uses the header information too. It's designed to act on a response object.
And of course, relying on an external source for data in a unit test is a horrible idea.
So how do I write a unit test for this?
urllib2 has a functions called build_opener() and install_opener() which you should use to mock the behaviour of urlopen()
import urllib2
from StringIO import StringIO
def mock_response(req):
if req.get_full_url() == "http://example.com":
resp = urllib2.addinfourl(StringIO("mock file"), "mock message", req.get_full_url())
resp.code = 200
resp.msg = "OK"
return resp
class MyHTTPHandler(urllib2.HTTPHandler):
def http_open(self, req):
print "mock opener"
return mock_response(req)
my_opener = urllib2.build_opener(MyHTTPHandler)
urllib2.install_opener(my_opener)
response=urllib2.urlopen("http://example.com")
print response.read()
print response.code
print response.msg
It would be best if you could write a mock urlopen (and possibly Request) which provides the minimum required interface to behave like urllib2's version. You'd then need to have your function/method which uses it able to accept this mock urlopen somehow, and use urllib2.urlopen otherwise.
This is a fair amount of work, but worthwhile. Remember that python is very friendly to ducktyping, so you just need to provide some semblance of the response object's properties to mock it.
For example:
class MockResponse(object):
def __init__(self, resp_data, code=200, msg='OK'):
self.resp_data = resp_data
self.code = code
self.msg = msg
self.headers = {'content-type': 'text/xml; charset=utf-8'}
def read(self):
return self.resp_data
def getcode(self):
return self.code
# Define other members and properties you want
def mock_urlopen(request):
return MockResponse(r'<xml document>')
Granted, some of these are difficult to mock, because for example I believe the normal "headers" is an HTTPMessage which implements fun stuff like case-insensitive header names. But, you might be able to simply construct an HTTPMessage with your response data.
Build a separate class or module responsible for communicating with your external feeds.
Make this class able to be a test double. You're using python, so you're pretty golden there; if you were using C#, I'd suggest either in interface or virtual methods.
In your unit test, insert a test double of the external feed class. Test that your code uses the class correctly, assuming that the class does the work of communicating with your external resources correctly. Have your test double return fake data rather than live data; test various combinations of the data and of course the possible exceptions urllib2 could throw.
Aand... that's it.
You can't effectively automate unit tests that rely on external sources, so you're best off not doing it. Run an occasional integration test on your communication module, but don't include those tests as part of your automated tests.
Edit:
Just a note on the difference between my answer and #Crast's answer. Both are essentially correct, but they involve different approaches. In Crast's approach, you use a test double on the library itself. In my approach, you abstract the use of the library away into a separate module and test double that module.
Which approach you use is entirely subjective; there's no "correct" answer there. I prefer my approach because it allows me to build more modular, flexible code, something I value. But it comes at a cost in terms of additional code to write, something that may not be valued in many agile situations.
You can use pymox to mock the behavior of anything and everything in the urllib2 (or any other) package. It's 2010, you shouldn't be writing your own mock classes.
I think the easiest thing to do is to actually create a simple web server in your unit test. When you start the test, create a new thread that listens on some arbitrary port and when a client connects just returns a known set of headers and XML, then terminates.
I can elaborate if you need more info.
Here's some code:
import threading, SocketServer, time
# a request handler
class SimpleRequestHandler(SocketServer.BaseRequestHandler):
def handle(self):
data = self.request.recv(102400) # token receive
senddata = file(self.server.datafile).read() # read data from unit test file
self.request.send(senddata)
time.sleep(0.1) # make sure it finishes receiving request before closing
self.request.close()
def serve_data(datafile):
server = SocketServer.TCPServer(('127.0.0.1', 12345), SimpleRequestHandler)
server.datafile = datafile
http_server_thread = threading.Thread(target=server.handle_request())
To run your unit test, call serve_data() then call your code that requests a URL that looks like http://localhost:12345/anythingyouwant.
Why not just mock a website that returns the response you expect? then start the server in a thread in setup and kill it in the teardown. I ended up doing this for testing code that would send email by mocking an smtp server and it works great. Surely something more trivial could be done for http...
from smtpd import SMTPServer
from time import sleep
import asyncore
SMTP_PORT = 6544
class MockSMTPServer(SMTPServer):
def __init__(self, localaddr, remoteaddr, cb = None):
self.cb = cb
SMTPServer.__init__(self, localaddr, remoteaddr)
def process_message(self, peer, mailfrom, rcpttos, data):
print (peer, mailfrom, rcpttos, data)
if self.cb:
self.cb(peer, mailfrom, rcpttos, data)
self.close()
def start_smtp(cb, port=SMTP_PORT):
def smtp_thread():
_smtp = MockSMTPServer(("127.0.0.1", port), (None, 0), cb)
asyncore.loop()
return Thread(None, smtp_thread)
def test_stuff():
#.......snip noise
email_result = None
def email_back(*args):
email_result = args
t = start_smtp(email_back)
t.start()
sleep(1)
res.form["email"]= self.admin_email
res = res.form.submit()
assert res.status_int == 302,"should've redirected"
sleep(1)
assert email_result is not None, "didn't get an email"
Trying to improve a bit on #john-la-rooy answer, I've made a small class allowing simple mocking for unit tests
Should work with python 2 and 3
try:
import urllib.request as urllib
except ImportError:
import urllib2 as urllib
from io import BytesIO
class MockHTTPHandler(urllib.HTTPHandler):
def mock_response(self, req):
url = req.get_full_url()
print("incomming request:", url)
if url.endswith('.json'):
resdata = b'[{"hello": "world"}]'
headers = {'Content-Type': 'application/json'}
resp = urllib.addinfourl(BytesIO(resdata), header, url, 200)
resp.msg = "OK"
return resp
raise RuntimeError('Unhandled URL', url)
http_open = mock_response
#classmethod
def install(cls):
previous = urllib._opener
urllib.install_opener(urllib.build_opener(cls))
return previous
#classmethod
def remove(cls, previous=None):
urllib.install_opener(previous)
Used like this:
class TestOther(unittest.TestCase):
def setUp(self):
previous = MockHTTPHandler.install()
self.addCleanup(MockHTTPHandler.remove, previous)
There is some way to remove HTTP Header 'Server: TwistedWeb/13.1.0' from responses from a Twisted based web application?
You can rewrite any header by calling the request.setHeader method.
class RootPage(Resource):
def getChild(self, name, request):
request.setHeader('server', 'MyVeryOwnServer/1.0')
return OtherResource(name)
The change applies to any resources on your site; you could put it in your Site class. You want that 404 or 500 errors would also return correct header; so you should set it as earlier as possible but not before it is set by twisted itself (in order to overwrite it):
#!/usr/bin/env python
import sys
from twisted.web import server, resource
from twisted.internet import reactor
from twisted.python import log
class Site(server.Site):
def getResourceFor(self, request):
request.setHeader('server', 'Server/1.9E377')
return server.Site.getResourceFor(self, request)
# example from http://twistedmatrix.com/
class HelloResource(resource.Resource):
isLeaf = True
numberRequests = 0
def render_GET(self, request):
self.numberRequests += 1
request.setHeader("content-type", "text/plain")
return "I am request #" + str(self.numberRequests) + "\n"
log.startLogging(sys.stderr)
reactor.listenTCP(8080, Site(HelloResource()))
reactor.run()
Default Server http header is specified in t.w.server.version.
I know this is an old question, but if you would like to remove the server http header. I am talking about the
request.setHeader('Server', 'SomeServer')
This is set by Twisted Web automagically if you don't specify a value. You can remove it by using the inner Headers class. For example,
request.responseHeaders.removeHeader('Server')
This will remove the Server Http Header.
I am using web.py to return a protocol buffer response from a post request and response time is critical. I have some writes to redis that I would like to do after the post response. rather than before.
r = redis.StrictRedis(host='localhost', port=6379, db=0)
class index:
def POST(self):
return pPbuffer
r.set('a','b')
So, how can I modify the code so I can I can return as quickly as possible but doing post cleanup (no pun intended).
Thanks
If you are using wsgi or something as server you could use yield to generate contents time after time and the browser will receive them in sort.
For your example:
class index:
def POST(self):
yield pPbuffer
r.set('a','b')
And this is a good example which is doing it this way.
Having a code inspired from http://code.djangoproject.com/wiki/XML-RPC :
from SimpleXMLRPCServer import SimpleXMLRPCDispatcher
from django.http import HttpResponse
dispatcher = SimpleXMLRPCDispatcher(allow_none=False, encoding=None) # Python 2.5
def rpc_handler(request):
"""
the actual handler:
if you setup your urls.py properly, all calls to the xml-rpc service
should be routed through here.
If post data is defined, it assumes it's XML-RPC and tries to process as such
Empty post assumes you're viewing from a browser and tells you about the service.
"""
if len(request.POST):
response = HttpResponse(mimetype="application/xml")
response.write(dispatcher._marshaled_dispatch(request.raw_post_data))
else:
pass # Not interesting
response['Content-length'] = str(len(response.content))
return response
def post_log(message = "", tags = []):
""" Code called via RPC. Want to know here the remote IP (or hostname). """
pass
dispatcher.register_function(post_log, 'post_log')
How could get the IP address of the client within the "post_log" definition?
I have seen IP address of client in Python SimpleXMLRPCServer? but can't apply it to my case.
Thanks.
Ok I could do it ... with some nifty tips ...
First, I created my own copy of SimpleXMLRPCDispatcher which inherit everything from it and overides 2 methods :
class MySimpleXMLRPCDispatcher (SimpleXMLRPCDispatcher) :
def _marshaled_dispatch(self, data, dispatch_method = None, request = None):
# copy and paste from /usr/lib/python2.6/SimpleXMLRPCServer.py except
response = self._dispatch(method, params)
# which becomes
response = self._dispatch(method, params, request)
def _dispatch(self, method, params, request = None):
# copy and paste from /usr/lib/python2.6/SimpleXMLRPCServer.py except
return func(*params)
# which becomes
return func(request, *params)
Then in my code, all to do is :
# ...
if len(request.POST):
response = HttpResponse(mimetype="application/xml")
response.write(dispatcher._marshaled_dispatch(request.raw_post_data, request = request))
# ...
def post_log(request, message = "", tags = []):
""" Code called via RPC. Want to know here the remote IP (or hostname). """
ip = request.META["REMOTE_ADDR"]
hostname = socket.gethostbyaddr(ip)[0]
That's it.
I know it's not very clean... Any suggestion for cleaner solution is welcome!