We Keep Coding

How to mock response from integrated service while testing server

The long story short is I am working on building a server that serves as something as a chat bot. The server uses google dialog flow. Right now I have an endpoint exposed that allows me to talk to my server, when I hit that endpoint, google auth, as well as google dialog flow gets called. I am attempting to mock the response of dialog flow while leaving the actual server to respond to the network call. As of now my test looks like this.
This is my base test file:
import unittest
import mock
class BaseTest(unittest.TestCase, object):
def __init__(self, *args, **kwargs):
super(BaseTest, self).__init__(*args, *kwargs)
def auto_patch(self, patch_target):
patcher = mock.patch(patch_target)
patched = patcher.start()
self.addCleanup(patcher.stop)
return patched
This is my test file:
import json
import uuid
from os import path
from tests.base_test import BaseTest
from agent.api_service import app
import requests_mock
import pytest
from hamcrest import assert_that, has_items, equal_to
CWD = path.dirname(path.realpath(__file__))
class TestAudio(BaseTest):
def test__interact__full_no_stt(self):
payload = json.load(open("tests/json_payloads/test__interact__full_audio.json"))
u_session_id = str(uuid.uuid1())
payload["session_id"] = u_session_id
#mock a 500 back from STT
with open("tests/json_payloads/stt_500.json", "r") as issues_file:
mock_response = issues_file.read()
with requests_mock.Mocker() as m:
m.register_uri('POST', 'https://speech.googleapis.com/v1/speech:recognize', text=mock_response)
request, response = app.test_client.post("/agent_service/interact", data=json.dumps(payload))
self.assertEqual(200, response.status)
This is my google stt file:
import json
import requests
from agent.exceptions import GoogleSTTException
from agent.integrations.google.google_auth_service import get_auth_token
from agent.integrations.google.google_stt_request import GoogleSTTRequest
from agent.integrations.google.google_stt_response import GoogleSTTResponse
def speech_to_text(audio_string):
try:
google_stt_request = GoogleSTTRequest(audio_string).to_payload()
request_headers = dict()
request_headers['Authorization'] = 'Bearer ' + get_auth_token()
request_headers['Content-Type'] = 'application/json'
url = 'https://speech.googleapis.com/v1/speech:recognize'
google_response = requests.post(url, data=json.dumps(google_stt_request), headers=request_headers)
response = GoogleSTTResponse(google_response.json())
return response
except Exception as e:
raise GoogleSTTException('Received an error invoking google stt {}'.format(e))
Does anyone have any ideas on how I can mock the response from the google stt call, without touching the google auth call or the server call itself? I have tried a handful of things and so far no luck. I either end up mocking nothing, or both the google stt and auth call.

So I ended up moving away from the original implementation, but this is what got me there.
#responses.activate
def test__interact__full_no_stt(self):
payload = json.load(open("tests/json_payloads/test__interact__full_audio.json"))
u_session_id = str(uuid.uuid1())
payload["session_id"] = u_session_id
#mock a 500 back from STT
responses.add(responses.POST,
'https://speech.googleapis.com/v1/speech:recognize',
json={'error': 'broken'}, status=500)
request, response = app.test_client.post("/agent_service/interact", data=json.dumps(payload))
self.assertEqual(200, response.status)
result = response.json
Responses makes this much easier, just be sure to include the annotation at the top of the test.

How do I sign the body of a requests.Request in an Auth object's __call__ method?

I'm trying to write a nice auth helper for kraken. I want it to be as automatic as possible, so it needs to:
add a nonce (time.time()*1000) to the POST body
calculate a signature over the POST body
put the signature into the headers
I wrote the obvious code based on this answer:
class KrakenAuth(AuthBase):
"""a requests-module-compatible auth module for kraken.com"""
def __init__(self, key, secret):
self.api_key = key
self.secret_key = secret
def __call__(self, request):
#print("Auth got a %r" % type(request))
nonce = int(1000*time.time())
request.data = getattr(request, 'data', {})
request.data['nonce'] = nonce
request.prepare()
message = request.path_url + hashlib.sha256(str(nonce) + request.body).digest()
hmac_key = base64.b64decode(self.secret_key)
signature = hmac.new(hmac_key, message, hashlib.sha512).digest()
signature = base64.b64encode(signature)
request.headers.update({
'API-Key': self.api_key,
'API-Sign': signature
})
return request
and them I'm calling it (from a wrapper method on another object) like:
def _request(self, method, url, **kwargs):
if not self._auth:
self._auth = KrakenAuth(key, secret)
if 'auth' not in kwargs:
kwargs['auth'] = self._auth
return self._session.request(method, URL + url, **kwargs)
...but it doesn't work. The commented-out print() statement shows that it's getting a PreparedRequest object not a Request object, and thus the call to request.prepare() is a call to PreparedRequest.prepare does nothing useful because there's no request.data because it's already been converted into a body attribute.

You can't access the data attribute of the request, because the authentication object is applied to a requests.PreparedRequest() instance, which has no .data attribute.
The normal flow for a Session.request() call (used by all the request.<method> and session.<method> calls), is as follows:
A Request() instance is created with all the same arguments as the original call
The request is passed to Session.prepare_request(), which merges in session-stored base values with the arguments of the original call first, then
A PreparedRequest() instance is created
The PreparedRequest.prepare() method is called on that prepared request instance, passing in the merged data from the Request instance and the session.
The prepare() method calls the various self.prepare_* methods, including PreparedRequest.prepare_auth().
PreparedRequest.prepare_auth() calls auth(self) to give the authentication object a chance to attach information to the request.
Unfortunately for you, at no point during this flow will the original data mapping be available to anyone else but PreparedRequest.prepare() and PreparedRequest.prepare_body(), and in those methods the mapping is a local variable. You can't access it from the authentication object.
Your options are then:
To decode the body again, and call prepare_body() with the updated mapping.
To not use an authentication object, but use the other path from my answer; to explicitly create a prepared request and manipulate data first.
To play merry hell with the Python stack and extract locals from the prepare() method that is two frames up. I really can't recommend this path.
To keep the authentication method encapsulated nicely, I'd go with decoding / re-encoding; the latter is simple enough by reusing PreparedRequest.prepare_body():
import base64
import hashlib
import hmac
import time
try:
# Python 3
from urllib.parse import parse_qs
except ImportError:
# Python 2
from urlparse import parse_qs
from requests import AuthBase
URL_ENCODED = 'application/x-www-form-urlencoded'
class KrakenAuth(AuthBase):
"""a requests-module-compatible auth module for kraken.com"""
def __init__(self, key, secret):
self.api_key = key
self.secret_key = secret
def __call__(self, request):
ctheader = request.headers.get('Content-Type')
assert (
request.method == 'POST' and (
ctheader == URL_ENCODED or
requests.headers.get('Content-Length') == '0'
)
), "Must be a POST request using form data, or empty"
# insert the nonce in the encoded body
data = parse_qs(request.body)
data['nonce'] = nonce
request.prepare_body(data, None, None)
body = request.body
if not isinstance(body, bytes): # Python 3
body = body.encode('latin1') # standard encoding for HTTP
message = request.path_url + hashlib.sha256(b'%s%s' % (nonce, body)).digest()
hmac_key = base64.b64decode(self.secret_key)
signature = hmac.new(hmac_key, message, hashlib.sha512).digest()
signature = base64.b64encode(signature)
request.headers.update({
'API-Key': self.api_key,
'API-Sign': signature
})
return request

How to get the response headers from a suds request

I'm using the python suds module and would like to retrieve the response headers (specifically Last-Modified) from a suds response.

With more effort than ought be necessary is the answer.
I've got suds version 0.3.9 here. I had to subclass the transport class in use and wrap the send method to store the last received headers on in the transport class.
import logging
logging.basicConfig(level=logging.INFO)
#logging.getLogger('suds.client').setLevel(logging.DEBUG)
#logging.getLogger('suds.transport').setLevel(logging.DEBUG)
#logging.getLogger('suds.xsd.schema').setLevel(logging.DEBUG)
#logging.getLogger('suds.wsdl').setLevel(logging.DEBUG)
from suds.client import Client
from suds.xsd.doctor import ImportDoctor, Import
from suds.transport.https import HttpAuthenticated
class MyTransport(HttpAuthenticated):
def __init__(self,*args,**kwargs):
HttpAuthenticated.__init__(self, *args, **kwargs)
self.last_headers = None
def send(self,request):
result = HttpAuthenticated.send(self, request)
self.last_headers = result.headers
return result
doctor = ImportDoctor(Import('http://schemas.xmlsoap.org/soap/encoding/'))
svc_url = 'https://server/Service?wsdl'
svc_user = 'username'
svc_pass = 'password'
client = Client(svc_url,doctor=doctor,transport=MyTransport())
# For some reason I can't be bothered to investigate, setting the username and password in
# client kwargs doesn't pass them to the custom transport:
client.set_options(location=svc_url.partition('?')[0],username=svc_user,password=svc_pass)
# call a method
client.service.SomeMethod()
# look at headers
client.options.transport.last_headers

Suds over https with cert

I have soap service under Apache with ssl, suds works greate without ssl.
I have client certificate (my.crt and user.p12 files).
How I need to configure suds client ot make it work with service over https?
without certs i see
urllib2.URLError: <urlopen error [Errno 1] _ssl.c:499: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure>

It sounds like you want to authenticate using a client certificate, not a server certificate as was stated in some of the comments. I had the same issue and was able to write a custom transport for SUDS. Here's the code that works for me.
You'll need your certificates in PEM format for this to work; OpenSSL can easily perform this conversion, though I don't remember the exact syntax.
import urllib2, httplib, socket
from suds.client import Client
from suds.transport.http import HttpTransport, Reply, TransportError
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
def __init__(self, key, cert):
urllib2.HTTPSHandler.__init__(self)
self.key = key
self.cert = cert
def https_open(self, req):
#Rather than pass in a reference to a connection class, we pass in
# a reference to a function which, for all intents and purposes,
# will behave as a constructor
return self.do_open(self.getConnection, req)
def getConnection(self, host, timeout=300):
return httplib.HTTPSConnection(host,
key_file=self.key,
cert_file=self.cert)
class HTTPSClientCertTransport(HttpTransport):
def __init__(self, key, cert, *args, **kwargs):
HttpTransport.__init__(self, *args, **kwargs)
self.key = key
self.cert = cert
def u2open(self, u2request):
"""
Open a connection.
#param u2request: A urllib2 request.
#type u2request: urllib2.Requet.
#return: The opened file-like urllib2 object.
#rtype: fp
"""
tm = self.options.timeout
url = urllib2.build_opener(HTTPSClientAuthHandler(self.key, self.cert))
if self.u2ver() < 2.6:
socket.setdefaulttimeout(tm)
return url.open(u2request)
else:
return url.open(u2request, timeout=tm)
# These lines enable debug logging; remove them once everything works.
import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger('suds.client').setLevel(logging.DEBUG)
logging.getLogger('suds.transport').setLevel(logging.DEBUG)
c = Client('https://YOUR_URL_HERE',
transport = HTTPSClientCertTransport('PRIVATE_KEY.pem',
'CERTIFICATE_CHAIN.pem'))
print c

Another workaround is to use requests library as transport which has better support for ssl. This is what I'm using now to access SOAP services through https using suds:-
import requests
from suds.transport.http import HttpAuthenticated
from suds.transport import Reply, TransportError
class RequestsTransport(HttpAuthenticated):
def __init__(self, **kwargs):
self.cert = kwargs.pop('cert', None)
# super won't work because not using new style class
HttpAuthenticated.__init__(self, **kwargs)
def send(self, request):
self.addcredentials(request)
resp = requests.post(request.url, data=request.message,
headers=request.headers, cert=self.cert)
result = Reply(resp.status_code, resp.headers, resp.content)
return result
And then you can instantiate suds client as:-
headers = {"Content-TYpe" : "text/xml;charset=UTF-8",
"SOAPAction" : ""}
t = RequestsTransport(cert='/path/to/cert', **credentials)
client = Client(wsdl_uri, location=send_url, headers=headers,
transport=t))
Update
We're now using Zeep, which use requests underneath.

Based on #k4ml answer, I've only added the open() which allows to fetch the WSDL using the certificate.
This method should fix the suds.transport.TransportError: HTTP Error 403: Forbidden when trying to fetch a WSDL (at Client creation) served behind a HTTPS service.
import requests
from suds.transport.http import HttpAuthenticated
from suds.transport import Reply, TransportError
class RequestsTransport(HttpAuthenticated):
def __init__(self, **kwargs):
self.cert = kwargs.pop('cert', None)
# super won't work because not using new style class
HttpAuthenticated.__init__(self, **kwargs)
def open(self, request):
"""
Fetches the WSDL using cert.
"""
self.addcredentials(request)
resp = requests.get(request.url, data=request.message,
headers=request.headers, cert=self.cert)
result = io.StringIO(resp.content.decode('utf-8'))
return result
def send(self, request):
"""
Posts to service using cert.
"""
self.addcredentials(request)
resp = requests.post(request.url, data=request.message,
headers=request.headers, cert=self.cert)
result = Reply(resp.status_code, resp.headers, resp.content)
return result
Side note, I've also made a suggested edit to k4ml's answer, but it can take ages before it gets approved.

Extending #k4ml solution, using cert + key
This will solve exceptions like:
requests.exceptions.SSLError: [SSL] PEM lib (_ssl.c:2599)
Solution:
import requests
from suds.client import Client
from suds.transport.http import HttpAuthenticated
from suds.transport import Reply, TransportError
class RequestsTransport(HttpAuthenticated):
def __init__(self, **kwargs):
self.cert = kwargs.pop('cert', None)
HttpAuthenticated.__init__(self, **kwargs)
def send(self, request):
self.addcredentials(request)
resp = requests.post(
request.url,
data=request.message,
headers=request.headers,
cert=self.cert,
verify=True
)
result = Reply(resp.status_code, resp.headers, resp.content)
return result
t = RequestsTransport(cert=('<your cert.pem path>', 'your key.pem path'))
headers = {"Content-Type": "text/xml;charset=UTF-8", "SOAPAction": ""}
client = Client(wsdl_url, headers=headers, transport=t)

SSL security feature is auto enabled python 2.7.9+ which breaks suds and other python libraries. I am sharing a patch which can fix this:
Locate you suds library and replace u2handlers function in suds/trasnport/http.py file with following line:
import ssl
def u2handlers(self):
"""
Get a collection of urllib handlers.
#return: A list of handlers to be installed in the opener.
#rtype: [Handler,...]
"""
handlers = []
unverified_context = ssl.create_default_context()
unverified_context.check_hostname = False
unverified_context.verify_mode = ssl.CERT_NONE
unverified_handler = urllib2.HTTPSHandler(context=unverified_context)
handlers.append(unverified_handler)
handlers.append(urllib2.ProxyHandler(self.proxy))
#handlers.append(urllib2.ProxyHandler(self.proxy))
return handlers
Note: It's not a recommended way of doing it.

How to get IP when using SimpleXMLRPCDispatcher in Django

Having a code inspired from http://code.djangoproject.com/wiki/XML-RPC :
from SimpleXMLRPCServer import SimpleXMLRPCDispatcher
from django.http import HttpResponse
dispatcher = SimpleXMLRPCDispatcher(allow_none=False, encoding=None) # Python 2.5
def rpc_handler(request):
"""
the actual handler:
if you setup your urls.py properly, all calls to the xml-rpc service
should be routed through here.
If post data is defined, it assumes it's XML-RPC and tries to process as such
Empty post assumes you're viewing from a browser and tells you about the service.
"""
if len(request.POST):
response = HttpResponse(mimetype="application/xml")
response.write(dispatcher._marshaled_dispatch(request.raw_post_data))
else:
pass # Not interesting
response['Content-length'] = str(len(response.content))
return response
def post_log(message = "", tags = []):
""" Code called via RPC. Want to know here the remote IP (or hostname). """
pass
dispatcher.register_function(post_log, 'post_log')
How could get the IP address of the client within the "post_log" definition?
I have seen IP address of client in Python SimpleXMLRPCServer? but can't apply it to my case.
Thanks.

Ok I could do it ... with some nifty tips ...
First, I created my own copy of SimpleXMLRPCDispatcher which inherit everything from it and overides 2 methods :
class MySimpleXMLRPCDispatcher (SimpleXMLRPCDispatcher) :
def _marshaled_dispatch(self, data, dispatch_method = None, request = None):
# copy and paste from /usr/lib/python2.6/SimpleXMLRPCServer.py except
response = self._dispatch(method, params)
# which becomes
response = self._dispatch(method, params, request)
def _dispatch(self, method, params, request = None):
# copy and paste from /usr/lib/python2.6/SimpleXMLRPCServer.py except
return func(*params)
# which becomes
return func(request, *params)
Then in my code, all to do is :
# ...
if len(request.POST):
response = HttpResponse(mimetype="application/xml")
response.write(dispatcher._marshaled_dispatch(request.raw_post_data, request = request))
# ...
def post_log(request, message = "", tags = []):
""" Code called via RPC. Want to know here the remote IP (or hostname). """
ip = request.META["REMOTE_ADDR"]
hostname = socket.gethostbyaddr(ip)[0]
That's it.
I know it's not very clean... Any suggestion for cleaner solution is welcome!