We are writing a client for a remote service that exposes SOAP web services and publishes a WSDL definition for those services.
We don't have access to the system during testing, so we'd like to write a mock service. We're using Python for the client, so ideally we'd want to use Python for the mock server, although I suppose it's not strictly necessary.
The dream would be to be able to generate stubs from the WSDL file which we could fill in, and then serve those using Paste Deploy as a WSGI server, although it doesn't have to be Paste Deploy or WSGI so long as it works reliably. The main thing is that we need to be generating the stubs from the "real" WSDL file so that we don't accidentally write a non-compliant mock server.
We're using suds for the client side library, and have looked at soaplib and ZSI. However, the wsgi2py stuff in soaplib says "do not use" at the top and ZSI seems like a lot to swallow. What do people generally use for this kind of thing?
Martin
As a mock server I would really recommend soapUI (http://www.soapui.org).
It takes a WSDL and generates the services and service methods automatically. You can then go on and define static returns or dynamic ones using Groovy scripts. Take a look here for the documentation of web service mocking.
soapUI comes in a free and paid pro version. I have used the free version with great success.
I would recommend you use soapUI for creating a mock service. It's very easy to install. It's just as easy to create a WS mock service.
It takes the WSDL file from your desired location, it creates the structure for the requests and if you wish, it also creates the mock web service with the intended structure drawn from the WSDL file.
When you create the new soapUI project, select the third checkbox option to create the mock web service.
If xsd schema files are required, make sure they are well referenced from the WSDL file.
It doesn't use python but if all you need is a test environment that you can send requests and get responses, that will be more than enough.
I hope that helps.
You could use this code for creating a suds mock client.
from suds.client import Client
class AlwaysCallable(object):
"""
Represents a chainable-access object and proxies calls to ClientMock.
"""
name = None
def __init__(self, client_cls):
self._client_cls = client_cls
def __call__(self, *args, **kwargs):
try:
hook = object.__getattribute__(self._client_cls, self.name)
except AttributeError:
pass
else:
return hook(self._client_cls, *args, **kwargs)
def __getattr__(self, item):
new = object.__getattribute__(self, '__class__')(self._client_cls)
new.name = item
return new
class ClientMock(Client):
"""
Abstract mock suds client.
"""
def __init__(self, url, **kwargs):
pass
def __getattr__(self, item):
return AlwaysCallable(self.__class__)
def __unicode__(self):
return 'Client mock'
def __str__(self):
return 'Client mock'
And next define a concrete ClientMock.
class UserCredentialsServiceClientMock(ClientMock):
"""
Mock object that implements remote side services.
"""
def GetUserInfo(cls, user_id):
"""
Stub for remote service.
"""
return UserInfo(id=user_id, name='Adam Smith')
Now you could use mock library to spoof a code, that uses suds.client.Client.
from unittest import TestCase
from mock import patch
from project.api import get_user_credentials
class UserCredentialsClientTestCase(TestCase):
def test_getting_user_credentials(self):
with patch('project.api.Client', new=UserCredentialsServiceClientMock):
self.assertEquals(get_user_credentials(1), 'Adam Smith')
Related
In the python standard environment quickstart, the endpoints method test_api_key returns a 503 Service Unavailable. The error occurs in the API Explorer when run with dev_appser.py and when the API is deployed. The code for it is:
import endpoints
from protorpc import message_types
from protorpc import messages
from protorpc import remote
class TestResponse(messages.Message):
content = messages.StringField(1)
#endpoints.api(name='practice', version='v1', description='My Practice API')
class PracticeApi(remote.Service):
#endpoints.method(
message_types.VoidMessage,
TestResponse,
path='test/getApiKey',
http_method='GET',
name='test_api_key')
def test_api_key(self, request):
return TestResponse(content=request.get_unrecognized_field_info('key'))
api = endpoints.api_server([PracticeApi])
I don't have a good understanding of .get_unrecognized_field_info('key') so I am not sure what the issue is? Thanks.
Firstly, I recommend reading Google Protocol RPC Library Overview, since it's Google Cloud Endpoints uses it extensively.
#endpoints.method allows you to configure a specific method in your API. Configuration options are documented in Google Cloud Platform documentation Creating an API with Cloud Endpoints Frameworks for App Engine, in the section, Defining an API method (#endpoints.method).
If you're restricting access to the test/getApiKey/test_api_key method, then you must configure the method with the api_key_required=True option. Restricting API Access with API Keys (Frameworks) discusses that further, but your method annotation should be:
#endpoints.method(
message_types.VoidMessage,
TestResponse,
path='test/getApiKey',
http_method='GET',
name='test_api_key',
api_key_required=True
)
Notice your method accepts a request parameter representing the HTTP request (i.e. client using your API):
def test_api_key(self, request):
However, the request parameter is actually Google Protocol RPC Message (Proto RPC) Message object and as such is very well defined. If additional fields exist in the ProtoRPC request parameter, beyond what is formally defined, they are still stored with the request object but must be retrieved using the following method:
def get_unrecognized_field_info(self, key, value_default=None,
variant_default=None):
"""Get the value and variant of an unknown field in this message.
Args:
key: The name or number of the field to retrieve.
value_default: Value to be returned if the key isn't found.
variant_default: Value to be returned as variant if the key isn't
found.
Returns:
(value, variant), where value and variant are whatever was passed
to set_unrecognized_field.
"""
Message class code on GitHub is quite well documented. .
No arguments will appear in the body of a request because you've configured the method with to be called with HTTP GET:
http_method='GET'
...you're correctly using the value message_types.VoidMessage.
In terms of your error, 503 is just a generic server error, can you provide any information from the StackDriver logs? They will point you to the exact line and error in your code.
There were three things that were creating the 503 error.
Firstly, I needed to make the method or entire Api require an Api Key. In this case I just applied it to the entire Api:
#endpoints.api(name='practice', version='v1', api_key_required=True)
class PracticeApi(remote.Service):
Secondly, after I generated the Api Key in the cloud console I needed to put the Key into the openapi.json file before deploying it.
Lastly, I was still getting a validation error:
ValidationError: Expected type <type 'unicode'> for field content, found (u'My Api Key', Variant(STRING, 9)) (type <type 'tuple'>)
The get_unrecognized_field_info() function returns a tuple of (value, variant). A tuple was not expected by the response so I updated the method to only show value:
def test_api_key(self, request):
return TestResponse(content=request.get_unrecognized_field_info('key')[0])
I'm building a (facebook connected) web server using Flask. Here's an example route
#app.route('/login', methods=['GET'])
def login():
graph = facebook.GraphAPI(request.args.get('access_token'))
profile = graph.get_object('me')
return jsonify(profile)
This works fine, but it seems bad practice to be continually hitting Facebook servers when I'm doing local dev.
What is the most effective way to re-route the facebook API to mocks or even a mock server? Internally, the facebook-sdk library uses requests to reach graph.facebook.com.
I've seen this question
python mock Requests and the response
and in particular, Dropbox's responses library, but it looks like you have to wrap each call in their decorator. It would work well for a unit testing suite, but I'm just interested in doing development against mock data and mock responses.
Update: In response to Thomas, I had been thinking about a similar solution. I tried adding:
if config_name == 'development':
print 'monkey'
import requests
def mock(*args, **kwargs):
print args, kwargs
return {}
requests.request = mock
But that didn't seem to change the behavior of the facebook-sdk library.
You could check if debug mode is on using app.config, and then (if debug mode is on), turn your requests into dummy requests. I'm not that familiar with the facebook API, so (unfortunately) I can't write out the dummy code for you.
How about using the flask-testing and mock libraries:
import requests
from flask.ext.testing import TestCase
from mock import Mock
class TestLogin(TestCase):
def setUp(self):
requests.request = Mock()
super(TestLogin, self).setUp()
def test_it(self):
resp = self.client.get('/login')
self.assertEquals('right stuff', resp.data)
I have a thorny problem that I can't seem to get to grips with. I am
currently writing unit tests for a django custom auth-backend. On our
system we actually have two backends: one the built-in django backend
and the custom backend that sends out requests to a Java based API
that returns user info in the form of XML. Now, I am writing unit
tests so I don't want to be sending requests outside the system like
that, I'm not trying to test the Java API, so my question is how can I
get around this and mock the side-effects in the most robust way.
The function I am testing is something like this, where the url
settings value is just the base url for the Java server that
authenticates the username and password data and returns the xml, and the service value is
just some magic for building the url query, its unimportant for
us:
#staticmethod
def get_info_from_api_with_un_pw(username, password, service=12345):
url = settings.AUTHENTICATE_URL_VIA_PASSWORD
if AUTH_FIELD == "username":
params = {"nick": username, "password": password}
elif AUTH_FIELD == "email":
params = {"email": username, "password": password}
params["service"] = service
encoded_params = urlencode([(k, smart_str(v, "latin1")) for k, v in params.items()])
try:
# get the user's data from the api
xml = urlopen(url + encoded_params).read()
userinfo = dict((e.tag, smart_unicode(e.text, strings_only=True))
for e in ET.fromstring(xml).getchildren())
if "nil" in userinfo:
return userinfo
else:
return None
So, we get the xml, parse it into a dict and if the key nil is present
then we can return the dict and carry on happy and authenticated.
Clearly, one solution is just to find a way to somehow override or
monkeypatch the logic in the xml variable, I found this answer:
How can one mock/stub python module like urllib
I tried to implement something like that, but the details there are
very sketchy and I couldn't seem to get that working.
I also captured the xml response and put it in a local file in the
test folder with the intention of finding a way to use that as a mock
response that is passed into the url parameter of the test function,
something like this will override the url:
#override_settings(AUTHENTICATE_URL_VIA_PASSWORD=(os.path.join(os.path.dirname(__file__), "{0}".format("response.xml"))))
def test_get_user_info_username(self):
self.backend = RemoteAuthBackend()
self.backend.get_info_from_api_with_un_pw("user", "pass")
But that also needs to take account of the url building logic that the
function defines, (i.e. "url + encoded_params"). Again, I could rename
the response file to be the same as the concatenated url but this is becoming
less like a good unit-test for the function and more of a "cheat", the whole
thing is just getting more and more brittle all the time with these solutions, and its really just a fixture anyway, which is also something I want to avoid if
at all possible.
I also wondered if there might be a way to serve the xml on the django development server and then point the function at that? It seems like a saner solution, but much googling gave me no clues if such a thing would be possible or advisable and even then I don't think that would be a test to run outside of the development environment.
So, ideally, I need to be able to somehow mock a "server" to
take the place of the Java API in the function call, or somehow serve
up some xml payload that the function can open as its url, or
monkeypatch the function from the test itself, or...
Does the mock library have the appropriate tools to do such things?
http://www.voidspace.org.uk/python/mock
So, there are two points to this question 1) I would like to solve my
particular problem in a clean way, and more importantly 2) what are
the best practices for cleanly writing Django unit-tests when you are
dependent on data, cookies, etc. for user authentication from a remote
API that is outside of your domain?
The mock library should work if used properly. I prefer the minimock library and I wrote a small base unit testcase (minimocktest) that helps with this.
If you want to integrate this testcase with Django to test urllib you can do it as follows:
from minimocktest import MockTestCase
from django.test import TestCase
from django.test.client import Client
class DjangoTestCase(TestCase, MockTestCase):
'''
A TestCase class that combines minimocktest and django.test.TestCase
'''
def _pre_setup(self):
MockTestCase.setUp(self)
TestCase._pre_setup(self)
# optional: shortcut client handle for quick testing
self.client = Client()
def _post_teardown(self):
TestCase._post_teardown(self)
MockTestCase.tearDown(self)
Now you can use this testcase instead of using the Django test case directly:
class MySimpleTestCase(DjangoTestCase):
def setUp(self):
self.file = StringIO.StringIO('MiniMockTest')
self.file.close = self.Mock('file_close_function')
def test_urldump_dumpsContentProperly(self):
self.mock('urllib2.urlopen', returns=self.file)
self.assertEquals(urldump('http://pykler.github.com'), 'MiniMockTest')
self.assertSameTrace('\n'.join([
"Called urllib2.urlopen('http://pykler.github.com')",
"Called file_close_function()",
]))
urllib2.urlopen('anything')
self.mock('urllib2.urlopen', returns=self.file, tracker=None)
urllib2.urlopen('this is not tracked')
self.assertTrace("Called urllib2.urlopen('anything')")
self.assertTrace("Called urllib2.urlopen('this is mocked but not tracked')", includes=False)
self.assertSameTrace('\n'.join([
"Called urllib2.urlopen('http://pykler.github.com')",
"Called file_close_function()",
"Called urllib2.urlopen('anything')",
]))
Here's the basics of the solution that I ended up with for the record. I used the Mock library itself rather than Mockito in the end, but the idea is the same:
from mock import patch
#override_settings(AUTHENTICATE_LOGIN_FIELD="username")
#patch("mymodule.auth_backend.urlopen")
def test_get_user_info_username(self, urlopen_override):
response = "file://" + os.path.join(os.path.dirname(__file__), "{0}".format("response.xml"))
# mock patch replaces API call
urlopen_override.return_value = urlopen(response)
# call the patched object
userinfo = RemoteAuthBackend.get_info_from_api_with_un_pw("user", "pass")
assert_equal(type(userinfo), dict)
assert_equal(userinfo["nick"], "user")
assert_equal(userinfo["pass"], "pass")
The problem is simple, we would like CherryPy to not log access log for a particular exposed method/API that gets called.
Basically when this API gets called, there are some parameters in the query string of the URL which are very sensitive and if leaked, would expose potential security. Naturally this is a /GET request and unfortunately it is the only way the parameters could be passed, since its a redirect(302) from an external service to this web server.
If it would not log the URL, that would serve the purpose as well.
So, is there a way that we can filter logging messages in access log by API's, URL's etc?
Thanks in advance for the help.
cherrypy uses Python's standard logging module by default, so you can just add a custom filter. This example will ignore any GET request with /foo as the path prefix:
import logging
class IgnoreURLFilter(logging.Filter):
# simple example of log message filtering
def __init__(self, ignore):
self.ignore = 'GET /' + ignore
def filter(self, record):
return self.ignore not in record.getMessage()
app = cherrypy.tree.mount( YourApplication() )
app.log.access_log.addFilter( IgnoreURLFilter('foo') )
cherrypy.engine.start()
Is there any example code of a cpython (not IronPython) client which can call Windows Communication Foundation (WCF) service?
I used suds.
from suds.client import Client
print "Connecting to Service..."
wsdl = "http://serviceurl.com/service.svc?WSDL"
client = Client(wsdl)
result = client.service.Method(variable1, variable2)
print result
That should get you started. I'm able to connect to exposed services from WCF and a RESTful layer. There needs to be some data massaging to help do what you need, especially if you need to bind to several namespaces.
TL;DR: For wsHttpBinding (SOAP 1.2) use zeep
In case someone is having trouble using suds (or suds-jurko for that matter) with WCF and wsHttpBinding (which is SOAP 1.2):
suds is pretty much dead (can't even pip install it on python 3)
suds-jurko seems kind-of dead. The 0.6 release has a very annoying infinite recursion bug (at least on the WSDL exposed by our service) which is fixed in the tip but that's not released and it's been 1.5years (at time of this writing in Feb'17) since the last commit.
It works on python 3 but doesn't support SOAP 1.2. Sovetnikov's answer is an attempt to get it working with 1.2 but I haven't managed to make it work for me.
zeep seems to be the current way to go and worked out of the box (I'm not affiliated with zeep, it just works for me and I spent several hours banging my head against a brick wall trying to make suds work). For zeep to work, the WCF service host configuration must include <security mode="None"/> under the wsHttpBinding node Actually zeep seems to support username and signature (x509) based WS-SE but I haven't tried that so can't speak to any problems around it.
WCF needs to expose functionality through a communication protocol. I think the most commonly used protocol is probably SOAP over HTTP. Let's assume that's
what you're using then.
Take a look at this chapter in Dive Into Python. It will show you how to
make SOAP calls.
I know of no unified way of calling a WCF service in Python, regardless of communication
protocol.
Just to help someone to access WCF SOAP 1.2 service with WS-Addressing using suds.
Main problem is to inject action name in every message.
This example for python 3 and suds port https://bitbucket.org/jurko/suds.
Example uses custom authentification based on HTTP headers, i leave it as is.
TODO: Automatically get api_direct_url from WSDL (at now it is hard coded).
from suds.plugin import MessagePlugin
from suds.sax.text import Text
from suds.wsse import Security, UsernameToken
from suds.sax.element import Element
from suds.sax.attribute import Attribute
from suds.xsd.sxbasic import Import
api_username = 'some'
api_password = 'none'
class api(object):
api_direct_url = 'some/mex'
api_url = 'some.svc?singleWsdl|Wsdl'
NS_WSA = ('wsa', 'http://www.w3.org/2005/08/addressing')
_client_instance = None
#property
def client(self):
if self._client_instance:
return self._client_instance
from suds.bindings import binding
binding.envns = ('SOAP-ENV', 'http://www.w3.org/2003/05/soap-envelope')
api_inst = self
class _WSAPlugin(MessagePlugin):
def marshalled(self, context):
api_inst._marshalled_message(context)
self._client_instance = Client(self.api_url,
plugins=[_WSAPlugin()],
headers={'Content-Type': 'application/soap+xml',
'login':api_username,
'password': api_password}
)
headers = []
headers.append(Element('To', ns=self.NS_WSA).setText(self.api_direct_url))
headers.append(Element('Action', ns=self.NS_WSA).setText('Blank'))
self._client_instance.set_options(soapheaders=headers)
cache = self._client_instance.options.cache
cache.setduration(days=10)
return self._client_instance
def _marshalled_message(self, context):
def _children(r):
if hasattr(r, 'children'):
for c in r.children:
yield from _children(c)
yield c
for el in _children(context.envelope):
if el.name == 'Action':
el.text = Text(self._current_action)
return
_current_action = None
def _invoke(self, method, *args):
try:
self._current_action = method.method.soap.action.strip('"')
return method(*args)
finally:
self._current_action = None
def GetRequestTypes(self):
return self._invoke(self.client.service.GetRequestTypes)[0]
def GetTemplateByRequestType(self, request_type_id):
js = self._invoke(self.client.service.GetTemplateByRequestType, request_type_id)
return json.loads(js)
def GetRequestStatus(self, request_guid):
return self._invoke(self.client.service.GetRequestStatus, request_guid)
def SendRequest(self, request_type_id, request_json):
r = json.dumps(request_json, ensure_ascii=False)
return self._invoke(self.client.service.SendRequest, request_type_id, r)
I do not know of any direct examples, but if the WCF service is REST enabled you could access it through POX (Plain Old XML) via the REST methods/etc (if the service has any). If you are in control of the service you could expose endpoints via REST as well.
if you need binary serialized communication over tcp then consider implementing solution like Thrift.
Even if there is not a specific example of calling WCF from Python, you should be able to make a fully SOAP compliant service with WCF. Then all you have to do is find some examples of how to call a normal SOAP service from Python.
The simplest thing will be to use the BasicHttpBinding in WCF and then you can support your own sessions by passing a session token with each request and response.