I have run into some trouble with the issue, that request.data sometimes is a dict (especially when testing) and sometimes a QueryDict instance (when using curl).
This is especially a problem because apparently there is a big difference when calling a view using curl like so:
curl -X POST --data "some_float=1.23456789012123123" "http://localhost:8000/myview"
Or using the django_webtest client like so:
class APIViewTest(WebTest):
def test_testsomething(self):
self.app.post(url=url, params=json.dumps({some_float=1.26356756467}))
And then casting that QueryDict to a dict like so
new_dict = dict(**request.data)
my_float = float(new_dict['some_float'])
Everything works fine in the tests, as there request.data is a dict, but in production the view crashes because new_dict['some_float'] is actually a list with one element, and not as expected a float.
I have considered fixing the issue like so:
if type(request.data) is dict:
new_dict = dict(**request.data)
else:
new_dict = dict(**request.data.dict())
which feels very wrong as the tests would only test line 2, and (some? all?) production code would run line 4.
So while I am wondering why QueryDict behaves in this way, I would rather know why and when response.data is a QueryDict in the first place. And how I can use django tests to simulate this behavior. Having different conditions for production and testing systems is always troublesome and sometimes unavoidable, but in this case I feel like it could be fixed. Or is this a specific issue related to django_webtest?
Your test isn't a reflection of your actual curl call.
In your test, you post JSON, which is then available as a dict from request.data. But your curl call posts standard form data, which is available as a QueryDict. This behaviour is managed by the parsers attribute of your view or the DEFAULT_PARSER_CLASSES settings - and further note that this is functionality specifically provided by django-rest-framework, not Django itself.
Really you should test the same thing as you are doing; either send JSON from curl or get your test to post form-data.
When your request content_type is "application/x-www-form-urlencoded", request.Data become QueryDict.
see FormParser class.
https://github.com/encode/django-rest-framework/blob/master/rest_framework/parsers.py
And
QueryDict has get lists method. but it can't fetch dict value.
convert name str to array.
<input name="items[name]" value="Example">
<input name="items[count]" value="5">
https://pypi.org/project/html-json-forms/
And define custom form paser.
class CustomFormParser(FormParser):
"""
Parser for form data.
"""
media_type = 'application/x-www-form-urlencoded'
def parse(self, stream, media_type=None, parser_context=None):
"""
Parses the incoming bytestream as a URL encoded form,
and returns the resulting QueryDict.
"""
parser_context = parser_context or {}
encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
data = QueryDict(stream.read(), encoding=encoding)
return parse_json_form(data.dict()) # return dict
And overwite DEFAULT_PARSER_CLASSES.
https://www.django-rest-framework.org/api-guide/settings/#default_parser_classes
Related
I'm using requests-cache to cache http responses in human-readable format.
I've patched requests using the filesystem backend, and the the serializer to json, like so:
import requests_cache
requests_cache.install_cache('example_cache', backend='filesystem', serializer='json')
The responses do get cached as json, but the response's body is encoded (I guess using the cattrs library, as described here).
Is there a way to make requests-cache save responses as-is?
What you want to do makes sense, but it's a bit more complicated than it appears. The response files you see are representations of requests.Response objects. Response._content contains the original bytes received from the server. The wrapper methods and properties like Response.json() and Response.text will then attempt to decode that content. For a Response object to work correctly, it needs to have the original binary response body.
When requests-cache serializes that response as JSON, the binary content is encoded in Base85. That's why you're seeing encoded bytes instead of JSON there. To have everything including the response body saved in JSON, there are a couple options:
Option 1
Make a custom serializer. If you wanted to be able to modify response content and have those changes reflected in responses returned by requests-cache, this would probably be the best way to do it.
This may be become a bit convoluted, because you would have to:
Handle response content that isn't valid JSON, and save as encoded bytes instead
During deserialization, if the content was saved as JSON, convert it back into bytes to recreate the original Response object
It's doable, though. I could try to come up with an example later, if needed.
Option 2
Make a custom backend. It could extend FileCache and FileDict, and copy valid JSON content to a separate file. Here is a working example:
import json
from os.path import splitext
from requests import Response
from requests_cache import CachedSession, FileCache, FileDict
class JSONFileCache(FileCache):
"""Filesystem backend that copies JSON-formatted response content into a separate file
alongside the main response file
"""
def __init__(self, cache_name, **kwargs):
super().__init__(cache_name, **kwargs)
self.responses = JSONFileDict(cache_name, **kwargs)
class JSONFileDict(FileDict):
def __setitem__(self, key: str, value: Response):
super().__setitem__(key, value)
response_path = splitext(self._path(key))[0]
json_path = f'{response_path}_content.json'
# Will handle errors and skip writing if content can't be decoded as JSON
with self._try_io(ignore_errors=True):
content = json.dumps(value.json(), indent=2)
with open(json_path, mode='w') as f:
f.write(content)
Usage example:
custom_backend = JSONFileCache('example_cache', serializer='json')
session = CachedSession(backend=custom_backend)
session.get('https://httpbin.org/get')
After making a request, you will see a pair of files like:
example_cache/680f2a52944ee079.json
example_cache/680f2a52944ee079_content.json
That may not be exactly what you want, but it's the easiest option if you only need to read the response content and don't need to modify it.
Hello I am new to the python world, and I am learning, I am currently developing a WebApp in Django and I am using ajax for sending requests, what happens is that in the view.py I get a JSON, from which I have not been able to extract the attributes individually to send to a SQL query, I have tried every possible way, I appreciate any help in advance.
def profesionales(request):
body_unicode = request.body.decode('utf-8')
received_json = json.loads(body_unicode)
data = JsonResponse(received_json, safe=False)
return data
Data returns the following to me
{opcion: 2, fecha_ini: "2021-02-01", fecha_fin: "2021-02-08", profesional: "168", sede: "Modulo 7", grafico: "2"}
This is the answer I get and I need to extract each of the values of each key into a variable
You can interpret this as dict.
for key in received_json:
print(key,received_json[key])
# do your stuff here
but if it's always a object with same keys (fixed keys), you can access directly:
key_data = received_json[key]
I have Pyramid/cornice resource, that requires a ?query=keyword in end of the url. But I don't know how to add this in a pyramid's dummyRequest object. Code works perfectly on browser and I will get correct response when using this url to get stuff: *url*/foo?query=keyword.
My class/resource is defined like this:
#resource(path='/bar/search/foo')
class SearchFooResource(object):
def __init__(self, request):
self.request = request
#view(renderer='json')
def get(self):
#get query string, it's a tuple
req = self.request.GET.items()
#do stuff with req
Now req should contain the all the query string 'stuffs' in a list that contains them as a tuple's, for example: [('query', 'bar'),('query', 'asd')]. But how do I make unittest to this resource? I can't seem to add anything to self.request.GET.items() method. When running unittest req is empty, and I will get this error: AttributeError: 'list' object has no attribute 'items'.
My current unittest:
def test_passing_GetFooBaarResource(self):
request = testing.DummyRequest()
request.GET = [('query', 'keyword')]
info = SearchFooResource.get(SearchFooResource(request))
self.assertEqual(info['foo'], 'baar')
In addition to what #matino has suggested, you can just use a plain dictionary (instead of a list of tuples you tried).
def test_passing_GetFooBaarResource(self):
request = testing.DummyRequest()
request.GET = {'query': 'keyword'}
info = SearchShowResource.get(SearchShowResource(request))
self.assertEqual(info['foo'], 'baar')
This will work in uncomplicated cases where you don't have multiple parameters with the same name (/someurl?name=foo&name=baz&name=bar).
If you need to test those more complicated queries you can replace your DummyRequest's GET attribute with a WebOb MultiDict
from webob.multidict import MultiDict
def test_passing_GetFooBaarResource(self):
request = testing.DummyRequest()
request.GET = MultiDict([('query', 'foo'), ('query', 'bar'), ('query', 'baz')])
info = SearchShowResource.get(SearchShowResource(request))
self.assertEqual(info['foo'], 'baar')
Then, normally, in your actual view method, if you need to handle multiple parameters with the same name you use request.GET.getall('query') which should return ['foo', 'bar', 'baz'].
In simpler cases you can just use request.GET['query'] or request.GET.get('query', 'default'). I mean, your use of request.GET.items() is a bit unusual...
According to the docs I think you need to pass it as params argument (not tested):
request = testing.DummyRequest(params={'query': 'keyword'})
I have a Pyramid web service, and code samples are as follows:
View declaration:
#view_config(route_name="services/Prices/GetByTicker/")
def GET(request):
ticker = request.GET('ticker')
startDate = request.GET('startDate')
endDate = request.GET('endDate')
period = request.GET('period')
Routing:
config.add_route('services/Prices/GetByTicker/', 'services/Prices/GetByTicker/{ticker}/{startDate}/{endDate}/{period}')
Now I know this is all screwed up but I don't know what the convention is for Pyramid. At the moment this works inasmuch as the request gets routed to the view successfully, but then I get a "Dictionary object not callable" exception.
The URL looks horrible:
#root/services/Prices/GetByTicker/ticker=APPL/startDate=19981212/endDate=20121231/period=d
Ideally I would like to be able to use a URL something like:
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
Any Pyramid bods out there willing to take five minutes to explain what I'm doing wrong?
from you sample code, i think you use the URL Dispatch
so it should be like this
config.add_route('services/Prices/GetByTicker/', 'services/Prices/GetByTicker/')
then the URL like:
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
will match it
--edit--
you don't have to use a name like "services/Prices/GetByTicker" for route_name,and you can get the GET params use request.params['key']
View declaration:
#view_config(route_name="services_Prices_GetByTicker")
def services_Prices_GetByTicker(request):
ticker = request.params['ticker']
startDate = request.params['startDate']
endDate = request.params['endDate']
period = request.params['period']
Routing:
config.add_route('services_Prices_GetByTicker', 'services/Prices/GetByTicker/')
The query string is turned into the request.GET dictionary. You are using parenthesis to call the dictionary instead of accessing items via the brackets. For a url such as
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
request.GET['ticker'] # -> 'APPL' or an exception if not available
request.GET.get('ticker') # -> 'APPL' or None if not available
request.GET.get('ticker', 'foo') # -> 'APPL' or 'foo' if not available
request.GET.getall('ticker') # -> ['APPL'] or [] if not available
The last option is useful if you expect ticker to be supplied multiple times.
request.params is a combination of request.GET and request.POST where the latter is a dictionary representing the request's body in a form upload.
Anyway, the answer is that request.GET('ticker') syntactically is not one of the options I mentioned, stop doing it. :-)
I'm trying to simulate some data in to the datastore to emulate POSTs.
What I'm looking for is a way to post named arguments but as one argument. So I can use the same method as a normal POST would do.
The method I want to invoke get params in two ways.
def HandlePost(params):
params.get('name')
params.get_all('collection')
class SavePartHandler(webapp.RequestHandler):
def post(self):
HandlePost(self.request)
I'm trying to find out what type the self.request are but can't find it anywhere in appengines source code.
My goal is to simulate multiple POST to fill the datastore by the methods users would.
EDIT:
Or is the anyway to change the behavior of dict so it could use the get_all method?
EDIT 2:
I'm using appengins webapp.
Out of curiosity is there a way to invoke a dummy webapp.RequestHandler and populate it with arguments? I'm browsing the source code to see how it's done and made a new instance of it but can't find how to populate it.
EDIT 3:
Updated method name so it's not confused with webapp request handlers.
Thanks to pthulin I'm almost there. Only thing left is how to simulate data that has the same key. Since using a dict will override other keys with the same name. Sometimes in HTML forms a post can contain multiple values for the same key that we get with self.request.get_all('key'). So how to create a dict (or something equal) that supports multiple key=value with same key.
..fredrik
I think what you want to do is prepare a webapp Request object and then call the post method of your handler:
from google.appengine.ext.webapp import Request
from google.appengine.ext.webapp import Response
from StringIO import StringIO
form = 'msg=hello'
handler.response = Response()
handler.request = Request({
'REQUEST_METHOD': 'POST',
'PATH_INFO': '/',
'wsgi.input': StringIO(form),
'CONTENT_LENGTH': len(form),
'SERVER_NAME': 'hi',
'SERVER_PORT': '80',
'wsgi.url_scheme': 'http',
})
handler.post()
TIP: 'msg=hello' above works fine in this case, but to pass multiple POST parameters you need to create a URL encoded string:
>>> form = {
... 'first_name': 'Per',
... 'last_name': 'Thulin',
... }
>>> from urllib import urlencode
>>> urlencode(form)
'first_name=Per&last_name=Thulin'
If you want to pass in multiple POST parameters with the same name, I think you need to do a little bit of the url encoding work yourself. For example:
>>> from urllib import urlencode
>>> form_inputs = [
... {'someparam': 'aaa'},
... {'someparam': 'bbb'},
... {'someparam': 'ccc'},
... ]
>>> '&'.join([urlencode(d) for d in form_inputs])
'someparam=aaa&someparam=bbb&someparam=ccc'
Then in the RequestHandler, you can extract the parameters with the Request.get_all method.
http://code.google.com/appengine/docs/python/tools/webapp/requestclass.html
http://code.google.com/appengine/docs/python/tools/webapp/requestclass.html#Request_get_all
For sure they are request objects that represents a http request. It should usually contain typical http request information like method , uri, message etc.
To find out what type self.request are, you could do introspection without the doc. if self.request is a instantiation of a class then you could obtain the class name through
print self.request.__class__
[Edit]:
The information is provided in the google app engine document
http://code.google.com/appengine/docs/python/tools/webapp/requestclass.html
Yes, that can be done. Here's the webapp response handler. The way to do it is to use the response.argument() which returns a list of the data arguments. Then iterate and get(each one). Just for clarity, I show how to handle named arguments that are known ahead of time and also multiple arbitrary arguments as you ask:
from google.appengine.ext import webapp
class postHandler(webapp.RequestHandler):
def post(self):
# Handle arguments one at a time
var1 = self.request.get('var1')
# or get them all and build a dict:
args = self.request.arguments()
arg_dict = {}
for arg in args:
arg_dict.update(arg: self.request.get(arg))
Here's some documentation: http://code.google.com/appengine/docs/python/tools/webapp/requestclass.html#Request_get_all