We Keep Coding

Django Views: When is request.data a dict vs a QueryDict?

I have run into some trouble with the issue, that request.data sometimes is a dict (especially when testing) and sometimes a QueryDict instance (when using curl).
This is especially a problem because apparently there is a big difference when calling a view using curl like so:
curl -X POST --data "some_float=1.23456789012123123" "http://localhost:8000/myview"
Or using the django_webtest client like so:
class APIViewTest(WebTest):
def test_testsomething(self):
self.app.post(url=url, params=json.dumps({some_float=1.26356756467}))
And then casting that QueryDict to a dict like so
new_dict = dict(**request.data)
my_float = float(new_dict['some_float'])
Everything works fine in the tests, as there request.data is a dict, but in production the view crashes because new_dict['some_float'] is actually a list with one element, and not as expected a float.
I have considered fixing the issue like so:
if type(request.data) is dict:
new_dict = dict(**request.data)
else:
new_dict = dict(**request.data.dict())
which feels very wrong as the tests would only test line 2, and (some? all?) production code would run line 4.
So while I am wondering why QueryDict behaves in this way, I would rather know why and when response.data is a QueryDict in the first place. And how I can use django tests to simulate this behavior. Having different conditions for production and testing systems is always troublesome and sometimes unavoidable, but in this case I feel like it could be fixed. Or is this a specific issue related to django_webtest?

Your test isn't a reflection of your actual curl call.
In your test, you post JSON, which is then available as a dict from request.data. But your curl call posts standard form data, which is available as a QueryDict. This behaviour is managed by the parsers attribute of your view or the DEFAULT_PARSER_CLASSES settings - and further note that this is functionality specifically provided by django-rest-framework, not Django itself.
Really you should test the same thing as you are doing; either send JSON from curl or get your test to post form-data.

When your request content_type is "application/x-www-form-urlencoded", request.Data become QueryDict.
see FormParser class.
https://github.com/encode/django-rest-framework/blob/master/rest_framework/parsers.py
And
QueryDict has get lists method. but it can't fetch dict value.
convert name str to array.
<input name="items[name]" value="Example">
<input name="items[count]" value="5">
https://pypi.org/project/html-json-forms/
And define custom form paser.
class CustomFormParser(FormParser):
"""
Parser for form data.
"""
media_type = 'application/x-www-form-urlencoded'
def parse(self, stream, media_type=None, parser_context=None):
"""
Parses the incoming bytestream as a URL encoded form,
and returns the resulting QueryDict.
"""
parser_context = parser_context or {}
encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
data = QueryDict(stream.read(), encoding=encoding)
return parse_json_form(data.dict()) # return dict
And overwite DEFAULT_PARSER_CLASSES.
https://www.django-rest-framework.org/api-guide/settings/#default_parser_classes

How to convert suds object to xml string

This is a duplicate to this question:
How to convert suds object to xml
But the question has not been answered: "totxt" is not an attribute on the Client class.
Unfortunately I lack of reputation to add comments. So I ask again:
Is there a way to convert a suds object to its xml?
I ask this because I already have a system that consumes wsdl files and sends data to a webservice. But now the customers want to alternatively store the XML as files (to import them later manually). So all I need are 2 methods for writing data: One writes to a webservice (implemented and tested), the other (not implemented yet) writes to files.
If only I could make something like this:
xml_as_string = My_suds_object.to_xml()
The following code is just an example and does not run. And it's not elegant. Doesn't matter. I hope you get the idea what I want to achieve:
I have the function "write_customer_obj_webservice" that works. Now I want to write the function "write_customer_obj_xml_file".
import suds
def get_customer_obj():
wsdl_url = r'file:C:/somepathhere/Customer.wsdl'
service_url = r'http://someiphere/Customer'
c = suds.client.Client(wsdl_url, location=service_url)
customer = c.factory.create("ns0:CustomerType")
return customer
def write_customer_obj_webservice(customer):
wsdl_url = r'file:C:/somepathhere/Customer.wsdl'
service_url = r'http://someiphere/Customer'
c = suds.client.Client(wsdl_url, location=service_url)
response = c.service.save(someparameters, None, None, customer)
return response
def write_customer_obj_xml_file(customer):
output_filename = r'C\temp\testxml'
# The following line is the problem. "to_xml" does not exist and I can't find a way to do it.
xml = customer.to_xml()
fo = open(output_filename, 'a')
try:
fo.write(xml)
except:
raise
else:
response = 'All ok'
finally:
fo.close()
return response
# Get the customer object always from the wsdl.
customer = get_customer_obj()
# Since customer is an object, setting it's attributes is very easy. There are very complex objects in this system.
customer.name = "Doe J."
customer.age = 42
# Write the new customer to a webservice or store it in a file for later proccessing
if later_processing:
response = write_customer_obj_xml_file(customer)
else:
response = write_customer_obj_webservice(customer)

I found a way that works for me. The trick is to create the Client with the option "nosend=True".
In the documentation it says:
nosend - Create the soap envelope but don't send. When specified, method invocation returns a RequestContext instead of sending it.
The RequestContext object has the attribute envelope. This is the XML as string.
Some pseudo code to illustrate:
c = suds.client.Client(url, nosend=True)
customer = c.factory.create("ns0:CustomerType")
customer.name = "Doe J."
customer.age = 42
response = c.service.save(someparameters, None, None, customer)
print response.envelope # This prints the XML string that would have been sent.

You have some issues in write_customer_obj_xml_file function:
Fix bad path:
output_filename = r'C:\temp\test.xml'
The following line is the problem. "to_xml" does not exist and I can't find a way to do it.
What's the type of customer? type(customer)?
xml = customer.to_xml() # to be continued...
Why mode='a'? ('a' => append, 'w' => create + write)
Use a with statement (file context manager).
with open(output_filename, 'w') as fo:
fo.write(xml)
Don't need to return a response string: use an exception manager. The exception to catch can be EnvironmentError.
Analyse
The following call:
customer = c.factory.create("ns0:CustomerType")
Construct a CustomerType on the fly, and return a CustomerType instance customer.
I think you can introspect your customer object, try the following:
vars(customer) # display the object attributes
help(customer) # display an extensive help about your instance
Another way is to try the WSDL URLs by hands, and see the XML results.
You may obtain the full description of your CustomerType object.
And then?
Then, with the attributes list, you can create your own XML. Use an XML template and fill it with the object attributes.
You may also found the magic function (to_xml) which do the job for you. But, not sure the XML format matches your need.

client = Client(url)
client.factory.create('somename')
# The last XML request by client
client.last_sent()
# The last XML response from Web Service
client.last_received()

Pyramid route matching and query parameters

I have a Pyramid web service, and code samples are as follows:
View declaration:
#view_config(route_name="services/Prices/GetByTicker/")
def GET(request):
ticker = request.GET('ticker')
startDate = request.GET('startDate')
endDate = request.GET('endDate')
period = request.GET('period')
Routing:
config.add_route('services/Prices/GetByTicker/', 'services/Prices/GetByTicker/{ticker}/{startDate}/{endDate}/{period}')
Now I know this is all screwed up but I don't know what the convention is for Pyramid. At the moment this works inasmuch as the request gets routed to the view successfully, but then I get a "Dictionary object not callable" exception.
The URL looks horrible:
#root/services/Prices/GetByTicker/ticker=APPL/startDate=19981212/endDate=20121231/period=d
Ideally I would like to be able to use a URL something like:
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
Any Pyramid bods out there willing to take five minutes to explain what I'm doing wrong?

from you sample code, i think you use the URL Dispatch
so it should be like this
config.add_route('services/Prices/GetByTicker/', 'services/Prices/GetByTicker/')
then the URL like:
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
will match it
--edit--
you don't have to use a name like "services/Prices/GetByTicker" for route_name,and you can get the GET params use request.params['key']
View declaration:
#view_config(route_name="services_Prices_GetByTicker")
def services_Prices_GetByTicker(request):
ticker = request.params['ticker']
startDate = request.params['startDate']
endDate = request.params['endDate']
period = request.params['period']
Routing:
config.add_route('services_Prices_GetByTicker', 'services/Prices/GetByTicker/')

The query string is turned into the request.GET dictionary. You are using parenthesis to call the dictionary instead of accessing items via the brackets. For a url such as
#root/services/Prices/GetByTicker/?ticker=APPL&startDate=19981212&endDate=20121231&period=d
request.GET['ticker'] # -> 'APPL' or an exception if not available
request.GET.get('ticker') # -> 'APPL' or None if not available
request.GET.get('ticker', 'foo') # -> 'APPL' or 'foo' if not available
request.GET.getall('ticker') # -> ['APPL'] or [] if not available
The last option is useful if you expect ticker to be supplied multiple times.
request.params is a combination of request.GET and request.POST where the latter is a dictionary representing the request's body in a form upload.
Anyway, the answer is that request.GET('ticker') syntactically is not one of the options I mentioned, stop doing it. :-)

File upload with Django via PUT

I am trying to implement a function in Django to upload an image from a client (an iPhone app) to an Amazon S3 server. The iPhone app sends a HttpRequest (method PUT) with the content of the image in the HTTPBody. For instance, the client PUTs the image to the following URL: http://127.0.0.1:8000/uploadimage/sampleImage.png/
My function in Django looks like this to handle such a PUT request and save the file to S3:
def store_in_s3(filename, content):
conn = S3Connection(settings.ACCESS_KEY, settings.PASS_KEY) # gets access key and pass key from settings.py
bucket = conn.create_bucket("somepicturebucket")
k = Key(bucket)
k.key = filename
mime = mimetypes.guess_type(filename)[0]
k.set_metadata("Content-Type", mime)
k.set_contents_from_string(content)
k.set_acl("public-read")
def upload_raw_data(request, name):
if request.method == 'PUT':
store_in_s3(name,request.raw_post_data)
return HttpResponse('Upload of raw data to S3 successful')
else:
return HttpResponse('Upload not successful')
My problem is how to tell my function the name of the image. In my urls.py I have the following but it won't work:
url(r'^uploadrawdata/(\d+)/', upload_raw_data ),
Now as far as I'm aware, d+ stands for digits, so it's obviously of no use here when I pass the name of a file. However, I was wondering if this is the correct way in the first place. I read this post here and it suggests the following line of code which I don't understand at all:
file_name = path.split("/")[-1:][0]
Also, I have no clue what the rest of the code is all about. I'm a bit new to all of this, so any suggestions of how to simply upload an image would be very welcome. Thanks!

This question is not really about uploading, and the linked answer is irrelevant. If you want to accept a string rather than digits in the URL, in order to pass a filename, you can just use w instead of d in the regex.
Edit to clarify Sorry, didn't realise you were trying to pass a whole file+extension. You probably want this:
r'^uploadrawdata/(.+)/$'
so that it matches any character. You should probably read an introduction to regular expressions, though.

Unable to view item in browser based on its key in Python GAE

I'm using python GAE with webapp.
I have a form for a user to create a object in the database, something like:
class SpamRecord(db.Model):
author = db.ReferenceProperty(Author, required=True)
text = db.StringProperty()
After it's created, the user is redirected to a page whose URL contains that object's key... using code such as:
spam = SpamRecord(author=author, text=text)
spam.put()
new_spam_key = spam.key()
self.redirect("/view_spam/%s" % new_spam_key)
And this mostly works, with me being able to view items at:
sitename.com/view_spam/ag1waWNreXByZXNlbnRzchQLEgxBbm5vdW5jZW1lbnQYy8oJDA
sitename.com/view_spam/ag1waWNreXByZXNlbnRzchQLEgxBbm5vdW5jZW1lbnQY_boJDA
However, there's an occasional key that won't work. Here are 2 recent examples of pages that won't load and return HTTP 404 not found errors:
sitename.com/view_spam/ag1waWNreXByZXNlbnRzchQLEgxBbm5vdW5jZW1lbnQY-5MJDA
sitename.com/view_spam/ag1waWNreXByZXNlbnRzchQLEgxBbm5vdW5jZW1lbnQY-boJDA
My html-mappings.py contains the following mapping:
(r"/view_spam/(\w+)", ViewSpamPage)
And the ViewSpamPage looks something like:
class ViewSpamPage(webapp.RequestHandler):
def get(self, spam_id):
self.response.out.write("Got here")
Can anyone offer any insight as to why this is occurring and how it may be prevented?
Thanks very much!

In regular expressions, \w doesn't match hyphens. (It will match underscores.) For that second pair of keys, this'll result in only passing part of the key to your handler.
In your URL pattern, try r"/view_spam/(.*)" instead.