i have a webservice which gets user requests and produces (multiple) solution(s) to this request.
I want to return a solution as soon as possible, and send the remaining solutions when they are ready.
In order to do this, I thought about using Django's Http stream response. Unfortunately, I am not sure if this is the most adequate way of doing so, because of the problem I will describe below.
I have a Django view, which receives a query and answers with a stream response. This stream returns the data returned by a generator, which is always a python dictionary.
The problem is that upon the second return action of the stream, the Json content breaks.
If the python dictionary, which serves as a response, is something like {key: val}, after the second yield the returned response is {key: val} {key: val}, which is not valid Json.
Any suggestions on how to return multiple Json objects at different moments in time?
Try decoding with something like
for example
import json
json.dumps( {key: val} {key: val}, separators=('}', ':')) #check it
The solution is on the client side (eg: browser). You need to interpret the JSON result when it comes. By definition of a stream you never know when it finished, so everytime you receive some data on the client side, you need to interpret it as a self contain JSON message.
Related
When I run the code:
import requests
import json
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140")
json_data = json.loads(catFact.text)
return json_data
print(get_fact())
The output is like
{'fact': "Cats are the world's most popular pets, outnumbering dogs by as many as three to one", 'length': 84}
However I just want the fact.
How do I get rid of the 'fact:' at the front and 'length:' at the back?
What you want is to access the key in the python dict you made with the json.loads call. We actually don't need the json library as requests can read and deserialize JSON itself.
This code also checks if the response was OK and fails with informative error message. It follows PEP 20 – The Zen of Python.
import requests
def get_fact():
# Get the facts dictionary in a JSON serialized form.
cat_fact_response = requests.get("https://catfact.ninja/fact?max_length=140")
# Let the response raise the exception if something bad happened to the cat facts server connection.
cat_fact_response.raise_for_status()
# Deserialize the json (make a Python dict from the text we got). requests can do that on it's own:
cat_fact_dict = cat_fact_response.json()
# Access the fact from the json from the dictionary
return cat_fact_dict['fact']
print(get_fact())
When called you get following output as wanted:
# python3 script.py
The cat's tail is used to maintain balance.
Short answer:
you need to use either get_fact()['fact'] or get_fact().get('fact'). The former will throw an exception if fact doesn't exist whereas the latter will return None.
Why:
In your code sample you fetch some json data, and then print out the entire bit of json. When you parse json, the output is a key/value map called a dictionary (or map or object in other languages). The dictionary in this case contains two keys: fact and length. If you only one want of the values, then you need to tell python that you want only a single value -- fact in this case.
Remember though: this wouldn't apply to every json object you read. Not every one is going to have a fact key.
What you are returning in get_fact is a complete JSON object which you are then printing.
To get just its property fact (without the length) use a reference to that key or property like:
return json_data["fact"]
Below is also a link to a tutorial on using JSON in Python:
w3schools: Python JSON
To extract fact field from the response, use:
import requests
import json
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140")
json_data = json.loads(catFact.text)
return json_data['fact'] # <- HERE
print(get_fact())
Output:
Cats have "nine lives" thanks to a flexible spine and powerful leg and back muscles
Note: you don't need json module here, use json() method of Response instance returned by requests:
import requests
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140").json()
return catFact['fact']
print(get_fact())
I want to stream a response using Django. In order to do so,
I have a view like this:
def flights(request, query):
req_obj = Flights.RequestObject(query)
return StreamingHttpResponse(req_obj.make_response(), content_type="application/json")
In order to produce the data for the stream, I have a generator function "make_response",
which is a method of the class "Flights", instantiated as "req_obj".
The generator function yields, at particular moments, a pure python dictionary.
def make_response(self):
for _ in range(0,3):
yield some_dict
time.sleep(1)
This results in the following behaviour:
after the first yield, the json content return is valid;
after the second (and following) yields, the json content returned is NOT valid;
if the dictionary returned is something like this
{"data": "some_data"}
,
after the second yield, the response the user receives is:
{"data": "some_data"}{"data": "some_data"}
, which is NOT valid json.
Any suggestions on how to solve this problem?
Have you tried something like
req_obj.update(req_obj.make_response())
which will update your initial dict with the values newly yielded from your method?
The solution is on the client side (eg: browser). You need to interpret the JSON result when it comes. By definition of a stream you never know when it finished, so every time you receive some data on the client side, you need to interpret it as a self contain JSON message.
So in your case, the client would probably need to append the received JSON dictionary to an array.
You might need to manually "start" and "end" the json output. Example:
def make_response(self):
yield '['
for _ in range(0,3):
yield some_dict
yield ',' # you might add extra magic here to detect if this is the last time
# the loop will execute so you can skip the ','.
# It will still be valid json regardless
time.sleep(1)
yield ']'
I am new to aws Lambda, I am struggling enough to find the solution dont know what to do. I have lambda handler function. my code is designed in such a way it works with the last value only. if I pass the data inside the console like:
Data=[['1700','340','2040']]
and pass it to lambda function. it works fine. But when I use API to pass the data as API request in configure test event and print the data , it change the json structure.
json data in test event:
{
"sum_net": 1700,
"sum_tax": 340,
"sum_gross": 2040
}
Handler function:
def handler(event,extract)
data= event.values()
return
result change the json object order. values are not the same as the manually defined dictionary. is there any way to keep the structure same.problem appears when json object changes to python dict. it changes the order of keys and values
Thank you in advance.
Extracted the answer from JSON order mixed up :
You may not rely on the ordering of elements within a JSON object.
From the JSON specification at http://www.json.org/
An object is an unordered set of
name/value pairs
As a consequence, lambda and any other JSON libraries are free to rearrange the order of the elements as they see fit.
The only way to change what lambda returns is to change lambda. If it is not an open-source AWS library and not open-source, it may not be easy at all.
If you know which is the value that you want from the handler, "sum_gross" I assume, you can directly get this value with the dictionary.get("key") method:
def handler(event,extract)
data = event.get("sum_gross");
return
more info about python dictionaries: http://www.tutorialspoint.com/python/dictionary_get.htm
So, I am using API calls to get some json data, and I am using a for loop to execute the call using multiple ids which will return different data based on the user's id. So basically we need to update the permissions for some of our users so the json data looks somewhat like this for the users without permissions:
{"meta":{"httpStatus":"401 - Unauthorized","error":{"errorMessage":"Insufficient
permissions.","errorCode":"ARH_8.5"},"requestId":null}}
and this for users with the permissions:
{"result":{"elements":[{"id":"dslkjafsadl","name":"Test","ownerId"
:"sdfadfsfsda","lastModified":"2016","isActive":true},
{"id":"dsafsad","name":"Test","ownerId":"sdfasdfa","lastModified":"2016","isActive":true}],"nextPage":null},"meta":{"httpStatus":"200 - OK","requestId":"3242343"}}
so my question is if I have a big response with many results both with and without permissions how could I only pull the ids from the users with permissions, and exclude the ones without the permissions.
Say the response is stored in the variable "data" and the ids that I am looking for will be stored in "requestId" this is the code I tried:
requestId = request_id = [element['requestId'] for element in data['result']['elements']]
from this code I receive the a TypeError, and it doesn't recognize 'result' because the first response is a user without the correct permissions.
Best way is to check the http status code from the original request. Say you use the requests library to perform the GET to whatever resource, status_code of the requests object will contain 401 for permission denied.
Just relying on the code you already have, you can include a conditional statement as:
requestId = request_id = [element['requestId'] for element in data['result']['elements'] if element.get('meta').get('httpStatus') == '200 - OK']
which will basically only take elements where 'meta'.'httpStatus' is the desired one.
For the requests library, take a look here: http://docs.python-requests.org/
If you have control over your JSON:
One potential solution would be to consturct your json such that there's a parent node which describes the permissions of the returned result. For example:
{"permissions": false,
{"meta":{"httpStatus":"401 - Unauthorized","error":
{"errorMessage":"Insufficient permissions.","errorCode":"ARH_8.5"},
"requestId":null}
}
}
And
{"permissions": true,
{"result":{"elements":[{"id":"dslkjafsadl","name":"Test","ownerId"
:"sdfadfsfsda","lastModified":"2016","isActive":true},{"
id":"dsafsad","name":"Test","ownerId":"sdfasdfa","la
stModified":"2016","isActive":true}],"nextPage":null},"meta":{"h
ttpStatus":"200 - OK","requestId":"3242343"}}}
With this setup, just read the first value, and then structure your json-parsing code accordingly.
If your don't have control over the JSON:
In this case, you have to figure out which request you're receiving before you can really process it. One way to do this would be to read the names of the top keys.
In the case of the correct response, you could do something like this:
parsed_json = json.loads(json_string)
# The key "result" is only in the sucess response, not in the error response
if "result" in parsed_json:
# Success
else:
# Error
I'm trying to figure out what's the correct URL format for the Intersango API (which is poorly documented). I'm programming my client in C#, but I'm looking at the Python example and I'm a little confused as to what is actually being placed in the body of the request:
def make_request(self,call_name,params):
params.append(('api_key',self.api_key)) // <-- How does this get serialized?
body = urllib.urlencode(params)
self.connect()
try:
self.connection.putrequest('POST','/api/authenticated/v'+self.version+'/'+call_name+'.php')
self.connection.putheader('Connection','Keep-Alive')
self.connection.putheader('Keep-Alive','30')
self.connection.putheader('Content-type','application/x-www-form-urlencoded')
self.connection.putheader('Content-length',len(body))
self.connection.endheaders()
self.connection.send(body)
response = self.connection.getresponse()
return json.load(response)
//...
I can't figure out this piece of code: params.append(('api_key',self.api_key))
Is it some kind of a dictionary, something that gets serialized to JSON, comma delimited, or exactly how does it get serialized? What would the body look like when the parameters are encoded and assigned to it?
P.S. I don't have anything that I can run the code with so I can debug it, but I'm just hoping that this is simple enough to understand for somebody that knows Python and they would be able to tell me what's happening on that line of code.
params is a list of 2-element lists. The list would look like ((key1, value1), (key2, value2), ...)
params.append(('api_key',self.api_key)) adds another 2-element list to the existing params list.
Finally, urllib.urlencode takes this list and converts it into a propert urlencoded string. In this case, it will return a string key1=value1&key2=value2&api_key=23423. If there are any special characters in your keys or values, urlencode will %encode them. See documentation for urlencode
I tried to get the C# code working, and it kept failing with exception {"The remote server returned an error: (417) Expectation Failed."}. I finally found what the problem is. You could read about it in depth here
In short, the way to make C# access Intersango API is to add following code:
System.Net.ServicePointManager.Expect100Continue = false;
This code needs to only run once. This is a global setting, so it affects your full application, so beware that something else could break as a result.
Here's a sample code:
System.Net.ServicePointManager.Expect100Continue = false;
var address = "https://intersango.com/api/authenticated/v0.1/listAccounts.php";
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var postBytes = Encoding.UTF8.GetBytes("api_key=aa75***************fd65785");
request.ContentLength = postBytes.Length;
var dataStream = request.GetRequestStream();
dataStream.Write(postBytes, 0, postBytes.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Piece of cake
instead of params.append(('api_key',self.api_key))
just write:
params['api_key']=self.api_key