I want to stream a response using Django. In order to do so,
I have a view like this:
def flights(request, query):
req_obj = Flights.RequestObject(query)
return StreamingHttpResponse(req_obj.make_response(), content_type="application/json")
In order to produce the data for the stream, I have a generator function "make_response",
which is a method of the class "Flights", instantiated as "req_obj".
The generator function yields, at particular moments, a pure python dictionary.
def make_response(self):
for _ in range(0,3):
yield some_dict
time.sleep(1)
This results in the following behaviour:
after the first yield, the json content return is valid;
after the second (and following) yields, the json content returned is NOT valid;
if the dictionary returned is something like this
{"data": "some_data"}
,
after the second yield, the response the user receives is:
{"data": "some_data"}{"data": "some_data"}
, which is NOT valid json.
Any suggestions on how to solve this problem?
Have you tried something like
req_obj.update(req_obj.make_response())
which will update your initial dict with the values newly yielded from your method?
The solution is on the client side (eg: browser). You need to interpret the JSON result when it comes. By definition of a stream you never know when it finished, so every time you receive some data on the client side, you need to interpret it as a self contain JSON message.
So in your case, the client would probably need to append the received JSON dictionary to an array.
You might need to manually "start" and "end" the json output. Example:
def make_response(self):
yield '['
for _ in range(0,3):
yield some_dict
yield ',' # you might add extra magic here to detect if this is the last time
# the loop will execute so you can skip the ','.
# It will still be valid json regardless
time.sleep(1)
yield ']'
Related
When I run the code:
import requests
import json
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140")
json_data = json.loads(catFact.text)
return json_data
print(get_fact())
The output is like
{'fact': "Cats are the world's most popular pets, outnumbering dogs by as many as three to one", 'length': 84}
However I just want the fact.
How do I get rid of the 'fact:' at the front and 'length:' at the back?
What you want is to access the key in the python dict you made with the json.loads call. We actually don't need the json library as requests can read and deserialize JSON itself.
This code also checks if the response was OK and fails with informative error message. It follows PEP 20 – The Zen of Python.
import requests
def get_fact():
# Get the facts dictionary in a JSON serialized form.
cat_fact_response = requests.get("https://catfact.ninja/fact?max_length=140")
# Let the response raise the exception if something bad happened to the cat facts server connection.
cat_fact_response.raise_for_status()
# Deserialize the json (make a Python dict from the text we got). requests can do that on it's own:
cat_fact_dict = cat_fact_response.json()
# Access the fact from the json from the dictionary
return cat_fact_dict['fact']
print(get_fact())
When called you get following output as wanted:
# python3 script.py
The cat's tail is used to maintain balance.
Short answer:
you need to use either get_fact()['fact'] or get_fact().get('fact'). The former will throw an exception if fact doesn't exist whereas the latter will return None.
Why:
In your code sample you fetch some json data, and then print out the entire bit of json. When you parse json, the output is a key/value map called a dictionary (or map or object in other languages). The dictionary in this case contains two keys: fact and length. If you only one want of the values, then you need to tell python that you want only a single value -- fact in this case.
Remember though: this wouldn't apply to every json object you read. Not every one is going to have a fact key.
What you are returning in get_fact is a complete JSON object which you are then printing.
To get just its property fact (without the length) use a reference to that key or property like:
return json_data["fact"]
Below is also a link to a tutorial on using JSON in Python:
w3schools: Python JSON
To extract fact field from the response, use:
import requests
import json
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140")
json_data = json.loads(catFact.text)
return json_data['fact'] # <- HERE
print(get_fact())
Output:
Cats have "nine lives" thanks to a flexible spine and powerful leg and back muscles
Note: you don't need json module here, use json() method of Response instance returned by requests:
import requests
def get_fact():
catFact = requests.get("https://catfact.ninja/fact?max_length=140").json()
return catFact['fact']
print(get_fact())
i have a webservice which gets user requests and produces (multiple) solution(s) to this request.
I want to return a solution as soon as possible, and send the remaining solutions when they are ready.
In order to do this, I thought about using Django's Http stream response. Unfortunately, I am not sure if this is the most adequate way of doing so, because of the problem I will describe below.
I have a Django view, which receives a query and answers with a stream response. This stream returns the data returned by a generator, which is always a python dictionary.
The problem is that upon the second return action of the stream, the Json content breaks.
If the python dictionary, which serves as a response, is something like {key: val}, after the second yield the returned response is {key: val} {key: val}, which is not valid Json.
Any suggestions on how to return multiple Json objects at different moments in time?
Try decoding with something like
for example
import json
json.dumps( {key: val} {key: val}, separators=('}', ':')) #check it
The solution is on the client side (eg: browser). You need to interpret the JSON result when it comes. By definition of a stream you never know when it finished, so everytime you receive some data on the client side, you need to interpret it as a self contain JSON message.
Code is importing another file, which is working perfectly.
But, there is a problem in the line where I try to import the csv file, with a column called 'account key', returning the TypeError above.
import file_import as fi
Function for collectively finding data necessary from a csv file.
def unique_students(csv_file):
unique_students_list = set()
for information in csv_file:
unique_students_list.add(csv_file["account_key"])
return len(unique_students_list)
#enrollment_num_rows = len(fi.enrollments)
#engagement_num_rows = len(fi.daily_engagement)
#submission_num_rows = len(fi.project_submissions)
#enrollment_num_unique_students = unique_students(fi.enrollments)
#engagement_num_unique_students = unique_students(fi.daily_engagement)
#submission_num_unique_students = unique_students(fi.project_submissions)
csv_file["account_key"]
Lists expect a numeric index. As far as I know, only dictionaries accept String indices.
I'm not entirely sure what this is supposed to do; I think your logic is flawed. You bind information in the for loop, then never use it. Even if the list did accept a string index, all it would do is populate the Set with the same information over and over since the for loop body remains the same same every loop. This would only work if you were expecting csv_file to be a custom container type that had side effects when indexed (like advancing some internal counter).
I am trying to retrieve all JSON like dictionaries from a long string.
For example,
{"uri": "something"} is referencing {"link": "www.aurl.com"}
I want to get {"uri": "something"} and {"link": "www.aurl.com"} as result. Is there a way to do this through regex in python?
Probably the "nicest" way to do this is to let a real JSON decoder do the work, not using horrible regexes. Find all open braces as "possible object start points", then try to parse them with JSONDecoder's raw_decode method (which returns the object parsed and number of characters consumed on success making it possible to skip successfully parsed objects efficiently). For example:
import json
def get_all_json(teststr):
decoder = json.JSONDecoder()
# Find first possible JSON object start point
sliceat = teststr.find('{')
while sliceat != -1:
# Slice off the non-object prefix
teststr = teststr[sliceat:]
try:
# See if we can parse it as a JSON object
obj, consumed = decoder.raw_decode(teststr)
except Exception:
# If we couldn't, find the next open brace to try again
sliceat = teststr.find('{', 1)
else:
# If we could, yield the parsed object and skip the text it was parsed from
yield obj
sliceat = consumed
This is a generator function, so you can either iterate the objects one by one e.g. for obj in get_all_json(mystr): or if you need them all at once for indexing, iterating multiple times or the like, all_objs = list(get_all_json(mystr)).
I'm trying to figure out what's the correct URL format for the Intersango API (which is poorly documented). I'm programming my client in C#, but I'm looking at the Python example and I'm a little confused as to what is actually being placed in the body of the request:
def make_request(self,call_name,params):
params.append(('api_key',self.api_key)) // <-- How does this get serialized?
body = urllib.urlencode(params)
self.connect()
try:
self.connection.putrequest('POST','/api/authenticated/v'+self.version+'/'+call_name+'.php')
self.connection.putheader('Connection','Keep-Alive')
self.connection.putheader('Keep-Alive','30')
self.connection.putheader('Content-type','application/x-www-form-urlencoded')
self.connection.putheader('Content-length',len(body))
self.connection.endheaders()
self.connection.send(body)
response = self.connection.getresponse()
return json.load(response)
//...
I can't figure out this piece of code: params.append(('api_key',self.api_key))
Is it some kind of a dictionary, something that gets serialized to JSON, comma delimited, or exactly how does it get serialized? What would the body look like when the parameters are encoded and assigned to it?
P.S. I don't have anything that I can run the code with so I can debug it, but I'm just hoping that this is simple enough to understand for somebody that knows Python and they would be able to tell me what's happening on that line of code.
params is a list of 2-element lists. The list would look like ((key1, value1), (key2, value2), ...)
params.append(('api_key',self.api_key)) adds another 2-element list to the existing params list.
Finally, urllib.urlencode takes this list and converts it into a propert urlencoded string. In this case, it will return a string key1=value1&key2=value2&api_key=23423. If there are any special characters in your keys or values, urlencode will %encode them. See documentation for urlencode
I tried to get the C# code working, and it kept failing with exception {"The remote server returned an error: (417) Expectation Failed."}. I finally found what the problem is. You could read about it in depth here
In short, the way to make C# access Intersango API is to add following code:
System.Net.ServicePointManager.Expect100Continue = false;
This code needs to only run once. This is a global setting, so it affects your full application, so beware that something else could break as a result.
Here's a sample code:
System.Net.ServicePointManager.Expect100Continue = false;
var address = "https://intersango.com/api/authenticated/v0.1/listAccounts.php";
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var postBytes = Encoding.UTF8.GetBytes("api_key=aa75***************fd65785");
request.ContentLength = postBytes.Length;
var dataStream = request.GetRequestStream();
dataStream.Write(postBytes, 0, postBytes.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Piece of cake
instead of params.append(('api_key',self.api_key))
just write:
params['api_key']=self.api_key