I am trying to send some JSON to my django app in a query string by using encodeURIComponent() my server enpoint receives the data just fine as I can print it to the python console.
print request.GET
The output of the following line is in this format
<QueryDict: {u'[my json array]': [u''}}>
I want to convert this to JSON so I can use to get some information but I've tried using json.loads and other means of manipulating the data with no luck.
My output should look like this
[{u'something': something}, {u'something1': something2}, {u'something3': something3}]
Any tips as to what I am doing wrong here?
QueryDict class is a subclass of regular Python dictionary, except that it handles multiple values for a same key (see MultiValueDict implementation).
If you want to dump it to a string, just use json.dumps():
json.dumps(my_query_dict)
There is also a relevant dict() method:
QueryDict.dict()
Returns dict representation of QueryDict.
Another way:
from django.http import QueryDict
# serialize
request.session["last_get_request"] = request.GET.urlencode()
# deserialize
last_get_request = QueryDict(request.session["last_get_request"])
I am using Python 2.7.13 and Django 1.11.2.You can get your data in a dictionary, so that you could access those data by using their related keys.
data = json.loads(request.GET.dict().keys()[0])
A block of code inside the function that I used to get the data. Output is also available at bottom. This will show the value of parts of the above statement.
But here I am using POST in place of GET as we are are posting data to the server.
So the above 1 line code is sufficient to get data as a dictionary in your case.
import json
# request.POST
print "request.POST = ", request.POST
print type(request.POST),"\n"
# DICTIONARY
print "request.POST.dict() = ", request.POST.dict()
print type(request.POST.dict()), "\n"
# LIST ALL KEYS(here is only 1)
print "request.POST.dict().keys() = ", request.POST.dict().keys()
print type(request.POST.dict().keys()), "\n"
# UNICODE
print "request.POST.dict().keys()[0] = ", request.POST.dict().keys()[0]
print type(request.POST.dict().keys()[0]), "\n"
# GETTING THE ORIGINAL DATA(as Dictionary)
data = json.loads(request.POST.dict().keys()[0])
# PRINTING DATA AND IT'S TYPE
print "json.loads(request.POST.dict().keys()[0]): ", data
print type(data), "\n"
# ITERATING OVER ITEMS in data dictionary
for key, value in data.iteritems():
print key, value
Let's see the output,
request.POST = <QueryDict: {u'{"fname":"Rishikesh Agrawani","email":"rishikesh0014051992#gmail.com","contact":"7353787704","message":"Have a nice day."}': [u'']}>
<class 'django.http.request.QueryDict'>
request.POST.dict() = {u'{"fname":"Rishikesh Agrawani","email":"rishikesh0014051992#gmail.com","contact":"7353787704","message":"Have a nice day."}': u''}
<type 'dict'>
request.POST.dict().keys() = [u'{"fname":"Rishikesh Agrawani","email":"rishikesh0014051992#gmail.com","contact":"7353787704","message":"Have a nice day."}']
<type 'list'>
request.POST.dict().keys()[0] = {"fname":"Rishikesh Agrawani","email":"rishikesh0014051992#gmail.com","contact":"7353787704","message":"Have a nice day."}
<type 'unicode'>
json.loads(request.POST.dict().keys()[0]): {u'message': u'Have a nice day.', u'contact': u'7353787704', u'email': u'rishikesh0014051992#gmail.com', u'fname': u'Rishikesh Agrawani'}
<type 'dict'>
message Have a nice day.
contact 7353787704
email rishikesh0014051992#gmail.com
fname Rishikesh Agrawani
If you want to handle multi values, you can do the following:
json.dumps({k: d.getlist(k) for k in d.keys()})
or use join for compactness:
json.dumps({k: ",".join(d.getlist(k)) for k in d.keys()})
or check if this is multi value, and only then show as list
json.dumps({k: (d.getlist(k) if len(d.getlist(k)) > 1 else d[k]) for k in d.keys()})
Related
the dataframe 'dataset' is automatically generated by PowerBI here is the result of my dataset.head(10).to_clipboard(sep=',', index=False)
coordinates,status
"[143.4865219,-34.7560602]",not started
"[143.4865241,-34.7561332]",not started
"[143.4865264,-34.7562088]",not started
"[143.4865286,-34.7562818]",not started
"[143.4865305,-34.7563453]",not started
"[143.4865327,-34.7564183]",not started
"[143.486535,-34.756494]",not started
"[143.4865371,-34.756567]",not started
"[143.486539,-34.7566304]",not started
"[143.4865412,-34.7567034]",not started
then to get the json
i do this data=dataset.to_json(orient='records')
which give me this results
[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"},
how i get this instead , no quotes on the coordinates values
[{"coordinates":[143.4865219,-34.7560602],"status":"not started"},{"coordinates":[143.4865241,-34.7561332],"status":"not started"},
edit
print(type(data))
<class 'str'>
You could use ast.literal_eval:
Safely evaluate an expression node or a string containing a Python
literal or container display. The string or node provided may only
consist of the following Python literal structures: strings, bytes,
numbers, tuples, lists, dicts, sets, booleans, and None.
This can be used for safely evaluating strings containing Python
values from untrusted sources without the need to parse the values
oneself.[...]
Your data seems to be a string, and not a list as Python would print it (it uses single quotes by default, the double quotes in your data seem to indicate that it is a string, ready to be saved in a json file for example). So, you have to convert it first to a Python object with json.loads:
from ast import literal_eval
import json
data = """[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]"""
data = json.loads(data)
for d in data:
d['coordinates'] = literal_eval(d['coordinates'])
print(data)
# [{'coordinates': [143.4865219, -34.7560602], 'status': 'not started'}, {'coordinates': [143.4865241, -34.7561332], 'status': 'not started'}]
import json
s = '[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]'
d = json.loads(s)
d[0]['coordinates'] = json.loads(d[0]['coordinates'])
Applying this concept to every value can be done as in
for dic in d:
for key, value in dic.items():
try:
temp = json.loads(value)
if isinstance(temp, list):
dic[key] = temp
except Exception:
pass
or if you are sure there will be a coordinates key in ever dictionary
and that key having a "list" value
for dic in d: dic['coordinates'] = json.loads(dic['coordinates'])
simply u can use eval function.
new =[]
l = '[{"coordinates":"[143.4865219,-34.7560602]","status":"not started"},{"coordinates":"[143.4865241,-34.7561332]","status":"not started"}]'
l=eval(l)
for each_element in l:
temp={}
for k,v in each_element.items():
if k =='coordinates' :
temp[k]=eval(v)
else:
temp[k]=v
new.append(temp)
print(temp)
I spent several hours on this, tried everything I found online, pulled some of the hair left on my head...
I have this JSON sent to a Flask webservice I'm writing :
{'jsonArray': '[
{
"nom":"0012345679",
"Start":"2018-08-01",
"Finish":"2018-08-17",
"Statut":"Validee"
},
{
"nom":"0012345679",
"Start":"2018-09-01",
"Finish":"2018-09-10",
"Statut":"Demande envoyée au manager"
},
{
"nom":"0012345681",
"Start":"2018-04-01",
"Finish":"2018-04-08",
"Statut":"Validee"
},
{
"nom":"0012345681",
"Start":"2018-07-01",
"Finish":"2018-07-15",
"Statut":"Validee"
}
]'}
I want to simply loop through the records :
app = Flask(__name__)
#app.route('/graph', methods=['POST'])
def webhook():
if request.method == 'POST':
req_data = request.get_json()
print(req_data) #-> shows JSON that seems to be right
##print(type(req_data['jsonArray']))
#j1 = json.dumps(req_data['jsonArray'])
#j2 = json.loads(req_data['jsonArray'])
#data = json.loads(j1)
#for rec in data:
# print(rec) #-> This seems to consider rec as one of the characters of the whole JSON string, and prints every character one by one
#for key in data:
# value = data[key]
# print("The key and value are ({}) = ({})".format(key, value)) #-> TypeError: string indices must be integers
for record in req_data['jsonArray']:
for attribute, value in rec.items(): #-> Gives error 'str' object has no attribute 'items'
print(attribute, value)
I believe I am lost between JSON object, python dict object, strings, but I don't know what I am missing. I really tried to put the JSON received through json.dumps and json.loads methods, but still nothing. What am I missing ??
I simply want to loop through each record to create another python object that I will feed to a charting library like this :
df = [dict(Task="0012345678", Start='2017-01-01', Finish='2017-02-02', Statut='Complete'),
dict(Task="0012345678", Start='2017-02-15', Finish='2017-03-15', Statut='Incomplete'),
dict(Task="0012345679", Start='2017-01-17', Finish='2017-02-17', Statut='Not Started'),
dict(Task="0012345679", Start='2017-01-17', Finish='2017-02-17', Statut='Complete'),
dict(Task="0012345680", Start='2017-03-10', Finish='2017-03-20', Statut='Not Started'),
dict(Task="0012345680", Start='2017-04-01', Finish='2017-04-20', Statut='Not Started'),
dict(Task="0012345680", Start='2017-05-18', Finish='2017-06-18', Statut='Not Started'),
dict(Task="0012345681", Start='2017-01-14', Finish='2017-03-14', Statut='Complete')]
The whole thing is wrapped in single quotes, meaning it's a string and you need to parse it.
for record in json.loads(req_data['jsonArray']):
Looking at your commented code, you did this:
j1 = json.dumps(req_data['jsonArray'])
data = json.loads(j1)
Using json.dumps on a string is the wrong idea, and moreover json.loads(json.dumps(x)) is just the same as x, so that just got you back where you started, i.e. data was the same thing as req_data['jsonArray'] (a string).
This was the right idea:
j2 = json.loads(req_data['jsonArray'])
but you never used j2.
As you've seen, iterating over a string gives you each character of the string.
I'm trying to process a log from Symphony using Pandas, but have some trouble with a malformed JSON which I can't parse.
An example of the log :
'{id:46025,
work_assignment:43313=>43313,
declaration:<p><strong>Bijkomende interventie.</strong></p>\r\n\r\n<p>H </p>\r\n\r\n<p><strong><em>Vaststellingen.</em></strong></p>\r\n\r\n<p><strong><em>CV. </em></strong>De.</p>=><p><strong>Bijkomende interventie.</strong></p>\r\n\r\n<p>He </p>\r\n\r\n<p><strong><em>Vaststellingen.</em></strong></p>\r\n\r\n<p><strong><em>CV. </em></strong>De.</p>,conclusions:<p>H </p>=><p>H </p>}'
What is the best way to process this?
For each part (id/work_assignment/declaration/etc) I would like to retrieve the old and new value (which are separated by "=>").
Use the following code:
def clean(my_log):
my_log.replace("{", "").replace("}", "") # Removes the unneeded { }
my_items = list(my_log.split(",")) # Split at the comma to get the pairs
my_dict = {}
for i in my_items:
key, value = i.split(":") # Split at the colon to separate the key and value
my_dict[key] = value # Add to the dictionary
return my_dict
Function returns a Python dictionary, which can then be converted to JSON using a serializer if needed, or directly used.
Hope I helped :D
I'm using ConfigParser which returns a dictionary of configuration data as such:
{'general': {'UserKey': 'thisisatestkey'}}
If I want to simply print the value of the UserKey key (in this case thisisatestkey), then I generally just do a print "Your key is: {0}".format(mydictvar.get('UserKey')).
If I just print out the raw dict to a string I get the above. If I use the print statement above I get result of None since there is no key in the root of the dict called UserKey. If I .get('general') I just get: {'UserKey': 'thisisatestkey'}
Obviously I could do a fore loop like so:
keydic = cp.get_config_data()
for m, k in keydic.iteritems():
for s, v in k.iteritems():
userkey = v
and then print userkey which works fine. But I want to know how I can just avoid having to do the entire for loop first and just print the darned value right inline? Thanks!
You can use
mydictvar['general']['UserKey']
Or, if keys might be missing
mydictvar.get('general', {}).get('UserKey')
mydictvar['general'] returns a dictionary object; you can then just apply [...] to that value to retrieve the next key.
This works in string formatting too:
>>> mydictvar = {'general': {'UserKey': 'thisisatestkey'}}
>>> print "Your key is: {0[general][UserKey]}".format(mydictvar)
Your key is: thisisatestkey
simply without loop:
>>> my_dict = {'general': {'UserKey': 'thisisatestkey'}}
>>> my_dict['general']['UserKey']
'thisisatestkey'
I'm trying to figure out how to query a JSON array in Python. Could someone show me how to do a simple search and print through a fairly complex array please?
The example I'm using is here: http://eu.battle.net/api/wow/realm/status
I'd like to see, for example, how to find the 'Silvermoon' server, and print say its 'population', then the 'controlling-faction' within the 'Wintergrasp' array.
The array snippet currently looks like this:
{"type":"pve","population":"high","queue":false,"wintergrasp":{"area":1,"controlling-faction":0,"status":0,"next":1382350068792},"tol-barad":{"area":21,"controlling-faction":0,"status":0,"next":1382349141932},"status":true,"name":"Silvermoon","slug":"silvermoon","battlegroup":"Cyclone / Wirbelsturm","locale":"en_GB","timezone":"Europe/Paris"}
At the moment I can access the main array, but don't seem to be able to access sub-arrays without copying the whole thing to another new variable which seems wasteful. I'd like to be able to do something like
import urllib2
import json
req = urllib2.Request("http://eu.battle.net/api/wow/realm/status", None, {})
opener = urllib2.build_opener()
f = opener.open(req)
x = json.load(f) # open the file and read into a variable
# search and find the Silvermoon server
silvermoon_array = ????
# print the population
print silvermoon_array.????
# access the Wintergrasp sub-array
wintergrasp_sub = ????
print wintergrasp_sub.???? # print the controlling-faction variable
This would really help me get to grips with how to access other things.
Python's interactive mode is a great way to progressively explore structured data. It's easy to find how to access, say, the silvermoon server data:
>>> data=json.load(urllib2.urlopen("http://eu.battle.net/api/wow/realm/status"))
>>> type(data)
<type 'dict'>
>>> data.keys()
[u'realms']
>>> type(data['realms'])
<type 'list'>
>>> type(data['realms'][0])
<type 'dict'>
>>> data['realms'][0].keys()
[u'status', u'wintergrasp', u'battlegroup', u'name', u'tol-barad', u'locale', u'queue', u'timezone', u'type', u'slug', u'population']
>>> data['realms'][0]['name']
u'Aegwynn'
>>> [realm['name'] for realm in data['realms']].index('Silvermoon')
212
>>> silvermoon= data['realms'][212]
>>> silvermoon['population']
u'high'
>>> type(silvermoon['wintergrasp'])
<type 'dict'>
>>> silvermoon['wintergrasp'].keys()
[u'status', u'next', u'controlling-faction', u'area']
>>> silvermoon['wintergrasp']['controlling-faction']
>>> silvermoon['population']
u'high'
If you don't know about them yet, you should read up on dictionary.keys, list.index and list comprehensions to understand what's going on.
After figuring out the structure of the data, you can finally rewrite the data access to be a bit more readable and efficient:
realms= data['realms']
realm_from_name= dict( [(realm['name'], realm) for realm in realms])
print realm_from_name['Silvermoon']['population']
print realm_from_name['Silvermoon']['wintergrasp']['controlling-faction']
As to copying the array to another variable being wasteful, you should know that python passes value by reference. That means that there's no copying involved when you assign something to a new variable. Here's a simple explanation of passing by value vs passing by reference
Finally, you seem to be excessively worried with performance. Python's philosophy is get it right first, optimize later. When you have this working correctly and if you need better performance, optimize it, if it's worth the time.
This does what you want:
# -*- coding: utf-8 -*-
import json
import urllib2
def searchListOfDicts(listOfDicts, attr, value):
"""
Loops through a list of dictionaries and returns matching attribute value pair
You can also pass it slug, silvermoon or type, pve
returns a list containing all matching dictionaries
"""
matches = [record for record in listOfDicts if attr in record and record[attr] == value]
return matches
def myjsonDict():
"""
Opens url, grabs json and puts it inside a dictionary
"""
req = urllib2.Request("http://eu.battle.net/api/wow/realm/status", None, {})
opener = urllib2.build_opener()
f = opener.open(req)
json_dict = json.load(f)
return json_dict
jsonDict = myjsonDict()
#we want to search inside realms list
silverMoonServers = searchListOfDicts(jsonDict["realms"], "name", "Silvermoon")
#access first dictionary that matched "name, Silvermoon" query
print silverMoonServers[0]
print silverMoonServers[0]["wintergrasp"]
print silverMoonServers[0]["wintergrasp"]["controlling-faction"]
In Python, json.loads maps json objects to python dictionaries, and Arrays to list, so further manipulation is done just like with regular python dict and list structures.
Here is how you can do it with requests and lamdbas:
import json
import requests
response = requests.get("http://eu.battle.net/api/wow/realm/status")
json_data = json.loads(response.text)
# loop through the list of realms to find the one you need (realm_name)
get_realm = lambda realm_name, jd: [r for r in jd['realms']
if r['name'] == realm_name]
# extract data you need, if there is a match in the list of realms,
# return None otherwise
get_your_data = lambda realm: (
realm[0]['name'],
realm[0]['wintergrasp']['controlling-faction']
) if realm else None
print get_your_data(get_realm('Silvermoon', json_data))