I have the following code
import requests
import json
import sys
credentials_User=sys.argv[1]
credentials_Password=sys.argv[2]
email=sys.argv[3]
def auth_api(login_User,login_Password,):
gooddata_user=login_User
gooddata_password=login_Password
body = json.dumps({
"postUserLogin":{
"login": gooddata_user,
"password": gooddata_password,
"remember":1,
"verify_level":0
}
})
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
url="https://reports.domain.com/gdc/account/login"
response = requests.request(
"POST",
url,
headers=headers,
data=body
)
sst=response.headers.get('Set-Cookie')
return sst
def query_api(cookie,email):
url="https://reports.domain.com/gdc/account/domains/domain/users?login="+email
body={}
headers={
'Content-Type': 'application/json',
'Accept': 'application/json',
'Cookie': cookie
}
response = requests.request(
"GET",
url,
headers=headers,
data=body
)
jsonContent=[]
jsonContent.append({response.text})
accountSettings=jsonContent[0]
print(accountSettings)
cookie=auth_api(credentials_User,credentials_Password)
profilehash=query_api(cookie,email)
The code itself works and sends a request to the Gooddata API.
The query_api() function returns JSON similar to below
{
"accountSettings": {
"items": [
{
"accountSetting": {
"login": "user#example.com",
"email": "user#example.com",
"firstName": "First Name",
"lastName": "Last Name",
"companyName": "Company Name",
"position": "Data Analyst",
"created": "2020-01-08 15:44:23",
"updated": "2020-01-08 15:44:23",
"timezone": null,
"country": "United States",
"phoneNumber": "(425) 555-1111",
"old_password": "secret$123",
"password": "secret$234",
"verifyPassword": "secret$234",
"authenticationModes": [
"SSO"
],
"ssoProvider": "sso-domain.com",
"language": "en-US",
"ipWhitelist": [
"127.0.0.1"
],
"links": {
"projects": "/gdc/account/profile/{profile_id}/projects",
"self": "/gdc/account/profile/{profile_id}",
"domain": "/gdc/domains/default",
"auditEvents": "/gdc/account/profile/{profile_id}/auditEvents"
},
"effectiveIpWhitelist": "[ 127.0.0.1 ]"
}
}
],
"paging": {
"offset": 20,
"count": 100,
"next": "/gdc/uri?offset=100"
}
}
}
The issue I am having is reading specific keys from this JSON Dict, I can use accountSettings=jsonContent[0] but that just returns the same JSON.
What I want to do is read the value of the project key within links
How would I do this with a dict?
Thanks
Based on your description, uyou have your value inside a list, (not a set. Foergt about set: sets are not used with JSON). Inside your list, you either your content as a single string, which then you'd have to parse with json.loads, or it is simply a well behaved nested data structure already extracted from JSON, but which is inside a single element list. This seems the most likely.
So, you should be able to do:
accountlink = jsonContent[0]["items"][0]["accountSetting"]["login"]
otherwise, if it is encoded as a a json string, you have to parse it first:
import json
accountlink = json.loads(jsonContent[0])["items"][0]["accountSetting"]["login"]
Now, given your question, I'd say your are on a begginer level as a programmer, or a casual user, just using Python to automatize something either way, I'd recommend you do try some exercising before proceeding: it will save you time (a lot of time). I am not trying to bully or mock anything here: this is the best advice I can offer you. Seek for tutorials that play around on the interactive mode, rather than trying entire programs at once that you'd just copy and paste.
Using the below code fixed the issue
jsonContent=json.loads(response.text)
print(type(jsonContent))
test=jsonContent["accountSettings"]["items"][0]
test2=test["accountSetting"]["links"]["self"]
print(test)
print(test2)
I believe this works because for some reason I didn't notice I was using .append for my jsonContent. This resulted in the data type being something other than it should have been.
Thanks to everyone who tried helping me.
Related
I have issue then i try get from correct information.
For example i have very big json output after request i made in post (i cant use get).
"offers": [
{
"rank": 1,
"provider": {
"id": 6653,
"isLocalProvider": false,
"logoUrl": "https://img.vxcdn.com/i/partner-energy/c_6653.png?v=878adaf9ed",
"userRatings": {
"additonalCustomerRatings": {
"price": {
"percent": 73.80
},
"service": {
"percent": 67.50
},
"switching": {
"percent": 76.37
},
"caption": {
"text": "Zusätzliche Kundenbewertungen"
}
},
I cant show it all because its very big.
Like you see "rank" 1 in this request exist 20 ranks with information like content , totalCost and i need pick them all. Like 6 rank content and totalCost, 8 rank content and totalCost.
So first off all in python i use code for getting what json data.
import requests
import json
url = "https://www.verivox.de/api/energy/offers/electricity/postCode/10555/custom?"
payload="{\"profile\":\"H0\",\"prepayment\":true,\"signupOnly\":true,\"includePackageTariffs\":true,\"includeTariffsWithDeposit\":true,\"includeNonCompliantTariffs\":true,\"bonusIncluded\":\"non-compliant\",\"maxResultsPerPage\":20,\"onlyProductsWithGoodCustomerRating\":false,\"benchmarkTariffId\":741122,\"benchmarkPermanentTariffId\":38,\"paolaLocationId\":\"71085\",\"includeEcoTariffs\":{\"includesNonEcoTariffs\":true},\"maxContractDuration\":240,\"maxContractProlongation\":240,\"usage\":{\"annualTotal\":3500,\"offPeakUsage\":0},\"priceGuarantee\":{\"minDurationInMonths\":0},\"maxTariffsPerProvider\":999,\"cancellationPeriod\":null,\"previewDisplayTime\":null,\"onlyRegionalTariffs\":false,\"sorting\":{\"criterion\":\"TotalCosts\",\"direction\":\"Ascending\"},\"includeSpecialBonusesInCalculation\":\"None\",\"totalCostViewMode\":1,\"ecoProductType\":0}"
headers = {
'Content-Type': 'application/json',
'Cookie': '__cfduid=d97a159bb287de284487ebdfa0fd097b41606303469; ASP.NET_SessionId=jfg3y20s31hclqywloocjamz; 0e3a873fd211409ead79e21fffd2d021=product=Power&ReturnToCalcLink=/power/&CustomErrorsEnabled=False&IsSignupWhiteLabelled=False; __RequestVerificationToken=vrxksNqu8CiEk9yV-_QHiinfCqmzyATcGg18dAqYXqR0L8HZNlvoHZSZienIAVQ60cB40aqfQOXFL9bsvJu7cFOcS2s1'
}
response = requests.request("POST", url, headers=headers, data=payload)
jsondata = response.json()
# print(response.text)
For it working fine, but then i try pick some data what i needed like i say before im getting
for Rankdata in str(jsondata['rank']):
KeyError: 'rank'
my code for this error.
dataRank = []
for Rankdata in str(jsondata['rank']):
dataRank.append({
'tariff':Rankdata['content'],
'cost': Rankdata['totalCost'],
'sumOfOneTimeBonuses': Rankdata['content'],
'savings': Rankdata['content']
})
Then i try do another way. Just get one or some data, but not working too.
data = response.json()
#print(data)
test = float((data['rank']['totalCost']['content']))
I know my code not perfect, but i first time deal with json what are so big and are so difficult. I will be very grateful if show my in my case example how i can pick rank 1 - rank 20 data and print it.
Thank you for your help.
If you look closely at the highest level in the json, you can see that the value for key offers is a list of dicts. You can therefore loop through it like this:
for offer in jsondata['offers']:
print(offer.get('rank'))
print(offer.get('provider').get('id'))
And the same goes for other keys in the offers.
import requests #package required to handle API requests
import json #built in package to handle JSON
import os #accesing variables from .bash_profile
i= 0 #const required to iteration of loop
f = open("/mnt/c/_KOD_/project/JSON/output.txt", "a", encoding="UTF-8" )
fraza=str
id_category=int
url = os.environ.get("PROD_LB")
print("Provide search phrase :")
input(fraza)
print("provide category:")
input(id_category)
payload = "\r\n{\r\n \"criteria\": {\r\n \"product_list.show\": true,\r\n \"product_list.show_if_below\": -1,\r\n \"product_list.limit\": -1,\r\n \"product_sum.show\": true,\r\n \"query.phrase\": \"", fraza ,"\",\r\n \"category.id_or_deeper\": [\"", id_category,"diod \"]\r\n }\r\n}"
headers = {
'Content-Type': "application/json",
'User-Agent': "PostmanRuntime/7.18.0",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Postman-Token': "75924d05-2bd0-4133-aed7-515aa644a535,bfdc06e9-6142-4822-a919-81390ba871e4",
'Host': "search.tme.eu:8443",
'Accept-Encoding': "gzip, deflate",
'Content-Length': "230",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
ress = json.loads(response.text)
list = ress["product_list"]
a = list.__len__() #no longer needed in for loop.
# this part below is quite junky and i have no clue how to revive it...
**For list, value in list.items() :
# while i < a:
f.write(list["symbol"])
f.write("\r\n")
# i +=1
f.close**
Payload above genere JSON response that look like
{
"request_id": 4454370058,
"product_list": [
{
"score": 19993,
"symbol": "M22-ES-MS2",
"id": 346733
},
.
.
.
{
"score": 19989,
"symbol": "M22-D-R-X0/KC11/I",
"id": 94432
}
],
"do_show": true,
"do_show_list": {
"do_show_products": true,
"do_show_parameters": false,
"do_show_parameter_values": false,
"do_show_flags": false
}
}
My goal is to save part of response to txt file that look like:
M22-ES-MS2
M22-D-R-X0/KC11/I
I was able to do this by saving respones to JSON file and from that to save to final format... however that was not optimal..
Currently im stuck with either of 2 errors
When i try to do this via While loop im receving :
in _encode_params
for k, vs in to_key_val_list(data):
ValueError: too many values to unpack (expected 2)
or after reading THIS
im stuck with
File "/mnt/c/_KOD_/project/JSON/request.py", line 39
For list, value in list.items() :
^
SyntaxError: invalid syntax
As I have no clue how this should work.
This is my most advanced python code I have ever created in my 3 weeks of programming.
you can load the json data using the json library and then just iterate over the products list printing the symbols.
data = """{
"request_id": 4454370058,
"product_list": [
{
"score": 19993,
"symbol": "M22-ES-MS2",
"id": 346733
},
{
"score": 19989,
"symbol": "M22-D-R-X0/KC11/I",
"id": 94432
}
],
"do_show": true,
"do_show_list": {
"do_show_products": true,
"do_show_parameters": false,
"do_show_parameter_values": false,
"do_show_flags": false
}
}"""
import json
json_data = json.loads(data)
for product in json_data['product_list']:
print(product['symbol'])
OUTPUT
M22-ES-MS2
M22-D-R-X0/KC11/I
list is a reserved name in python, try changing the name list to something else.
You can put key, value instead of list, value.
Also the For should be lower case for.
I make a request to a backend API and get the data back in json format
The response looks something like this. Kindly note that the body key values will be different and there are over 100's of them. The data1.json looks like this
[
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\"}]",
code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
},
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\",\}]",
code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
},
...
]
I need to get rid of the
"\" in all of the body key in the json response
Concatenate the key[body'] into one array
ideally it should look something like this.
[
{"body":"[{"task_ids":[],"accounts":[],"entity_ids":[12814],"guid":"2DFEB337-5F5D-4DF5-84CF-E951D237D448","id":"0034030fb97251b3","subject":"Uploaded Application",]","[{"task_ids":[],"accounts":[],"entity_ids":[12814],"guid":"2DFEB337-5F5D-4DF5-84CF-E951D237D448","id":"0034030fb97251b3","subject":"Uploaded Application",]",..}
]
I have tried replace and a lot of methods but none of them are replacing the \ so I cannot even go to step 2. What I did find out was if I save it to a text file the backslashes are replaced but then I cannot again send the response back as a json object. The code to get the data1.json file so far looks like this.
data = json.loads(r.text)
with open('data1.json', 'w') as outfile:
json.dump(data, outfile, sort_keys = True, indent = 4,
ensure_ascii = False)
Any suggestions on how to achieve the first points as in my desired output? Thanks.
(For starters, what you gave is invalid JSON, and json will either fail to parse it completely or produce something bogus. You need to make sure you extract the response correctly, and if this is really what the response is, have the sending party fix it.)
Now, about the question as asked:
You don't need to do anything special. That's just how JSON represents values that themselves contain JSON's special characters ("escapes" them with a backslash).
If you load the data via a proper JSON parser (e.g. json.loads()), it will undo that escaping, and in e.g. data[0]['body'], you will see proper data.
Of course, since that string is JSON itself, you will need to further parse it with json, too, if you need to split it into its meaningful parts...
The JSON data is incorrectly formatted and invalid JSON (missing quotes in "key" strings (e.g. code": 200,), invalid syntax in second dictionary body object, as mentioned in comment (e.g. "Uploaded Application\",\}]")).
However, after fixing these, a simple str.replace() statement can be used to get the expected JSON format. Then, simply parse the JSON content and build the desired list:
import json
data = '''[
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\"}]",
"code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
},
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\"}]",
"code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
}
]'''
r = json.loads(data.replace('\\', '').replace('"[', "[").replace("]\"", "]"))
l = []
for d in r:
l.append(d)
Now inspect the contents of l:
>>> l
[{u'body': [{u'entity_ids': [12814], u'accounts': [], u'task_ids': [], u'guid': u'2DFEB337-5F5D-4DF5-84CF-E951D237D448', u'id': u'0034030fb97251b3', u'subject': u'Uploaded Application'}], u'headers': {u'Content-Type': u'application/json'}, u'code': 200, u'name': u'0', u'msg': u'OK'},
{u'body': [{u'entity_ids': [12814], u'accounts': [], u'task_ids': [], u'guid': u'2DFEB337-5F5D-4DF5-84CF-E951D237D448', u'id': u'0034030fb97251b3', u'subject': u'Uploaded Application'}], u'headers': {u'Content-Type': u'application/json'}, u'code': 200, u'name': u'0', u'msg': u'OK'}]
I have a response that I receive from foursquare in the form of json. I have tried to access the certain parts of the object but have had no success. How would I access say the address of the object? Here is my code that I have tried.
url = 'https://api.foursquare.com/v2/venues/explore'
params = dict(client_id=foursquare_client_id,
client_secret=foursquare_client_secret,
v='20170801', ll=''+lat+','+long+'',
query=mealType, limit=100)
resp = requests.get(url=url, params=params)
data = json.loads(resp.text)
msg = '{} {}'.format("Restaurant Address: ",
data['response']['groups'][0]['items'][0]['venue']['location']['address'])
print(msg)
Here is an example of json response:
"items": [
{
"reasons": {
"count": 0,
"items": [
{
"summary": "This spot is popular",
"type": "general",
"reasonName": "globalInteractionReason"
}
]
},
"venue": {
"id": "412d2800f964a520df0c1fe3",
"name": "Central Park",
"contact": {
"phone": "2123106600",
"formattedPhone": "(212) 310-6600",
"twitter": "centralparknyc",
"instagram": "centralparknyc",
"facebook": "37965424481",
"facebookUsername": "centralparknyc",
"facebookName": "Central Park"
},
"location": {
"address": "59th St to 110th St",
"crossStreet": "5th Ave to Central Park West",
"lat": 40.78408342593807,
"lng": -73.96485328674316,
"labeledLatLngs": [
{
"label": "display",
"lat": 40.78408342593807,
"lng": -73.96485328674316
}
],
the full response can be found here
Like so
addrs=data['items'][2]['location']['address']
Your code (at least as far as loading and accessing the object) looks correct to me. I loaded the json from a file (since I don't have your foursquare id) and it worked fine. You are correctly using object/dictionary keys and array positions to navigate to what you want. However, you mispelled "address" in the line where you drill down to the data. Adding the missing 'a' made it work. I'm also correcting the typo in the URL you posted.
I answered this assuming that the example JSON you linked to is what is stored in data. If that isn't the case, a relatively easy way to see exact what python has stored in data is to import pprint and use it like so: pprint.pprint(data).
You could also start an interactive python shell by running the program with the -i switch and examine the variable yourself.
data["items"][2]["location"]["address"]
This will access the address for you.
You can go to any level of nesting by using integer index in case of an array and string index in case of a dict.
Like in your case items is an array
#items[int index]
items[0]
Now items[0] is a dictionary so we access by string indexes
item[0]['location']
Now again its an object s we use string index
item[0]['location']['address]
i am using this python script to feed my data to elasticsearch 6.0. How can i store the variable Value with type float in Elasticsearch?
I can't use the metric options for the visualization in Kibana, because all the data is stored automatically as string
from elasticsearch import Elasticsearch
Device=""
Value=""
for key, value in row.items():
Device = key
Value = value
print("Dev",Device, "Val:", Value)
doc = {'Device':Device, 'Measure':Value , 'Sourcefile':filename}
print(' doc: ', doc)
es.index(index=name, doc_type='trends', body=doc)
Thanks
EDIT:
After the advice of #Saul, i could fix this problem with the following code:
import os,csv
import time
from elasticsearch import Elasticsearch
#import pandas as pd
import requests
Datum = time.strftime("%Y-%m-%d_")
path = '/home/pi/Desktop/Data'
os.chdir(path)
name = 'test'
es = Elasticsearch()
#'Time': time ,
#url = 'http://localhost:9200/index_name?pretty'
doc = {
"mappings": {
"doc": {
"properties": {
"device": { "type": "text" },
"measure": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
#headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
#r = requests.post(url, data=json.dumps(data), headers=headers)
r= es.index(index=name, doc_type='trends', body=doc)
print(r)
You need to send a HTTP Post request using python request, as follows:
url = "http://localhost:9200/index_name?pretty”
data = {
"mappings": {
"doc": {
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)
Please replace index_name in the URL with the name of the index you are defining in to elasticsearch engine.
If you want to delete the index before creating it again, please do as follows:
url = "http://localhost:9200/index_name”
data = { }
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.delete(url, data=json.dumps(data), headers=headers)
please replace index_name in the URL with your actual index name. After deleting the index, create it again with the first code example above including the mappings that you would need. Enjoy.
Elasticsearch defines field types in the index mapping. It looks like you probably have dynamic mapping enabled, so when you send data to Elasticsearch for the first time, it makes an educated guess about the shape of your data and the field types.
Once those types are set, they are fixed for that index, and Elasticsearch will continue to interpret your data according to those types no matter what you do in your python script.
To fix this you need to either:
Define the index mapping before you load any data. This is the better option as it gives you complete control over how your data is interpreted. https://www.elastic.co/guide/en/elasticsearch/reference/6.0/mapping.html
Make sure that, the first time you send data into the index, you use the correct data types. This will rely dynamic mapping generation, but it will typically do the right thing.
Defining the index mapping is the best option. It's common to do that once off, in Kibana or with curl, or if you create a lot of indices, with a template.
However if you want to use python, you should look at the create or put_mapping functions on IndicesClient