Currently my process works as follows:
Builds and sends requests to API
Receives responses
Parses JSON responses
Writes parsed values to csv
I am receiving a JSON response from the Google Directions API. My code works fine 99% of the time, but fails if I don't receive a JSON array as expected. Since I write the responses in bulk to csv after this loop, if an error occurs, I loose all of the results.
The code as is follows:
import requests
import csv
import json
import urlparse
import hashlib
import base64
import hmac
import sys
import time
from pprint import pprint
url_raw = 'https://maps.googleapis.com/maps/api/directions/json'
Private_Key = ''
client = ''
decodedkey = base64.urlsafe_b64decode(Private_Key)
with open('./Origins.csv', 'rU') as csvfile:
reader = csv.DictReader(csvfile)
origincoords = ['{Y},{X}'.format(**row) for row in reader]
with open('./Destinations.csv', 'rU') as csvfile:
reader = csv.DictReader(csvfile)
destinationcoords = ['{Y},{X}'.format(**row) for row in reader]
results=[]
session = requests.session()
for origin, destination in zip(origincoords, destinationcoords):
params ={'origin': origin, 'destination': destination, 'client': client}
request = requests.Request('GET', url_raw, params=params).prepare()
parsed_url = urlparse.urlparse(request.url)
Signature = hmac.new(decodedkey, '{}?{}'.format(parsed_url.path, parsed_url.query), hashlib.sha1).digest()
request.prepare_url(request.url, {'signature': base64.urlsafe_b64encode(Signature)})
response = session.send(request)
directions = response.json()
time.sleep(0.0)
results.append(directions)
pprint(results)
output = open('duration_distance_results.csv', 'w+')
writer = csv.DictWriter(output, delimiter=',', fieldnames=['duration(s)', 'distance(m)'])
writer.writeheader()
for route in results:
for leg in route['routes'][0]['legs']:
params = {
"duration(s)": leg['duration']['value'],
"distance(m)": leg['distance']['value'],
}
print(params)
writer.writerow(params)
If I don't get a JSON response back in the array expected, I get a list index out of range error and through this loose what had already been done up to this point.
Ideally, I think it would be more robust if I wrote to csv within the loop, as each result is received, rather than doing that after all of the results have been received. Or, as I have attempted, use if statements -
I have attempted an if statement, where if a value exists, write the value, if it doesn't, write error. However, I get the following error "NameError: name 'leg' is not defined"
if leg in route['routes'][0]['legs'] == value:
for route in results:
for leg in route['routes'][0]['legs']:
params = {
"duration(s)": leg['duration']['value'],
"distance(m)": leg['distance']['value'],
}
print(params)
writer.writerow(params)
else:
for value in results:
error = {
"duration(s)": 'error',
"distance(m)": 'error',
}
print(error)
writer.writerow(error)
I attempted an if statement which looks at the status message returned from Google. However, this fails with the error "for error_message in results['results'][0]['error_message']:
TypeError: list indices must be integers, not str"
for error_message in results['results'][0]['error_message']:
params = { "error_message": error_message}
if error_message == value:
for route in results:
for leg in route['routes'][0]['legs']:
params = {
"duration(s)": leg['duration']['value'],
"distance(m)": leg['distance']['value'],
}
print(params)
writer.writerow(params)
else:
for value in results:
error = {
"duration(s)": 'error',
"distance(m)": 'error',
}
print(error)
writer.writerow(error)
pprint(results)
As is obvious, Im learning slowly here. Comments on the best way to handle errors would be much appreciated.
The first obvious thing to do would be to check the HTTP status code for the response object. It's not an explicit part of the Google API documentation but obviously something else than a 200 ("Found") HTTP status code means you have a problem and you cannot even expect to get anything useful in the response's body. HTTP response codes are documented in the HTTP RFC.
Then if you read the API's documentation (https://developers.google.com/maps/documentation/directions/), you'll find out that the returned json dict has a 'status' key, which values are documented too. So the next obvious thing to check this status code, and behave appropriately, ie:
for result in results:
if result["status"] == "OK":
for leg in result['routes'][0]['legs']:
params = {
"duration(s)": leg['duration']['value'],
"distance(m)": leg['distance']['value'],
}
print(params)
writer.writerow(params)
else:
# not OK, let's check if we have an explicit error message:
if "error_message" in result:
print result["error_message"]
# handle the error case...
Related
I cleaned some keys of a dictionary and tried to add them into a new dict, so i can only work with them. But when i try to encode and decode keys such as Radaufhängung or Zündanlage and add them into the new dict i get an error. My question is if there is a way to go around thir or if there is a better solution to handle this (line: 49)?
my code:
import requests
import json
import time
from requests.exceptions import HTTPError
attempts = 0
def get_data_from_url(url):
try:
response = requests.get(url)
# If the response was successful, no Exception will be raised
response.raise_for_status()
except HTTPError:
return "HTTPError"
else:
response_dict = json.loads(response.text)
return response_dict
url = get_data_from_url("http://160.85.252.148/")
#(1) upon no/invalid response, the query is resubmitted with exponential backoff waiting time in between requests to avoid server overload;
while url == "HTTPError":
attempts += 1
time.sleep(attempts * 1.5)
url = get_data_from_url("http://160.85.252.148/")
print(url)
#(2) material records with missing or invalid cost are ignored;
valid_values = {}
for key in url:
if type(url[key]) == int or type(url[key]) == float and url[key].isdigit() == True:
#(3) wrongly encoded umlauts are repaired.
key = key.encode('latin1').decode('utf8')
#key = key.replace('é', 'Oe').replace('ä', 'ae').replace('ü', 'ue')
valid_values[key]=abs(url[key])
print(valid_values)
You're trying to access the dictionary with a modified key. Store the original key and use it when accessing the dictionary:
okey = key
key = key.encode('latin1').decode('utf8')
...
valid_values[key]=abs(url[okey])
Trying to find a way to iterate over the roleprivs and having issues getting to that level of the yaml from python.
testrole.yaml
info:
rolename: "testDeveloper"
desc: "Test Developer Role"
roletype: "user"
roleprivs:
admin-appliance:
name: "Administrate Appliance" # Informational Only Not used in code
description: "admin-appliance" # Informational Only Not used in code
code: "admin-appliance"
access: "full"
admin-backupSettings:
name: "Administrate Backup Settings" # Informational Only Not used in code
description: "admin-appliance" # Informational Only Not used in code
code: "admin-backupSettings"
access: "full"
I have a few different needs / use cases.
Part 1 of the script below - grab all the files in a directory and take the rolename, desc, and roletype and create a role.
Get the Role ID of the newly created role that was above.
HELP Needed - going back to the original yaml file and iterating over it and getting only the roleprivs..code and roleprivs..code --> role type would be something like admin-appliance. Keeping in mind that there are like 50 some odd features that need to be updated with the type of access.
The question:
How do i get the code and access in the yaml file into python variables?
def genericRoleCreate(baseURL, bearerToken):
print("initial")
files = glob.glob(ROLES_DIR)
logger.debug('Roles Dir '+ROLES_DIR)
for file in files:
yaml_file = file
logger.debug(yaml_file)
with open(yaml_file) as f:
try:
result=yaml.safe_load(f)
authority = result['info']['rolename']
desc = result['info']['desc']
roletype = result['info']['roletype']
url = baseURL+"/api/roles"
payload= json.dumps({"role":{"authority": authority, "description": desc, "roletype": roletype}})
headers = {'Content-Type': 'application/json','Authorization': 'Bearer ' +bearerToken}
roleResult = requests.request("POST", url, verify=False, headers=headers, data=payload)
logger.debug(roleResult.text)
except yaml.YAMLError as exc:
logger.error(exc)
# Getting Role ID
try:
with open(yaml_file) as f:
result = yaml.safe_load(f)
authority = result['info']['rolename']
url = baseURL+"/api/roles?phrase="+authority
headers = {'Content-Type': 'application/json','Authorization': 'Bearer ' +bearerToken}
roleResult = requests.request("GET", url, verify=False, headers=headers )
#print(roleResult.text)
roleID = json.loads(roleResult.text)
role = roleID['roles'][0]['id']
#logger.debug(role)
logger.info("Get Role ID")
print(role)
#return role
#logger.debug("Role ID: "+role)
except Exception as e:
logger.error('Exception occurred', exc_info=True)
logger.error('Error getting roleID')
# Start Updating
#role = getRoleId(baseURL, bearerToken)
try:
with open(yaml_file) as f:
result = yaml.safe_load(f)
except Exception as e:
logger.error("Broken")
strRoleID = str(role)
url = baseURL+"/api/roles/"+strRoleID+"/update-permission"
#logger.debug(result)
keys = list(result.keys())
for features in keys:
#logger.debug(keys)
code = result[features]['code']
access = result[features]['access']
payload = json.dumps({
"permissionCode": code,
"access": access
})
headers = {'Content-Type': 'application/json','Authorization': 'Bearer ' +bearerToken}
requests.request("PUT", url, verify=False, headers=headers, data=payload)
lets keep in mind i do know that i should be breaking that big nasty thing into multiple functions - i have it broken down in other areas - but compiling everything in a single function at the time.
I have been trying multiple iterations of how to get to the feature level. I have looked at many examples and can't seem to figure out how to drop to a level.
update 1
try:
with open(yaml_file, 'r') as f:
result = yaml.safe_load(f)
except Exception as e:
logger.error("Broken")
strRoleID = str(role)
url = baseURL+"/api/roles/"+strRoleID+"/update-permission"
#logger.debug(result)
keys = list(result['roleprivs'].keys())
#code2 = {roleprivs for roleprivs in result['roleprivs'].keys()}
#print(code2)
#return inventory, sites
for features in keys:
print(features)
The code above produces the output:
admin-appliance
admin-backupSettings
now the question is how do i go one step deeper in the chain and get code and access into a variable in python.
I ended up solving the problem after a few hours of testing and researching...The main problem i was encountering was how do i get to the next level after roleprivs. I could easily print the main elements under the roleprivs, but getting the code and access elements were a bit of a challenge.
I also kept running into an error where the indices needed to be an integer. This cleans up some of the keys that i was doing before and puts it into a one liner. Should help clean up a few for loops that i have been working with.
for k, v in result['roleprivs'].items():
print("Code "+result['roleprivs'][k]['code'])
print("Access: "+result['roleprivs'][k]['access'])
access = result['roleprivs'][k]['access']
code = result['roleprivs'][k]['code']
payload = json.dumps({
"permissionCode": code,
"access": access
})
headers = {'Content-Type': 'application/json','Authorization': 'Bearer ' +bearerToken}
requests.request("PUT", url, verify=False, headers=headers, data=payload)
From the original code i may have multiple roles in the ./config/roles/ directory. I needed to make sure i can read all and iterate in a for loop for each one. This solved it for me.
Final output:
2021-10-13 01:25:50,212:84:logger:role:genericRoleCreate:DEBUG:./config/roles/testDeveloper.yaml
2021-10-13 01:25:50,487:110:logger:role:genericRoleCreate:INFO:Get Role ID
8
Code admin-appliance
Access: full
Code admin-backupSettings
Access: full
Request Method: PATCH
There is a Query String Parameter section
Your code cannot be run to reproduce your error. Check what you get as a response here:
r = requests.patch(url, headers=self._construct_header(),data=body)
response = getattr(r,'_content').decode("utf-8")
response_json = json.loads(response)
If you pass invalid json to json.loads(), then an error occurs with a similar message.
import json
response = b'test data'.decode("utf-8")
print(response)
response_json = json.loads(response)
print(response_json)
Output:
test data
Traceback (most recent call last):
...
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
EDIT:
In your case, to avoid an error, you need to add an if-else block. After receiving the response, you need to check what exactly you received.
r = requests.patch(url, headers=self._construct_header(),data=body)
# if necessary, check content type
print(r.headers['Content-Type'])
response = getattr(r,'_content').decode("utf-8")
if r.status_code == requests.codes.ok:
# make sure you get the string "success"
# if necessary, do something with the string
return response
else:
# if necessary, check what error you have: client or server errors, etc.
# or throw an exception to indicate that something went wrong
# if necessary, make sure you get the error in json format
# you may also get an error if the json is not valid
# since your api returns json formatted error message:
response_dict = json.loads(response)
return response_dict
In these cases, your function returns the string "success" or dict with a description of the error.
Usage:
data = {
'correct_prediction': 'funny',
'is_accurate': 'False',
'newLabel': 'funny',
}
response = aiservice.update_prediction(data)
if isinstance(response, str):
print('New Prediction Status: ', response)
else:
# provide error information
# you can extract error description from dict
I'm using python to retrieve some data back from a rest API. I'm looping through a list to get data back for each record that is passed in like so:
safety_codes = ['M516RHJ', 'M16AJAR', 'Z49EJ57', 'Z1035TH', 'S0X6DJU9', 'S9099LP', 'S912AZSD', 'S72AEFH', 'S61ABKJ', 'W4XXATF']
#Building API variables:
api_data = dict()
rest_url = "https://www.gencodedemosite.com/restws/empcodes="
response_type = '&format=json'
header_details = {"KEY": "1101079000335WAXMEMU14532"}
#Making API call:
for i in generic_emp_codes:
response = requests.get(rest_url+'{}'.format(i)+response_format, headers = header_details)
data = json.loads(response.text)
api_data.update({i: data})
So long as the emp_code is valid I will return some data. But the moment one of those codes are bad, I get the following response and all of the other data is lost.
{
"error": "No record found for given employee safety code."
}
How do I filter out the these responses so that my api call does not fall over? Essentially, if codes M516RHJ and M16AJAR return that data. But if Z49EJ57 returns an error message, move that aside and keep the the data for M516RHJ and M16AJAR, then move on to the next code and repeat the process.
json_rdd = sc.parallelize(api_data)
json_df = spark.read.json(json_rdd)
json_df.printSchema()
root
|-- _corrupt_record: string (nullable = true)
Once I get that error message all of my data becomes corrupted.
results = []
bad_codes = []
#Making API call:
for i in generic_emp_codes:
response = requests.get(rest_url+'{}'.format(i)+response_format, headers = header_details)
data = json.loads(response.text)
if type(data)==dict and 'error' in data.keys():
bad_codes.append(data)
else:
results.append(data)
I've been trying to use the Poloniex APIg. I get a key and secret from my account, exactly like this. Then, following the examples, I should use one of the following forms:
import poloniex
polo = poloniex.Poloniex('yourApiKeyHere','yourSecretKeyHere123')
# or
polo.APIKey = 'yourApiKeyHere'
polo.Secret = 'yourSecretKeyHere123'
By using either one of them, I get this error:
TypeError: key: expected bytes or bytearray, but got 'str'
I've tried:
polo.Secret = b'yourSecretKeyHere123'
And get:
TypeError: Unicode-objects must be encoded before hashing
So I tried:
polo.Secret = 'yourSecretKeyHere123'.encode('utf-8')
I'm a bit out of my depth here with the encoding and would also expect the API just take my secret key as a string. What am I missing?
This code will works (assuming both your Secret & APIKey exists and also that API are not IP or withdraw-only restricted):
import urllib
import urllib2
import json
import time
import hmac,hashlib
req={}
APIKey = "<my_API_key>"
Secret = "<my_secret>"
command="returnBalances"
req['command'] = command
req['nonce'] = int(time.time()*1000)
post_data = urllib.urlencode(req)
sign = hmac.new(Secret, post_data, hashlib.sha512).hexdigest()
#print sign
headers = {
'Sign': sign,
'Key': APIKey
}
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/tradingApi', post_data, headers))
jsonRet = json.loads(ret.read())
print jsonRet