We Keep Coding

keyerror when adding key in dict

I cleaned some keys of a dictionary and tried to add them into a new dict, so i can only work with them. But when i try to encode and decode keys such as Radaufhängung or Zündanlage and add them into the new dict i get an error. My question is if there is a way to go around thir or if there is a better solution to handle this (line: 49)?
my code:
import requests
import json
import time
from requests.exceptions import HTTPError
attempts = 0
def get_data_from_url(url):
try:
response = requests.get(url)
# If the response was successful, no Exception will be raised
response.raise_for_status()
except HTTPError:
return "HTTPError"
else:
response_dict = json.loads(response.text)
return response_dict
url = get_data_from_url("http://160.85.252.148/")
#(1) upon no/invalid response, the query is resubmitted with exponential backoff waiting time in between requests to avoid server overload;
while url == "HTTPError":
attempts += 1
time.sleep(attempts * 1.5)
url = get_data_from_url("http://160.85.252.148/")
print(url)
#(2) material records with missing or invalid cost are ignored;
valid_values = {}
for key in url:
if type(url[key]) == int or type(url[key]) == float and url[key].isdigit() == True:
#(3) wrongly encoded umlauts are repaired.
key = key.encode('latin1').decode('utf8')
#key = key.replace('é', 'Oe').replace('ä', 'ae').replace('ü', 'ue')
valid_values[key]=abs(url[key])
print(valid_values)

You're trying to access the dictionary with a modified key. Store the original key and use it when accessing the dictionary:
okey = key
key = key.encode('latin1').decode('utf8')
...
valid_values[key]=abs(url[okey])

Why json output so small?

This output should be way longer than it is in here.
I start with a GET request, I parse a JSON list and extract the id, which I then call on the second function, that will give me a second ID which then I will use to call on the 3rd function. But, I am only getting one entry whereas I should be getting way more entries.
The code is the following:
from requests.auth import HTTPBasicAuth
import requests
import json
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def countries():
data = requests.get("https://localhost:8543/api/netim/v1/countries/", verify=False, auth=HTTPBasicAuth("admin", "admin"))
rep = data.json()
return [elem.get("id","") for elem in rep['items']]
def regions():
for c in countries():
url = requests.get("https://localhost:8543/api/netim/v1/countries/{}/regions".format(c), verify=False, auth=HTTPBasicAuth("admin", "admin"))
response = url.json()
return [cid.get("id","") for cid in response['items']]
def city():
for r in regions():
api = requests.get("https://localhost:8543/api/netim/v1/regions/{}/cities".format(r), verify=False, auth=HTTPBasicAuth("admin", "admin"))
resolt = api.json()
return(json.dumps([{"name":r.get("name",""),"id":r.get("id", "")} for r in resolt['items']], indent=4))
city()
print(city())
The output is the following :
[
{
"name": "Herat",
"id": "AF~HER~Herat"
}
]
I should have a huge list, so I am not sure what am I missing?

You need to go through all the iterations of your loop and collect the results, then jsonify the and return them.
data = []
for r in regions():
api = requests.get("https://localhost:8543/api/netim/v1/regions/{}/cities".format(r), verify=False, auth=HTTPBasicAuth("admin", "admin"))
resolt = api.json()
data.extend([{"name":r.get("name",""),"id":r.get("id", "")} for r in resolt['items']])
return json.dumps(data, indent=4)
This would be a fix for city() but you have the same problem in all your functions. return immediately exits the function and does not do anything else, effectively all your for loops are doing 1 iteration.
I'll update my example here to give you a better idea what's occurring.
Your functions are basically this:
def test_fn():
for i in [1,2,3,4]:
return i
# output:
1
# We never see 2 or 3 or 4 because we return before looping on them.
What you want:
def test_fn():
results = []
for i in [1,2,3,4]:
results.append(i)
return results
# output
[1,2,3,4]
It seems like you understand that the for loop is going to take some action once for each element in the list. What you're not understanding is that return ends the function NOW. No more for loop, no more actions, and in your code, you immediately return inside the for loop, stopping any further action.

Setting default to NULL with format() returns unexpected string

In my API call defined below to retrieve the last 24 hrs of data, the normal request url would be:
https://api.foobar.com/data
That is why I have set the next_page parameter default to NULL.
However, sometimes the API will return a unique URL at the end of the json (such as https://api.foobar.com/data?page%237hfaj39), which indicates another page exists and another get_data request needs to be made to retrieve the remainder.
In that case, the {next_page} parameter will be set to whatever this unique url returned would be.
My problem is after adding the {next_page} parameter, the default get_data url somehow gets 4 unwanted characters - %7B%7D appended so that the request looks like
https://api.foobar.com/data%7B%7D and of course the API does not respond.
In UTF-8 encoding %7B%7D are two brackets {}
Why does this happen and what am I doing wrong here in terms of formatting? Using None in place of {} also does not work.
The code:
def make_request(url, params={}, headers={}):
r = requests.get(url, params=params, headers=headers)
print r.url
if(not r.status_code is 200):
print "Error access API" + r.text
exit()
return r.json()
def get_data(access_token, next_page={}):
end_time = int(round(time.time() * 1000))
start_time = end_time - (seconds_in_day * 1000)
headers = {'Authorization': 'Bearer ' + access_token, 'start_time': str(start_time), 'end_time': str(end_time)}
url = 'https://api.foobar.com/data{next_page}'.format(next_page=next_page)
return make_request(url, headers=headers)
Note: the API call works when the next_page parameter is removed

With next_page={}, you will get unexpected formatting results. If you try the following:
>>> '{}'.format({})
'{}'
As you can see, instead of the desired '', you get a string with two brackets. This is because:
>>> str({})
'{}'
A similar thing happens with None:
>>> '{}'.format(None)
'None'
>>> str(None)
'None'
To fix this, instead of next_page={}, try next_page='', because .format() will do this:
>>> '{}'.format('')
''

SHA 256 of what?

https://www.binance.com/restapipub.html
I have been trying to code a trading bot. I have figured out the data and decision making part of the program. Now I need to code the making order part of the program.
I checked their website and found that I need to supply the sha256 of
clientsecret|totalparams
and that
totalParams is defined as the query string concatenated with the request body
So far this is what I have:
import requests
headers = {
'X-MBX-APIKEY': MY_API_KEY,
}
data = [
('symbol', 'LTCBTC'),
('side', 'BUY'),
('type', 'LIMIT'),
('timeInForce', 'GTC'),
('quantity', '1'),
('price', '0.1'),
('recvWindow', '6000000'),
('timestamp', '1499827319559'),
('signature', NO_IDEA ),
]
requests.post('https://www.binance.com/api/v1/order', headers=headers, data=data)
I need to figure out what the signature and by extension totalparams would be.

The documentation just wants you to use the request body, the query string on the url, and the client secret together in one string (the query string and request body are concatenated together, and then the client secret is prepended with a | character).
You can use a prepared request; this gives you access to the query string and request body before sending:
import requests
import hashlib
from urllib.parse import urlparse
def calculate_signature(secret, data=None, params=None):
# the actual URL doesn't matter as this request is never sent.
request = requests.Request('POST', 'http://example.com',
data=data, params=params)
prepped = request.prepare()
query_string = urlparse(prepped.url).query
# neither the URL nor the body are encoded to bytes yet
total_params = query_string + prepped.body
return hashlib.sha256('{}|{}'.format(secret, total_params).encode('ASCII')).hexdigest()
MY_API_KEY = 'XXX'
CLIENT_SECRET = 'XXX'
headers = {
'X-MBX-APIKEY': MY_API_KEY,
}
data = [
('symbol', 'LTCBTC'),
('side', 'BUY'),
('type', 'LIMIT'),
('timeInForce', 'GTC'),
('quantity', '1'),
('price', '0.1'),
('recvWindow', '6000000'),
('timestamp', '1499827319559'),
]
data.append(
('signature', calculate_signature(CLIENT_SECRET, data=data)))
response = requests.post('https://www.binance.com/api/v1/order', data=data, headers=headers)
The prepared request object was only used to give you the request body to sign. Their API is a little convoluted in that they then expect you to append the signature to the request body itself rather than in a header (which is what most REST APIs do).
The calculate_signature() function produces the same results as the documentation:
>>> from urllib.parse import parse_qsl
>>> documentation_secret = 'NhqPtmdSJYdKjVHjA7PZj4Mge3R5YNiP1e3UZjInClVN65XAbvqqM6A7H5fATj0j'
>>> requestBody = parse_qsl('symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.1&recvWindow=6000000&timestamp=1499827319559')
>>> calculate_signature(documentation_secret, requestBody)
'24b39c6588d0f2378a2b641e68c00e87bc81d997146ca3c5482337857a045041'
>>> queryString = parse_qsl('symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC')
>>> requestBody = parse_qsl('quantity=1&price=0.1&recvWindow=6000000&timestamp=1499827319559')
>>> calculate_signature(documentation_secret, requestBody, queryString)
'77eb3b3727bc8c523646e2a35f52a8eb4cc4418b24c113f3ea0b3b59248579d4'

hashlib provides various hash function including sha256, e.g.:
import hashlib
hashlib.sha256('|'.join([clientsecret, totalparams]).encode('utf-8')).hexdigest()

from the page you linked there is example of how to calculate it:
echo -n "NhqPtmdSJYdKjVHjA7PZj4Mge3R5YNiP1e3UZjInClVN65XAbvqqM6A7H5fATj0j|symbol=LTCBTC&side=BUY&type=LIMIT&timeInForce=GTC&quantity=1&price=0.1&recvWindow=6000000&timestamp=1499827319559" | sha256sum
24b39c6588d0f2378a2b641e68c00e87bc81d997146ca3c5482337857a045041 -
simple function to calculate the sig without too many requests manipulations (because we know data is list of tuples it will be ordered and passed down correctly, if it was dict it might not preserve the order)
import hashlib
from urllib.parse import urlencode
data = [...] # your params
def calc_sig(data, your_secret_key):
sig_content = '%s|%s' % (your_secret_key, urlencode(data))
return hashlib.sha256(sig_content).hexdigest()

Getting string instead of list when trying to get value of JSON response

I have the following JSON response:
{"response": [100, {"name": "Bill"}, {"name": "John"}]}
All I need is to iterate through the list.
So my plan is to get the list first and then iterate through it.
But when I tried to get the list with
list_dict.json().get("response")
I got the string:
100{"name": "Bill"}{"name": "John"}
How could I get the list?
UPDATE: Here is the related code views.py
from django.http import HttpResponse
from lib.api import Api
import requests, json
def verify(request):
api = Api(access_token=access_token)
list_dict = api.get_all(owner_id=owner_id)
result = list_dict.json().get("response")
return HttpResponse(result)
Here is the api.py
import requests
class Api:
def __init__(self, access_token='', **kwargs):
self.access_token = access_token
def get_all(self, owner_id=''):
api_url_template = 'http://api.example.com/method/get.All?owner_id={0}&access_token={1}'
api_url = api_url_template.format(owner_id, self.access_token)
response = requests.get(api_url)
return response

That's just the result of passing a list to HttpResponse. If you want to see the full response as a string, pass it as a string:
return HttpResponse(str(result))
Of course, to use it as a list within your code you don't need to convert it to a string, you can just use it as is.