We Keep Coding

Python HTTP Requests

The following figure shows the TMDb endpoint documentation by which a client can request a movie or TV show be added to a user's watchlist, where the user has previously established a session_id.
Based on the developer documentation, and using the (fictional) values of the various pieces that need to be part of the request and that are presented in the following table:
Question !!
Can you please provide the HTTP for this request? You can present the request in a line-oriented fashion, and without needing to note linefeeds or carriage returns.

If you are asking about writing code, the following lines could be useful for you:
import requests
account_id = 12345
url = 'https://YOUR_URL.com/accout/' + account_id + '/watchlist'
payload = {'media_type': 'movie', 'media_id': 777, 'watchlist': 'true'}
x = requests.post(url, data = payload)
print(x.text)
Where is:
requests.post(url, data={key: value}, json={key: value} -> string, args)

AWS HTTP API - same request but different response in Python Requests vs Dart HTTP

I am trying to use AWS DynamoDB in a Flutter app, and given the lack of an official AWS SDK for Dart I am forced to use the low level HTTP REST API.
The method for signing an AWS HTTP request is quite tedious, but using an AWS supplied sample as a guide, I was able to convert the Python to Dart pretty much line-for-line relatively easily. The end result was both sets of code producing the same auth signatures.
My issue came when I actually went to sent the request. The Python works as expected but sending a POST with Dart's HTTP package gives the error
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.
I'll spare you the actual code for generating the auth signature, as the issue can be replicated simply by sending the same request hard-coded. See the Python and Dart code below.
Note: A valid response will return
Signature expired: 20190307T214900Z is now earlier than
20190307T215809Z (20190307T221309Z - 15 min.)
as the request signature uses current date and is only valid for 15 mins.
*****PYTHON CODE*****
import requests
headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'}
endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/'
request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}'
r = requests.post(endpoint, data=request_parameters, headers=headers)
print('Response status: %d\n' % r.status_code)
print('Response body: %s\n' % r.text)
*****DART CODE*****
import 'package:http/http.dart' as http;
void main(List<String> arguments) async {
var headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'};
var endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/';
var request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}';
http.post(endpoint, body: request_parameters, headers: headers).then((response) {
print("Response status: ${response.statusCode}");
print("Response body: ${response.body}");
});
}
The endpoint, headers and body are literally copy and pasted between the two sets of code.
Is there some nuance to how Dart HTTP works that I am missing here? Is there some map/string/json conversion of the headers or request_paramaters happening?
One thing I did note is that in the AWS provided example it states
For DynamoDB, the request can include any headers, but MUST include
"host", "x-amz-date", "x-amz-target", "content-type", and
"Authorization". Except for the authorization header, the headers must
be included in the canonical_headers and signed_headers values, as
noted earlier. Order here is not significant. Python note: The 'host'
header is added automatically by the Python 'requests' library.
But
a) When I add 'Host':'dynamodb.ap-southeast-2.amazonaws.com' to the headers in the Dart code I get the same result
and
b) If I look at r.request.headers after the Python requests returns, I can see that it has added a few new headers (Content-Length etc) automatically, but "Host" isn't one of them.
Any ideas why the seemingly same HTTP request works for Python Requests but not Dart HTTP?

Ok this is resolved now. My issue was in part a massive user-error. I was using a new IDE and when I generated the hardcoded example I provided I was actually still executing the previous file. Stupid, stupid, stupid.
But...
I was able to sort out the actual issue that caused me raise the question in the first place. I found that if you set the content type to "application/json" in the headers, the dart HTTP package automatically appends "; charset=utf-8". Because this value is part of the auth signature, when AWS encodes the values from the header to compare to the user-generated signature, they don't match.
The fix is simply to ensure that when you are setting the header content-type, make sure that you manually set it to "application/json; charset=utf-8" and not "application/json".
Found a bit more discussion about this "bug" after the fact here.

Requests package and API documentation

I'm having trouble understanding where to add parameters defined by API documentation. Take BeeBole's documentation for example, which specifies that to get an absence by ID, the following request is required:
{
"service": "absence.get",
"id": "absence_id"
}
They provide only one URL in the documentation:
BeeBole is accepting HTTP POST resquests in a json-doc format to the following URL:
https://beebole-apps.com/api/v2
How would this be implemented in the context of Python requests? The following code I've tried returns 404:
import requests
payload = {
"service": "absence.get",
"id": "absence_id"
}
auth = {
"username": "API_token",
"password": "x"
}
url = "https://beebole-apps.com/api/v2"
req = requests.get(url, params=payload, auth=auth).json()

BeeBole is accepting HTTP POST resquests in a json-doc format to the following URL: https://beebole-apps.com/api/v2
The JSON document format here is the part you missed; you need to pass the information as a JSON encoded body of the request. The params argument you used only sets the URL query string (the ?... part in a URL).
Use
import requests
payload = {
"service": "absence.get",
"id": "absence_id"
}
auth = ("API_token", "x")
url = "https://beebole-apps.com/api/v2"
req = requests.get(url, json=payload, auth=auth).json()
The json= part ensures that the payload dictionary is encoded to JSON and sent as a POST body. This also sets the Content-Type header of the request.
I've also updated the API authentication, all that the auth keyword needs here is a tuple of the username and password. See the Basic Authentication section.
You may want to wait with calling .json() on the response; check if the response was successful first:
req = requests.get(url, json=payload, auth=auth)
if not req.ok:
print('Request not OK, status:', req.status_code, req.reason)
if req.content:
print(req.text)
else:
data = req.json()
if data['status'] == 'error':
print('Request error:', data['message'])
This uses the documented error responses.

From the site documentation it would appear that this particular vendor has chosen an unusual API. Most people use different endpoints to implement different operations, but BeeBole appears to implement everything off the one endpoint, and then selects the operation by examining the "service" key in the request data.
Try
response - request.post('https://beebole-apps.com/api/v2',
json={"service": "company.list"},
headers={"authorization": TOKEN)
From the documentation I can't guarantee that will put the request in the right format, but at least if it doesn't work it should give you some clue as to how to proceed. Establishing the correct value of TOKEN is described under "Authorization" in the BeeBole documentation.
It's an unusual way to offer an API, but it seems workable.

Unsure how to authenticate my api key Kucoin

I am learning how to use the Kucoin and am having trouble with the authenticating myself to the API server.
I am trying to load all of the active orders however keep getting a 401 error.
The Kucoin API documentation states that I need to add this:
{
"KC-API-KEY": "59c5ecfe18497f5394ded813",
"KC-API-NONCE" : 1506219855000 //Client timestamp (exact to
milliseconds), before using the calibration time, the server does not
accept calls with a time difference of more than 3 seconds
"KC-API-SIGNATURE" :
"fd83147802c361575bbe72fef32ba90dcb364d388d05cb909c1a6e832f6ca3ac"
//signature after client encryption
}
as a parameter to headers of request. I am unsure what this means. Any help will be appreciated.

Creating the header can be a little tricky.
For the nonce value, or any millisecond timestamp value, I've found the best way to generate this is like this
import time
int(time.time() * 1000)
The signature requires you to order the parameters alphabetically in a query string format, combine that with the path and nonce and then hash the string using sha256 with your secret key.
If you'd like to implement it yourself feel free to copy the code from here, it's split over a few functions and should be quite readable https://github.com/sammchardy/python-kucoin/blob/0ece729c406056a428a57853345c9931d449be02/kucoin/client.py#L117
Or alternatively you may be best off just using that library. (Note: I'm the author and maintainer of python-kucoin)

Here are my working codes in Python 3:
import requests
import json
import hmac
import hashlib
import base64
from urllib.parse import urlencode
import time
api_key = 'xxxxx'
api_secret = 'xx-xxx-xx'
api_passphrase = 'xxx' #note that this is *not* trading password
base_uri = 'https://api.kucoin.com'
def get_headers(method, endpoint):
now = int(time.time() * 1000)
str_to_sign = str(now) + method + endpoint
signature = base64.b64encode(hmac.new(api_secret.encode(), str_to_sign.encode(), hashlib.sha256).digest()).decode()
passphrase = base64.b64encode(hmac.new(api_secret.encode(), api_passphrase.encode(), hashlib.sha256).digest()).decode()
return {'KC-API-KEY': api_key,
'KC-API-KEY-VERSION': '2',
'KC-API-PASSPHRASE': passphrase,
'KC-API-SIGN': signature,
'KC-API-TIMESTAMP': str(now)
}
#List Accounts
method = 'GET'
endpoint = '/api/v1/accounts'
response = requests.request(method, base_uri+endpoint, headers=get_headers(method,endpoint))
print(response.status_code)
print(response.json())
Output
200
{'code': '200000', 'data': [{'available': blah,blah,blah }]}

How do I sign a POST request using HMAC-SHA512 and the Python requests library?

I'm trying to use Python to access the trading API at poloniex.com, a cryptocurrency exchange. To do this I must follow this prescription:
All calls to the trading API are sent via HTTP POST to https://poloniex.com/tradingApi and must contain the following headers:
Key - Your API key.
Sign - The query's POST data signed by your key's "secret" according to the HMAC-SHA512 method.
Additionally, all queries must include a "nonce" POST parameter. The nonce parameter is an integer which must always be greater than the previous nonce used.
Here is what I have so far. My current issue is that I do not know how to compile the POST url so that it can be signed without sending the incomplete request first. This obviously doesn't work.
import requests
import hmac
import hashlib
import time
headers = { 'nonce': '',
'Key' : 'myKey',
'Sign': '',}
payload = { 'command': 'returnCompleteBalances',
'account': 'all'}
secret = 'mySecret'
headers['nonce'] = int(time.time())
response = requests.post( 'https://poloniex.com/tradingApi', params= payload, headers= headers )
headers['Sign'] = hmac.new( secret, response.url, hashlib.sha512)

Create a prepared request; you can add headers to that after the body has been created:
import requests
import hmac
import hashlib
request = requests.Request(
'POST', 'https://poloniex.com/tradingApi',
data=payload, headers=headers)
prepped = request.prepare()
signature = hmac.new(secret, prepped.body, digestmod=hashlib.sha512)
prepped.headers['Sign'] = signature.hexdigest()
with requests.Session() as session:
response = session.send(prepped)
I changed your params argument to data; for a POST request it is customary to send the parameters in the body, not the URL.
For the nonce, I'd use a itertools.count() object, seeded from the current time so restarts don't affect it. According to the Poloniex API documentation (which you quoted in your question), the nonce is part of the POST body, not the headers, so put it in the payload dictionary:
from itertools import count
import time
# store as a global variable
NONCE_COUNTER = count(int(time.time() * 1000))
# then every time you create a request
payload['nonce'] = next(NONCE_COUNTER)
Using int(time.time()) would re-use the same number if you created more than one request per second. The example code provided by Poloniex uses int(time.time()*1000) to make it possible to create a request every microsecond instead, but using your own monotonically increasing counter (seeded from time.time()) is far more robust.
You can also encapsulate the digest signing process in a custom authentication object; such an object is passed in the prepared request as the last step in preparation:
import hmac
import hashlib
class BodyDigestSignature(object):
def __init__(self, secret, header='Sign', algorithm=hashlib.sha512):
self.secret = secret
self.header = header
self.algorithm = algorithm
def __call__(self, request):
body = request.body
if not isinstance(body, bytes): # Python 3
body = body.encode('latin1') # standard encoding for HTTP
signature = hmac.new(self.secret, body, digestmod=self.algorithm)
request.headers[self.header] = signature.hexdigest()
return request
Use this with your requests calls:
response = requests.post(
'https://poloniex.com/tradingApi',
data=payload, headers=headers, auth=BodyDigestSignature(secret))
The argument passed in is the secret used in the HMAC digest; you can also pass in a different header name.