We Keep Coding

Accomplishing Oauth2.0 authorization with refresh token through Python (Google API service creation)

I'm trying to access Google API services through a headless Linux server using Oauth2. I read through all the answers on this post: How do I authorise an app (web or installed) without user intervention? but none of them showed how to use the refresh token to generate an access token in python. pinnoyyid had a javascript example (https://stackoverflow.com/a/19766913/15713034) that went something like this:
function get_access_token_using_saved_refresh_token() {
// from the oauth playgroundfunction get_access_token_using_saved_refresh_token() {
// from the oauth playground
const refresh_token = "1/0PvMAoF9GaJFqbNsLZQg-f9NXEljQclmRP4Gwfdo_0";
// from the API console
const client_id = "559798723558-amtjh114mvtpiqis80lkl3kdo4gfm5k.apps.googleusercontent.com";
// from the API console
const client_secret = "WnGC6KJ91H40mg6H9r1eF9L";
// from https://developers.google.com/identity/protocols/OAuth2WebServer#offline
const refresh_url = "https://www.googleapis.com/oauth2/v4/token";
let refresh_request = {
body:`grant_type=refresh_token&client_id=${encodeURIComponent(client_id)}&client_secret=${encodeURIComponent(client_secret)}& refresh_token=${encodeURIComponent(refresh_token)}`;,
method: "POST",
headers: new Headers({
'Content-Type': 'application/x-www-form-urlencoded'
})
}
JavaScript isn't really my best language, but I could decipher they were sending a POST request to the google server. So I tried to recreate the request in Python with the requests package:
import requests
result = requests.post("https://www.googleapis.com/oauth2/v4/token", body={'grant_type':'refresh-token', 'client_id':client_id, 'client_secret':client_secret, 'refresh_token': refresh_token}, headers={'Content-Type': 'application/x-www-form-urlencoded'})
And when I look at result it shows it has a 200 status code (success) but when I try to examine the response, there's nothing easy to read and I can't parse the result in JSON to get the access token. The other approach I tried was to spin up a Flask server using Google's suggested code: https://developers.google.com/identity/protocols/oauth2/web-server#python_5 but that doesn't work either because when I try to return the credentials from one of the functions (object that contains the access code) that won't return JSON no matter what. I'd prefer the post request method since it is cleaner and uses less code. Thanks!

In Python, one approach is to use requests-oauthlib to perform the Backend Application Flow. This is useful when you don't have a front-end to redirect someone to, in order to approve fetching a token.

This website (https://community.atlassian.com/t5/Bitbucket-questions/Refresh-Tokens-using-Python-requests/qaq-p/1213162) says solution could be something like this:
import requests
auth = ("<consumer_id>", "<consumer_secret>")
params = {
"grant_type":"refresh_token",
"refresh_token":"<your_refresh_token_here>"
}
url = "https://www.googleapis.com/oauth2/v4/token"
ret = requests.post(url, auth=auth, data=params) #note data=params, not params=params

Since none of the solutions above worked, I had to finally just give up and use a service account.

Not persistent connection for flutter/dart http.Client()

I have a running django-server that works with sessions. A simple example from my views.py that should be enough to reproduce my problem is given here:
def test(request):
print("Session objects(")
for k,v in request.session.items():
print(k,v)
print(")")
request.session["a"] = "b"
So this does just print everything in the current session and after that saving some dummy-data in the session. If I do access this via my browser the first time the output is
Session objects(
)
so the session is empty just like expected. Then after refreshing the site the output is:
Session objects(
a b
)
also as expected, so everything seems to work just fine.
But now I want to use the site with my flutter app. For that I used the flutter packacke import 'package:http/http.dart' as http like this:
var client = http.Client();
String host = ...; // just the ip:port to my host
void my_request() async {
var response = await client.get(host + "/path/to/test/");
response = await client.get(host + "/path/to/test/");
}
So everything this should do is requesting my site twice just like i did before in the browser manually. But now my server just logges twice:
Session objects(
)
So obviously the client has a not persistent connection where the session is not preserved. But according to the doc https://pub.dev/packages/http this should work
If you're making multiple requests to the same server, you can keep open a persistent connection by using a Client rather than making one-off requests
is this a problem with my flutter/dart app or is the problem on my server? Is it maybe a big in the flutter package?
note: I first thought this could be a problem with csrf-authentication so deactivated it on my server, but this doesn't change anything ...

You don't need a 3rd-party library. It's a fairly small amount of code. So, after a first authorized request, server will respond with a cookie in the response headers, under the key set-cookie. The useful information is in the value of set-cookie key, from the beginning, to the 1st occurrence of the ; character (more on this later). For example, the value of set-cookie might look like this:
sessionid=asfweggv7bet4zw11wfpb4u415yx; expires=Fri, 06 Nov 2020 11:14:40 GMT;
You need to save it and use every time in your next authorized requests.
To save it, I created a method, which you should call after the first authorized response. You can call it after every response (if you have generic response handling), since it won't mess with the existing cookie if the server didn't send a new one.***
import 'package:flutter_secure_storage/flutter_secure_storage.dart';
const kCookie = 'my_fancy_cookie';
// ...
void _storeCookie(http.Response response) async {
String rawCookie = response.headers['set-cookie'];
if (rawCookie != null) {
int index = rawCookie.indexOf(';');
String cookie = (index == -1) ? rawCookie : rawCookie.substring(0, index);
await FlutterSecureStorage().write(key: kCookie, value: cookie);
}
}
And then before I send my request, I add the cookie to headers:
// endpoint, payload and accessToken are defined earlier
cookie = await FlutterSecureStorage().read(key: kCookie);
http.Response response = await http.post(
endpoint,
body: json.encode(payload),
headers: {
'Content-Type': 'application/json',
'Accept': 'application/json',
'Authorization': 'Bearer ' + accessToken,
'cookie': cookie,
}
);
Remember to clear the cookie from secure storage after logout. :)
*** - Servers might change the session id (to reduce things like clickjacking that we don't need to consider yet), so it's good to keep extracting the cookie from every response.

The browser is saving/sending the Django session cookie. The flutter app http requests are not.
"Persistent" connection does not mean it will handle cookies for you. You will need to do that yourself or find a third-party library that does it.

AWS HTTP API - same request but different response in Python Requests vs Dart HTTP

I am trying to use AWS DynamoDB in a Flutter app, and given the lack of an official AWS SDK for Dart I am forced to use the low level HTTP REST API.
The method for signing an AWS HTTP request is quite tedious, but using an AWS supplied sample as a guide, I was able to convert the Python to Dart pretty much line-for-line relatively easily. The end result was both sets of code producing the same auth signatures.
My issue came when I actually went to sent the request. The Python works as expected but sending a POST with Dart's HTTP package gives the error
The request signature we calculated does not match the signature you
provided. Check your AWS Secret Access Key and signing method. Consult
the service documentation for details.
I'll spare you the actual code for generating the auth signature, as the issue can be replicated simply by sending the same request hard-coded. See the Python and Dart code below.
Note: A valid response will return
Signature expired: 20190307T214900Z is now earlier than
20190307T215809Z (20190307T221309Z - 15 min.)
as the request signature uses current date and is only valid for 15 mins.
*****PYTHON CODE*****
import requests
headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'}
endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/'
request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}'
r = requests.post(endpoint, data=request_parameters, headers=headers)
print('Response status: %d\n' % r.status_code)
print('Response body: %s\n' % r.text)
*****DART CODE*****
import 'package:http/http.dart' as http;
void main(List<String> arguments) async {
var headers = {'Content-Type':'application/json',
'X-Amz-Date':'20190307T214900Z',
'X-Amz-Target':'DynamoDB_20120810.GetItem',
'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'};
var endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/';
var request_parameters = '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}';
http.post(endpoint, body: request_parameters, headers: headers).then((response) {
print("Response status: ${response.statusCode}");
print("Response body: ${response.body}");
});
}
The endpoint, headers and body are literally copy and pasted between the two sets of code.
Is there some nuance to how Dart HTTP works that I am missing here? Is there some map/string/json conversion of the headers or request_paramaters happening?
One thing I did note is that in the AWS provided example it states
For DynamoDB, the request can include any headers, but MUST include
"host", "x-amz-date", "x-amz-target", "content-type", and
"Authorization". Except for the authorization header, the headers must
be included in the canonical_headers and signed_headers values, as
noted earlier. Order here is not significant. Python note: The 'host'
header is added automatically by the Python 'requests' library.
But
a) When I add 'Host':'dynamodb.ap-southeast-2.amazonaws.com' to the headers in the Dart code I get the same result
and
b) If I look at r.request.headers after the Python requests returns, I can see that it has added a few new headers (Content-Length etc) automatically, but "Host" isn't one of them.
Any ideas why the seemingly same HTTP request works for Python Requests but not Dart HTTP?

Ok this is resolved now. My issue was in part a massive user-error. I was using a new IDE and when I generated the hardcoded example I provided I was actually still executing the previous file. Stupid, stupid, stupid.
But...
I was able to sort out the actual issue that caused me raise the question in the first place. I found that if you set the content type to "application/json" in the headers, the dart HTTP package automatically appends "; charset=utf-8". Because this value is part of the auth signature, when AWS encodes the values from the header to compare to the user-generated signature, they don't match.
The fix is simply to ensure that when you are setting the header content-type, make sure that you manually set it to "application/json; charset=utf-8" and not "application/json".
Found a bit more discussion about this "bug" after the fact here.

Streamlabs API 405 response code

I'm trying to use Streamlabs API. Streamlabs API uses Oauth2 for creating apps. So first I send whoever's using my app to an authorization link containing my app's client id and the scopes I want to use.
(Something like this: streamlabs.com/api/v1.0/authorize?client_id=CLIENT-ID-HERE&redirect_uri=REDIRECT-URI&response_type=code&scope=SOME+SCOPES+HERE)
Once I've done that I receive a code at the redirect uri specified. I then use that code to get the access token for permanent access to the connected user's account. I then receive the access token from a POST request that works perfectly... Now I run into the problem. When getting the temporary code before the access token I specified the scopes: "donations.read +donations.create+alerts.write+alerts.create".
When authorizing, the app asks for permission to the different scopes. The scope in focus is "alerts.write" so that I can send test alerts using POST requests. But this doesn't work for some reason. To send a test alert I have to send a POST request to this url: "https://streamlabs.com/api/alerts/send_test_alert"
I've tried doing that in two different ways.
1:
import requests
url = "https://streamlabs.com/api/alerts/send_test_alert"
data = {
"access_token":"UserAccessTokenHere",
"type":"donation"
}
response = requests.post(url=url, data=data)
print(response.text)
2:
import requests
url = "https://streamlabs.com/api/alerts/send_test_alert?access_token=UserAccessTokenHere&type=donation"
response = requests.post(url=url)
print(response.text)
If I do print(response) it prints "Response [405]".
But if I do print(response.text) I get a long HTML document for this page: Error response page
Any ideas what's going wrong with my Python requests? send_test_alert documentation here: Link

I've contacted support and looks like you've made the same error as me.
You're not actually sending a request to the right URL.
You are a sending a request to: "https://streamlabs.com/api/alerts/send_test_alert"
You should be using the URL: "https://streamlabs.com/api/v1.0/alerts/send_test_alert"

Invalid response (404) with Twitter API in Python (and in general)

tl;dr:
I am trying to set headers in a Python HTTP request for the first time and am getting a 404. I would appreciate any help (see bottom).
I have been experimenting with the Twitter API and have not been having much luck. Eventually I am trying to get all of the media (photos) a user has posted (20 or 50 or whatever per fetch)
In my experience with other APIs, this process would go as follows: Get The userID, Make a get request to some endpoint using that userId, get a JSON feed response.
It seems to be much more complicated in Twitter.
For instance, I do not see any URLs where I can attach an access token or client ID. Instead, in their documentation they show a place where I can retrieve my
Comsumer Key, Consumer Secret, Access Token, and, Access Token Secret
If I enter my request URI and "query", it generates an oAuth Signature, which in this case consists of a
Signature base string, Authorization header and cURL command
This is where things get confusing. It says
Important: This will only be valid for a few minutes. Also remember the cURL command will actually execute the request.
So:
Question 1: right away I am wondering how I can use these credentials to retrieve media over an entire day or a weekend if they become invalid only a few minutes later?
Question 2: Using their "exploring API" console, I can test this query (where I am trying to get the user ID for the Ford" twitter account I use GET https://api.twitter.com/1.1/users/lookup.json?screen_name=hyundai
Typing that into the browser alone returns a 404
{"errors": [{"message": "Bad Authentication data","code": 215}]}
But using their little console APP I can pick "O Auth 1 authentication" (using a twitter app I made) and I get the JSON response I want. Examining the request object in the console shows:
GET /1.1/users/lookup.json?screen_name=hyundai HTTP/1.1
Authorization:
OAuth oauth_consumer_key="555SECRET555",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1421370510",oauth_nonce="1869828628",oauth_version="1.0",oauth_token="333DONTHACKMEPLEASE333",oauth_signature="444SECRET444"
Host:
api.twitter.com
X-Target-URI:
https://api.twitter.com
Connection:
Keep-Alive
tl;dr Version:
So, I thought this was the headers object I would need to send from Python (3) to make the identical request. So here is that code:
import urllib.request
header = {
"Authorization" : "OAuth",
"oauth_consumer_key" :"555SECRET555",
"oauth_signature_method": "HMAC-SHA1",
"oauth_timestamp" : "1421362844",
"oauth_nonce":"1201915763",
"oauth_version": "1.0",
"oauth_token":"333CHANGINGTHIS33333",
"oauth_signature":"222CHANGEDTHIS222",
"Host": "api.twitter.com",
"X-Target-URI": "https://api.twitter.com",
"Connection": "Keep-Alive"
}
endpoint = 'https://api.twitter.com/1.1/users/lookup.json?screen_name=hyundai'
q = urllib.request.Request(endpoint, headers=header)
a = urllib.request.urlopen(q)
print(a.read().decode('utf-8'))
But I get a bad, 404 response.
Have I formatted my headers wrong here or is there another way to do this?

If you capture the network traffic from your request (use http not https), you will see that the headers you send are not the same as the header that are expected. This is why you are getting a 404 response.
What you want is something like
header = {
"Authorization": 'OAuth oauth_consumer_key="555SECRET555",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1421362844",oauth_nonce="1201915763",oauth_version="1.0",oauth_token="333CHANGINGTHIS33333",oauth_signature="222CHANGEDTHIS222"',
"Host": "api.twitter.com",
"X-Target-URI": "https://api.twitter.com",
"Connection": "Keep-Alive"
}
Of course, you could always use an OAuth library, such as RAuth or similar, see
Python: OAuth Library for a discussion