We Keep Coding

Python-Authlib: setting an Oauth2 Async client with (service principal) and (user, password)

I come here for help after being stuck a few hours creating an OAuth2 Async httpx client.
It's one of those situations that feels so close, and still doesn't come through in the end.
I'm using this oauth2 session/client documentation:
What I need:
Creating an oauth2_async_client with authentication implemented at the creation -hopefully-.
The flow isn't that much different than what is described in the documentation.
Understanding the usage of fetch_token in the client.
About the flow:
I have two sets of credentials: a service principal (id, secret), and (user, password) that are called inside data.
The Postman Authentication request looks like this:
// Postman call
auth_enc = btoa(sp_key+":"+sp_secret)
var a_request = {
url : auth_login,
method : 'POST',
header : { "Authorization": "Basic "+auth_enc,
"Content-Type":"application/x-www-form-urlencoded"},
body: {
mode: "urlencoded",
urlencoded : [
{ key: 'username' , value: user },
{ key: 'password' , value: pass },
{ key: 'grant_type', value:'password'},
] } }
pm.sendRequest(a_request, (error, response) => {
if (error === null) {
pm.environment.set("ERP_TOKEN", response.json()["access_token"])
};
} );
What I tried:
class ERPClientX(OauthAsyncClient):
def __init__(self, config, **kwargs):
self.config = config
clienters = {
'client_id' : config['client_id'],
'client_secret': config['client_secret'],
'token_endpoint_auth_method': 'client_secret_basic'}
authers = {
'url' : self.config['auth-url'],
'data' : self.config['auth-data']), # Keys: {grant_type, username, password}
'headers': {'Content-Type': "application/x-www-form-urlencoded"}}
super().__init__(**clienters, **kwargs)
self.create_authorization_url(**authers)
self.base_url = self.config['base-url']
self.headers.update(self.std_headers)
std_headers = { # For the general request, not auth.
'Accept-Encoding' : 'gzip, deflate',
'Content-Type' : 'application/json',
'Accept' : 'application/json',
'Format' : 'json'}
async def inside_call(self, call_params):
sub_url = f"/call-me-maybe/{params['number']}/"
the_resp = await self.get(sub_url) # prepended by base_url defined above.
# Do something with result.
results_ls = the_resp.json()['d']['results']
for each_result in results_ls:
each_result.pop('__metadata')
results_df = (pd.DataFrame(results_ls)
.assign(ts_call=dt.now().strftime("%Y-%m-%d %H:%M:%S")))
return results_df
The error I get:
TypeError: 'str' object cannot be interpreted as an integer
...
307
308 super().__init__(**clienters, **kwargs)
--> 309 uri, _ = self.create_authorization_url(**authers)
310
When I didn't include the data and headers parameters in create_authorization_url -authers dict, I got another error:
<MissingTokenError \"missing_token\".
The documentation has several examples of using fetch_token but haven't been able to get my head around it.
Thank you for your help.

Facing Issue Sending Gmail Using Google Gmail API

It shows credentials are missing when I try to use the Google Gmail API to send messages. I want to send an email to my other Gmail account using the Google Gmail API.
import sys
import requests
import base64
import sys
from email.mime.text import MIMEText
AccessToken = ""
params = {
"grant_type": "refresh_token",
"client_id": "xxxxxxxxxxxxxxx",
"client_secret": "xxxxxxxxxxxxxxx",
"refresh_token": "xxxxxxxxxxxxxxxxxxxx",
}
authorization_url = "https://www.googleapis.com/oauth2/v4/token"
r = requests.post(authorization_url, data=params)
if r.ok:
AccessToken = str((r.json()['access_token']))
EmailFrom = "Test1#gmail.com"
EmailTo = "test2#gmail.com"
def create_message(sender, to, subject, message_text):
message = MIMEText(message_text, 'html')
message['to'] = to
message['from'] = sender
message['subject'] = subject
raw = base64.urlsafe_b64encode(message.as_bytes())
raw = raw.decode()
body = {'raw': raw}
return body
body = create_message(EmailFrom, EmailTo, "Just wanna Say Waka Waka!", "Waka Waka!")
url = "https://gmail.googleapis.com/gmail/v1/users/me/messages/send"
header = {
'Authorization': 'Bearer ' + AccessToken,
'Content-Type': 'application/json',
'Accept': 'application/json'
}
r = requests.post(
url,
header,
body
)
print("\n")
print(r.text)
print("\n")
Error:
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "CREDENTIALS_MISSING",
"domain": "googleapis.com",
"metadata": {
"method": "caribou.api.proto.MailboxService.SendMessage",
"service": "gmail.googleapis.com"
}
}
]
}
}

I'm not a google api user, but I've used oauth several times, and your setup is a bit different than what I usually use or what I see from a quick sniff of Google's documention. For example, I use client-creds instead of a refresh token. For what I'm seeing above in your code, no reason to refresh an old token, when you can just mint another one. Compared to yours, usually with oauth, we'll do something like auth=(client_id, client_secret)
Lastly, before you change anything big, when you changed your header to place the AccessToken variable in quotes, did you use an f-string? What is the output of
print(header)
after you have defined it? Is it what you expect? If you didn't format it, it's going to have the variable's name rather than value.
If that's all ok, I'd try to write it according to OAuth standards that I've used several times. Not telling you how to do it, but you could try something like:
def getKey():
url = "https://www.googleapis.com/oauth2/v4/token"
client_id = "*yourgoogleclientid*"
client_secret = "*yourgoogleclientsecret*"
data = {
'grant_type': 'client_credentials'
}
r = requests.post(url, json=data, auth=(client_id, client_secret))
key = r.json()['access_token']
return key
def getWhatever(key, url):
header = {
'Authorization': f'Bearer {key} '
}
params = {
'whatever params': 'you might need'
}
response = requests.get(url, headers=header, params=params)
* parse, process, return, whatever you'd like to do with the response.*
now to use it....
if __name__ == '__main__':
myKey = getKey()
whatImLookingFor = getWhatever(myKey, "*https://google_api_endpoint*")

Generating Accèss Token- OAuth2.0

I am working with the Spotify API. I am trying to generate an access token. I followed along https://www.youtube.com/watch?v=xdq6Gz33khQ this video to generate the token.
I am getting an error
{'error': 'invalid_client'}
The code I have written is:
import base64
from wsgiref import headers
import requests
import json
client_id = "09e0b9beeba74aee986546f496823d60"
client_secret = "be1c93f2a446477e8416235b2a3f442c"
# searching for token which will help in authorization
client_creds = f"{client_id} : {client_secret}"
client_creds_b64 = base64.b64encode(client_creds.encode())
token_url = "https://accounts.spotify.com/api/token"
method = "POST"
token_data = {
"grant_type": "client_credentials"
}
token_header = {
"Authorization" : f"Basic {client_creds}" # <base64 encoded client_id:client_secret>
}
r = requests.post(token_url, data=token_data, headers=token_header)
print(r.json())
Not sure why I get this error. It could be the link I am using for the token url but can't find what to replace it with.

change
client_creds = f"{client_id} : {client_secret}"
to
client_creds = f"{client_id}:{client_secret}"
change
"Authorization" : f"Basic {client_creds}" # <base64 encoded client_id:client_secret>
to
"Authorization": f"Basic {client_creds_b64.decode()}"

Downloading custom Nessus scan report using Nessus API

I have python code that successfully downloads a Nessus scan report in csv format, but I need to add some additional fields to the downloaded report. I include parameters in the request payload to include some fields, but the scan that is downloaded does not include those fields.
I've tried changing the value of the reportedContents params to actual Boolean types with the True keyword.
Also, I changed the format to pdf and it exports a PDF file that is just a title page and a page with a blank table of contents.
The downloaded csv file has data in it, but only includes the default headers (i.e.):
Plugin ID,CVE,CVSS v2.0 Base Score,Risk,Host,Protocol,Port,Name,Synopsis,Description,Solution,See Also,Plugin Output
The raw output of the POST request looks like:
POST https://localhost:8834/scans/<scan_id>/export
X-ApiKeys: accessKey=accessKey;secretKey=secretKey
Content-Type: application/x-www-form-urlencoded
Content-Length: 122
format=csv&reportContents.vulnerabilitySections.exploitable_with=true&reportContents.vulnerabilitySections.references=true
def download_scan(scan_num):
# Post an export request
headers = {
'X-ApiKeys': 'accessKey=accessKey;secretKey=secretKey',
'Content-Type': 'application/x-www-form-urlencoded'
}
data = {
'format': 'csv',
'reportContents.vulnerabilitySections.exploitable_with': 'true',
'reportContents.vulnerabilitySections.references': 'true'
}
res = requests.post(url + '/scans/{id_num}/export'.format(id_num = scan_num), data=data, verify=False, headers=headers)
if res.status_code == 200:
export = json.loads(res.text)
file_id = export.get('file')
# Continually check the scan status until the status is ready
while True:
# Check file status
res = requests.get(url + '/scans/{id_num}/export/{file_num}/status'.format(id_num = scan_num, file_num = file_id), verify=False, headers=headers)
if res.status_code == 200:
status = json.loads(res.text)['status']
if status == 'ready':
break
# Download the scan
res = requests.get(url + '/scans/{scan_num}/export/{file_num}/download'.format(scan_num = scan_num, file_num = file_id), verify=False, headers=headers)
# If the scan is successfully downloaded, get the attachment file
if res.status_code == 200:
attachment = res.content
print("Scan downloaded!!!")
else:
raise Exception("Download request failed with status code: " + str(res))
return attachment
def main():
# Download the scan based on the scan_id. I have a helper function that returns the id that I am omitting here
try:
scan = download_scan(scan_id)
except Exception as e:
print(e)
quit()
with open("scan.csv", "wb") as f:
f.write(scan)
f.close()
if __name__ == "__main__":
main()

I'm having the exact same issue but with PowerShell. Neither my additional columns nor filters appear to be working. Was wondering if you'd had any joy getting this to work?
If I change the scan_id I get the correct different results, which suggests it is receiving the JSON but ignoring the columns and filters.
My JSON is as follows...
{
"scan_id": 3416,
"format": "csv",
"reportContents.vulnerabilitySections.cvss3_base_score": true,
"filters": {
"filter.0.quality": "gt",
"filter.0.filter": "cvss2_base_score",
"filter.0.value": "6.9",
"filter.1.quality": "neq",
"filter.1.filter": "cvss2_base_score",
"filter.1.value": ""
}
}

I managed to fix it, my problem was that I was using Python's requests module and it's data={} keyword, which defaults to header content-type: application-x-www-form-urlencoded, it generates reports with strictly 13 fields regardless of your payload.
To make it actually consider your payload, use the header "content-type": "application/json", in your code implicitly and json={} in your payload instead of data={}.
WILL NOT WORK:
requests.post(
nessus_url + f"/scans/{scan_id}/export",
data={
"format": "csv",
"template_id": "",
"reportContents": {
"csvColumns": {
"id": True,
"cve": True,
"cvss": True,
**other_columns,
}
}
},
verify=False,
headers={
"X-ApiKeys": f"accessKey={credentials['access_key']}; secretKey={credentials['secret_key']}",
},
)
WILL WORK:
requests.post(
nessus_url + f"/scans/{scan_id}/export",
json={
"format": "csv",
"template_id": "",
"reportContents": {
"csvColumns": {
"id": True,
"cve": True,
"cvss": True,
**other_columns
}
}
},
verify=False,
headers={
"X-ApiKeys": f"accessKey={credentials['access_key']}; secretKey={credentials['secret_key']}",
"content-type": "application/json",
},
)

openstack cannot retrieve authentication token from API

I am trying to retrieve the authentication token from the API using requests library from python. Here is the attempt I have made so far:
def get_token():
data = {
"auth" : {
"identity" : {
"methods" : [ "password" ],
"password": {
"user" : {
"name" : OS_USERNAME,
"domain": { "name": "Default" },
"password": OS_PASSWORD
}
}
}
}
}
r = requests.post(
OS_AUTH_URL+'/auth/tokens',
headers = HEADERS,
json = data, # https://stackoverflow.com/questions/9733638
verify = False
)
print(r.content)
j = json.loads(r.content)
return j['token']['user']['id']
I get token in the response :
{
"token": {
"issued_at": "2018-07-03T11:03:59.000000Z",
"audit_ids": [
"Fg1ywtZBQ1CkigCw70If9g"
],
"methods": [
"password"
],
"expires_at": "2018-07-03T12:03:59.000000Z",
"user": {
"password_expires_at": null,
"domain": {
"id": "default",
"name": "Default"
},
"id": "e0dc5beb383a46b98dad824c5d76e719",
"name": "admin"
}
}
}
However, when I am reusing this token to get, for instance, the list of projects :
def get_tenantID():
r = requests.get(
OS_AUTH_URL+'/auth/projects',
headers = HEADERS,
verify = False
)
return r
r = get_token()
HEADERS['X-Auth-Project-Id'] = 'admin'
HEADERS['X-Auth-Token'] = r
r = get_tenantID()
I get this error as if I would not be authenticated:
<Response [401]>
{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
Googling around and using openstack token issue command showed me that usually, token are more like:
gAAAAABaCo1F1CIMVlsTBeuqYH8tm2qR29tbkmUL4vZuhCNPXJI39TQ2YzL6Twoj8fNcAyLe3WhCYW2O1YpbBF0G8mo4bt7Kf0IRsoDOoJ6uWa3RYyJ5SQNoB_5n8EnVXbKPxFYOZ_iFBnaVtL1_XDrGbwsrlDeyy8lZTDdLsqY52pUhFR-7Uow
which is not what I get with get_token.
What am I doing wrong?
Many thanks!

When you use the auth API directly, the token issued comes in the X-Subject-Token header.
Thus, to retrieve in your python example, you could access the response.headers dict like this:
token = r.headers['X-Subject-Token']
More info about authentication in the Keystone v3 docs

1. fetch authentication token as mentioned below:
r = requests.post(
OS_AUTH_URL+'/auth/tokens',
headers = HEADERS,
json = data,
verify = False
)
token = r.headers[X-Subject-Token]
2. Pass this token in header for further request:
{
'X-Auth-Token': token
}