My goal is to reproduce/replicate the functionality of gcloud compute addresses create without depending on the gcloud binary.
I am trying to use python to authenticate a POST to a googleapis compute endpoint per the documentation at https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address about reserving a static external ip address
But my POST's return 401 every time.
I have created a JWT from google.auth.jwt python module and when I decode it the JWT has all the strings embedded that I would expect to be there.
I've also tried combinations of the following OAuth scopes to be included in the JWT:
- "https://www.googleapis.com/auth/userinfo.email"
- "https://www.googleapis.com/auth/compute"
- "https://www.googleapis.com/auth/cloud-platform"
this is my function for getting a JWT using the information in my service account's JSON key file
def _generate_jwt( tokenPath, expiry_length=3600 ):
now = int(time.time())
tokenData = load_json_data( tokenPath )
sa_email = tokenData['client_email']
payload = {
'iat': now,
# expires after 'expiry_length' seconds.
"exp": now + expiry_length,
'iss': sa_email,
"scope": " ".join( [
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/userinfo.email"
] ),
'aud': "https://www.googleapis.com/oauth2/v4/token",
'email': sa_email
}
# sign with keyfile
signer = google.auth.crypt.RSASigner.from_service_account_file( tokenPath )
jwt = google.auth.jwt.encode(signer, payload)
return jwt
once I have the JWT then I make the following post that fails, 401, ::
gapiURL = 'https://www.googleapis.com/compute/v1/projects/' + projectID + '/regions/' + region + '/addresses'
jwtToken = _generate_jwt( servicetoken )
headers = {
'Authorization': 'Bearer {}'.format( jwtToken ),
'content-type' : 'application/json',
}
post = requests.post( url=gapiURL, headers=headers, data=data )
post.raise_for_status()
return post.text
I received a 401 no matter how many combinations of scopes I used in the JWT or permissions I provided to my service account. What am I doing wrong?
edit: many thanks to #JohnHanley for pointing out that I'm missing the next/second POST to https://www.googleapis.com/oauth2/v4/token URL in GCP's auth sequence. So, you get a JWT to get an 'access token.'
I've changed my calls to use the python jwt module rather than the google.auth.jwt module in-combo with the google.auth.crypt.RSASigner. So the code is a bit simpler and I put it in a single method
## serviceAccount auth sequence for google :: JWT -> accessToken
def gke_get_token( serviceKeyDict, expiry_seconds=3600 ):
epoch_time = int(time.time())
# Generate a claim from the service account file.
claim = {
"iss": serviceKeyDict["client_email"],
"scope": " ".join([
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/userinfo.email"
]),
"aud": "https://www.googleapis.com/oauth2/v4/token",
"exp": epoch_time + expiry_seconds,
"iat": epoch_time
}
# Sign claim with JWT.
assertion = jwt.encode( claim, serviceKeyDict["private_key"], algorithm='RS256' ).decode()
data = urllib.urlencode( {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": assertion
} )
# Request the access token.
result = requests.post(
url="https://www.googleapis.com/oauth2/v4/token",
headers={
"Content-Type": "application/x-www-form-urlencoded"
},
data=data
)
result.raise_for_status()
return loadJsonData(result.text)["access_token"]
In Google Cloud there are three types of "tokens" that grant access:
Signed JWT
Access Token
Identity Token
In your case you created a Signed JWT. A few Google services accept this token. Most do not.
Once you create a Signed JWT, then next step is to call a Google OAuth endpoint and exchange for an Access Token. I wrote an article that describes this in detail:
Google Cloud – Creating OAuth Access Tokens for REST API Calls
Some Google services now accept Identity Tokens. This is called Identity Based Access Control (IBAC). This does not apply to your question but is the trend for the future in Google Cloud Authorization. An example is my article on Cloud Run + Cloud Storage + KMS:
Google Cloud – Go – Identity Based Access Control
The following example Python code shows how to exchange tokens:
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
Related
The account that is running cloudrun has been linked to the billing account, but still the cloud billing python apis throw errors. What else needs to be done in the service account ?
#1. Created json token for service account and passed into the access_bills() method
#2. The service account has role for Billing Access View
#3. Copied these methods as advised in comments from John Hanley's blog:
def load_private_key(json_cred):
''' Return the private key from the json credentials '''
return json_cred['private_key']
def create_signed_jwt(pkey, pkey_id, email, scope):
'''
Create a Signed JWT from a service account Json credentials file
This Signed JWT will later be exchanged for an Access Token
'''
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
expires_in = 3600
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# Note: this token expires and cannot be refreshed. The token must be recreated
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256", # Google uses SHA256withRSA
"typ": "JWT"
}
# JWT Payload
payload = {
"iss": email, # Issuer claim
"sub": email, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scope # Permissions
}
# Encode the headers and payload and sign creating a Signed JWT (JWS)
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
return sig
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
def access_bills(sa_json):
cred = json.loads(sa_json)
private_key = load_private_key(cred)
# scopes = "https://www.googleapis.com/auth/cloud-platform" # this does not work, gets 404
scopes = "https://www.googleapis.com/auth/cloud-billing.readonly"
s_jwt = create_signed_jwt(
private_key,
cred['private_key_id'],
cred['client_email'],
scopes)
token, err = exchangeJwtForAccessToken(s_jwt)
if token is None:
logger.error("Error: {}".format(err))
exit(1)
logger.info("Token response: {}".format(token))
# the token is obtained and prints in the log
headers = {
"Host": "www.googleapis.com",
"Authorization": "Bearer " + token,
"Content-Type": "application/json"
}
try:
url = "https://cloudbilling.googleapis.com/v1/billingAccounts/01C8DC-336472-E177E1" # account name is "Billing Core"
response = requests.get(url=url, headers=headers)
logger.info("Response: {}".format(response))
# logs -> app - INFO - Response: <Response [404]>
return {
'statusCode': 200,
'body': 'Success'
}
except Exception as e:
logger.error("Error")
raise e
It gives 404 error as shown in the comment log after trying on that url.
Okay I found that the only way it works as of now is via big query export from billing account, sort out dataViewer permission and run the corresponding sql from python application, it works.
In Google Cloud documentation, the command line to get the list and number of instances that are running on a project in Google Cloud is given as following :
gcloud compute instances list
or
GET https://compute.googleapis.com/compute/v1/projects/{project}/zones/{zone}/instances
How can I get the equivalent function in Python using Google Cloud ?
There is documentation on this here: https://cloud.google.com/compute/docs/tutorials/python-guide
In short:
import googleapiclient.discovery
project="your project"
zone="your zone"
compute = googleapiclient.discovery.build('compute', 'v1')
instances = compute.instances().list(project=project, zone=zone).execute()
for instance in instances:
print(' - ' + instance['name'])
Below is an example I wrote that uses the REST API and does not use one of the Google Cloud SDKs.
This example will teach you the low level details of services accounts, authorization, access tokens and the Compute Engine REST API.
'''
This program lists lists the Google Compute Engine Instances in one zone
'''
import time
import json
import jwt
import requests
import httplib2
# Project ID for this request.
project = 'development-123456'
# The name of the zone for this request.
zone = 'us-west1-a'
# Service Account Credentials, Json format
json_filename = 'service-account.json'
# Permissions to request for Access Token
scopes = "https://www.googleapis.com/auth/cloud-platform"
# Set how long this token will be valid in seconds
expires_in = 3600 # Expires in 1 hour
def load_json_credentials(filename):
''' Load the Google Service Account Credentials from Json file '''
with open(filename, 'r') as f:
data = f.read()
return json.loads(data)
def load_private_key(json_cred):
''' Return the private key from the json credentials '''
return json_cred['private_key']
def create_signed_jwt(pkey, pkey_id, email, scope):
'''
Create a Signed JWT from a service account Json credentials file
This Signed JWT will later be exchanged for an Access Token
'''
# Google Endpoint for creating OAuth 2.0 Access Tokens from Signed-JWT
auth_url = "https://www.googleapis.com/oauth2/v4/token"
issued = int(time.time())
expires = issued + expires_in # expires_in is in seconds
# Note: this token expires and cannot be refreshed. The token must be recreated
# JWT Headers
additional_headers = {
'kid': pkey_id,
"alg": "RS256",
"typ": "JWT" # Google uses SHA256withRSA
}
# JWT Payload
payload = {
"iss": email, # Issuer claim
"sub": email, # Issuer claim
"aud": auth_url, # Audience claim
"iat": issued, # Issued At claim
"exp": expires, # Expire time
"scope": scope # Permissions
}
# Encode the headers and payload and sign creating a Signed JWT (JWS)
sig = jwt.encode(payload, pkey, algorithm="RS256", headers=additional_headers)
return sig
def exchangeJwtForAccessToken(signed_jwt):
'''
This function takes a Signed JWT and exchanges it for a Google OAuth Access Token
'''
auth_url = "https://www.googleapis.com/oauth2/v4/token"
params = {
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": signed_jwt
}
r = requests.post(auth_url, data=params)
if r.ok:
return(r.json()['access_token'], '')
return None, r.text
def gce_list_instances(accessToken):
'''
This functions lists the Google Compute Engine Instances in one zone
'''
# Endpoint that we will call
url = "https://www.googleapis.com/compute/v1/projects/" + project + "/zones/" + zone + "/instances"
# One of the headers is "Authorization: Bearer $TOKEN"
headers = {
"Host": "www.googleapis.com",
"Authorization": "Bearer " + accessToken,
"Content-Type": "application/json"
}
h = httplib2.Http()
resp, content = h.request(uri=url, method="GET", headers=headers)
status = int(resp.status)
if status < 200 or status >= 300:
print('Error: HTTP Request failed')
return
j = json.loads(content.decode('utf-8').replace('\n', ''))
print('Compute instances in zone', zone)
print('------------------------------------------------------------')
for item in j['items']:
print(item['name'])
if __name__ == '__main__':
cred = load_json_credentials(json_filename)
private_key = load_private_key(cred)
s_jwt = create_signed_jwt(
private_key,
cred['private_key_id'],
cred['client_email'],
scopes)
token, err = exchangeJwtForAccessToken(s_jwt)
if token is None:
print('Error:', err)
exit(1)
gce_list_instances(token)
I am automating the deployment of an infrastructure containing an Azure Databricks instance. To be able to use the Azure Blob Storage from within Databricks I want to create a Secret Scope via the Databricks REST API 2.0 in my DevOps Pipeline running a Python job.
When I try to create the secret scope, I get the response
{"message":"Authentication is temporarily unavailable. Please try again later.", "error_code": "TEMPORARILY_UNAVAILABLE"}
I was already able to create a databricks access token using the API, i.e. the endpoint /token/create worked perfectly.
I am authenticating to databricks using the Code from the answer to this question: https://stackoverflow.com/a/61826488/2196531
This is how I am able to create a token and how I try to generate the scope:
import requests
import adal
import json
# set variables
clientId = "<Service Principal Id>"
tenantId = "<Tenant Id>"
clientSecret = "<Service Principal Secret>"
subscription_id = "<Subscription Id>"
resource_group = "<Resource Group Name>"
databricks_workspace = "<Databricks Workspace Name>"
dbricks_url = "<Databricks Azure URL>"
# Acquire a token to authenticate against Azure management API
authority_url = 'https://login.microsoftonline.com/'+tenantId
context = adal.AuthenticationContext(authority_url)
token = context.acquire_token_with_client_credentials(
resource='https://management.core.windows.net/',
client_id=clientId,
client_secret=clientSecret
)
azToken = token.get('accessToken')
# Acquire a token to authenticate against the Azure Databricks Resource
token = context.acquire_token_with_client_credentials(
resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
client_id=clientId,
client_secret=clientSecret
)
adbToken = token.get('accessToken')
# Format Request API Url
dbricks_api = "https://{}/api/2.0".format(dbricks_url)
# Request Authentication
dbricks_auth = {
"Authorization": "Bearer {}".format(adbToken),
"X-Databricks-Azure-SP-Management-Token": azToken,
"X-Databricks-Azure-Workspace-Resource-Id": ("/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Databricks/workspaces/{}".format(subscription_id, resource_group, databricks_workspace) )
}
# Creating a databricks token
payload = {
"comment": "This token is created by API call"
}
requests.post(f"{dbricks_api}/token/create", headers=dbricks_auth, json=payload)
# works
# Creating a databricks secret scope
payload = {
"scope": "my-databricks-secret-scope",
"initial_manage_principal": "users"
}
requests.post(f"{dbricks_api}/secrets/scopes/create", headers=dbricks_auth, json=payload)
# returns {"message":"Authentication is temporarily unavailable. Please try again later.", "error_code": "TEMPORARILY_UNAVAILABLE"}
Databricks is running in westeurope.
Python 3.8.5 x64
Packages used in the Code snippet
adal-1.2.4
requests-2.24.0
Is there a problem with the databricks API or am I doing something wrong?
According to my test, when we use the Databricks Rest API to create Secret Scope, we should use the person access token.
For example
Create a service principal
az login
az ad sp create-for-rbac -n "MyApp"
Code
import requests
import adal
import json
# set variables
clientId = "<Service Principal Id>"
tenantId = "<Tenant Id>"
clientSecret = "<Service Principal Secret>"
subscription_id = "<Subscription Id>"
resource_group = "<Resource Group Name>"
databricks_workspace = "<Databricks Workspace Name>"
dbricks_url = "<Databricks Azure URL>"
# Acquire a token to authenticate against Azure management API
authority_url = 'https://login.microsoftonline.com/'+tenantId
context = adal.AuthenticationContext(authority_url)
token = context.acquire_token_with_client_credentials(
resource='https://management.core.windows.net/',
client_id=clientId,
client_secret=clientSecret
)
azToken = token.get('accessToken')
# Acquire a token to authenticate against the Azure Databricks Resource
token = context.acquire_token_with_client_credentials(
resource="2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
client_id=clientId,
client_secret=clientSecret
)
adbToken = token.get('accessToken')
# Format Request API Url
dbricks_api = "https://{}/api/2.0".format(dbricks_url)
# Request Authentication
dbricks_auth = {
"Authorization": "Bearer {}".format(adbToken),
"X-Databricks-Azure-SP-Management-Token": azToken,
"X-Databricks-Azure-Workspace-Resource-Id": ("/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Databricks/workspaces/{}".format(subscription_id, resource_group, databricks_workspace) )
}
# Creating a databricks token
payload = {
"lifetime_seconds": 3600, # the token lifetime
"comment": "This token is created by API call"
}
data =requests.post(f"{dbricks_api}/token/create", headers=dbricks_auth, json=payload)
dict_content = json.loads(data.content.decode('utf-8'))
token = dict_content.get('token_value')
payload = {
"scope": "my-databricks-secret-scope",
"initial_manage_principal": "users"
}
res=requests.post(f"{dbricks_api}/secrets/scopes/create", headers={
"Authorization": "Bearer {}".format(token),
}, json=payload)
print(res.status_code)
Currently Now I am developing code to forward emails onto another email_id.
My code is as below.
flow = flow_from_clientsecrets(settings.CLIENTSECRETS_LOCATION, ' '.join(settings.SCOPES))
flow.redirect_uri = settings.REDIRECT_URI
credentials = flow.step2_exchange(authorization_code)
url = 'https://www.googleapis.com/gmail/v1/users/moon#mydomain.com/settings/forwardingAddresses'
body = {
"forwardingEmail": "test#mydomain.com",
}
headers = {'Authorization': 'Bearer ' + credentials.access_token}
api_call_response = requests.post(url, headers=headers, json=body)
I have used below scopes
SCOPES = [
'https://www.googleapis.com/auth/gmail.readonly',
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/gmail.settings.sharing',
'https://mail.google.com/',
]
I am getting below error:
Access restricted to service accounts that have been delegated domain-wide authority
But when I am doing same with postman where I am generating access token using Get New Access Token button api works perfectly.
Postman SS access token:
My question is when generating access token from postman It works properly but with code it generating access restriction error.
Thanks.
Is it possible to use an Authorization: Bearer … header to make a request through Identity Aware Proxy to my protected application? (Using a service account, of course. From outside GCP.)
I would like to not perform the OIDC token exchange, is this supported?
If so, does anyone have any examples?
So far, I have the following but it doesn't work:
iat = time.time()
exp = iat + 3600
payload = {'iss': account['client_email'],
'sub': account['client_email'],
'aud': '/projects/NNNNN/apps/XXXXXXX',
'iat': iat,
'exp': exp}
additional_headers = {'kid': account['private_key']}
signed_jwt = jwt.encode(payload, account['private_key'], headers=additional_headers,
algorithm='RS256')
signed_jwt = signed_jwt.decode('utf-8')
This produces: Invalid IAP credentials: JWT signature is invalid.
this is not currently supported. IAP is expecting a signature generated by the Google accounts infrastructure using its private key, so that's why the signature check is failing. Could you tell me more about why you'd like to avoid the OIDC token exchange? --Matthew, Google IAP Engineering