I am currently using the following code to get the OAUTH Token
command = 'gcloud auth print-access-token'
result = str(subprocess.Popen(command, universal_newlines=True, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE).communicate())
The result variable has the OAUTH Token. This technique uses my current logged in gcloud config.
However, I am looking out for a way to get the OAUTH Token without using command line.
I am using this OAUTH Token to make CDAP calls to get the Google Dataflow Pipeline Execution Details.
I checked some google blogs. This is the one I think should try but it asks to create consent screen and it will require one time activity to provide consent to the scopes defined and then it should work.
Google Document
Shall I follow steps in above document and check OR is there any other way we can get the OAUTH Token?
Is there a way to get authentication done by service account instead of google user account and get the OAUTH Token?
For automated process, service account is the recommended way. You can use the google-oauth library for this. You can generate an access token like this
# With default credential (your user account or the Google Cloud Component service account.
# Or with the service account key file defined in the GOOGLE_APPLICATION_CREDENTIALS env var -> for platform outside GCP)
credentials, project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
# With service account key file (not recommended)
# credentials = service_account.Credentials.from_service_account_file('service-account.json',
# scopes=["https://www.googleapis.com/auth/cloud-platform"])
from google.auth.transport import requests
credentials.refresh(requests.Request())
print(credentials.token)
However, if you want to call Google cloud APIs, I recommend you to use authorized request object
Here an example of BigQuery call. You can use service account key file to generate your credential as in my previous example.
base_url = 'https://bigquery.googleapis.com'
credentials, project_id = google.auth.default(scopes=['https://www.googleapis.com/auth/cloud-platform'])
project_id = 'MyProjectId'
authed_session = AuthorizedSession(credentials)
response = authed_session.request('GET', f'{base_url}/bigquery/v2/projects/{project_id}/jobs')
print(response.json())
EDIT
When you want to use Google APIs, a service account key file is not needed (and I recommend you to not use it) on your computer and on GCP component. The Application Default Credential is always sufficient.
When you are in your local environment, you must run the command gcloud auth application-default login. With this command, you will register your personal account as default credential when you run locally your app. (of course, you need to have your user account email authorized on the component that you call)
When you are on GCP environment, each component have a default service account (or you can specify one with you configure your component). Thanks to the component "identity", you can use the default credential. (of course, you need to have the service account email authorized on the component that you call)
ONLY when you run an app automatically and outside GCP, you need a service account key file (for example, in your CI/CD other that Cloud Build, or in an app deployed on other Cloud Provider or on premise)
Why service account key file is not recommended? It's at least my recommendation because this file is ..... a file!! That's the problem. You have a way to authenticate a service account in a simple file: you have to store it securely (it's a secret and an authentication method!!), you can copy it, you can send it by email, you can even commit it in a public GIT repository... In addition, Google recommend to rotate them every 90 days, so it's a nightmare to rotate, to trace and to manage
Related
I'm hosting a Flask web app on Cloud Run. I'm also using Secret Manager to store Service Account keys. (I previously downloaded a JSON file with the keys)
In my code, I'm accessing the payload then using os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = payload to authenticate. When I deploy the app and try to visit the page, I get an Internal Service Error. Reviewing the logs, I see:
File "/usr/local/lib/python3.10/site-packages/google/auth/_default.py", line 121, in load_credentials_from_file
raise exceptions.DefaultCredentialsError(
google.auth.exceptions.DefaultCredentialsError: File {"
I can access the secret through gcloud just fine with: gcloud secrets versions access 1 --secret="<secret_id>" while acting as the Service Account.
Here is my Python code:
# Grabbing keys from Secret Manager
def access_secret_version():
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = "projects/{project_id}/secrets/{secret_id}/versions/1"
# Access the secret version.
response = client.access_secret_version(request={"name": name})
payload = response.payload.data.decode("UTF-8")
return payload
#app.route('/page/page_two')
def some_random_func():
# New way
payload = access_secret_version() # <---- calling the payload
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = payload
# Old way
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "service-account-keys.json"
I'm not technically accessing a JSON file like I was before. The payload variable is storing entire key. Is this why it's not working?
Your approach is incorrect.
When you run on a Google compute service like Cloud Run, the code runs under the identity of the compute service.
In this case, by default, Cloud Run uses the Compute Engine default service account but, it's good practice to create a Service Account for your service and specify it when you deploy it to Cloud Run (see Service accounts).
This mechanism is one of the "legs" of Application Default Credentials when your code is running on Google Cloud, you don't specify the environment variable (you also don't need to create a key) and Cloud Run service acquires the credentials from the Metadata service:
import google.auth
credentials, project_id = google.auth.default()
See google.auth package
It is bad practice to define|set an environment variable within code. By their nature, environment variables should be provided by the environment. Doing this with APPLICATION_DEFAULT_CREDENTIALS means that your code always sets this value when it should only do this when the code is running off Google Cloud.
For completeness, if you need to create Credentials from a JSON string rather than from a file contain a JSON string, you can use from_service_account_info (see google.oauth2.service_account)
I have a Python code that was using SendGrid API to send emails, but now I want to migrate to Google in order to send business emails. It also runs in Docker containers.
I followed Gmail Python Quickstart in order to use Gmail API in my Python code and the problem is that when trying to send email, it shows an authorization link in Docker logs in order to get token, etc.
Is there a way to complete authorization in the background without any further interaction or use an API key just like SendGrid to programmatically authenticate your application?
I am a service provider and want to send emails such as reset password links, confirmation code, etc. automatically; the code is deployed on a Linux host. I have access to workspace account, and I already have verified my domain.
The tutorial you are following is designed for an installed application. Hence the InstalledAppFlow.
It states it at the top of the file.
Authorization credentials for a desktop application. To learn how to create credentials for a desktop application, refer to Create credentials.
This means when your code runs it is going to pop up the consent screen on the machine the code is running on, in this instance Docker.
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
You need to create it using a web application so that your users can consent to your application accessing their data and their Gmail account.
Automated send emails service account option.
You were not clear as who you are sending emails for. As you are using send grid, it implies to me this is some kind of automated system. Which would mean that you are trying to send emails on behalf of a Gmail account that you control.
In that case you would most often want to use a service account. Service accounts allow for server-to-server interaction between Google APIs. However, service accounts will only work with Gmail if this is a Google workspace Gmail account and you can set up domain-wide delegation.
There is an example in the documentation. Just change it to Gmail scopes. The key point is the create_delegated which must be a user on your domain.
from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
# Email of the Service Account
SERVICE_ACCOUNT_EMAIL = '<some-id>#developer.gserviceaccount.com'
# Path to the Service Account's Private Key file
SERVICE_ACCOUNT_PKCS12_FILE_PATH = '/path/to/<public_key_fingerprint>-privatekey.p12'
def create_directory_service(user_email):
"""Build and returns an Admin SDK Directory service object authorized with the service accounts
that act on behalf of the given user.
Arguments:
user_email: The email of the user. Needs permissions to access the Admin APIs.
Returns:
Admin SDK directory service object.
"""
credentials = ServiceAccountCredentials.from_p12_keyfile(
SERVICE_ACCOUNT_EMAIL,
SERVICE_ACCOUNT_PKCS12_FILE_PATH,
'notasecret',
scopes=['https://www.googleapis.com/auth/admin.directory.user'])
credentials = credentials.create_delegated(user_email)
return build('admin', 'directory_v1', credentials=credentials)
Standard Gmail solution
You can run your application once, and then when you place it in the Docker container, make sure that you include the token.json file that was created this is the file that contains the credentials that grant the application access to your account.
If you open it you will find an access token and a refresh token within. The refresh token will give your application the ability to request a new access token whenever it needs one.
I'm using the azure python sdk to programmatically connect to azure services via linux.
I can login successfully via az('login')
login found
However, when I try to create credentials and get a token, I get an error that my grant is expired:
creds = DefaultAzureCredential()
token = creds.get_token('https://database.windows.net/.default')
VisuaLStudioCodeCredential.get_token failed: Azure Active Directory error ' (invalid_grant) AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on '2021-11-04T15:21:38,19517642' and the TokensValidFrom date (before which tokens are not valid) for this user is '2022-01-0819:52:17.0000000z'
How do I refresh this? I tried using az('account clear'), then az('login') but I get the same result. Is there a method specific for DefaultAzureCredential to get a refreshed token?
DefaultAzureCredential class tries to acquire a token using multiple methods in a particular order and VS Code logged in user has a higher precedence than Azure CLI logged in user.
When you use az login, you are logging in using Azure CLI and it seems you are already logged in into VS Code using some other credentials which does not have proper permissions and this is why you are getting this error.
To fix this issue, you can exclude VS Code credentials to be considered by setting exclude_visual_studio_code_credential to true. So your code would be something like:
creds = DefaultAzureCredential(exclude_visual_studio_code_credential=true)
That way DefaultAzureCredential will not take your VS Code credentials to acquire the token.
I'm running a flask app that will access Bigquery on behalf of users using a service account they upload.
To store those service account credentials, I thought the following might be a good set up:
ENV Var: Stores my credentials for accessing google secrets manager
Secret & secret version: in google secrets manager for each user of the application. This will access the user's own bigquery instance on behalf of the user.
--
I'm still learning about secrets, but this seemed more appropriate than any way of storing credentials in my own database?
--
The google function for accessing secrets is:
def access_secret_version(secret_id, version_id=version_id):
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
# Access the secret version.
response = client.access_secret_version(name=name)
# Return the decoded payload.
return response.payload.data.decode('UTF-8')
However, this returns JSON as a string. When then using this for big query:
credentials = access_secret_version(secret_id, version_id=version_id)
BigQuery_client = bigquery.Client(credentials=json.dumps(credentials),
project=project_id)
I get the error:
File "/Users/Desktop/application_name/venv/lib/python3.8/site-
packages/google/cloud/client/__init__.py", line 167, in __init__
raise ValueError(_GOOGLE_AUTH_CREDENTIALS_HELP)
ValueError: This library only supports credentials from google-auth-library-python.
See https://google-auth.readthedocs.io/en/latest/ for help on authentication with
this library.
Locally I'm storing the credentials and accessing them via a env variable. But as I intend for this application to have multiple users, from different organisations I don't think that scales.
I think my question boils down to two pieces:
Is this a sensible method for storing and accessing credentials?
Can you authenticate to Bigquery using a string rather than a .json file indicated here
Trying to let a python command line to access(HTTP GET) IAP-protected application.
With https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iap/make_iap_request.py
But 403 err occurs Exception: Service account xxx#xxxxxxxxx-production.iam.gserviceaccount.com does not have permission to access the IAP-protected application.
Following methods have been tried, but still issue persists as above
1. remove/create the service account
2. re-create the json key file
3. grant the service account project owner permission
if __name__ == '__main__':
os.environ[
"GOOGLE_APPLICATION_CREDENTIALS"] = '/Users/foo/bar.json'
print make_iap_request('https://foo.com/', 'xxxxtheidforthatiapservice.apps.googleusercontent.com')
I think you need to follow the Authenticating from a service account procedure:
Use an OpenID Connect (OIDC) token to authenticate a service account
to a Cloud IAP-secured resource.
Add the service account to the access list for the Cloud IAP-secured project.
Generate a JWT-based access token. This uses a target_audience additional claim that requires a client ID. To find
your client ID, follow the steps below:
a. Go to the Cloud IAP page.
b. Find the resource you want to access, then click More > Edit
OAuth Client.
edit OAuth client on the More menu
c. On the Credentials page that appears, note the client ID.
Request an OIDC token for the Cloud IAP-secured client ID.
Include the OIDC token in an Authorization: Bearer header to make the authenticated request to the Cloud IAP-secured resource.
I faced the same problem.
You have to run the script authenticated as a service account used to access the IAP-protected application programmatically. You can do so either by running script directly from VM which uses that service account as default, or you will need to download SA credentials and do it locally Obtaining an OIDC token from a local service account key file
Besides that, you also need to assign the IAP-secured Web App User role to the desired Service Account for that particular IAP-protected app.
After that, rerun the script.