I am trying to authenticate google API without a config file, I can't even find proof it is possible other than old code in my service that wasn't used in years.
My class receive this dict:
self._connection_data = {
"type": args,
"project_id": args,
"private_key_id": args,
"private_key": args,
"client_email": args,
"client_id": args,
"auth_uri": args,
"token_uri": args,
"auth_provider_x509_cert_url": args,
"client_x509_cert_url": args
}
and the code is -
from google.cloud import bigquery
from google.oauth2 import service_account
def _get_client(self):
credentials = service_account.Credentials.from_service_account_info(self._connection_data)
return bigquery.Client(project=self._project_id, credentials=credentials, location='US')
I receive the error
'{"error":"invalid_grant","error_description":"Invalid grant: account not found"}
however, everything works when I use a helper file for the configs called config.json and an OS environmentnt variable:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "config.json"
self.job_config = bigquery.QueryJobConfig()
self.job_config.use_legacy_sql = True
return bigquery.Client()
I don't want a solution with the env variable, I would like to use the Credentials class without a file path
Well In the end I managed to make my code work without any need for the global variable or a file path. I had a problem with my configured credentials...
This is the code -
# init class here
self.job_config = bigquery.QueryJobConfig()
self.job_config.use_legacy_sql = True
def _get_client(self):
credentials = service_account.Credentials.from_service_account_info(self._connection_data)
return bigquery.Client(project=self._project_id, credentials=credentials)
# function to get columns
query_job = self._get_client().query(query, job_config=self.job_config)
results = query_job.result(timeout=self._current_timeout)
The only part I was missing was to send the QueryJobConfig class with legacy SQL set to true in all of my queries.
Unfortunately, there are no other methods to authenticate your API request without either using an environment variable or specifying the key file path. There are some ways of authenticating your request with GCP using a key json file. Before anything, you should set up your service account and download the json file with your key, as described here.
Then, the first method is using default credentials, according to the documentation:
If you don't specify credentials when constructing the client, the
client library will look for credentials in the environment.
That means, you just need to set your environment variable. Then, the Google Client Library will determine the credentials implicitly. In addition, it also allows you to provide credentials separately from your application, which eases the process of making changes in the code. You can set the environment variable as follows:
export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"
After setting it, you would be able to run the following code:
def implicit():
from google.cloud import storage
# If you don't specify credentials when constructing the client, the
# client library will look for credentials in the environment.
storage_client = storage.Client()
# Make an authenticated API request
buckets = list(storage_client.list_buckets())
print(buckets)
Secondly, you can specify the file path within your code using the [google.oauth2.service_account][3] module. It is stated in the documentation that:
An OAuth 2.0 client identifies the application and lets end users
authenticate your application with Google. It allows your application
to access Google Cloud APIs on behalf of the end user.
In order to use the module, you can use one of both codes:
#It creates credentials using your .json file and the Credentials.from_service_account_file constructor
credentials = service_account.Credentials.from_service_account_file(
'service-account.json')
Or
#If you set the environment variable, you can also use
#info = json.loads(os.environ['GOOGLE_APPLICATION_CREDENTIALS_JSON_STRING'])
#Otherwise, you specify the path inside json.load() as below
service_account_info = json.load(open('service_account.json'))
credentials = service_account.Credentials.from_service_account_info(
service_account_info)
Finally, I encourage you to check the Authentication strategies in the documentation.
Related
I'm hosting a Flask web app on Cloud Run. I'm also using Secret Manager to store Service Account keys. (I previously downloaded a JSON file with the keys)
In my code, I'm accessing the payload then using os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = payload to authenticate. When I deploy the app and try to visit the page, I get an Internal Service Error. Reviewing the logs, I see:
File "/usr/local/lib/python3.10/site-packages/google/auth/_default.py", line 121, in load_credentials_from_file
raise exceptions.DefaultCredentialsError(
google.auth.exceptions.DefaultCredentialsError: File {"
I can access the secret through gcloud just fine with: gcloud secrets versions access 1 --secret="<secret_id>" while acting as the Service Account.
Here is my Python code:
# Grabbing keys from Secret Manager
def access_secret_version():
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = "projects/{project_id}/secrets/{secret_id}/versions/1"
# Access the secret version.
response = client.access_secret_version(request={"name": name})
payload = response.payload.data.decode("UTF-8")
return payload
#app.route('/page/page_two')
def some_random_func():
# New way
payload = access_secret_version() # <---- calling the payload
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = payload
# Old way
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "service-account-keys.json"
I'm not technically accessing a JSON file like I was before. The payload variable is storing entire key. Is this why it's not working?
Your approach is incorrect.
When you run on a Google compute service like Cloud Run, the code runs under the identity of the compute service.
In this case, by default, Cloud Run uses the Compute Engine default service account but, it's good practice to create a Service Account for your service and specify it when you deploy it to Cloud Run (see Service accounts).
This mechanism is one of the "legs" of Application Default Credentials when your code is running on Google Cloud, you don't specify the environment variable (you also don't need to create a key) and Cloud Run service acquires the credentials from the Metadata service:
import google.auth
credentials, project_id = google.auth.default()
See google.auth package
It is bad practice to define|set an environment variable within code. By their nature, environment variables should be provided by the environment. Doing this with APPLICATION_DEFAULT_CREDENTIALS means that your code always sets this value when it should only do this when the code is running off Google Cloud.
For completeness, if you need to create Credentials from a JSON string rather than from a file contain a JSON string, you can use from_service_account_info (see google.oauth2.service_account)
I'm running a flask app that will access Bigquery on behalf of users using a service account they upload.
To store those service account credentials, I thought the following might be a good set up:
ENV Var: Stores my credentials for accessing google secrets manager
Secret & secret version: in google secrets manager for each user of the application. This will access the user's own bigquery instance on behalf of the user.
--
I'm still learning about secrets, but this seemed more appropriate than any way of storing credentials in my own database?
--
The google function for accessing secrets is:
def access_secret_version(secret_id, version_id=version_id):
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
# Access the secret version.
response = client.access_secret_version(name=name)
# Return the decoded payload.
return response.payload.data.decode('UTF-8')
However, this returns JSON as a string. When then using this for big query:
credentials = access_secret_version(secret_id, version_id=version_id)
BigQuery_client = bigquery.Client(credentials=json.dumps(credentials),
project=project_id)
I get the error:
File "/Users/Desktop/application_name/venv/lib/python3.8/site-
packages/google/cloud/client/__init__.py", line 167, in __init__
raise ValueError(_GOOGLE_AUTH_CREDENTIALS_HELP)
ValueError: This library only supports credentials from google-auth-library-python.
See https://google-auth.readthedocs.io/en/latest/ for help on authentication with
this library.
Locally I'm storing the credentials and accessing them via a env variable. But as I intend for this application to have multiple users, from different organisations I don't think that scales.
I think my question boils down to two pieces:
Is this a sensible method for storing and accessing credentials?
Can you authenticate to Bigquery using a string rather than a .json file indicated here
I'm trying to run a google app script function remotely from a python flask app. This function creates google calendar events with inputs from a google sheet. I referred to this documentation from Google in order to set up the python script to run the appscript function. I followed every step required to deploy the app script project as an executable API and connected it to a google developer project and made OAuth 2.0 ID credentials as well.
From the API executable documentation, I got the following code and modified it to run as an object which can be called from the main server file.
from __future__ import print_function
from googleapiclient import errors
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file as oauth_file, client, tools
class CreateGCalEvent:
def main(self):
"""Runs the sample.
"""
SCRIPT_ID = 'my app script deployment ID was put here'
# Set up the Apps Script API
SCOPES = [
'https://www.googleapis.com/auth/script.scriptapp',
'https://www.googleapis.com/auth/drive.readonly',
'https://www.googleapis.com/auth/drive',
]
store = oauth_file.Storage('token.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('app_script_creds.json', SCOPES)
creds = tools.run_flow(flow, store)
service = build('script', 'v1', credentials=creds)
# Create an execution request object.
request = {"function": "getFoldersUnderRoot"}
try:
# Make the API request.
response = service.scripts().run(body=request,
scriptId=SCRIPT_ID).execute()
if 'error' in response:
# The API executed, but the script returned an error.
# Extract the first (and only) set of error details. The values of
# this object are the script's 'errorMessage' and 'errorType', and
# an list of stack trace elements.
error = response['error']['details'][0]
print("Script error message: {0}".format(error['errorMessage']))
if 'scriptStackTraceElements' in error:
# There may not be a stacktrace if the script didn't start
# executing.
print("Script error stacktrace:")
for trace in error['scriptStackTraceElements']:
print("\t{0}: {1}".format(trace['function'],
trace['lineNumber']))
else:
# The structure of the result depends upon what the Apps Script
# function returns. Here, the function returns an Apps Script Object
# with String keys and values, and so the result is treated as a
# Python dictionary (folderSet).
folderSet = response['response'].get('result', {})
if not folderSet:
print('No folders returned!')
else:
print('Folders under your root folder:')
for (folderId, folder) in folderSet.items():
print("\t{0} ({1})".format(folder, folderId))
except errors.HttpError as e:
# The API encountered a problem before the script started executing.
print(e.content)
Here is where the error comes. It can neither locate token.json nor the app_script_creds.json.
Now with a service account and any normal OAuth2.0 ID, when I create it, I will be given the option to download the credentials.json but here, this is all I seem to be getting, an App Script ID with no edit access or credentials to download as JSON. I created another OAuth ID in the same project as shown in the screenshot which has the edit access and json ready for download. When I used that json file inside the python script, It told me that it was expecting redirect uris, which I don't know for what it is or where to redirect to.
What do I need to do to get this working?
I adapted some code that I used for connecting to the App Scripts API. I hope it works for you too. The code is pretty much the same thing as this.
You can use from_client_secrets_file since you're already loading these credentials from the file. So, what the code does is look for a token file first. If the token file is not there, it logs in the user (prompting using the Google authorization screen) and stores the new token in the file as pickle.
Regarding the credentials in the Google console you need to pick the Desktop application when creating them because that is basically what a server is.
Note: with this, you can only have one user that will be doing all of these actions. This is because the server script will start a local server on the server machine to authenticate you, your client code will not see any of this.
import logging
import pickle
from pathlib import Path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
class GoogleApiService:
def __init__(self, , scopes):
"""
Args:
scopes: scopes required by the script. There needs to be at least
one scope specified.
"""
self.client_secrets= Path('credentials/credentials.json')
self.token_path = Path('credentials/token.pickle')
self.credentials = None
self.scopes = scopes
def get_service(self):
self.__authenticate()
return build('script', 'v1', credentials=self.credentials)
def __authenticate(self):
log.debug(f'Looking for existing token in {self.token_path}')
if self.token_path.exists():
with self.token_path.open('rb') as token:
self.credentials = pickle.load(token)
if self.__token_expired():
self.credentials.refresh(Request())
# If we can't find any token, we log in and save it
else:
self.__log_in()
self.__save_token()
def __log_in(self):
flow = InstalledAppFlow.from_client_secrets_file(
self.client_secrets,
self.scopes
)
self.credentials = flow.run_local_server(port=0)
def __save_token(self):
with self.token_path.open('wb') as token:
pickle.dump(self.credentials, token)
def __token_expired(self):
return self.credentials and self.credentials.expired and \
self.credentials.refresh_token
# Example for Google Apps Scripts
def main():
request = {'function': 'some_function', 'parameters': params}
gapi_service = GoogleApiService()
with gapi_service.get_service() as service:
response = service.scripts().run(
scriptId=self.script_id,
body=request
).execute()
if response.get('error'):
message = response['error']['details'][0]['errorMessage']
raise RuntimeError(message)
else:
return response['response']['result']
I have read the AWS documentation but I couldn't find an example of using Temporary Security Credentials to authenticate to AWS with Python.
I would like an example of using a temporary security credentials provided by the AWS Security Token Service (AWS STS) to sign a request.
There are several ways you can use STS to get temporary credential. The two most common ones would be:
get_session_token - used to get temp credentials for existing IAM user or account
assume_role - used to get credentials when assuming iam role
In both cases the call to these function will give you temp credentials, e.g.:
{
"Credentials": {
"AccessKeyId": "AddsdfsdfsdxxxxxxKJ",
"SecretAccessKey": "TEdsfsdfSfdsfsdfsdfsdclkb/",
"SessionToken": "FwoGZXIvYXdzEFkaDGgIUSvDdfgsdfgsdfgsMaVYgsSxO8OqRfjHc4se90WbaspOwCtdgZNgeasdfasdfasdf5wrtChz2QCTnR643exObm/zOJzXe9TUkcdODajHtxcgR8r+unzMo+7WxgQYyKGN9kfbCqv3kywk0EvOBCapusYo81fpv8S7j4JQxEwOGC9JZQL6umJ8=",
"Expiration": "2021-02-17T11:53:31Z"
}
}
Having these credentials, you create new boto3 session, e.g.:
new_session = boto3.session.Session(<temp credentails>)
The new_session will allow you to make new boto3 client or resource, e.g.:
ec2 = new_session.client('ec2')
s3r = new_session.resource('s3')
And then you can use these new clients/resource as you would normally use them.
I am using tkinter to create gui application that returns the security groups. Currently if you want to change your credentials (e.g. if you accidentally entered the wrong ones) you would have to restart the application otherwise boto3 would carry on using the old credentials.
I'm not sure why it keeps using the old credentials because I am running everything again using the currently entered credentials.
This is a snippet of the code that sets the environment variables and launches boto3. It works perfectly fine if you enter the right credentials the first time.
os.environ['AWS_ACCESS_KEY_ID'] = self.accessKey
os.environ['AWS_SECRET_ACCESS_KEY'] = self.secretKey
self.sts_client = boto3.client('sts')
self.assumedRoleObject = self.sts_client.assume_role(
RoleArn=self.role,
RoleSessionName="AssumeRoleSession1"
)
self.credentials = self.assumedRoleObject['Credentials']
self.ec2 = boto3.resource(
'ec2',
region_name=self.region,
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
)
The credentials variables are set using:
self.accessKey = str(self.AWS_ACCESS_KEY_ID_Form.get())
self.secretKey = str(self.AWS_SECRET_ACCESS_KEY_Form.get())
self.role = str(self.AWS_ROLE_ARN_Form.get())
self.region = str(self.AWS_REGION_Form.get())
self.instanceID = str(self.AWS_INSTANCE_ID_Form.get())
Is there a way to use different credentials in boto3 without restarting the program?
You need boto3.session.Session to overwrite the access credentials.
Just do this
reference http://boto3.readthedocs.io/en/latest/reference/core/session.html
import boto3
# Assign you own access
mysession = boto3.session.Session(aws_access_key_id='foo1', aws_secret_access_key='bar1')
# If you want to use different profile call foobar inside .aws/credentials
mysession = boto3.session.Session(profile_name="fooboar")
# Afterwards, just declare your AWS client/resource services
sqs_resource=mysession.resource("sqs")
# or client
s3_client=mysession.client("s3")
Basically, little change to your code. you just pass in the session instead of direct boto3.client/boto3.resource
self.sts_client = mysession.client('sts')
Sure, just create different sessions from botocore.session.Session object for each set of credentials:
import boto3
s1 = boto3.session.Session(aws_access_key_id='foo1', aws_secret_access_key='bar1')
s2 = boto3.session.Session(aws_access_key_id='foo2', aws_secret_access_key='bar2')
Also you can leverage set_credentials method to keep 1 session an change creds on the fly:
import botocore
session - botocore.session.Session()
session.set_credentials('foo', 'bar')
client = session.create_client('s3')
client._request_signer._credentials.access_key
u'foo'
session.set_credentials('foo1', 'bar')
client = session.create_client('s3')
client._request_signer._credentials.access_key
u'foo1'
The answers given by #mootmoot and #Vor clearly state the way of dealing with multiple credentials using a session.
#Vor's answer
import boto3
s1 = boto3.session.Session(aws_access_key_id='foo1', aws_secret_access_key='bar1')
s2 = boto3.session.Session(aws_access_key_id='foo2', aws_secret_access_key='bar2')
But some of you would be curious about
why does the boto3 client or resource behave in that manner in the first place?
Let's clear out a few points about Session and Client as they'll actually lead us to the answer to the aforementioned question.
Session
A 'Session' stores configuration state and allows you to create service clients and resources
Client
if the credentials are not passed explicitly as arguments to the boto3.client method, then the credentials configured for the session will automatically be used. You only need to provide credentials as arguments if you want to override the credentials used for this specific client
Now let's get to the code and see what actually happens when you call boto3.client()
def client(*args, **kwargs):
return _get_default_session().client(*args, **kwargs)
def _get_default_session():
if DEFAULT_SESSION is None:
setup_default_session()
return DEFAULT_SESSION
def setup_default_session(**kwargs):
DEFAULT_SESSION = Session(**kwargs)
Learnings from the above
The function boto3.client() is really just a proxy for the boto3.Session.client() method
If you once use the client, the DEFAULT_SESSION is set up and for the next consecutive creation of clients it'll keep using the DEFAULT_SESSION
The credentials configured for the DEFAULT_SESSION are used if the credentials are not explicitly passed as arguments while creating the boto3 client.
Answer
The first call to boto3.client() sets up the DEFAULT_SESSION and configures the session with the oldCredsAccessKey, oldCredsSecretKey, the already set values for env variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACESS_KEY respectively.
So even if you set new values of credentials in the environment, i.e do this
os.environ['AWS_ACCESS_KEY_ID'] = newCredsAccessKey
os.environ['AWS_SECRET_ACCESS_KEY'] = newCredsSecretKey
The upcoming boto3.client() calls still pick up the old credentials configured for the DEFAULT_SESSION
NOTE
boto3.client() call in this whole answer means that no arguments passed to the client method.
References
https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3.html#client
https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/session.html#Session
https://ben11kehoe.medium.com/boto3-sessions-and-why-you-should-use-them-9b094eb5ca8e