My GAE app is trying to manipulate files stored on Google Cloud Storage.
The files are stored in the default bucket for my app. I already managed to read/write files to that bucket using the GCS Python Client Library (https://developers.google.com/appengine/docs/python/googlecloudstorageclient/).
Unfortunately it does not support copy. Instead, I'm trying the JSON API with the API Client Library (https://google-api-client-libraries.appspot.com/documentation/storage/v1/python/latest/storage_v1.objects.html) and service account (https://developers.google.com/api-client-library/python/guide/google_app_engine#ServiceAccounts)
So far I'm getting an error 403 when requesting the cloud storage url.
Here's the code:
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/devstorage.read_write')
http = credentials.authorize(httplib2.Http(memcache))
service = discovery.build('storage', 'v1', http=http, developerKey='api_key_generated_from_the_dev_console')
bucket_name = app_identity.get_default_gcs_bucket_name()
# I'm planning to batch multiple requests, although there is just one in this example
batch = BatchHttpRequest()
# process_list_response outputs the exception if any
batch.add(service.objects().list(bucket=bucket_name), callback=process_list_response)
batch.execute(http=http)
Here's the log:
URL being requested:
https://www.googleapis.com/discovery/v1/apis/storage/v1/rest?userIp=x.x.x.x
Attempting refresh to obtain initial access_token
URL being requested:
https://www.googleapis.com/storage/v1/b/xxx.appspot.com/o?alt=json
HttpError 403 when requesting
https://www.googleapis.com/storage/v1/b/xxx-dev.appspot.com/o?alt=json
returned "Access Not Configured. Please use Google Developers Console
to activate the API for your project."
Here's what I've done in the dev console:
Google Cloud Storage and Google Cloud Storage JSON API are switched
to ON.
I created an API key which I use to build the service (is it necessary since I also use Oauth?)
Under Permissions, I added a member for my app with the email xxx#appspot.gserviceaccount.com
How can I make this work?
Posting this as an answer as it seems that my edit (we work together) was silently rejected, and a comment is too limited. This is not an answer but that is expanding the question.
Simpler example with a single http request. It seems that the JSON API is simply not working outside the API explorer. The XML/REST API works and returns a list of files in the bucket.
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/devstorage.read_write')
http = credentials.authorize(httplib2.Http(memcache))
bucket_name = app_identity.get_default_gcs_bucket_name()
# This works (200 with list of files in the content)
request_url = 'http://commondatastorage.googleapis.com/' + bucket_name
response, content = http.request(request_url, method="GET")
# This doesn't work (403, Access not configured)
request_url = 'https://www.googleapis.com/storage/v1/b/' + bucket_name + '/o?alt=json'
response, content = http.request(request_url, method="GET")
# This doesn't work (403, Access not configured), the key and project id header seem useless.
request_url = 'https://www.googleapis.com/storage/v1/b/' + bucket_name + '/o?alt=json&key=' + API_KEY
response, content = http.request(request_url, method="GET", headers={'x-goog-project-id': PROJECT_ID})
Also, looking at the code of AppAssertionCredentials, we can see:
kwargs: optional keyword args, including:
service_account_id: service account id of the application. If None or
unspecified, the default service account for the app is used.
self.service_account_id = kwargs.get('service_account_id', None)
Passing anything as service_account_id argument results in an exception:
Traceback (most recent call last):
File "/base/data/home/apps/.../1.37.../backup.py", line 61, in get
response, content = http.request(request_url, method="GET")
File "/base/data/home/apps/.../1.377.../oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/.../1.37.../oauth2client/client.py", line 491, in new_request
self._refresh(request_orig)
File "/base/data/home/apps/.../1.37.../oauth2client/appengine.py", line 197, in _refresh
raise AccessTokenRefreshError(str(e))
AccessTokenRefreshError
I have tested to pass the value returned by app_identity.get_service_account_name(), that doesn't work. (even though the documentation says it will use "the default service account for the app" if it is not set).
I have tested to pass the service account email found in the developer console that has the form: 3....-v0....#developer.gserviceaccount.com. Same token exception.
So, why are we getting a 403 Access not configured when the Cloudstorage JSON API is clearly enabled under our api/services?
And why is passing a service_account_id to AppAssertionCredentials failing with a AccessTokenRefreshError?
Edit:
The solution was ridiculous: turn OFF the Google Cloud Storage API, and turn it back ON.
I assume that the app was a "legacy" app, and doing so made the last bullet point 12 work here: https://developers.google.com/appengine/docs/python/googlecloudstorageclient/activate
Related
I'm hosting a Flask web app on Cloud Run. I'm also using Secret Manager to store Service Account keys. (I previously downloaded a JSON file with the keys)
In my code, I'm accessing the payload then using os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = payload to authenticate. When I deploy the app and try to visit the page, I get an Internal Service Error. Reviewing the logs, I see:
File "/usr/local/lib/python3.10/site-packages/google/auth/_default.py", line 121, in load_credentials_from_file
raise exceptions.DefaultCredentialsError(
google.auth.exceptions.DefaultCredentialsError: File {"
I can access the secret through gcloud just fine with: gcloud secrets versions access 1 --secret="<secret_id>" while acting as the Service Account.
Here is my Python code:
# Grabbing keys from Secret Manager
def access_secret_version():
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = "projects/{project_id}/secrets/{secret_id}/versions/1"
# Access the secret version.
response = client.access_secret_version(request={"name": name})
payload = response.payload.data.decode("UTF-8")
return payload
#app.route('/page/page_two')
def some_random_func():
# New way
payload = access_secret_version() # <---- calling the payload
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = payload
# Old way
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "service-account-keys.json"
I'm not technically accessing a JSON file like I was before. The payload variable is storing entire key. Is this why it's not working?
Your approach is incorrect.
When you run on a Google compute service like Cloud Run, the code runs under the identity of the compute service.
In this case, by default, Cloud Run uses the Compute Engine default service account but, it's good practice to create a Service Account for your service and specify it when you deploy it to Cloud Run (see Service accounts).
This mechanism is one of the "legs" of Application Default Credentials when your code is running on Google Cloud, you don't specify the environment variable (you also don't need to create a key) and Cloud Run service acquires the credentials from the Metadata service:
import google.auth
credentials, project_id = google.auth.default()
See google.auth package
It is bad practice to define|set an environment variable within code. By their nature, environment variables should be provided by the environment. Doing this with APPLICATION_DEFAULT_CREDENTIALS means that your code always sets this value when it should only do this when the code is running off Google Cloud.
For completeness, if you need to create Credentials from a JSON string rather than from a file contain a JSON string, you can use from_service_account_info (see google.oauth2.service_account)
Background
I have a Google Cloud Composer 1 environment running on GCP.
I have written a Google Cloud Function that, when run in the cloud, successfully triggers a DAG Run in my Composer environment. My code is based on and almost identical to the code in the Trigger DAGs with Cloud Functions guide from GCP documentation.
Here is the section of code most relevant to my question (source):
from google.auth.transport.requests import Request
from google.oauth2 import id_token
import requests
def make_iap_request(url, client_id, method='GET', **kwargs):
"""Makes a request to an application protected by Identity-Aware Proxy.
Args:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
method: The request method to use
('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
Returns:
The page body, or raises an exception if the page couldn't be retrieved.
"""
# Set the default timeout, if missing
if 'timeout' not in kwargs:
kwargs['timeout'] = 90
# Obtain an OpenID Connect (OIDC) token from metadata server or using service
# account.
open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
raise Exception('Service account does not have permission to '
'access the IAP-protected application.')
elif resp.status_code != 200:
raise Exception(
'Bad response from application: {!r} / {!r} / {!r}'.format(
resp.status_code, resp.headers, resp.text))
else:
return resp.text
Challenge
I want to be able to run the same Cloud Function locally on my dev machine. When I try to do that, the function crashes with this error message:
google.auth.exceptions.DefaultCredentialsError: Neither metadata server or valid service account credentials are found.
This makes sense because the line that throws the error is:
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
Indeed when running locally the Metadata Server is not available and I don't know how to make valid service account credentials available to the call to fetch_id_token().
Question
My question is - What do I need to change in order to be able to securely obtain the OpenID Connect token when I run my function locally?
I've been able to run my code locally without changing it. Below are the details though I'm not sure this is the most secure option to get it done.
In the Google Cloud Console I browsed to the Service Accounts module.
I clicked on the "App Engine default service account" to see its details.
I switched to the "KEYS" tab.
I clicked on the "Add Key" button and generated a new JSON key.
I downloaded the JSON file and placed it outside of my source code folder.
Finally, on my dev machine*, I set the GOOGLE_APPLICATION_CREDENTIALS environment variable to be the path to where I placed the JSON file. More details here: https://cloud.google.com/docs/authentication/production
Once I did this, the call to id_token.fetch_id_token() picked up the service account details from the key file and returned the token successfully.
* - In my case I set the environment variable inside my PyCharm Debug Configuration.
I have a Flask app in Google App Engine Standard Environment Python, and I also have a Cloud Function with an HTTP trigger which accepts a JSON body including the URL of a file. The CF downloads the file at that URL then saves it to a GCS bucket. The GAE service account has Cloud Function Invoker permissions, yet when using urlfetch.fetch() in my GAE code to trigger CF, the App Engine code gets a 403 Forbidden error unless I make the CF trigger callable by anyone.
How do I successfully call/trigger CF's from GAE in Python? I assume the answer is one of these:
Set IAM permissions on GAE service account to {enlighten me here}
Add authentication headers in urlfetch.fetch() like so {different enlightenment}
Make CF triggerable from anywhere, but hard code some secret key so the CF code itself handles authentication.
It's well documented here: Cloud Functions Authentication
In short you have to provide your service account credentials in the authentication header.
To get your credentials use the Google Auth Client library. If you are testing from local you should create a service account JSON and load it to the environment variable GOOGLE_APPLICATION_CREDENTIALS but on App Engine it will work from scratch.
After you have gotten your token, pass it as an auth header like so:
auth_req = google.auth.transport.requests.Request()
auth_token = google.oauth2.id_token.fetch_id_token(auth_req, cloud_function_url)
response = requests.post(cloud_function_url, json=payload, headers={"Authorization" : f"Bearer {auth_token}"})
We are using Google Drive API in our Google App Engine application.
This weekend we noticed that it has problems with updating spreadsheet title. We are getting the following error:
HttpError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files/1_X51WMK0U12rfPKc2x60E_EuyqtQ8koW-NSRZq7Eqdw?quotaUser=5660071165952000&fields=title&alt=json returned "The authenticated user has not granted the app 593604285024 write access to the file 1_X51WMK0U12rfPKc2x60E_EuyqtQ8koW-NSRZq7Eqdw">
Other calls to Google Drive API succeed. We just have the problem with this one. Also this functionality worked properly for a long time. Is it possible that some update on Google side has broken this?
The minimal code to reproduce the issue is:
class TestDriveUpdate(webapp2.RequestHandler):
def get(self):
credentials = StorageByKeyName(Credentials,
'103005000283606027776',
'credentials').get()
spreadsheet_key = '1_X51WMK0U12rfPKc2x60E_EuyqtQ8koW-NSRZq7Eqdw'
quota_user = '5660071165952000'
body = {"title": 'Test'}
fields = "title"
http = httplib2.Http(timeout=60)
credentials.authorize(http)
gdrive = apiclient.discovery.build('drive', 'v2', http=http)
response = gdrive.files().update(
fileId=spreadsheet_key,
body=body,
fields=fields,
quotaUser=quota_user
).execute()
self.response.write("OK")
Based from this documentation, error occurs when the requesting app is not on the ACL for the file and the user never explicitly opened the file with this Drive app. Found this SO question which states that the scope strings must match exactly between your code and the Admin Console, including trailing slashes, etc. Make sure also that Drive Apps are allowed on the domain ("Allow users to install Google Drive apps").
I'm using python with google cloud speech api I did all the steps in "How to use google speech recognition api in python?" on ubuntu and on windows as well and when I trying to run the simple script from here - "https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/speech/api/speech_rest.py"
I get the next error:
<HttpError 403 when requesting https://speech.googleapis.com/$discovery/rest?version=v1beta1 returned "Google Cloud Speech API has not been used in project google.com:cloudsdktool before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/speech.googleapis.com/overview?project=google.com:cloudsdktool then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.">
what is weird is that I don't have project by the name "cloudsdktool"
I run "gcloud init", and linked the json file that I got when I created service account key with "gcloud auth activate-service-account --key-file=jsonfile" command,
I tried in linux to create google credentials environment variable and still I get the same massage
So I found two ways to fix that problem:
1 - if using google cloud sdk and the cloud speech is in beta version you need to run 'gcloud beta init' instead of 'gcloud init' and then provide the json file
2 - if you don't want to use the cloud sdk from google you can pass the json file straight in python app
here are the methods for this:
from oauth2client.client import GoogleCredentials
GoogleCredentials.from_stream('path/to/your/json')
then you just create scope on the creds and authorizing or if using grpc(streaming) you pass it to the header just like in the example.
here are the changed script for the grpc:
def make_channel(host, port):
"""Creates an SSL channel with auth credentials from the environment."""
# In order to make an https call, use an ssl channel with defaults
ssl_channel = implementations.ssl_channel_credentials(None, None, None)
# Grab application default credentials from the environment
creds = GoogleCredentials.from_stream('path/to/your/json').create_scoped([SPEECH_SCOPE])
# Add a plugin to inject the creds into the header
auth_header = (
'Authorization',
'Bearer ' + creds.get_access_token().access_token)
auth_plugin = implementations.metadata_call_credentials(
lambda _, cb: cb([auth_header], None),
name='google_creds')
# compose the two together for both ssl and google auth
composite_channel = implementations.composite_channel_credentials(
ssl_channel, auth_plugin)
return implementations.secure_channel(host, port, composite_channel)