We Keep Coding

Roles Required to write to Cloud Storage (GCP) from python (pandas)

I have a question for the GCP connoisseurs among you.
I have an issue that I can upload to a bucket via UI and gsutil - but if I try to do this via python
df.to_csv('gs://BUCKET_NAME/test.csv')
I get a 403 insufficient permission error.
My guess at the moment is that python does this via an API and requires an extra role - to make things more confusing I am already project owner of the project of the bucket and compared to other team members did not really find lacking permissions for this specific bucket.
I use python 3.9.1 via pyenv and pandas '1.4.2'
Anyone had the same issue/ knows what role I am missing?
I checked that I have in principal rights to upload both via UI and gsutil
I used the same virtual python environemnt to read and write from bigquery to check that I can in principle use GCP data in python - this works
I have the following Roles on the Bucket
Storage Admin, Storage Object Admin, Storage Object Creator, Storage Object Viewer

gsutil and gcloud share credentials.
These credentials are not shared with other code running locally.
The quick-fix but sub-optimal solution is to:
gcloud auth application-default login
And run the code again.
It will then use your gcloud (gsutil) user credentials configured to run as if you were using a Service Account.
These credentials are stored (on Linux) in ${HOME}/.config/gcloud/application_default_credentials.json.
A better solution is to create a Service Account specifically for your app and grant it the minimal set of IAM permissions that it will need (BigQuery, GCS, ...).
For testing purposes (!) you can download the Service Account key locally.
You can then auth your code using Google's Application Default Credentials (ADC) by (on Linux):
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/key.json
python3 your_app.py
When you deploy code that leverages ADC to a Google Cloud compute service (Compute Engine, Cloud Run, ...), it can be deployed unchanged because the credentials for the compute resource will be automatically obtained from the Metadata service.
You can Google e.g. "Google IAM BigQuery" to find the documentation that lists the roles:
IAM roles for BigQuery
IAM roles for Cloud Storage

Cloud Composer + Airflow: Setting up DAWs to trigger on HTTP (or should I use Cloud Functions?)

Ultimately, what I want to do is have a Python script that runs whenever a HTTP request is created, dynamically. It'd be like: App 1 runs and sends out a webhook, Python script catches the webhook immediately and does whatever it does.
I saw that you could do this in GCP with Composer and Airflow.
But I'm having several issues following these instrutions https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf:
Running this in Cloud Shell to grant blob signing permissions:
gcloud iam service-accounts add-iam-policy-binding
your-project-id#appspot.gserviceaccount.com
--member=serviceAccount:your-project-id#appspot.gserviceaccount.com
--role=roles/iam.serviceAccountTokenCreator
When I put in my project ID, I get a "Gaia id not found for your-project-id#appspot.gserviceaccount.com"
When I run the airflow_uri = environment_data['config']['airflowUri'] bit, I get a key error on 'config'.
Is there a better way to do what I'm trying to do (i.e. run Python scripts dynamically)?

The reason for getting: Gaia id not found for email <project-id>#appspot.gserviceaccount.com error is not enabling all needed APIs in your project. Please follow the steps:
Create or select Google Cloud Platform Project you wish to work with.
Enable the Cloud Composer, Google Cloud Functions and Cloud Identity and Google Identity and Access Management (IAM) APIs. You can find it in Menu -> Products -> Marketplace and typing the name of corresponding API.
Grant blob signing permissions to the Cloud Functions Service Account. In order for GCF to authenticate to Cloud IAP, the proxy that protects the Airflow webserver, you need to grant the Appspot Service Account GCF the Service Account Token Creator role. Do so by running the following command in your Cloud Shell, substituting the name of your project for <your-project-id>:
gcloud iam service-accounts add-iam-policy-binding \
<your-project-id>#appspot.gserviceaccount.com \
--member=serviceAccount:<your-project-id>#appspot.gserviceaccount.com \
--role=roles/iam.serviceAccountTokenCreator
I tested the scenario, firstly without enabling APIs and I've retrieved the same error as you. After enabling the APIs, error disappear and IAM policy has been updated correctly.
There is already well described Codelabs tutorial, which shows the workflow of triggering the DAG with Google Cloud Functions.

How to get credentials in Google AppEngine Python37

I started new app in AppEngine Python3.7 stadard.
I am trying to get credentials using the following snippet, but its failing.
Anybody able to get credentials in GAE standard Python37 ?
Input:
from google.auth import app_engine
credentials = app_engine.Credentials()
Output:
The App Engine APIs are not available

When using App Engine Standard with python 3.7, none of the google.xxx libraries are available. You have to build your own, or use standard Python libraries. This goes for: auth, users, images, search, mail, taskqueue, memcache, urlfetch, deferred, etc., and even the ndb datastore interface.
For datastore, you use google-cloud-datastore or some 3rd party.
For others, you use a standard Python library, e.g.: google.auth => rauth, google.appengine.api.memcache => python-memcached
Read more here: https://cloud.google.com/appengine/docs/standard/python3/python-differences
That page recommends Google Identity Platform or Firebase Authentication to do authorization.

Integrating MQTT with GCP using IOT adapter and google pub/sub api in python

Integration with Cloud Pub/Sub APIs from App Engine Standard
I am working on developing a Google app engine app in standard Python environment. For some portions of the code, I need to integrate with Google Cloud pub/sub APIs.
As mentioned here, Pub/Sub can only be integrated in the App Engine flexible environment (BTW it is also only in alpha). Can someone please describe how to integrate with Pub/Sub in the App Engine Standard environment?
My use case description
I am trying to integrate MQTT with google app engine by using Agosto IOT broker. I will be using MQTT for clients (Currently mobile platforms) and on server side, I plan to use pub/sub for receiving/sending the messages and saving relevant data to the database.

You might want to try instead to use the new Google Cloud IoT Core product (full disclosure, I worked on it) instead of hosting MQTT on App Engine. Cloud IoT Core lets you connect to a Google-provided MQTT bridge that will put your data into Google Cloud PubSub. You can use Google Cloud DataFlow to move your data from PubSub to your data warehouse for analytics or can use your own database as the output from DataFlow.
The connection details for communicating with the Google Cloud IoT Core MQTT bridge are discussed in detail in the documentation but the important connection properties you will need to be aware of are the hostname (mqtt.googleapis.com) port (8883 or 443) and the MQTT password / client ID which are going to be based on the devices you've provisioned for the service.
Your actual MQTT client will need to be chosen depending on which programming language you're trying to access the MQTT bridge. If you're trying from Android, you could start from the Java MQTT client sample and would probably end up with something like the Android Things Cloud IoT sensor hub connector from the AndroidThings team.

TL;DR - App Engine standard does not support the newer Google Cloud Client libraries. You will instead need to use the older Google Cloud API Client libraries to communicate with Cloud Pub/Sub.
Cloud API Client libraries (older) vs Google Cloud Client libraries (newer)
The Cloud Pub/Sub client library documentation you're pointing to advises you to use the older Google API Client libraries (which is supported on App Engine Standard environment) instead of Google Cloud Client libraries (which is supported on App Engine Flexible environment but not on standard)
The client libraries are explained in detail here.
Google API Client libraries for Cloud Pub/Sub
Here are all the list of APIs which are supported using Google API Client libraries. Cloud Pub/Sub APIs are also part of this list.
Using Google API Client libraries with App Engine Standard
If you scroll down that page, there is a section describing how this API library can be used in App Engine Standard environment. In short, you will need to bundle the library along with your application just like other third-party libraries you use.
App Engine
Because the Python client libraries are not installed in the App
Engine Python runtime environment, they must be vendored into your
application just like third-party libraries.
This warning that you will see on the page, advises you to use the regular Cloud Client library if possible. But since App Engine Standard does not support it, you can ignore it for that use case.
While this library is still supported, we suggest trying the newer
Cloud Client Library for Google Cloud Pub/Sub, especially for new
projects. See Google Cloud Pub/Sub Libraries for installation and
usage details.
Examples using google-api-python-client library to invoke PubSub APIs
Using credentials from a service account json file
The following example shows you how you can use a service account to authenticate with Google Cloud PubSub APIs and invoke them. The information about how to use credentials from a Service account is available here.
You will need to have the following python packages pre-installed for this example to work: google-api-python-client and oauth2client.
If you're using pip, you can do:
pip install google-api-python-client oauth2client
Example which I have tested personally:
from googleapiclient import discovery
from httplib2 import Http
from oauth2client.service_account import ServiceAccountCredentials
# BEGIN CONFIG
PRIVATE_KEY_JSON = 'path/to/service_account_private_key.json'
API_SCOPES = ['https://www.googleapis.com/auth/pubsub']
PROJECT_NAME = 'FILL_IN_PROJECT_NAME_HERE'
# END CONFIG
# The format of project name expected by PubSub
PROJECT = 'projects/{0}'.format(PROJECT_NAME)
# Create a ServiceAccountCredentials object by reading the credentials from
# your JSON file.
credentials = ServiceAccountCredentials.from_json_keyfile_name(
PRIVATE_KEY_JSON, scopes=API_SCOPES)
# Build the Cloud PubSub API object which you will be using for
# invoking the corresponding APIs using the credentials object
# you created previously
pubsub = discovery.build('pubsub', 'v1', credentials=credentials)
# List all topics the specified project
topics = pubsub.projects().topics().list(
project=PROJECT).execute()
print topics
# Add a new topic
topic_name = 'TOPIC_NAME_TO_ADD'
added_topic_response = pubsub.projects().topics().create(
name='{0}/topics/{1}'.format(PROJECT, topic_name), body={}).execute()
print added_topic_response
Using credentials from a service account within an App Engine app
There is some info here regarding how to use Service account credentials from your App Engine apps.
The above example will work for the most part for invoking PubSub APIs, except for the part where you will initialize the credentials object. That part can be replaced roughly as described below:
Service Accounts
If your App Engine application needs to call an API to access data
owned by the application's project, you can simplify OAuth 2.0 by
using Service Accounts. These server-to-server interactions do
not involve a user, and only your application needs to authenticate
itself. Use the AppAssertionCredentials class to create a
Credentials object without using a Flow object.
In the following code snippet, a Credentials object is created and an
Http object is authorized:
import httplib2
from google.appengine.api import memcache
from oauth2client.contrib.appengine import AppAssertionCredentials
...
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/devstorage.read_write')
http = credentials.authorize(httplib2.Http(memcache))
pubsub = discovery.build('pubsub', 'v1', http=http)
...
Once you have an authorized Http object, you can pass it to the
build() or execute() functions as you normally would.
Using Application Default Credentials
You can also utilize Application Default credentials for your local testing and also works within App Engine environment.
from oauth2client.client import GoogleCredentials
...
credentials = GoogleCredentials.get_application_default()
pubsub = discovery.build('pubsub', 'v1', credentials=credentials)
...

How can I log in to an arbitrary user in appengine for use with the Drive SDK?

I have an application that needs to log into a singular Drive account and perform operations on the files automatically using a cron job. Initially, I tried to use the domain administrator login to do this, however I am unable to do any testing with the domain administrator as it seems that you cannot use the test server with a domain administrator account, which makes testing my application a bit impossible!
As such, I started looking at storing arbitray oauth tokens--especially the refresh token--to log into this account automatically after the initial setup. However, all of the APIs and documentation assume that multiple individual users are logging in manually, and I cannot find functionality in the oauth APIs that allow or account for logging into anything but the currently logged in user.
How can I achieve this in a way that I can test my code on a test domain? Can I do it without writing my own oauth library and doing the oauth requests by hand? Or is there a way to get the domain administrator authorization to work on a local test server?

You can load the credentials for a single account into your datastore using the Remote API, which can be enabled in your app.yaml file:
builtins:
- remote_api: on
By executing
remote_api_shell.py -s your_app_id.appspot.com
from the command line you'll have access to a shell which can execute in the environment of your application. Before doing this, make sure you have your application deployed (more on local development below) and make sure the source for google-api-python-client is included by pip-installing it and running enable-app-engine-project /path/to/project to add it to your App Engine project.
Once you are in the remote shell (after executing the remote command above), perform the following:
from oauth2client.appengine import CredentialsModel
from oauth2client.appengine import StorageByKeyName
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run
KEY_NAME = 'your_choice_here'
CREDENTIALS_PROPERTY_NAME = 'credentials'
SCOPE = 'https://www.googleapis.com/auth/drive'
storage = StorageByKeyName(CredentialsModel, KEY_NAME, CREDENTIALS_PROPERTY_NAME)
flow = OAuth2WebServerFlow(
client_id=YOUR_CLIENT_ID,
client_secret=YOUR_CLIENT_SECRET,
scope=SCOPE)
run(flow, storage)
NOTE: If you have not deployed your application with the google-api-python-client code, this will fail, because your application won't know how to make the same imports you made on your local machine, e.g. from oauth2client.appengine import CredentialsModel.
When run is called, your web browser will open and prompt you to accept the OAuth access for the client you've specified with CLIENT_ID and CLIENT_SECRET and after successfully completing, it will save an instance of CredentialsModel in the datastore of the deployed application your_app_id.appspot.com and it will store it using the KEY_NAME your provided.
After doing this, any caller in your application -- including your cron jobs -- can access those credentials by executing
storage = StorageByKeyName(CredentialsModel, KEY_NAME, CREDENTIALS_PROPERTY_NAME)
credentials = storage.get()
Local Development:
If you'd like to test this locally, you can run your application locally via
dev_appserver.py --port=PORT /path/to/project
and you can execute the same commands using the remote API shell and pointing it at your local application:
remote_api_shell.py -s localhost:PORT
Once here, you can execute the same code you did in the remote api shell and similarly an instance of CredentialsModel will be stored in the datastore of your local development server.
As above, if you don't have the correct google-api-python-client modules included, this will fail.
EDIT: This used to recommend using the Interactive Console at:
http://localhost:PORT/_ah/admin/interactive
but it was discovered that this doesn't work because socket does not work properly in the App Engine local development sandbox.

This article explains how to interact with Google Drive on behalf of users of your domain by having the Domain Administrator delegate domain-wide authority to a Service Account
This other article explains how to interact with a Drive owned by your application using a Service Account.
Note that both methods use a JWT based Service Accounts and which currently need a modified version of the google-api-python-client in order to work on App Engine.
Unlike Google App Engine Service account, JWT based Service Accounts should work with the development server.