I have prepared automation in GCP cloud, automation is prepared in Python SDK. Script is deploying VPC firewall rules (I used documentation to prepare it - GCP Python SDK firewall deployment). Automation works as expected, firewall rules are being created in the Google environment, but how can I check if the deployment completed successfully? I know that I can use list method to create list of existing firewall rules, then compere it with rules I wanted do deploy, but if there any native method to verify the deployment status?
OK, I prepared that code, and it works in my environment.
import json
import time
import googleapiclient
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
from colormodule import color
credentials = GoogleCredentials.get_application_default()
def create_vpc_firewall_rule(credentials, discovery, project, firewall_body):
service = discovery.build('compute', 'v1', credentials=credentials)
request = service.firewalls().insert(project=project, body=firewall_body)
response = request.execute()
return response
def firewall_body(traffic_direction_name, network_name, ports, json, project_name):
rule_name_input = f'firewall-rule-test-{traffic_direction_name}'
network_input = f'projects/{project_name}/global/networks/{network_name}'
json_string = {
"name": rule_name_input,
"allowed": [
{
"IPProtocol": "tcp",
"ports": ports
}
],
"network": network_input,
"direction": "EGRESS",
"destinationRanges": "192.168.0.23/32",
"priority": 1000,
"targetTags": [
"testwindows"
]
}
data = json.dumps(json_string)
return data
def wait_for_operation_fw_deployment(compute, project_id, operation, fw_rule_name, colors):
print('')
print('Waiting for operation to finish...')
print('')
while True:
result = compute.globalOperations().get(
project=project_id,
operation=operation).execute()
if result['status'] == 'DONE':
print(f'Deployment of {colors.OKCYAN}%s{colors.ENDC} Firewall rule has been completed.'
% fw_rule_name)
print('')
time.sleep(2)
if 'error' in result:
raise Exception(result['Deployment has returned an error.'])
return result
time.sleep(2)
firewall_body = eval(firewall_body(traffic_direction_name,network_name, ports, json, project_name))
operation = create_vpc_firewall_rule(credentials, discovery, project_name, firewall_body)
compute = googleapiclient.discovery.build('compute', 'v1')
wait_for_operation_fw_deployment(compute, project_id, operation['name'], fw_rule_name, colors)
Related
I've created a simple Python/flask website which connects to a Google Calendar via the Google Calendar API. I followed the instructions at https://developers.google.com/calendar/api/quickstart/python, and it works well...
The whole thing is internal to my company.
I copied the credentials.json and token.json along with my website to an internal device I'm using as a webserver. Again, all works fine.
Except, at some stage, I am under the impression that my credentials will expire, and the web server will need to re-authenticate with my personal google account.
How do I get around this?
Thanks
The reason your refresh token is expiring is that your app is still in the testing phase. Go to google cloud console under the consent screen and set it to production. Your tokens will stop expiring.
web app vs installed app
You should note is the sample you are following states
and uses the code for an installed application
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
So by following this example you are not creating a web application you are creating an installed application.
Using flask you can design the following scheme.
You should have a /auth endpoint that generates authorization_url that you can redirect to perform authorization in Google API.
You should have a /callback endpoint that handles requests when your authorization in Google API is completed. In this callback, you can store your credentials in a flask session.
Before making events request, you should check whether your stored credentials are still valid. If not, you should call /auth again.
In GCP app console, when you create credentials for your app you should choose "web application".
from flask import Flask, redirect, request, url_for
from google_auth_oauthlib.flow import Flow
app = Flask(__name__)
app.secret_key = os.environ.get("SECRET_KEY")
app.config["SESSION_TYPE"] = "filesystem"
# this callback URL should match one saved in GCP app console "Authorized redirection URIs" section
CALLBACK_URL = os.environ.get("CALLBACK_URL") # you can use `url_for('callback')` instead
API_CLIENT_ID = os.environ.get("API_CLIENT_ID")
API_CLIENT_SECRET = os.environ.get("API_CLIENT_SECRET")
SCOPES = ["https://www.googleapis.com/auth/calendar"]
class CalendarClient:
API_SERVICE = "calendar"
API_VERSION = "v3"
def __init__(self, client_id: str, client_secret: str, scopes: Sequence[str]):
self._client_id = client_id
self._client_secret = client_secret
self._scopes = scopes
self._client_config = {
"web": {
"client_id": client_id,
"client_secret": client_secret,
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
}
}
def get_flow(self, callback_url: str) -> Flow:
return Flow.from_client_config(
self._client_config, self._scopes, redirect_uri=callback_url
)
def get_auth_url(self, callback_url: str) -> str:
flow = self.get_flow(callback_url)
auth_url, _ = flow.authorization_url(
access_type="offline", include_granted_scopes="true"
)
return auth_url
def get_credentials(self, code: str, callback_url: str) -> Credentials:
flow = self.get_flow(callback_url)
flow.fetch_token(code=code)
return flow.credentials
#app.route("/callback")
def callback():
credentials = client.get_credentials(
code=request.args.get("code"),
callback_url=CALLBACK_URL,
)
session["credentials"] = {
"token": credentials.token,
"refresh_token": credentials.refresh_token,
"token_uri": credentials.token_uri,
"client_id": credentials.client_id,
"client_secret": credentials.client_secret,
"scopes": credentials.scopes,
}
return credentials.to_json()
#app.route("/auth")
def auth():
return redirect(client.get_auth_url(CALLBACK_URL))
Full codebase: https://github.com/jorzel/flask-google-calendar
here is the way using youtube api to upload videos to the channel.
but my question is every time i run the code, it ask me to "Please visit this URL to authorize this application"
# -*- coding: utf-8 -*-
# Sample Python code for youtube.videos.insert
# NOTES:
# 1. This sample code uploads a file and can't be executed via this interface.
# To test this code, you must run it locally using your own API credentials.
# See: https://developers.google.com/explorer-help/guides/code_samples#python
# 2. This example makes a simple upload request. We recommend that you consider
# using resumable uploads instead, particularly if you are transferring large
# files or there's a high likelihood of a network interruption or other
# transmission failure. To learn more about resumable uploads, see:
# https://developers.google.com/api-client-library/python/guide/media_upload
import os
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
from googleapiclient.http import MediaFileUpload
scopes = ["https://www.googleapis.com/auth/youtube.upload"]
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtube"
api_version = "v3"
client_secrets_file = "C:/Users/User/computer science/moviepy/client_secret_660339349555-der7rru7no2g1v4oe1o1g7qqm27ujodo.apps.googleusercontent.com.json"
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
credentials = flow.run_console()
youtube = googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
request = youtube.videos().insert(
part = "snippet,status",
body={
"snippet": {
"title": "test",
"description": "四次是鬼"
},
"status": {
"privacyStatus": "private"
}
},
# TODO: For this request to work, you must replace "YOUR_FILE"
# with a pointer to the actual file you are uploading.
media_body=MediaFileUpload("D:/Downloads/ytdl/wtf8.mp4")
)
response = request.execute()
#print(response)
if __name__ == "__main__":
main()
is there any way to prevent this situation?
i hope someone can solve my question, thanks.
I'm trying to set up a deployment street in Google Cloud Build. To do this, I want to:
Run unit test
Deploy to Cloud Run without traffic
Run integration tests
Migrate traffic in Cloud Run
I've got this mostly set up, but my integration tests include a couple of calls to Cloud Run to validate that authenticated calls return 200 and unauthenticated return 401. The thing I'm having difficulties with is to make signed requests from Cloud Build. When deploying by hand and running integration tests, they work, but not from Cloud Build.
Ideally, I would like to use the Cloud Build Service Account for invoking Cloud Run like I usually do in AWS, but I can't find how to get access to that from the Cloud Runner. So instead, I retrieve a credentials file from Secret Manager. This credentials file is from a newly created Service Account with Cloud Run Invoker role:
steps:
- name: gcr.io/cloud-builders/gcloud
id: get-github-ssh-secret
entrypoint: 'bash'
args: [ '-c', 'gcloud secrets version access latest --secret=name-of-secret > /root/service-account/credentials.json' ]
volumes:
- name: 'service-account'
path: /root/service-account
...
- name: python:3.8.7
id: integration-tests
entrypoint: /bin/sh
args:
- '-c'
- |-
if [ $_STAGE != "prod" ]; then
python -m pip install -r requirements.txt
python -m pytest test/integration --disable-warnings ;
fi
volumes:
- name: 'service-account'
path: /root/service-account
For the integration tests, I've created a class called Authorizer and I have __get_authorized_header_for_cloud_build and __get_authorized_header_for_cloud_build2 as attempts:
import json
import time
import urllib
from typing import Optional
import google.auth
import requests
from google import auth
from google.auth.transport.requests import AuthorizedSession
from google.oauth2 import service_account
import jwt
class Authorizer(object):
cloudbuild_credential_path = "/root/service-account/credentials.json"
# Permissions to request for Access Token
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
def get_authorized_header(self, receiving_service_url) -> dict:
auth_header = self.__get_authorized_header_for_current_user() \
or self.__get_authorized_header_for_cloud_build(receiving_service_url)
return auth_header
def __get_authorized_header_for_current_user(self) -> Optional[dict]:
credentials, _ = auth.default()
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
if hasattr(credentials, "id_token"):
authorized_header = {"Authorization": f'Bearer {credentials.id_token}'}
auth_req.session.close()
print("Got auth header for current user with auth.default()")
return authorized_header
def __get_authorized_header_for_cloud_build2(self, receiving_service_url) -> dict:
credentials = service_account.Credentials.from_service_account_file(
self.cloudbuild_credential_path, scopes=self.scopes)
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
return {"Authorization": f'Bearer {credentials.token}'}
def __get_authorized_header_for_cloud_build(self, receiving_service_url) -> dict:
with open(self.cloudbuild_credential_path, 'r') as f:
data = f.read()
credentials_json = json.loads(data)
signed_jwt = self.__create_signed_jwt(credentials_json, receiving_service_url)
token = self.__exchange_jwt_for_token(signed_jwt)
return {"Authorization": f'Bearer {token}'}
def __create_signed_jwt(self, credentials_json, run_service_url):
iat = time.time()
exp = iat + 3600
payload = {
'iss': credentials_json['client_email'],
'sub': credentials_json['client_email'],
'target_audience': run_service_url,
'aud': 'https://www.googleapis.com/oauth2/v4/token',
'iat': iat,
'exp': exp
}
additional_headers = {
'kid': credentials_json['private_key_id']
}
signed_jwt = jwt.encode(
payload,
credentials_json['private_key'],
headers=additional_headers,
algorithm='RS256'
)
return signed_jwt
def __exchange_jwt_for_token(self, signed_jwt):
body = {
'grant_type': 'urn:ietf:params:oauth:grant-type:jwt-bearer',
'assertion': signed_jwt
}
token_request = requests.post(
url='https://www.googleapis.com/oauth2/v4/token',
headers={
'Content-Type': 'application/x-www-form-urlencoded'
},
data=urllib.parse.urlencode(body)
)
return token_request.json()['id_token']
So when running locally, the __get_authorized_header_for_current_user is being used and works. When running in Cloud Build, __get_authorized_header_for_cloud_build is used. But even when temporarily disabling __get_authorized_header_for_current_user and let cloudbuild_credential_path reference to a json-file on my local pc, it keep getting 401s. Even when I give the service account from the credentials-file Owner rights. Another attempt is __get_authorized_header_for_cloud_build where I try to get the token more by myself instead of a package, but still 401.
For completeness, the integration test look somewhat like this:
class NameOfViewIntegrationTestCase(unittest.TestCase):
base_url = "https://**.a.run.app"
name_of_call_url = base_url + "/name-of-call"
def setUp(self) -> None:
self._authorizer = Authorizer()
def test_name_of_call__authorized__ok_result(self) -> None:
# Arrange
url = self.name_of_call_url
# Act
response = requests.post(url, headers=self._authorizer.get_authorized_header(url))
# Arrange
self.assertTrue(response.ok, msg=f'{response.status_code}: {response.text}')
Any idea what I'm doing wrong here? Let me know if you need any clarification on something. Thanks in advance!
Firstly, your code is too complex. If you want to leverage the Application Default Credential (ADC) according with the runtime environment, only these lines are enough
from google.oauth2.id_token import fetch_id_token
from google.auth.transport import requests
r = requests.Request()
print(fetch_id_token(r,"<AUDIENCE>"))
On Google Cloud Platform, the environment service account will be used thanks to the metadata server. On your local environment, you need to set the environment variable GOOGLE_APPLICATION_CREDENTIALS with as value the path of the service account key file
Note: you can generate id_token only with service account credential (on GCP or on your environment), it's not possible with your user account
The problem here, it's that doesn't work on Cloud Build. I don't know why, but it's not possible to generate an id_token with the Cloud Build metadata server. So, I wrote an article on this with a possible workaround
I am setting up RBAC with Airflow, and testing locally to start. I have provisioned an AWS Cognito User Group via the console. Additionally, I have a webserver_config.py file I have mounted to my Airflow docker container to set up OAuth with RBAC.
Relevant section in my webserver_config.py file:
COGNITO_URL = os.getenv('COGNITO_URL')
CONSUMER_KEY = os.getenv('COGNITO_CLIENT_KEY')
SECRET_KEY = os.getenv('COGNITO_CLIENT_SECRET')
# When using OAuth Auth, uncomment to setup provider(s) info
# Google OAuth example:
OAUTH_PROVIDERS = [{
'name':'AWS Cognito',
'whitelist': ['#company.com'], # optional
'token_key':'access_token',
'icon':'fa-amazon',
'remote_app': {
'base_url': os.path.join(COGNITO_URL, 'oauth2/idpresponse'),
# 'base_url': COGNITO_URL,
'request_token_params':{
'scope': 'email profile'
},
'access_token_url': os.path.join(COGNITO_URL, 'oauth2/token'),
'authorize_url': os.path.join(COGNITO_URL, 'oauth2/authorize'),
'request_token_url': None,
'consumer_key': CONSUMER_KEY,
'consumer_secret': SECRET_KEY,
}
}]
Variables are as follows:
COGNITO_URL: The domain name I have created in the "App Integration" section of my user pool
COGNITO_CLIENT_KEY: The app client id for my app in the "App Clients" section of my user pool
COGNITO_CLIENT_SECRET: The app client secret for my app in the "App Clients" section of my user pool
In the Cognito UI, I have the following settings for my App Client:
enter image description here
Basically, I have set the endpoints as they should be on my local machine when testing. I have fiddled with both the http://localhost:8083/oauth2/idpresponse and http://localhost:8083/admin (normal home page for Airflow) routes and received the same error.
I think that the issue is that the URI the client is trying to request and the URI specified do not match. I tried following the advice at https://stackoverflow.com/a/53602884/13717098, but when I extracted that URI and saved it in the Cognito console, I continue to get the same error. I am looking for help identifying the URI needed. The request I've identified per the linked post is: /oauth2/authorize?response_type=code&client_id=269vguq386076suj80vpq4ctmj&redirect_uri=http%3A%2F%2Flocalhost%3A8083%2Foauth-authorized%2FAWS%2520Cognito&scope=email+profile&state=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJuZXh0IjpbImh0dHA6Ly9sb2NhbGhvc3Q6ODA4My9ob21lIl19.CcuxpZyuVIqW0GtnNL219Xkg1IftE0tzFiVilR6b4us I would appreciate any help with identifying the URI and/or its associated patterns.
Edited for spacing.
Flask builder library uses the name of the config object as value in redirect_uri.
Set callback value to: http://localhost:8083/oauth-authorized/AWS%20Cognito instead of http://localhost:8080/oauth2/idresponse in AWS Cognito client. This should solve the redirection issue.
The real problem will start for userinfo endpoint as AWS cognito uses OpenID auth pattern.
aws-cognito-client
EDIT
AWS Cognito has oauth2/userinfo endpoint for receiving user information. To retrieve the userinfo, you're supposed to send openid scope along with your request. Following is my webserver_config.py.
from airflow.www_rbac.security import AirflowSecurityManager
from flask_appbuilder.security.manager import AUTH_OAUTH
import os
import json
class CognitoSecurity(AirflowSecurityManager):
def oauth_user_info(self, provider, response=None):
if provider == "aws_cognito":
me = self.appbuilder.sm.oauth_remotes[provider].get("userInfo")
data = json.loads(me.raw_data)
print("User info from aws_cognito: {0}".format(data))
return {"username": data.get("username"), "email": data.get("email")}
else:
return {}
AUTH_TYPE = AUTH_OAUTH
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Admin"
COGNITO_URL = ""
CONSUMER_KEY = ""
SECRET_KEY = ""
OAUTH_PROVIDERS = [{
'name':'aws_cognito',
'whitelist': ['#positsource.com'], # optional
'token_key':'access_token',
'url': COGNITO_URL,
'icon': 'fa-amazon',
'remote_app': {
'base_url': os.path.join(COGNITO_URL, 'oauth2/idpresponse'),
'request_token_params': {
'scope': 'email profile openid'
},
'access_token_url': os.path.join(COGNITO_URL, 'oauth2/token'),
'authorize_url': os.path.join(COGNITO_URL, 'oauth2/authorize'),
'request_token_url': None,
'consumer_key': CONSUMER_KEY,
'consumer_secret': SECRET_KEY,
}
}]
SECURITY_MANAGER_CLASS = CognitoSecurity
This should get the airflow webserver working with AWS cognito. Roles and permissions management can be done by you.
I am having issues trying to create a calendar event using the Google Calendar API with an AWS Lambda function call.
The AWS CloudWatch logs are not giving me a whole lot of information to work with for debugging. I am very new to working with the Google API and can't figure out what is going wrong with my code.
The code I am using is here:
from __future__ import print_function
from oauth2client import service_account
from apiclient import discovery
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage
import httplib2
import boto3
import json
import logging
import requests
def lambda_handler(event, context):
logger = logging.getLogger()
logger.setLevel(logging.INFO)
ses = boto3.client('ses')
email_address = 'myemail#gmail.com' # change it to your email address
SCOPES = ['https://www.googleapis.com/auth/calendar']
SERVICE_ACCOUNT_FILE = 'aws-iot-f3edea2d0394.json'
credentials = service_account.ServiceAccountCredentials.from_json_keyfile_name(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
http = httplib2.Http()
http = credentials.authorize(http)
service = discovery.build('calendar', 'v3', http=http)
calendarId = 'myemail#gmail.com'
event = {
"summary": "aws iot test",
"location": "etc",
"description": "aws iot test - pushed from button",
"start": {
"dateTime": "2018-01-31T09:00:00-07:00",
"timeZone": "America/Los_Angeles"
},
"end": {
"dateTime": "2018-01-31T17:00:00-07:00",
"timeZone": "America/Los_Angeles"
}
}
event = service.events().insert(calendarId='primary', body=event).execute()
The calendar event is not being created, and I cannot figure out why. The only information I am getting from the AWS CloudWatch logs has the last event here:
[INFO] 2018-01-23T17:09:28.116Z 29b2e36a-0060-11e8-8c9e-1725945b009e URL being requested: POST https://www.googleapis.com/calendar/v3/calendars/primary/events?alt=json
This is not very informative. Can anyone help out with maybe getting AWS to give me more information, or perhaps point out what I am doing wrong in the Python code? Thank you!
I was able to figure out the issue. Google left out some administrative details needed for the integration to work, and I was able to track down a similar issue someone else was having at Google Calendar Api v3 asp.net Code 200, yet no event inserted.
Turns out you need to enable the Google service account to make changes to your calendar.