I have a Google App Engine Standard Environment application written in Python 3, using Flask as the framework, and firestore in native mode as the database. All of the database calls are done in the App Engine code, hidden behind Flask end points/views/handlers. Client browsers do not execute any javascript that directly call the firestore database. Client side javascript is basically 'dumb' code used for cosmetics. The only time client side javascript does "anything" is when a user creates a new account or logs in using the firebase auth ui.
Having said so, I noticed that some online resources mention that it is absolutely necessary to secure the firestore database since anything that is not disallowed by security rules are basically allowed (i.e. the firestore database is insecure by default), however, I suspect that this is only the case for apps that have thick clients (i.e. the client side code or javascript is in charge of doing the heavy lifting of querying and writing to firestore).
So my question is, is writing these security rules necessary only for mobile/web clients and not for firestore databases accessed only by server side code? Or is it necessary for all firestore projects to define these security rules? If so, then I would appreciate any pointers as to where to find reasonable default security rules to start securing my firestore database.
I am including a caricature of my flask main.py file for reference.
# main.py
from google.cloud import firestore
from mylibrary import function_that_fetches_user_data
from mylibrary2 import function_that_writes_user_content
def validate_cookie(protected_function):
def wrapper(*args, **kwargs):
# handle cookie validation
# run protected function
return wrapper
# The dashboard is meant to display user data and user content to the user.
# It is not meant to be seen by other users.
#app.route("/user_dashboard")
#validate_cookie
def dashboard():
user_id = get_uid_from_cookie
firestore_client = firestore.Client()
user_data = function_that_fetches_user_data(user_id, firestore_client)
return render_template('dashboard.html', user_data)
# The write function creates user content that should only be accessible to the author
# and the system/app.
#app.route("/write_user_content")
#validate_cookie
def write_user_content():
user_id = get_uid_from_cookie
firestore_client = firestore.Client()
result = function_that_writes_user_content(user_id, firestore_client)
return render_template('success.html', result)
Security rules are only necessary to control access coming from web and mobile clients. Backend SDK accessing Firestore actually bypass security rules altogether, so writing any rules at all won't change the behavior of your backend code at all.
If you simply do not directly access the database from web or mobile, then you can set the security rules to reject all access, and that's fine.
match /{document=**} {
allow read, write: if false;
}
Related
I'm trying to run a query on bigquery in a Django project and get results. While working successfully in localhost, it does not redirect to the verification link at all when I take it to the live server.
I think I need to change the redirect_uri value as I read it. I added this in Da appflow variable but the url doesn't change. I am using the same query below with the example query in google's document, I am submitting my own query because it contains private information, but it is exactly the same query.
I have added to Authorized redirect URIs, and I put the api in production mode.;
The resulting redirect url is output as localhost in this way;
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=123-nml31ekr2n0didomei5.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&state=XF1WdtCoR4HaICwzSKk9A1giBrSzBv&access_type=offline
def query_stackoverflow():
launch_browser = True
project = 'xx-prod'
appflow = flow.InstalledAppFlow.from_client_secrets_file("static/client_secret_518684-nmpoqtgo5flvcgnl31ekr2ni5.apps.googleusercontent.com.json", scopes=["https://www.googleapis.com/auth/bigquery"], redirect_uri=["https://xx.com/"])
if launch_browser:
appflow.run_local_server()
else:
appflow.run_console()
credentials = appflow.credentials
client = bigquery.Client(project=project, credentials=credentials)
client = bigquery.Client()
query_job = client.query(
"""
SELECT
CONCAT(
'https://stackoverflow.com/questions/',
CAST(id as STRING)) as url,
view_count
FROM `bigquery-public-data.stackoverflow.posts_questions`
WHERE tags like '%google-bigquery%'
ORDER BY view_count DESC
LIMIT 10"""
)
results = query_job.result() # Waits for job to complete.
for row in results:
print("{} : {} views".format(row.url, row.view_count))
On live server google return auth url like this;
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=51864584-nmpoqtgo5flvcgnln0didomei5.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&state=W2uMZwzaYMEpFzExodRCf2wA4&access_type=offline
The first problem is that it does not automatically redirect to the link as in localhost, the second problem is that when I open this link manually, the link cannot be reached after mail verification.
From what i can see your code is using installed app flow. This means that the consent screen is going to open up on the machine its running on. If you have this running on a server, are you logging into the server and running it or are you in fact creating a web application?
flow.InstalledAppFlow
web app
If you are making a web application then you should be following this sample.
API access on behalf of your clients (web flow)
You will need to convert it to work with big query.
import google.oauth2.credentials
import google_auth_oauthlib.flow
# Initialize the flow using the client ID and secret downloaded earlier.
# Note: You can use the GetAPIScope helper function to retrieve the
# appropriate scope for AdWords or Ad Manager.
flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file(
'client_secret.json',
scope=[oauth2.GetAPIScope('adwords')])
# Indicate where the API server will redirect the user after the user completes
# the authorization flow. The redirect URI is required.
flow.redirect_uri = 'https://www.example.com/oauth2callback'
The code for a web application is slightly different then that of an installed application.
Context:
Node Server in Google App Engine (GAE) that effectively houses a backend for a frontend that is also served by the same app engine instance
Hence why iAP is enabled (for selected web app users only)
Has various endpoints for the frontend to call via reverse-proxy (as I understand it's called)
Google Cloud Function(GCF) within the same project that (funny enough) is actually being called by the node server to initiate the cloud function that then needs to call an endpoint within the GAE node server.
....k wait I might've just found another way to solve the problem but I'll get to that at the end.
I created a VPC Connector for GCF to access a VM instance that I created to talk to external networks. GAE (Flex) is able to do so natively. Not sure if this is relevant but wanted to throw it in the mix.
Short term solution:
Since I need to call the GCF from the GAE node server first, I can just provide it with the relevant data as needed.
Long term solution:
Ideally, the GCF should be called by any other services that might or might not have the data, so it would be ideal to have the GCF call out the GAE endpoint to get the data.
So far:
import urllib
import google.auth.transport.requests
import google.oauth2.id_token
req = urllib.request.Request('https://the-gcp-project-id.appspot.com/api/theEndpoint')
auth_req = google.auth.transport.requests.Request()
id_token = google.oauth2.id_token.fetch_id_token(auth_req, 'https://appengine.googleapis.com')
log.info("Authorization: " + f"Bearer {id_token}")
# req.add_header("Authorization", f"Bearer {id_token}")
# response = urllib.request.urlopen(req)
# # return response.read()
# log.info(response.read())
import requests as reqs
response = reqs.post('https://the-gcp-project-id.appspot.com/theEndpoint', json={'test':'123'}, headers={"Authorization" : f"Bearer {id_token}"})
log.info(response)
This doesn't seem to actually trigger the endpoint though. As far as I know the service account for the cloud function should have the same permissions as the app engine service account.
Can anyone point me in the right direction on this?
I'm running a flask app that will access Bigquery on behalf of users using a service account they upload.
To store those service account credentials, I thought the following might be a good set up:
ENV Var: Stores my credentials for accessing google secrets manager
Secret & secret version: in google secrets manager for each user of the application. This will access the user's own bigquery instance on behalf of the user.
--
I'm still learning about secrets, but this seemed more appropriate than any way of storing credentials in my own database?
--
The google function for accessing secrets is:
def access_secret_version(secret_id, version_id=version_id):
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
# Access the secret version.
response = client.access_secret_version(name=name)
# Return the decoded payload.
return response.payload.data.decode('UTF-8')
However, this returns JSON as a string. When then using this for big query:
credentials = access_secret_version(secret_id, version_id=version_id)
BigQuery_client = bigquery.Client(credentials=json.dumps(credentials),
project=project_id)
I get the error:
File "/Users/Desktop/application_name/venv/lib/python3.8/site-
packages/google/cloud/client/__init__.py", line 167, in __init__
raise ValueError(_GOOGLE_AUTH_CREDENTIALS_HELP)
ValueError: This library only supports credentials from google-auth-library-python.
See https://google-auth.readthedocs.io/en/latest/ for help on authentication with
this library.
Locally I'm storing the credentials and accessing them via a env variable. But as I intend for this application to have multiple users, from different organisations I don't think that scales.
I think my question boils down to two pieces:
Is this a sensible method for storing and accessing credentials?
Can you authenticate to Bigquery using a string rather than a .json file indicated here
Authenticating requests, especially with Google's API's is so incredibly confusing!
I'd like to make authorized HTTP POST requests through python in order to query data from the datastore. I've got a service account and p12 file all ready to go. I've looked at the examples, but it seems no matter which one I try, I'm always unauthorized to make requests.
Everything works fine from the browser, so I know my permissions are all in order. So I suppose my question is, how do I authenticate, and request data securely from the Datastore API through python?
I am so lost...
You probably should not be using raw POST requests to use Datastore, instead use the gcloud library to do the heavy lifting for you.
I would also recommend the Python getting started page, as it has some good tutorials.
Finally, I recorded a podcast where I go over the basics of using Datastore with Python, check it out!
Here is the code, and here is an example:
#Import libraries
from gcloud import datastore
import os
#The next few lines will set up your environment variables
#Replace "YOUR_RPOEJCT_ID_HERE" with the correct value in code.py
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "key.json"
projectID = "YOUR_RPOEJCT_ID_HERE"
os.environ["GCLOUD_TESTS_PROJECT_ID"] = projectID
os.environ["GCLOUD_TESTS_DATASET_ID"] = projectID
datastore.set_default_dataset_id(projectID)
#Let us build a message board / news website
#First, create a fake email for our fake user
email = "me#fake.com"
#Now, create a 'key' for that user using the email
user_key = datastore.Key('User', email)
#Now create a entity using that key
new_user = datastore.Entity( key=user_key )
#Add some fields to the entity
new_user["name"] = unicode("Iam Fake")
new_user["email"] = unicode(email)
#Push entity to the Cloud Datastore
datastore.put( new_user )
#Get the user from datastore and print
print( datastore.get(user_key) )
This code is licensed under Apache v2
How can I access Google App Engine endpoints API for Python (not web, android, ios)?
I read this tutorial but it not explains it enough to understand this.
As I found on serve side I can use such code to identify user:
#endpoints.method(message_types.VoidMessage, Greeting,
path='hellogreeting/authed', http_method='POST',
name='greetings.authed')
def greeting_authed(self, request):
current_user = endpoints.get_current_user()
email = (current_user.email() if current_user is not None
else 'Anonymous')
return Greeting(message='hello %s' % (email,))
Full code of API example
How can I connect from Python client to this API and call 'hellogreeting/authed' with authentication current_user != None.
Can you share some code how to do it?
app_id = 'xxx'
user = 'xxx'
password = 'xxx'
callAPI(app_id, user, password, 'hellogreeting/authed')
You need to configure your App Engine instance to be able to serve your API. I would recommend you create a separate module dedicated to your API, like explained in these docs: https://developers.google.com/appengine/docs/python/endpoints/api_server.
Once everything is correctly set up on the server side, you can call your API using something like: http://your-module.your-app.appspot.com/_ah/spi/hellogreeting/authed.
If you're using the development server, things are a little bit different for accessing modules, but once you know which port number the App Engine development server has assigned to your API module, you can reach it locally using: http://localost:<api_module_port_#>/_ah/spi/hellogreeting/authed.
Hope this helped.