I have a big Query Database connected to a Google Sheet in which I have a read only access
My request is that I want to get data from a table and this request is working perfectly fine in the Big Query editor but I want to create a Google Cloud function to have an API and access this request directly from URL
I have ceated a Service Account using this command:
gcloud iam service-accounts create connect-to-bigquery
gcloud projects add-iam-policy-binding test-24s --member="serviceAccount:connect-to-bigquery#test-24s.iam.gserviceaccount.com" --role="roles/owner"
and I have created a Google cloud function as follow :
Creating Cloud Function
Service account settings
Here is my code for main.py file :
from google.cloud import bigquery
def hello_world(request):
client = bigquery.Client()
query = "SELECT order_id, MAX(status) AS last_status FROM `test-24s.Dataset.Order_Status` GROUP BY order_id"
query_job = client.query(query)
print("The query data:")
for row in query_job:
print("name={}, count ={}".format(row[0], row["last_status"]))
return f'The query run successfully'
And for the requirements.txt file :
# Function dependencies, for example:
# package>=version
google-cloud-bigquery
The function deploys successfully however when I try to test it I get this error :
Error: function terminated. Recommended action: inspect logs for termination reason. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging Details:
500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
And when reading the log file I found this error
403 Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.
Please help me to solve this I already tried all the solutions that I found on the net without any success
Based on this: "Permission denied while getting Drive credentials" - I would say that your service account's IAM permissions are not 'transient' => while that service account probably has relevant access to the BigQuery, it does not have access to the underlined spreadsheet maintained on the Drive...
I would try - either
extend the scope of the service account's credentials (if possible, but that may not be very straightforward). Here is an article by Zaar Hai with some details - Google Auth — Dispelling the Magic and a comment from Guillaume - "Yes, my experience isn't the same";
or (preferably from my point of view)
make a copy (may be with regular updates) of the original spreadsheet based table as a native BigQuery table, and use the later in your cloud function. A side effect of this approach - a significant performance improvement (and cost savings).
Related
There is a GCP project that contains a bucket that I have read and write permissions to, but I don't know the name of the project nor am I part of the project. None of the contents of this bucket are public.
I have successfully authenticated my user locally using gcloud auth application-default login.
I can successfully download from this bucket using gsutil cat gs://BUCKET/PATH.
However, if I use the google.cloud.storage Python API, it fails at the point of identifying the project, presumably because I don't have access to the project itself:
from google.cloud import storage
client = storage.Client()
storage.Blob.from_string("gs://BUCKET/PATH", client=client).download_as_text()
The billing account for the owning project is disabled in state closed: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
I can't use storage.Client.create_anonymous_client() since this is only relevant for public buckets, but I suspect that I could fix this by changing the credentials argument to Client().
Can anyone help me download the file from Google Cloud in this case?
If you have permission, you can find the project number for a given bucket with the bucket get API call. See this guide for how to do it with various client libraries.
I am trying to get Firestore working in emulator-mode with Python on my local Linux PC. Is it possible to use anonymous credentials for this so I don't have to create new credentials?
I have tried two methods of getting access to the Firestore database from within a Python Notebook, after having run firebase emulators:start from the command-line:
First method:
from firebase_admin import credentials, firestore, initialize_app
project_id = 'my_project_id'
cred = credentials.ApplicationDefault()
initialize_app(cred, {'projectId': project_id})
db = firestore.client()
This raises the following exception:
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
Second method:
from google.auth.credentials import AnonymousCredentials
from google.cloud.firestore import Client
cred = AnonymousCredentials()
db = Client(project=project_id, credentials=cred)
This works, but then I try and access a document from the database that I have manually inserted into the database using the web-interface, using the following Python code:
doc = db.collection('my_collection').document('my_doc_id').get()
And then I get the following error, which perhaps indicates that the anonymous credentials don't work:
PermissionDenied: 403 Missing or insufficient permissions.
Thoughts
It is a surprisingly complicated system and although I have read numerous doc-pages, watched tutorial videos, etc., there seems to be an endless labyrinth of configurations that need to be setup in order for it to work. It would be nice if I could get Firestore working on my local PC with minimal effort, to see if the database would work for my application.
Thanks!
Method 2 works if an environment variable is set. In the following change localhost:8080 to the Firestore server address shown when the emulator is started using firebase emulators:start
import os
os.environ['FIRESTORE_EMULATOR_HOST'] = 'localhost:8080'
I don't know how to make it work with Method 1 above using the firebase_admin Python package. Perhaps someone else knows.
Also note that the emulated Firestore database will discard all its data when the server is shut down. To persist and reuse the data start the emulator with a command like this:
firebase emulators:start --import=./emulator_data --export-on-exit
I'm working on a project with python3 and BigQuery and i encountered a strange behavior in google-cloud-bigquery library.
When i exec simple query like:
select * from my.project where field1="foo" and field2="bar"
The program running well but when I try to use a more "complex" query like:
with a as (...),b as(...)
select * from a as a_el
select * from b as b_el
join b_el on a_el.field1 = b_el.field1 ...
The program crash with:
google.api_core.exceptions.PermissionDenied: 403 request failed: the user does not have 'bigquery.readsession.create' permission
So, I know this is a permission problem but my question is: what's the difference under the hood? Why extra permission are needed? The tables are in the same project, so where is the problem?
I use query() method for execute queries
The google-cloud-bigquery python library leverages the BigQuery Storage read api to download query results faster. The permissions are part of the roles/bigquery.user predefined role. Likely you've got a custom role on the project that excludes the readsession permissions, which are permissions specific to the BigQuery storage API. This sounds like a case where the results from the first query are small enough that the storage API session doesn't get created.
More information about permissions can be found on the access control page.
I am new to Google cloud and would like to know if there is a way to retrieve cloudSQL (MySQL) instance metadata and error logs using Python.
I installed the Google cloud SDK and ran the following commands to retrieve metadata and I got detailed metadata like IP, region, disk, etc.
gcloud sql instances describe my-instance-id
I need to check this data periodically for compliance. I have worked on AWS and I use boto3 Python package for these kind of tasks. I googled for boto3 equivalent in Google but the docs for Google API client are really confusing to me.
I also need to fetch MySQL error logs from cloudSQL instance (for alerting in case any errors are found).
Can anyone show me how to perform these operations using google API for python or point me in the right direction?
Here is a sample code on how to retrieve the Cloud SQL MySQL error logs using Cloud Logging API. For testing I logged in with a wrong password to generate error logs.
The filter used is a sample filter in the Cloud Logging docs.
from google.cloud.logging import Client
projectName = 'your-project-here'
myFilter = 'resource.type = "cloudsql_database" AND log_id("cloudsql.googleapis.com/mysql.err")'
client = Client(project = projectName)
entries = client.list_entries(filter_ = myFilter)
for entry in entries:
print(entry)
Output snippet:
Here's how to get SQL instance metadata:
import json
from googleapiclient import discovery
service = discovery.build('sqladmin', 'v1beta4')
req = service.instances().list(project="project-name")
resp = req.execute()
print(json.dumps(resp, indent=2))
credit to #AKX, found the answer at cloud.google.com/sql/docs/mysql/admin-api/libraries#python
No luck on the 2nd part tough i.e. retrieving MySQL error log
I am having trouble trying to set up a connector to Kusto using the Kusto client library for Python.
I managed to make it work using the with_aad_device_authentication method, by doing the following:
KCSB = KustoConnectionStringBuilder.with_aad_device_authentication(KUSTO_CLUSTER)
KCSB.authority_id = AAD_TENANT_ID
client = KustoClient(KCSB)
KUSTO_QUERY = "Table | take 10"
RESPONSE = client.execute(KUSTO_DATABASE, KUSTO_QUERY)
which requires me to authenticate by going to a web page and entering a code provided by the library.
However, when I am trying to connect to the database using the with_aad_application_key_authentication method it throws me
KustoServiceError: (KustoServiceError(...), [{'error': {'code': 'Forbidden', 'message': 'Caller is not authorized to perform this action', '#type': 'Kusto.DataNode.Exceptions.UnauthorizedDatabaseAccessException' ...
which I don't understand since I have granted my application with the following permissions: Azure Data Explorer (with Multifactor Autentication) and Azure Data Explorer.
I have been struggling on this for a while and I couldn't come up with a solution. Does anyone have any idea of what could be the problem here?
There are two possible reasons:
1) You did not give the app permission on the database itself. Permissions on the Azure data explorer resource (we call it the 'control plane') using the "access control (IAM)" button allow your app to do management operations on the cluster (such as adding and removing databases), while permissions in the database itself allows doing operations within the database such as creating tables and doing queries (we call it the 'data plane'). Please note that you can also provide permissions to all databases in the cluster by clicking on "permissions" button in the cluster blade.
In order to fix it, click on the database in Azure portal and once you are in the database blade, click on the 'permissions' button and give the app permission (admin, user, viewer etc.). see screenshot below.
2) You did not provide the any of the three required datapoints correctly (appId, appKey and authority id)
Here is the relevant screenshot for adding permission in a specific database:
Adding some more context.
Granting your application delegated permissions to ADX only allows your application to perform user authentication for ADX resources, but it does not grant the application itself any roles on your specific ADX resource.
The answer above guides you how to do that.