I'm finding a way to automate the authentication from AAD with python.
In the past i used username and password for login to OneDrive and it worked properly. Here is the code:
pca = msal.PublicClientApplication(CLIENT_ID, authority=AUTHORITY_URL)
token = pca.acquire_token_by_username_password(USERNAME, PASSWORD, SCOPES)
Now, that i have two factor authentication i cannot use the same code to access OneDrive account.
I've searched the internet but all the solutions that i found requires to open the browser, and i can't do this because python script is a chronjob and runs in the late night. I need a solution that works in "background" without any action required.
Possibly it's better if there is a solution with MSAL library due to some permissions that i should request if i would change the library.
Thanks for the help!
You might consider using the Client-Credentials Grant flow or OAuth2. You would have to modify your cronjob to move away from fetching a token on behalf of the user and update it to acquire a token as the application using the application's identity (the app registration done in AAD). Upgrading to client-credentials flow, which is actually designed for scenarios similar to yours, will help you in situations where you cannot afford user interaction and you want the service to work in the backend.
For more information on Client-Credentials flow, check here.
Also, you can refer to the following python app that implements client-credentials flow:
Call Microsoft Graph API using App Client Secret
Call Microsoft Graph API using App Client Certificate:
I need to use Cloud Vision API in my python solution, I've been relying on an API key for a while now, but at the moment I'm trying to give my Compute Engine's default service account the scope needed to call Vision, with little luck so far.
I have enabled vision API in my project via cloud console, but I still get that 403 error:
Request had insufficient authentication scopes.
I would set access individually for each API from my gce's edit details tab, but couldn't find Vision listed along the other API's.
The only way I managed to correctly receive a correct response from Vision API is by flagging the "Allow full access to all Cloud APIs" checkbox, again from my gce's edit details tab, but that doesn't sound too secure to me.
Hopefully there are better ways to do this, but I couldn't find any on Vision's documentation on authentication, nor in any question here on stack overflow (some had a close topic, but none of the proposed answers quite fitted my case, or provided a working solution).
Thank you in advance for your help.
EDIT
I'm adding the list of every API I can individually enable in my gce's default service account from cloud console:
BigQuery; Bigtable Admin; Bigtable Data; Cloud Datastore; Cloud Debugger; Cloud Pub/Sub; Cloud Source Repositories; Cloud SQL; Compute Engine; Service Control; Service Management; Stackdriver Logging API; Stackdriver Monitoring API; Stackdriver Trace; Storage; Task queue; User info
None of them seems useful to my needs, although the fact that enabling full access to them all solves my problem is pretty confusing to me.
EDIT #2
I'll try and state my question(s) more concisely:
How do I add https://www.googleapis.com/auth/cloud-vision to my gce instance's default account?
I'm looking for a way to do that via any of the following: GCP console, gcloud command line, or even through Python (at the moment I'm using googleapiclient.discovery.build, I don't know if there is any way to ask for vision api scope through the library).
Or is it ok to enable all the scopes as long as limit the roles via IAM? And if that's the case how do I do that?
I really can't find my way around the documentation, thank you once again.
Google Cloud APIs (Vision, Natural Language, Translation, etc) do not need any special permissions, you should just enable them in your project (going to the API Library tab in the Console) and create an API key or a Service account to access them.
Your decision to move from API keys to Service Accounts is the correct one, given that Service Accounts are the recommended approach for authentication with Google Cloud Platform services, and for security reasons, Google recommends to use them instead of API keys.
That being said, I see that you are using the old Python API Client Libraries, which make use of the googleapiclient.discovery.build service that you mentioned. As of now, the newer idiomatic Client Libraries are the recommended approach, and they superseded the legacy API Client Libraries that you are using, so I would strongly encourage to move in that direction. They are easier to use, more understandable, better documented and are the recommended approach to access Cloud APIs programatically.
Getting that as the starting point, I will divide this answer in two parts:
Using Client Libraries
If you decided to follow my advice and migrate to the new Client Libraries, authentication will be really easy for you, given that Client Libraries use Application Default Credentials (ADC) for authentication. ADC make use of the default service account for Compute Engine in order to provide authentication, so you should not worry about it at all, as it will work by default.
Once that part is clear, you can move on to create a sample code (such as the one available in the documentation), and as soon as you test that everything is working as expected, you can move on to the complete Vision API Client Library reference page to get the information about how the library works.
Using (legacy) API Client Libraries
If, despite my words, you want to stick to the old API Client Libraries, you might be interested in this other documentation page, where there is some complete information about Authentication using the API Client Libraries. More specifically, there is a whole chapter devoted to explaining OAuth 2.0 authentication using Service Accounts.
With a simple code like the one below, you can use the google.oauth2.service_account module in order to load the credentials from the JSON key file of your preferred SA, specify the required scopes, and use it when building the Vision client by specifying credentials=credentials:
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ['https://www.googleapis.com/auth/cloud-vision']
SERVICE_ACCOUNT_FILE = '/path/to/SA_key.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
vision = googleapiclient.discovery.build('vision', 'v1', credentials=credentials)
EDIT:
I forgot to add that in order for Compute Engine instances to be able to work with Google APIs, it will have to be granted with the https://www.googleapis.com/auth/cloud-platform scope (in fact, this is the same as choosing the Allow full access to all Cloud APIs). This is documented in the GCE Service Accounts best practices, but you are right that this would allow full access to all resources and services in the project.
Alternatively, if you are concerned about the implications of allowing "access-all" scopes, in this other documentation page it is explained that you can allow full access and then perform the restriction access by IAM roles.
In any case, if you want to grant only the Vision scope to the instance, you can do so by running the following gcloud command:
gcloud compute instances set-service-account INSTANCE_NAME --zone=INSTANCE_ZONE --scopes=https://www.googleapis.com/auth/cloud-vision
The Cloud Vision API scope (https://www.googleapis.com/auth/cloud-vision) can be obtained, as for any other Cloud API, from this page.
Additionally, as explained in this section about SA permissions and access scopes, SA permissions should be compliant with instance scopes; that means that most restrictive permission would apply, so you need to have that in mind too.
To set the access scopes from the python client libraries with the same effect as that radio button in the GUI:
instance_client = compute_v1.InstancesClient()
instance.service_accounts = [
compute_v1.ServiceAccount(
email="$$$$$$$-compute#developer.gserviceaccount.com",
scopes=[
"https://www.googleapis.com/auth/compute",
"https://www.googleapis.com/auth/cloud-platform",
],
)
]
With a tutorial for creating instances from python here
I am trying to use Google cloud's natural language API at work, and I believe my corporate firewall is blocking communication between python and google cloud.
After entering the following in the terminal:
gcloud auth application-default login
My browser opens up to log into my google account successfully. After I log in, however, I get
ERROR: There was a problem with web authentication. Try running a
gain with --no-launch-browser.
ERROR: (gcloud.auth.application-default.login) Could not reach th
e login server. A potential cause of this could be because you ar
e behind a proxy. Please set the environment variables HTTPS_PROX
Y and HTTP_PROXY to the address of the proxy in the format "proto
col://address:port" (without quotes) and try again.
Example: HTTPS_PROXY=https://192.168.0.1:8080
I believe I need to contact my IT department to add an exception to our firewall. Does anyone know what the address / port for google cloud's natural language processing API is?
I can't directly answer your question but I can provide some general guidance that might workaround your issue.
The command
gcloud auth application-default login
Is a convenience helper for running sample code locally but it's really not the best auth strategy for a variety of reasons. It uses a special client ID that won't always have all your quota.
The way I would recommend using the API is Service Accounts. You can create a Service Account in the Cloud Console under API credentials, and then download a JSON key. Then you set the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to your file, and it will automatically work assuming you are using Application Default Credentials (which most samples and client libraries use).
On App Engine, and Compute Engine (assuming you created the VM with the correct scopes) Service Accounts exist by default so you don't even need to download the JSON and set the environment variable.
The other way you can use the API is just creating an API Key, then hit the HTTP endpoints with ?key=api-key at the end of the URL. API Keys are also less than ideal (no idea who client is, no scopes), but are a simple option.
In your case, I'd recommend using JSON service account keys and the environment variable, but it's worth reading the official authentication guide.
Is there any way using OAuth to download all content of a google bucket directory to a local directory.
I found two ways using (get request object) from storage API and gsutil. But since API uses direct name downloading I have to first parse all the list of bucket content and then send get request and then download it. I find gsutil more convenient but for this, I have to hard code details for the credential.
Basically, i am developing a client related application where I have to download the big query table data to the client local server
Can anyone help me for this
Unless your application knows ahead of time the object names that you want to download, you'll need to perform a list followed by GETs for each object.
You can use the gcloud-python client library to do this. You can configure your client application has the OAuth2 credentials and the library should handle the rest of the necessary authentication for you. See the documentation here for the basics of authentication, and [here](https://googlecloudplatform.github.io/google-cloud-python/stable/storage-blobs.html for interacting with Google Cloud Storage objects.
I have a website and I need to test it with 250 users. However, I am using google login via OAuth2. The website is hosted on Google App Engine.
I am stuck at this login part. After we log in we get and access token from Google that is passed to Google APIs via the Authorization: Bearer header. We use the access token in the application to get user details and access other google apps for that user. I don't know how to get that access token using my external script.
One option is to mock / stub this part of your application out during testing. For instance, you can provide a certain header that tells your application that you're in test mode and instead of calling the real google APIs, it calls a mock API instead. If your application is setup for dependency injection this could be trivial, otherwise, it may involve monkey-patching or similar.
Another option is to use an OAuth2 Service Account and acquire access tokens for a bunch of users in a test Google Apps domain. Your test script can do this and then just pass the access tokens just as a client normally would.