I have a Python script that accesses Google cloud platform, I also set up the service account, I can request & save the json file through the cloud console webpage after I login my Google account, and sets the GOOGLE_APPLICATION_CREDENTIALS to that json file, so the Python script can have access.
Now I want to share it with others, I have requirements.txt for the Python scrip to install the gcloud-api library, but I don't want to enforce others to install gcloud-sdk. And I don't want to share that json file with others. I would like to let others run the script, and if that json credential file is not found, the script will ask them to:
login gcloud
generate and save json credential, e.g., to a default directory
sets GOOGLE_APPLICATION_CREDENTIALS to that json file
All the step better be done without browser. Is there a way to use Python to do such thing? I did some research & googling but no luck.
I believe I can do this anyway by Python invoking curl or using requests, but just wonder if there is a simpler way to do this.
UPDATE
Thanks to the comments but I just want to release to others a Python script file.
I read through the service account and the work identity federation, I don‘t have infra to setup identity provider. I believe that based on my reading and the comments, if I want to use something like oauth, I need to register my script as a client on Google. I am not sure if this is feasible or considered as a good practice...
Related
I'm trying to automate report downloading from Google Play (thru Cloud Storage) using GC Python client library. From the docs, I found that it's possible to do it using gsutil. I found this question has been answered here, but I also found that Client infers credentials from environment and I plan to do this on automation platform with (assumed) no gcloud credentials set.
I've found that you can generate gsutil boto file then use it as credential, but how can I load this into the client library?
This is not exactly a direct answer to your question, but the best way would be to create a service account in GCP, and then use the service account's JSON keyfile to interact with GCS. See this documentation on how to generate said keyfile.
NOTE: You should treat this keyfile as a password as it will have the access you give it in the step below. So no uploading to public github repos for example.
You'll also have to give the serviceaccount the permission Storage Object Viewer, or one with more permissions.
NOTE: Always use the least needed to due to security considerations.
The code for this is extremely simple. Note that this is extremely similar to the methods mentioned in the link for generating the keyfile, the exception being the way the client is instantiated.
requirements.txt
google-cloud-storage
code
from google.cloud import storage
cred_json_file_path = 'path/to/file/credentials.json'
client = storage.Client.from_service_account_json(cred_json_file_path)
If you want to use the general Google API Python client library you can use this library to do a similar instantiation of a credentials object using the JSON keyfile, but for GCS the google-cloud-storage library is very much preferred as it does some magic behind the scenes, as the API python client library is a very generic one that (theoretically) be useable with all Google API's.
gsutil will look for a .boto file in the home directory of the user invoking it, so ~/.boto, for Linux and macOS, and in %HOMEDRIVE%%HOMEPATH% for Windows.
Alternately, you can set the BOTO_CONFIG environment variable to the path of the .boto file you want to use. Here's an example:
BOTO_CONFIG=/path/to/your_generated_boto_file.boto gsutil -m cp files gs://bucket
You can generate a .boto file with a service account by using the "-e" flag with the config command: gsutil config -e.
Also note that if gsutil is installed with the gcloud command, gcloud will share its authentication config with gsutil unless you disable that behavior with this command: gcloud config set pass_credentials_to_gsutil false.
https://cloud.google.com/storage/docs/boto-gsutil
I'm using selenium in order to extract some data (as a json file). This json is the final output of the script.
I've managed to do it locally so far in two different ways:
With a local webdriver (for Chrome).
With a Docker container.
However, I need it to be accessible from anywhere, in systems that don't have either webdrivers/Docker installed.
I have thought about deploying the script to Heroku and work around that idea, but I have no idea how to handle the data in this situation.
I think that cloud services are meant for these situations.
A storage account (S3 in Amazon or Blob Storage for Azure) allows you to acces the data from anywhere, and without almost any limitation of space, using its API or by using their SDK's.
Also you can specify access policies if your data should not be publicly accessible.
As you have already developed your script into a Docker conatiner, your are ready to run it in almost every cloud provider (for example in Amazon ECR).
I have a python script for some automation that contains a username and password.
This script triggers in a azure pipeline on GitHub.
But the idea of pushing credentials to GitHub is not good so I would like to store those credentials in a variable so it can be used by the script when the pipeline triggers.
Just to make the example clear, the python script holding the credentials looks like this:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class$='visible-lg'] input#signInFormUsername"))).send_keys('<USERNAME>')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[class$='visible-lg'] input#signInFormPassword"))).send_keys('<PASSWORD')
Those are the 2 important values that I would like to remove from my plain script and store them in an env variable so when the pipeline triggers the python script, will know from were to fetch those credentials.
Can anyone advice on the best practice for this kind of scenarios?
Thank you very much and if you need more info just let me know
I highly recommend github secrets. From the documentation:
Secrets are encrypted environment variables that you create in an organization, repository, or repository environment. The secrets that you create are available to use in GitHub Actions workflows. GitHub uses a libsodium sealed box to help ensure that secrets are encrypted before they reach GitHub and remain encrypted until you use them in a workflow.
I am working on a Google Colab notebook that requires the user to mount google drive using the colab.drive python library. They then input relative paths on the local directory tree (/content/drive/... by default on that mount) to files of interest for analysis. Now, I want to use a Google Sheet they can create as a configuration file. There is lots of info on how to authenticate gspread and fetch a sheet from its HTTPS url, but I can't find any info on how to access the .gsheet file using gspread that is already mounted on the local filesystem of the colab runtime.
There are many tutorials using this flow: https://colab.research.google.com/notebooks/io.ipynb#scrollTo=yjrZQUrt6kKj , but I don't want to make the user authenticate twice (having already done so for the initial mount), and i don't want to make them input some files as relative path, some as HTTPS URL.
I had thought this would be quite like using gspread to work with google sheets that I might have on my locally mounted drive as well. But, I haven't seen this workflow anywhere either. Any pointers in that direction might help me out as well.
Thank you!
Instead of adding .gsheet on colab's drive you can try storing it in the user's drive and later fetch from there when needed. So until that kernel is running you won't have to re-authenticate the user.
I'm also not finding anything to authenticate into colab from other device. So you would consider modifying your flow a bit.
I need to download a file on a corporate Sharepoint site using CPython. Existing codebase prevents me from using Ironpython without porting the code, so .NET's WebClient library is out. I also want to download the file without prompting the user to save and without prompting the user for network credentials. I tried other libraries, but they all had short-comings:
urllib2 plus python-ntlm: requires user/pass to be provided
COM automation of Internet Explorer: requires the user to click 'Save'
subprocess using wget or cURL: couldn't get either to authenticate without requesting user/pass
I couldn't find anything in pywin32 that looks like it hooks into urllib2 or provides equivalent functionality. So, is there a way to download the file without requesting credentials and without prompting the user to click 'Save'?
I ended up finding some VB code from a Microsoft support page that uses a function from urlmon.dll I replicated it with a single line of ctypes code and it accomplished exactly what I needed it to do.
ctypes.windll.urlmon.URLDownloadToFileA(0,url,local_file_name,0,0)
url is the location of the resource (in this case, an Excel file on a Sharepoint site)
local_file_name is the local path and name of the file to be saved.
This passed credentials across the wire with no prompts.