I'm a newbie so please bear with me.
I'm trying to set up a Google Cloud Function that can access a single GMail account on the same domain, download some emails and push them to Cloud Storage. I honestly would like to just use an email & password in the script (using Google KMS /w environment variables?) but I understand that isn't possible, and OAuth2 is required.
I've set up an OAuth Client in GCP and I have run the GMail API Python Quickstart guide. Running it locally I am prompted to allow access, and the token is saved so subsequent runs work without prompts.
I deployed the Cloud Function with the pickle file to test if the refresh token will still work, planning to figure out how to use KMS to make this more secure later on. But there's an issue loading the pickle:
UnpicklingError: invalid load key, '\xef'
Which makes it seem like the pickle gets compressed/corrupted on upload.
Is this even a sane approach? How could I do this? The email address is mine, so I was hoping I could just authenticate once and be done with it.
It's not possible for me to use a domain-delegated Service Account by the way- nor use IMAP.
Since you can't do domain delegated service accounts, you can try following this guide on setting up server side authorization. This sounds like what you want since it requires that the user authorizes the app once and then reuses that token. Here is a codelab that will take you through the auth part.
Alternatively, you can use push notifications. It sounds like your current design is to poll the Gmail API to look for new emails periodically, which also involves authorizing access to the account in the Cloud Function. However, if you take advantage of Push Notifications, you can both get the data in real time, and avoid having to authorize the Cloud Function to read the Gmail API. See guide here.
However, probably the easiest solution is to use app scripts. If you set up your cloud function to be triggered via an HTTP target, you can write an app script to ping that URL with the messages you want to send to GCS. Docs here.
function getEmails() {
let inbox = GmailApp.getInboxThreads(0, 50);
// Inbox Threads
for (i=0; i < inbox.length; i++){
let threadMessages = inbox[i].getMessages();
// Thread Messages
for (j=0; j < threadMessages.length; j++){
let message = threadMessages[j].getBody();
let subject = threadMessages[j].getSubject();
var options = {
'method' : 'post',
'contentType': 'application/json',
'payload' : JSON.stringify({"message": message, "subject": subject})
};
UrlFetchApp.fetch('YOUR_FUNCTION_TRIGGER_URL', options);
}
}
}
Related
I used to query my financial data through Power Query in Power BI. Recently I've switched to doing it through a python script running on Google Cloud functions, triggered by Cloud Scheduler. (is this the best way?) It saves a csv file to GCStorage.
The party that provides the data I'm after is switching to oAuth 2.0 using either implicit or authorization code flow. I believe this means that somewhere in this flow a browser is opened where username and password must be entered. Also I need to give a redirect uri to this party, I'm not sure how to implement this in my current setup.
Anyone have an idea? More info about the API can be found here. https://accounting.twinfield.com/webservices/documentation/#/ApiReference/Authentication/OpenIdConnect
Usually the Authorization Code flow would be the way to go in your kind of application.
You will send a authentication request to their API(redirecting the user). They will authenticate the User and redirect the user back to your application, using the redirect URI you provided.
You can get an access token or ID token from their token endpoint using the code, your client id and your client secret.
I am making an application in GAE (python) which allows users (who must be logged in with a Google Account) to upload files to Google Cloud Storage directly. The upload would happen asynchronously (a button would execute some javascript code to send the XHR PUT request with the file).
I decided to use the JSON API, and I'm trying to do a multipart upload. According to the docs I need to include an Authorization header with a token. I was wondering, since I don't need access to anything in the user's account (like calendar or that kind of thing), what would be the best way to achieve this?
Is it possible to create a temporary access token from the application default credentials and send that to the user within the html file to use it then from the js function? If not, what should I do?
You would need to ask the user to grant you the google cloud storage write scopes (which I would strongly recommend any google cloud storage user not grant to any random application). You would also need to grant the end users write permission on your bucket (which also means they can delete everything in the bucket). Unless you know all of your end users up front, this means you would need to make the bucket public write.. which I doubt is something you want to do.
I strongly recommend using delegating access to your service account instead, though unfortunately the JSON api does not currently support any form of authentication delegation. However, the XML API does support Signed URLs.
https://cloud.google.com/storage/docs/access-control/signed-urls
You only need to use it for the client-side uploads, everything else can use the JSON api.
There are three options:
Just sign a simple PUT request
https://cloud.google.com/storage/docs/xml-api/put-object-upload#query_string_parameters
Use a form POST and sign a policy document
https://cloud.google.com/storage/docs/xml-api/post-object#policydocument
Initiate a resumable upload server side and pass the upload URL back to the client. I would only recommend this option if being able to resume the upload is important (e.g. large uploads).
https://cloud.google.com/storage/docs/access-control/signed-urls#signing-resumable
I have to develop a service (using flask and dropbox API) in order to synchronize a server with my dropbox account. (This service has to be run in back in background as a daemon)
First, I have begun with "authentication" : in the beginning I used OAuth 2 (but it was an issue that every time, the client has to confirm the authorization)
So, now I am using an authentication with a generated access token :
dbx = dropbox.Dropbox('ACCESS TOKEN')
So I have some questions:
1) Is it recommended and secure to use such authentication ?! otherwise , what it the best solution for this ?
2) What is the advantage of using the microframework flask in that case , because until now I'm just using native Python language
thanks
1) Yes, using an OAuth 2 access token is the right way to authorize API calls to the Dropbox API.
Note that you don't have to process the authorization flow each time though. You can store and re-use the access token once you retrieve it once.
I would like to write a simple desktop application for personal use that uses the Spotify Web API to build playlists.
As far as I can tell, however, there's no way to use the API without providing a callback URI, which I don't have, seeing as I don't have a domain or server of any kind (other than my personal computer).
Is there a way to use the API without a URI?
If not, what is the best way to set up a callback URI? I don't have much of any experience working with web applications or client / server stuff, and the APIs I've used in the past haven't required any kind of callback.
Some background first, this answer became a bit longer than what I anticipated.
You need an access token. There are three ways to retrieve an access token; Authentication Code flow, Client Credentials flow, and Implicit Grant flow. These are all part of the oAuth 2.0 specification, each with its own purpose.
Since you're going to modify a user's account, you need that user's permission. Now, if you didn't do any actions that required user permissions, you could've used the Client Credentials flow. That's probably the easiest flow to learn since it just requires a request from your server to Spotify's server, and the response contains an access token. No callback/redirect URI is necessary.
As I'm sure you've read, the Authentication Code flow and Implicit Grant flow both require a callback URI. This is because a flow that includes a user, and the callback URI is where Spotify redirects the user after they have entered their password on Spotify's site.
The Authentication Code flow and Implicit Grant flow has benefits and drawbacks. Access tokens retrieved through the Authentication Code flow can be refreshed, but both return tokens that are valid for one hour. This means that a user that's authenticating using the Implicit Grant flow must reauthenticate after an hour. However, the Authentication Code flow does require some backend work, since it needs to make a request to exchange a code given from Spotify's server for an access token. The Implicit Grant flow is more straight forward - you get the access token and off you go.
The callback URI can be a localhost address, so if your desktop application would spin up a web server locally you could handle the callback on the same machine that the application is running on. (It's probably a good idea to not run the web server on port 80 since that might be used by something else.)
There's skeleton code for each of this authentication flows available on Github, see web-api-auth-examples. Read more about the flows in our Authorization Guide. If you choose to run a web server on the user's machine, I recommend that you use the Implicit Grant flow since that doesn't include any server-to-server requests, so you won't have to expose your client_secret exposed in the code. (As opposed to the Authorization Code flow.)
Also, since you're coding in Python, I recommend that you have a look at spotipy, a wrapper around the Web API packed with convenient methods that'll save you some time. If you do go ahead with the Implicit Grant flow, you should have a look at spotify-web-api-js, which has a similar purpose. Note that these wrappers are not at all required to work with the Web API but they'll make your life easier.
I'm working on converting a Python script using the Google gdata API client + user/pass authentication to something more suitable for production (an API key). I am pretty frustrated with the muddled state of their documentation on authentication. I admittedly don't have a great grasp of OAuth2, but it seems like it's way more complicated for my usage case, which is: Hit Google Analytics every 24 hours to get the X most popular articles on our site.
In this scenario, we're not dealing with modifying someone's personal data, and all activity is centered on one account. It doesn't seem like OAuth2 is worth the complexity for something so simple.
I see that on the Google API Console (https://code.google.com/apis/console/), I've registered there and notice that there's a "Simple API Access" section with one key beneath the "Client ID for web applications" (which appears to be OAuth2). There's also the Google domain update page, https://www.google.com/accounts/UpdateDomain, but that appears to be OAuth related.
Is there any way to use this Simple API Access key (not OAuth) for retrieving analytics data with the Python gdata client, and if so, does anyone have any authentication examples? I already have the data retrieval stuff working once authenticated, but I'm using the user/pass approach, which is not appropriate for production.
Greg,
If you are already using the library gdata-python-client, this is relatively easy to do if you are the only user that your application will be authorizing.
The general mechanisms were detailed in a blog post in September, 2011, but I'll describe them here for completeness.
Part 1: Go to the APIs console and start a new project.
Part 2: From the project, go to "Services" and enable "Analytics API"
Part 3: From the project, go to "API Access" and click "Create an OAuth 2.0 client ID..." (you'll need to provide a product name, though the value you provide won't matter). When asked for the application type, select "Installed Application" and then "Create client ID". Since you will be the only user, you will only need one refresh token, and you can get this by authorizing from a desktop application a single time.
Part 4: Get your client id and client secret from the APIs console and then create an empty token:
import gdata.gauth
CLIENT_ID = 'id-from-apis-console'
CLIENT_SECRET = 'secret-from-apis-console'
SCOPE = 'https://www.google.com/analytics/feeds/' # Default scope for analytics
token = gdata.gauth.OAuth2Token(
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
scope=SCOPE,
user_agent='application-name-goes-here')
I got the scope from GData FAQ, though I'm not sure if it is correct.
Part 5: Use the token to create authorization URL for you to visit:
url = token.generate_authorize_url(redirect_uri='urn:ietf:wg:oauth:2.0:oob')
Since your application is an "Installed Application", your redirect URI is the default 'urn:ietf:wg:oauth:2.0:oob'. (Also note, the blog post had a typo and used the keyword argument redirect_url.)
Part 6: Visit the url and authorize your application to make requests on behalf of your account. After authorizing, you'll be redirected to a page with a code on it. This code will be used to exchange for an access token and a long-lived refresh token. The code has a life of 10 minutes and the access token has a life of an hour. The refresh token will allow you to get new access tokens for signing requests in perpetuity (or until you revoke the permission from your account).
Part 7: Use the code to get an access token:
code = 'random-string-from-redirected-page'
token.get_access_token(code) # This returns the token, but also changes the state
This again differs slightly from the blog post, because we are using an installed application.
Part 8: With the token you can now make all requests you want to make to the analytics client:
import gdata.analytics.client
client = gdata.analytics.client.AnalyticsClient()
token.authorize(client)
This is the big money right here. When an access token expires, the API requests signed with that token are rejected. However, by authorizing the client as above, when the said requests fail, the token attempts to use the refresh token to obtain a new access token. If it successfully obtains a new access token, the client resends the original API request, signed with the new access token.
I don't know anything about the Analytics API so I won't provide any more details there.
Future Use Note 1: Saving information for future use. You can re-use this from different places and after this use very easily. There are methods called token_to_blob and token_from_blob provided by the library that allow turning a token into a string and converting out of a string:
saved_blob_string = gdata.gauth.token_to_blob(token)
Once you have done this, you can store the string in a file and kill your running Python process. When you'd like to use it again:
saved_blob_string = retrieve_string_from_file() # You'll need to implement this
token = gdata.gauth.token_from_blob(saved_blob_string)
Future Use Note 2: This token will be able to be used to authorize a client and perform all your magic again and again, so long as you have the refresh token around. If for some reason you would like to get an access token again without calling token.generate_authorize_url, you'll need to manually set this on the object:
token.redirect_uri = 'urn:ietf:wg:oauth:2.0:oob'
Future Use Note 3: Also, if you lose your refresh token and would like to get another one without having to go to the browser to revoke the original, you can use the approval_prompt parameter to get a new refresh token by visiting the url generated by:
url = token.generate_authorize_url(
redirect_uri='urn:ietf:wg:oauth:2.0:oob',
approval_prompt='force')