We Keep Coding

Office365-REST-Python-Client Access Token issue

I've found many examples of using the Office365-REST-Python-Client however, none of them are correctly obtaining the access token. I've registered an app under the Azure Portal, granted it API permissions using 'Application permissions', created a secret and used the client_secret and client_id in my settings dictionary to use in the below code.
def read_folder_and_files(context, list_title):
"""Read a folder example"""
list_obj = context.web.lists.get_by_title(list_title)
folder = list_obj.root_folder
context.load(folder)
context.execute_query()
print("List url: {0}".format(folder.properties["ServerRelativeUrl"]))
files = folder.files
context.load(files)
context.execute_query()
for cur_file in files:
print("File name: {0}".format(cur_file.properties["Name"]))
folders = context.web.folders
context.load(folders)
context.execute_query()
for folder in folders:
print("Folder name: {0}".format(folder.properties["Name"]))
if __name__ == '__main__':
ctx_auth = AuthenticationContext(url=settings['url'])
if ctx_auth.acquire_token_for_app(client_id=settings['client_credentials']['client_id'],
client_secret=settings['client_credentials']['client_secret']):
ctx = ClientContext(settings['url'], ctx_auth)
read_folder_and_files(ctx, "Documents")
# read_folder_and_files_alt(ctx, "Documents")
# upload_file_into_library(target_library, name, content)
# download_file(ctx)
else:
print(ctx_auth.get_last_error())
When I run the above code I get the following error:
File "/usr/local/lib/python3.7/site-packages/office365/runtime/auth/acs_token_provider.py", line 76, in get_authorization_header
return 'Bearer {0}'.format(self.access_token["access_token"])
KeyError: 'access_token'
My end goal is to upload files to a Sharepoint Document Libary with metadata from a python data pipeline. Sharepoint is not hosted locally and is included in our 365 licences.
Kind Regards

So it looks like this error can happen when you're not getting an access token.
I fixed this by ditching the client and secret in my Azure Portal and instead generated them in the SharePoint site under the following URL:
URL: https://[tenant].sharepoint.com/_layouts/15/appregnew.aspx
To find out what you should use in the space of [tenant] look at your SharePoint URL and pick out the text between 'https://' and '.sharepoint.com'. This is assuming your SharePoint is hosted by Microsoft.
Click the generate buttons, use a relevant Title and unless you know better just enter localhost for the App Domain and Redirect URL. (My project is just a simple upload script). Take a copy of the Client ID and Secret.
If you want your App to have full access then navigate to:
https://[tenant]-admin.sharepoint.com/_layouts/15/appinv.aspx
There is another link 'https://[tenant].sharepoint.com/_layouts/15/appinv.aspx' but this won't let you apply for full control permissions.
Paste in the client id into the App id, where would be the fun in using the same field name, or linking the form together? Click lookup and use the below XML to grant full control.
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
</AppPermissionRequests>
Click create and on the next page click trust. You will need to be logged in as a site owner with full admin permissions to grant this control.
I found all this info here which goes into more detail if you need it:
https://www.anupams.net/app-only-policy-with-tenant-level-permissions-in-sharepoint-online/
Bonus Info:
Our Sharepoint has two 'sites' so passing the base URL of 'https://[tenant].sharepoint.com' took me to the wrong site by default and meant that the document libraries I was looking didn't exist. To fix this using Office365-REST-Python-Client in your settings dictionary make sure the URL setting has the fill path to your site like this:
https://[tenant]-admin.sharepoint.com/sites/[site]
Hope this helps, this info cost me far to much time!

Uploading file with python returns Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>

blob.upload_from_filename(source) gives the error
raise exceptions.from_http_status(response.status_code, message, >response=response)
google.api_core.exceptions.Forbidden: 403 POST >https://www.googleapis.com/upload/storage/v1/b/bucket1-newsdata->bluetechsoft/o?uploadType=multipart: ('Request failed with status >code', 403, 'Expected one of', )
I am following the example of google cloud written in python here!
from google.cloud import storage
def upload_blob(bucket, source, des):
client = storage.Client.from_service_account_json('/path')
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket)
blob = bucket.blob(des)
blob.upload_from_filename(source)
I used gsutil to upload files, which is working fine.
Tried to list the bucket names using the python script which is also working fine.
I have necessary permissions and GOOGLE_APPLICATION_CREDENTIALS set.

This whole things wasn't working because I didn't have permission storage admin in the service account that I am using in GCP.
Allowing storage admin to my service account solved my problem.

As other answers have indicated that this is related to the issue of permission, I have found one following command as useful way to create default application credential for currently logged in user.
Assuming, you got this error, while running this code in some machine. Just following steps would be sufficient:
SSH to vm where code is running or will be running. Make sure you are user, who has permission to upload things in google storage.
Run following command:
gcloud auth application-default login
This above command will ask to create token by clicking on url. Generate token and paste in ssh console.
That's it. All your python application started as that user, will use this as default credential for storage buckets interaction.
Happy GCP'ing :)

This question is more appropriate for a support case.
As you are getting a 403, most likely you are missing a permission on IAM, the Google Cloud Platform support team will be able to inspect your resources and configurations.

This is what worked for me when the google documentation didn't work. I was getting the same error with the appropriate permissions.
import pathlib
import google.cloud.storage as gcs
client = gcs.Client()
#set target file to write to
target = pathlib.Path("local_file.txt")
#set file to download
FULL_FILE_PATH = "gs://bucket_name/folder_name/file_name.txt"
#open filestream with write permissions
with target.open(mode="wb") as downloaded_file:
#download and write file locally
client.download_blob_to_file(FULL_FILE_PATH, downloaded_file)

Google Drive API 403 error when updating spreadsheet title

We are using Google Drive API in our Google App Engine application.
This weekend we noticed that it has problems with updating spreadsheet title. We are getting the following error:
HttpError: <HttpError 403 when requesting https://www.googleapis.com/drive/v2/files/1_X51WMK0U12rfPKc2x60E_EuyqtQ8koW-NSRZq7Eqdw?quotaUser=5660071165952000&fields=title&alt=json returned "The authenticated user has not granted the app 593604285024 write access to the file 1_X51WMK0U12rfPKc2x60E_EuyqtQ8koW-NSRZq7Eqdw">
Other calls to Google Drive API succeed. We just have the problem with this one. Also this functionality worked properly for a long time. Is it possible that some update on Google side has broken this?
The minimal code to reproduce the issue is:
class TestDriveUpdate(webapp2.RequestHandler):
def get(self):
credentials = StorageByKeyName(Credentials,
'103005000283606027776',
'credentials').get()
spreadsheet_key = '1_X51WMK0U12rfPKc2x60E_EuyqtQ8koW-NSRZq7Eqdw'
quota_user = '5660071165952000'
body = {"title": 'Test'}
fields = "title"
http = httplib2.Http(timeout=60)
credentials.authorize(http)
gdrive = apiclient.discovery.build('drive', 'v2', http=http)
response = gdrive.files().update(
fileId=spreadsheet_key,
body=body,
fields=fields,
quotaUser=quota_user
).execute()
self.response.write("OK")

Based from this documentation, error occurs when the requesting app is not on the ACL for the file and the user never explicitly opened the file with this Drive app. Found this SO question which states that the scope strings must match exactly between your code and the Admin Console, including trailing slashes, etc. Make sure also that Drive Apps are allowed on the domain ("Allow users to install Google Drive apps").

Pyrax : This server could not verify that you are authorized to access the document you requested

I am using Rackspace cloud storage for my media uploads and implemented with django-cumulus. The problem is that, file uploads are working well but when I tried to access my uploaded file with the container URI my browser shows an error as "This server could not verify that you are authorized to access the document you requested.". Whether I need to change any setting for the container from my Rackspace account ?

Google AppEngine to Fusion Tables with Service Accounts

Late to the game on migrating to the /v1 Fusion Table API but no holding off any longer.
I'm using Python on AppEngine and trying to connect to Google Fusion Tables with Google Service Accounts (the more complicated cousin of OAuth2 for server side apps that uses JSON Web Tokens)
I found another question that pointed me to some documentation for using Service Accounts with Google Prediction API.
Fusion Table and Google Service Accounts
So far I've got
import httplib2
from oauth2client.appengine import AppAssertionCredentials
from apiclient.discovery import build
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/fusiontables')
http = credentials.authorize(httplib2.Http(memcache)) #Http(memcache)
service = build("fusiontables", "v1", http=http)
# list the tables
tables = service.table().list().execute() # <-- ERROR 401 invalid credentials here
Does anyone have an example of connecting to Fusion Tables on AppEngine using Service Accounts they might be able to share? Or something nice online?
Thanks

This actually does work. The important parts are you have to give the app engine service account access to your fusion table. If you are writing then the account needs write access. For help see: https://developers.google.com/api-client-library/python/start/installation (look for Getting started: Quickstart)
Your app engine service account will be something like your-app-id#appspot.gserviceaccount.com
You must also make the app engine service account a team member in the api console and give it "can edit" privilege.
SCOPE='https://www.googleapis.com/auth/fusiontables'
PROJECT_NUMBER = 'XXXXXXXX' # REPLACE WITH YOUR Project ID
# Create a new API service for interacting with Fusion Tables
credentials = AppAssertionCredentials(scope=SCOPE)
http = credentials.authorize(httplib2.Http())
logging.info('QQQ: accountname: %s' % app_identity.get_service_account_name())
service = build('fusiontables', 'v1', http=http, developerKey='YOUR KEY HERE FROM API CONSOLE')
def log(value1,value2=None):
tableid='YOUR TABLE ID FROM FUSION TABLES'
now = strftime("%Y-%m-%d %H:%M:%S", gmtime())
service.query().sql(sql="INSERT INTO %s (Temperature,Date) values(%s,'%s')" % (tableid,value1,now)).execute()

to clarify Ralph Yozzo's answer: you need to add the value of 'client_email' from the json file you downloaded when you created your service_account credentials (the same file you load when using ServiceAccountCredentials.from_json_keyfile_name('service_acct.json') with the new oauth2client library), to your table's sharing dialog screen (click 1 then enter the email address in 2)

Since Fusion Tables' tables are owned by individual Gmail accounts rather than the service account associated with an API console project, the AppAssertionCredentials probably won't work. It would make for an interesting feature request, though:
http://code.google.com/p/fusion-tables/issues/list

The best online resource I have found for help connecting Python AppEngine to Fusion Tables API with Oauth2 is
Google APIs Client Library for Python
The slide presentation is helpful to understanding the online samples, why decorators are used.
Also useful for understanding whether to use the app's Service Acount or User Accounts to authenticate is:
Using OAuth 2.0 to Access Google APIs
Consider installing the Google APIs Client Library for Python
Apart from the scope, the Oauth2 is more or less common to all Google APIs not just fusion tables.
Once oauth2 is working, see the Google Fusion Tables API

In case you want it to work from another host than Google App Engine or Google Compute Engine (e.g. from localhost for testing) then you should use ServiceAccountCredentials created from a json key file that you can generate and download from your service account page.
scopes = ['https://www.googleapis.com/auth/fusiontables']
keyfile = 'PATH TO YOUR SERVICE ACCOUNT KEY FILE'
FTID = 'FUSION TABLE ID'
credentials = ServiceAccountCredentials.from_json_keyfile_name(keyfile, scopes)
http_auth = credentials.authorize(Http(memcache))
service = build('fusiontables', 'v2', http=http_auth)
def insert(title, description):
sqlInsert = "INSERT INTO {0} (Title,Description) values('{1}','{2}')".format(FTID, title, description)
service.query().sql(sql=sqlInsert).execute()
Refer to Google's page on service accounts for explanations.