We Keep Coding

Accessing Office 365 ProPlus OneDrive folder using the official Python SDK

We are currently trying to access a folder of an Office 365 ProPlus tenant using the official OneDrive SDK for Python (https://github.com/OneDrive/onedrive-sdk-python). One of our clients would like to use a OneDrive folder as a way of storing and sharing programmatically generated files, therefore, we would like to provide basic file operations.
We have a working solution for a personal OneDrive account, however, when we try to apply the same approach for their OneDrive, we face an issue during the authentication process.
We asked them to register the application in the Azure AD following the steps in the official documentation. Next, they sent us the redirect URI, client ID and client secret that we included in our script. We are trying to use the following code:
redirect_uri = 'REDIRECT_URI'
client_secret = 'CLIENT_SECRET'
client_id='CLIENT_ID'
discovery_uri = 'https://api.office.com/discovery/'
auth_server_url='https://login.microsoftonline.com/common/oauth2/authorize'
auth_token_url='https://login.microsoftonline.com/common/oauth2/token'
http_provider = onedrivesdk.HttpProvider()
auth_provider = onedrivesdk.AuthProvider(http_provider,
client_id,
auth_server_url=auth_server_url,
auth_token_url=auth_token_url)
auth_url = auth_provider.get_auth_url(redirect_uri)
code = GetAuthCodeServer.get_auth_code(auth_url, redirect_uri)
However, we get the following error message when executing the last line:
Traceback (most recent call last):
File "onedrive-test.py", line 25, in
code = GetAuthCodeServer.get_auth_code(auth_url, redirect_uri)
File "/home/username/.local/lib/python3.6/site-packages/onedrivesdk/helpers/GetAuthCodeServer.py",
line 60, in get_auth_code
s = GetAuthCodeServer((host_address, port), code_acquired, GetAuthCodeRequestHandler)
File "/home/username/.local/lib/python3.6/site-packages/onedrivesdk/helpers/GetAuthCodeServer.py",
line 76, in init
HTTPServer.init(self, server_address, RequestHandlerClass)
File "/usr/lib/python3.6/socketserver.py", line 453, in init
self.server_bind()
File "/usr/lib/python3.6/http/server.py", line 136, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.6/socketserver.py", line 467, in server_bind
self.socket.bind(self.server_address)
socket.gaierror: [Errno -2] Name or service not known
We also tried opening the auth_url manually, which took us one step further, but still could not authenticate the application with the following error:
AADSTS50020: User account 'USER ACCOUNT' from identity provider
'live.com' does not exist in tenant 'TENANT NAME' and cannot access
the application 'CLIENT ID' in that tenant. The account needs to be
added as an external user in the tenant first. Sign out and sign in
again with a different Azure Active Directory user account.
We have two questions:
What might casue the first error? This is the comment (see below) that can be found in the readme of the SDK about using the GetAuthCodeServer class. It seems to us that the server cannot be run. Are there any not explicitly defined dependencies that we should be aware of before trying to run the webserver? (We are running the script on Ubuntu 18.10)
If you want to remove some of that manual work, you can
use the helper class GetAuthCodeServer. That helper class spins up a
webserver, so this method cannot be used on all environments.
With respect to the second issue, can you recommend proper material for configuring OneDrive for Business for our use-case? We went through a lot of documentation, but after long hours of research, we still could not find the correct way to fix that issue, especially since we do not have direct acces to the tenant and we cannot easily experiment with things. We would need to give a step-by-step cookbook to our client to set up everything on their side.
Any help would be much appreciated! :)

BadParametersError: Invalid signature when using OVH Python wrapper

I'm using OVH API along with python wrapper:
https://pypi.python.org/pypi/ovh
When trying to execute this code:
import ovh
client = ovh.Client()
# Print nice welcome message
print "Welcome", client.get('/me')['firstname']
I get this error:
Traceback (most recent call last):
File "index.py", line 6, in <module>
print "Welcome", client.get('/me')['firstname']
File "/home/rubinhozzz/.local/lib/python2.7/site-packages/ovh/client.py", line 290, in get
return self.call('GET', _target, None, _need_auth)
File "/home/rubinhozzz/.local/lib/python2.7/site-packages/ovh/client.py", line 419, in call
raise BadParametersError(json_result.get('message'))
ovh.exceptions.BadParametersError: Invalid signature
My info is saved in the ovh.conf as the documentation suggests.
[default]
; general configuration: default endpoint
endpoint=ovh-eu
[ovh-eu]
application_key=XXXlVy5SE7dY7Gc5
application_secret=XXXdTEBKHweS5F0P0tb0lfOa8GoQPy4l
consumer_key=pscg79fXXX8ESMIXXX7dR9ckpDR7Pful
It looks that I can connect but when trying to use the services like for instance "/me", the error raises!

It is difficult to reproduce the issue because it requires an application key and it seems that it is only granted to existing customers of OVH. I couldn't even see a link to an account registration page on their site.
By looking at the code of the call() method in /ovh/client.py, it seems that their server doesn't recognise the format or the content off the signature sent by your script. According to the inline documentation the signature is generated from these parameters:
application_secret
consumer_key
METHOD
full request url
body
server current time (takes time delta into account)
Since your call is identical to the example code provided on the OVH Python package web page, the last four parameters should be valid. In that case it looks like either the application secret or the customer key (or both) in your config file are not correct.
See also the documentation on OVH site under the 'Signing requests' heading. They explain how the signature is made and what it should look like.
Perhaps try to re-create a new application API to obtain new key and secret and make sure you copy them without any additional character.

authentification for Google Storage using Python

I want to build an app which has easy interactions with google storage, i.e., list files in bucket, download a file, and upload a file.
Following this tutorial, I decided to use a service account (not a user one) for authentification and followed the procedure. I created a public/private key on my console and download the key on my machine. Then I created the .boto file which points to this private key, and finally launched this program and it worked:
import boto
import gcs_oauth2_boto_plugin
uri = boto.storage_uri('txxxxxxxxxxxxxx9.appspot.com', 'gs')
for obj in uri.get_bucket():
print '%s://%s/%s' % (uri.scheme, uri.bucket_name, obj.name)
As you can see, the package gcs_oauth2_boto_plugin is not used in the code, so I decided to get rid of it. But magically, when I comment the import gcs_oauth2_boto_plugin line and run the program again, I get this error:
C:\Users\...\Anaconda3\envs\snakes\python.exe C:/Users/.../Dropbox/Prog/s3_manifest_builder/test.py
Traceback (most recent call last):
File "C:/Users/.../Dropbox/Prog/s3_manifest_builder/test.py", line 10, in <module>
for obj in uri.get_bucket():
File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\storage_uri.py", line 181, in get_bucket
conn = self.connect()
File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\storage_uri.py", line 140, in connect
**connection_args)
File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\gs\connection.py", line 47, in __init__
suppress_consec_slashes=suppress_consec_slashes)
File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\s3\connection.py", line 190, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\connection.py", line 569, in __init__
host, config, self.provider, self._required_auth_capability())
File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\auth.py", line 987, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
So my questions are:
1- how can you explain that deleting an import which IS NOT USED in the code makes it fail?
2- more generally, to be sure to understand the authentification process, if I want to run my app on a machine, I must be sure to have the .boto file (which points to my service account private key) generated previously? Or is there a cleaner/easier way to give access to my application to Google Storage for in/out interactions?
For instance, I only have to provide public and private key as strings to my program when I want to connect to a S3 bucket with boto. I don't needto generate a .boto file, importing packages etc..., which makes it so much easier to use, isn't it?

1- how can you explain that deleting an import which IS NOT USED in the code makes it fail?
The first hint is that the module is named a "plugin", although exactly how that's implemented isn't clear on the surface. It intuitively makes some sense that not importing a module would lead to an exception of this kind, though. Initially, I thought it was a bad practice of doing stateful activity on a global during the init of importing that module. In some ways, that is what it was, but only because class hierarchies are "state" in the meta-programmable python.
It turns out (as in many cases) that inspecting the location that stacktrace was thrown from (boto.auth.get_auth_handler()) provides the key to understanding the issue.
(see the linked source for commented version)
def get_auth_handler(host, config, provider, requested_capability=None):
ready_handlers = []
auth_handlers = boto.plugin.get_plugin(AuthHandler, requested_capability)
for handler in auth_handlers:
try:
ready_handlers.append(handler(host, config, provider))
except boto.auth_handler.NotReadyToAuthenticate:
pass
if not ready_handlers:
checked_handlers = auth_handlers
names = [handler.__name__ for handler in checked_handlers]
raise boto.exception.NoAuthHandlerFound(
'No handler was ready to authenticate. %d handlers were checked.'
' %s '
'Check your credentials' % (len(names), str(names)))
Note the reference to the class AuthHandler, which is defined in boto.auth_handler.
So, you can see that we need to look at the contents of boto.plugin.get_plugin(AuthHandler, requested_capability):
def get_plugin(cls, requested_capability=None):
if not requested_capability:
requested_capability = []
result = []
for handler in cls.__subclasses__():
if handler.is_capable(requested_capability):
result.append(handler)
return result
So, it becomes clear, at last finally when we see that the class definition of the class OAuth2Auth in gcs_oauth2_boto_plugin.oauth2_plugin, in which it is declared as a subclass of boto.auth_handler.AuthHandler, signaling its auth capabilities to the boto framework via the following member variable:
capability = ['google-oauth2', 's3']
2- more generally, to be sure to understand the authentification process, if I want to run my app on a machine, I must be sure to have the .boto file (which points to my service account private key) generated previously? Or is there a cleaner/easier way to give access to my application to Google Storage for in/out interactions?
This has a more generalized answer: You can use a .boto file, although you can also use service account credentials, and you could even use the REST API and go through an oauth2 flow to get the tokens needed to send in the Authorization header. The various methods of auth to cloud storage are in the documentation. The tutorial/doc you linked shows some methods, you've used .boto for another method. You can read about the Cloud Storage REST API (JSON) here and you can read about python oauth2 flows of various kinds here.

Google API quickstart.py error KeyError: '_module'

Using Gmail API. My client secret file is downloaded and working for Ruby. When I try the quickstart.py (python) version I get this error
File "quickstart.py", line 70, in <module>
main()
File "quickstart.py", line 55, in main
credentials = get_credentials()
File "quickstart.py", line 38, in get_credentials
credentials = store.get()
File "/Library/Python/2.7/site-packages/oauth2client/client.py", line 374, in get
return self.locked_get()
File "/Library/Python/2.7/site-packages/oauth2client/file.py", line 79, in locked_get
credentials = Credentials.new_from_json(content)
File "/Library/Python/2.7/site-packages/oauth2client/client.py", line 281, in new_from_json
module = data['_module']
KeyError: '_module'
I have not changed the file at all, just added the client_secret.json to that working directory and also install the google-api-python-client. My python code came from here: https://developers.google.com/gmail/api/quickstart/python

Try replacing creds = store.get() with creds = None temporarily. If this works, you can refactor your code to always start with flow-based credentials instantiation. This worked for me. It seems Google samples are out of sync with their oauth2client.

oauth2client is trying to load credentials from a json file with an incorrect structure.
Maybe the Ruby client uses a different file format but I'd be surprised. Are you sure you didn't save client_secret.json as ~/.credentials/gmail-quickstart.json accidentally?
Regardless, removing ~/.credentials/gmail-quickstart.json and re-authenticating will generate a new credentials file with the correct structure.

I am learning Python myself and had a similar problem, but with the Calendar API example. It turned out that it was a typo with regards to the SCOPE.
## Typo - Invalid definition
SCOPES = 'https://ww.googleapies.com/auth/calendar.readonly'
## Correct Value for SCOPE
SCOPES = 'https://www.googleapis.com/auth/calendar'
Also, Matt's answer help point me in the right direction. The gmail-quickstart.json is not the same thing as the client_secret.json. The client_secret.json allows you to make a request for an OAuth2 token. While the gmail-quickstart.json contains the issued token and meta-data associated with it. The gmail-quickstart.json isn't created until you successfully login.
One last thought, in order to log in successfully, the quickstart.py app launched an instance of my web-browser (Firefox) and went to the Google login screen. In order for Firefox to run properly, I had to set my DISPLAY variable properly first.
$ export DISPLAY=:0
$ xhost +
access control disabled, clients can connect from any host

In this GitHub issue: error "KeyError: '_module'" when running gdrive_upload.py
sputnik-dev answared on 10 Jan 2016:
If someone have the same issue : auth_token.txt and client_secret.json are not the same! Don't link the client_secret.json from google API console. The file will be automatically created by the script.
Wrong way: gauth.SaveCredentialsFile("client_secret.json")
Right way: gauth.SaveCredentialsFile("<any random name>.json")
Extra:
PyDrive code that automate google drive api authetication. Use the browser just one time to authenticate and never more. It saves your credential data on mycreds.json :)
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
gauth.LoadCredentialsFile("mycreds.json")
if gauth.credentials is None:
gauth.LocalWebserverAuth()
elif gauth.access_token_expired:
gauth.Refresh()
else:
gauth.Authorize()
gauth.SaveCredentialsFile("mycreds.json")

I solved this by moving the client_secret.json to the same directory as the py file that is trying to read it (quickstart.py), mine was on the Desktop while i had saved the json to Documents. I saved the json to the Desktop too and boy, It flew!.
I dont know why it doesnt work when they are in different directories, defining a custom credential_path doesn't help.

I followed several recommendations listed here:
moving the client_secret.json to the same directory as the py file that is trying to read it, as recommended by simic0de.
Working with skirill idea, instead of completely eliminating the 'credentials=store.get()', I decided to handle the exception, so it would work with flow-based credentials instantiation.
The final code is:
try:
credentials = store.get()
except:
print('Working with flow-based credentials instantiation')
You can write your own code in the except line. This approach will allow the store.get() command to work when conditions are met.

Finding gapps users groups using the python admin-sdk libraries via the Directory API?

I'm porting our old user management scripts from the Google Provisioning API (which used the python gdata libraries) to the Google Directory API (the python admin-sdk libaries). So far most things have gone fine, however I've run into issues when attempting to do a discovery on what groups a user belongs to (which I need to remove membership from before a user deletion). Even stripping the code down to the barest essentials (replaced e-mails/credentials for public consumption):
#!/usr/bin/python
import httplib2
from apiclient import errors
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
SERVICE_ACCOUNT_EMAIL = 'XXXXXXXXXXXXXXXXXXXXXX#developer.gserviceaccount.com'
SERVICE_ACCOUNT_PKCS12_FILE_PATH = '/blah/blah/XXXXXXXX-privatekey.p12'
f = file(SERVICE_ACCOUNT_PKCS12_FILE_PATH, 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(SERVICE_ACCOUNT_EMAIL, key,
scope='https://www.googleapis.com/auth/admin.directory.user', sub='serviceaccount#our.tld')
service.users()
members = service.members().get(memberKey = 'serviceaccount#our.old', groupKey = 'googlegroup#our.tld').execute()
print members
This returns a 403 permissions error:
Traceback (most recent call last):
File "group_tests.py", line 39, in <module>
members = service.members().get(memberKey = 'serviceaccount#our.tld', groupKey = 'googlegroup#our.tld').execute()
File "/XXX/bin/gapps/lib/python2.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
return wrapped(*args, **kwargs)
File "/XXX/bin/gapps/lib/python2.6/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/groups/googlegroup%40our.tld/members/serviceaccount%40our.tld?alt=json returned "Insufficient Permission">
I don't recognize if the scope is wrong here, and if so what it should be? This service account is already set with permission for the following scopes (in Security>Advanced Security>API>Manage API client access):
https://www.googleapis.com/auth/admin.directory.user
https://www.googleapis.com/auth/admin.directory.group
Or should I be using groups instead of members? Like:
members = service.groups().get(memberKey = 'serviceaccount#our.old', groupKey = 'googlegroup#our.tld').execute()
Any pointers appreciated, I've been goggling around for any help on this for a week now to no avail.

Got this working:
First off, the scope in the credentials definition was incorrect.
admin.directory.user
changed to:
admin.directory.group
Also had the wrong initialization for the "build":
service.users()
changed to:
service.groups()
And the query statement itself was completely wrong, I went back to the reference doc and kept trying different changes until it took:
members = service.groups().list(domain = 'our.tld',userKey = 'user_whos_groups_i_want#our.tld',pageToken=None,maxResults=500).execute()
Hopefully this will be useful to someone else running into the same issue later. Please be aware that not all permission errors google will throw back are literally because of permissions, it may be your own code has conflicting scopes that you're trying to use.