In the python google api documentation regarding mocks it is stated:
As you develop and test your application, it is a good idea to save actual API responses in files like books-discovery.json or books-android.json for use in testing.
Where do I get these json from? In particular the response to mock the build() command.
EDIT:
The class I want to test, calendar.py:
from google.oauth2 import service_account
from googleapiclient.discovery import build
class Calendar:
def __init__(self, credentials_file, calendar_id) -> None:
credentials = service_account.Credentials.from_service_account_file(
credentials_file, scopes=["https://www.googleapis.com/auth/calendar"]
)
self.service = build("calendar", "v3", credentials=credentials)
self._id = calendar_id
How to retrive the json response of build to save it in calendar-discovery.json?
Modified class to simplify mocking
import google_auth_httplib2
from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.http import build_http
class Calendar:
def __init__(self, credentials_file, calendar_id) -> None:
credentials = service_account.Credentials.from_service_account_file(
credentials_file, scopes=["https://www.googleapis.com/auth/calendar"]
)
http = google_auth_httplib2.AuthorizedHttp(credentials, http=build_http())
self.service = build("calendar", "v3", http=http)
self._id = calendar_id
The test:
def test_calendar_initialization(mocker):
mock = mocker.patch("meal_planner.calendar.build_http")
mock.return_value = HttpMock("calendar-discovery.json", {"status": 200})
calendar = Calendar(credentials_file="credential.json", calendar_id="id")
List all the discovery APIs: https://discovery.googleapis.com/discovery/v1/apis
Search for "name": "calendar"
Look for "discoveryRestUrl": "https://calendar-json.googleapis.com/$discovery/rest?version=v3"
Get the calendar discovery document
curl https://calendar-json.googleapis.com/$discovery/rest?version=v3 > calendar-discovery.json`
Reference:
https://developers.google.com/discovery
Related
I have this piece of code to extract some metrics about my YouTube channel and create a pandas dataframe from them.
import os
import google.oauth2.credentials
import google_auth_oauthlib.flow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
import json
SCOPES = ['https://www.googleapis.com/auth/yt-analytics.readonly']
API_SERVICE_NAME = 'youtubeAnalytics'
API_VERSION = 'v2'
CLIENT_SECRETS_FILE = 'client_secrets.json'
def get_service():
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
def execute_api_request(client_library_function, **kwargs):
response = client_library_function(
**kwargs
).execute()
with open('data.json', 'w') as fp:
json.dump(response, fp)
if __name__ == '__main__':
# Disable OAuthlib's HTTPs verification when running locally.
# *DO NOT* leave this option enabled when running in production.
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
youtubeAnalytics = get_service()
execute_api_request(
youtubeAnalytics.reports().query,
ids='channel==MINE',
startDate='2014-01-01',
endDate='2019-02-26',
metrics='averageViewDuration,views,likes,dislikes,subscribersGained,subscribersLost',
dimensions='day',
sort='day',
filters = 'country==US'
)
## Now, convert the json to dataframe
import json
import pandas as pd
with open('data.json') as json_data:
d = json.load(json_data)
colnames = [d['columnHeaders'][i]['name'] for i in range(0,len(d['columnHeaders']))]
Results = pd.DataFrame(d['rows'],columns = colnames)
Results.to_csv("Youtube_data.csv")
By running this code, a windows opens and asks me to login into youtube and then provide me the authorization code. Entering this authorization code finishes the running of above python program. However, you should repeat this authorization process each time you are running this program.
Is there anyway to bypass this repeated authorization such that this process can be automated?
You need to use the oauth2client.file.Storage class to store and retrieves the credentials object as (badly) explained here: https://developers.google.com/api-client-library/python/guide/aaa_oauth
You will need to modify your get_service function with something like this:
from oauth2client import client, file
def get_service():
flow = client.flow_from_clientsecrets(CLIENT_SECRETS_FILE, SCOPES)
storage = file.Storage(API_SERVICE_NAME + '.dat')
credentials = storage.get()
http = credentials.authorize(http=httplib2.Http())
service = build(API_SERVICE_NAME, API_VERSION, http=http)
return service
Hope this helps
What I'm traing to do is to get all the members of a single group, and then download all the emails headers they have in gmail. To do this I started with google gmail api and members api, but i get stuck. I realice that I can't do it even if I have admin role.
What i found is that the right way to do it (I think...) is to use a service account for impersonate user. What i done:
Create a new project.
Create service account credentials.
Add Domain-wide Delegation to the service account.
Enable scopes for this proyect.
The code to create the service:
from __future__ import print_function
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
from google.oauth2 import service_account
import requests
import json
class Service:
SCOPES = ['https://mail.google.com/',
'https://sites.google.com/feeds',
'https://www.google.com/m8/feeds',
'https://www.googleapis.com/auth/activity',
'https://www.googleapis.com/auth/calendar',
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/gmail.settings.basic',
'https://www.googleapis.com/auth/gmail.settings.sharing',
'https://www.googleapis.com/auth/plus.login',
'https://www.googleapis.com/auth/plus.me',
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/admin.directory.group',
'https://www.googleapis.com/auth/gmail.metadata'
]
secret_file = 'client-secret.json'
service = ''
service_name = ''
service_version = ''
delegated_credentials = ''
credentials = ''
def __init__(self, servicename, serviceversion):
self.service_name = servicename
self.service_verion = serviceversion
self.newService()
def newService(self):
self.credentials = service_account.Credentials.from_service_account_file(self.secret_file, scopes=self.SCOPES)
self.credentials = self.credentials.with_subject('admin#domain.com')
self.service = build(self.service_name, self.service_verion, credentials=self.credentials)
return self.service
if __name__ == '__main__':
main()
Members class:
from __future__ import print_function
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
from service import Service
import requests
import json
class Members:
service = ''
group_name = ''
members = []
def __init__(self, groupname):
self.group_name = groupname
self.service = Service('admin', 'directory_v1')
def get_members(self):
print('Buscando los miembros del grupo')
results = self.service.service.members().list(groupKey=self.group_name).execute()
for member in results['members']:
print('Descargando datos de: ', member['email'])
self.members.append(member['id'])
return self.members
if __name__ == '__main__':
main()
app.py:
from __future__ import print_function
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
from members import Members
from user import User
from service import Service
import requests
def main():
member = Members("group#ldomain.com")
members_list = member.get_members()
for member in members_list:
print(member)
user = User(member)
messages = user.get_user_inbox()
user.get_message_info(messages)
if __name__ == '__main__':
main()
The error I have:
"Client is unauthorized to retrieve access tokens using this method,
or client not authorized for any of the scopes requested."
Following is the code to access a google storage bucket using the httplib2 library
import json
from httplib2 import Http
from oauth2client.client import SignedJwtAssertionCredentials
from googleapiclient.discovery import build
from pprint import pprint
client_email = 'my.iam.gserviceaccount.com'
json_file = 'services.json'
cloud_storage_bucket = 'my_bucket'
files = 'reviews/reviews_myapp_201603.csv'
private_key = json.loads(open(json_file).read())['private_key']
credentials = SignedJwtAssertionCredentials(client_email,
private_key,'https://www.googleapis.com/auth/devstorage.read_only')
storage = build('storage', 'v1', http=credentials.authorize(Http()))
pprint(storage.objects().get(bucket=cloud_storage_bucket, object=files).execute())
Can someone tell me if I can make the http request using the Python Requests library here?
If yes, how?
Yes, you can use the HTTP header Authorization: Bearer <access_token> with requests or any library you want.
Service account
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file(
'services.json',
scopes=['https://www.googleapis.com/auth/devstorage.read_only'],
)
# Copy access token
bearer_token = credentials.token
User account credentials
import json
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
flow = InstalledAppFlow.from_client_secrets_file(
'test.json',
'https://www.googleapis.com/auth/devstorage.read_only'
)
# Construct cache path for oauth2 token
oauth2_cache_path = 'test-oauth2.json'
credentials = None
try:
# Try to load existing oauth2 token
with open(oauth2_cache_path, 'r') as f:
credentials = Credentials(**json.load(f))
except (OSError, IOError) as e:
pass
if not credentials or not credentials.valid:
credentials = flow.run_console()
with open(oauth2_cache_path, 'w+') as f:
f.write(json.dumps({
'token': credentials.token,
'refresh_token': credentials.refresh_token,
'token_uri': credentials.token_uri,
'client_id': credentials.client_id,
'client_secret': credentials.client_secret,
'scopes': credentials.scopes,
}))
# Copy access token
bearer_token = credentials.token
Use requests lib
import requests
# Send request
response = requests.get(
'https://www.googleapis.com/storage/v1/<endpoint>?access_token=%s'
% bearer_token)
# OR
response = requests.get(
'https://www.googleapis.com/storage/v1/<endpoint>',
headers={'Authorization': 'Bearer %s' % bearer_token})
Use googleapiclient lib
I recommend you to use build() method and not requests directly because the google library do some checks before sending your API call (like checking params, endpoint, auth and the method you use). This library also raise exceptions when error is detected.
from googleapiclient.discovery import build
storage = build('storage', 'v1', credentials=credentials)
print(storage.objects().get(bucket='bucket', object='file_path').execute())
More informations here : https://developers.google.com/identity/protocols/OAuth2WebServer#callinganapi (click on "HTTP/REST" tab)
I suggest to use the official Google Auth library which is already implementing Requests Library. See this link for more information.
Here is a code to try (given that you have a service account file with required permissions):
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
service_account_file = 'service_account.json'
scopes = ['https://www.googleapis.com/auth/devstorage.full_control']
credentials = service_account.Credentials.from_service_account_file(
service_account_file, scopes=scopes)
session = AuthorizedSession(credentials)
bucket_name = 'YOUR-BUCKET-NAME'
response = session.get(f'https://storage.googleapis.com/storage/v1/b/{bucket_name}')
print(response.json())
We are trying to start with Google Analytics Management API by running an example script in Python from the documentation:
https://developers.google.com/analytics/devguides/config/mgmt/v3/user-management
Unfortunately, it is not working and we receive a bug like:
It was sugested to add authorization to that script, so we use the code from:
https://developers.google.com/analytics/devguides/reporting/core/v3/quickstart/service-py
like:
scope = ['https://www.googleapis.com/auth/analytics.manage.users']
key_file_location = '...'
service_account_email= '...'
# Authenticate and construct service.
service = get_service('analytics', 'v3', scope, key_file_location,
service_account_email)
But still the problem remains unsolved.
What is wrong?
Is it the authorization process? But the HelloAnalytisc.py script is working and we can authorize and get data.
Our whole script is here:
"""A simple example of Google Analytics batched user permissions."""
import argparse
from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
import httplib2
from oauth2client import client
from oauth2client import file
from oauth2client import tools
import numpy as np
import pandas as pd
import json
from googleapiclient.errors import HttpError
from googleapiclient.http import BatchHttpRequest
def call_back(request_id, response, exception):
"""Handle batched request responses."""
print (request_id)
if exception is not None:
if isinstance(exception, HttpError):
message = json.loads(exception.content)['error']['message']
print ('Request %s returned API error : %s : %s ' %
(request_id, exception.resp.status, message))
else:
print (response)
def add_users(users, permissions):
"""Adds users to every view (profile) with the given permissions.
Args:
users: A list of user email addresses.
permissions: A list of user permissions.
Note: this code assumes you have MANAGE_USERS level permissions
to each profile and an authorized Google Analytics service object.
"""
# Get the a full set of account summaries.
account_summaries = analytics.management().accountSummaries().list().execute()
# Loop through each account.
for account in account_summaries.get('items', []):
account_id = account.get('id')
# Loop through each user.
for user in users:
# Create the BatchHttpRequest object.
batch = BatchHttpRequest(callback=call_back)
# Loop through each property.
for property_summary in account.get('webProperties', []):
property_id = property_summary.get('id')
# Loop through each view (profile).
for view in property_summary.get('profiles', []):
view_id = view.get('id')
# Construct the Profile User Link.
link = analytics.management().profileUserLinks().insert(
accountId=account_id,
webPropertyId=property_id,
profileId=view_id,
body={
'permissions': {
'local': permissions
},
'userRef': {
'email': user
}
}
)
batch.add(link)
# Execute the batch request for each user.
batch.execute()
if __name__ == '__main__':
# Define the auth scopes to request.
scope = ['https://www.googleapis.com/auth/analytics.manage.users']
key_file_location = '...'
service_account_email='...'
# Construct a list of users.
emails = ['ona#gmail.com', 'emi#gmail.com', 'sue#gmail.com', 'liz#gmail.com']
# call the add_users function with the list of desired permissions.
add_users(emails, ['READ_AND_ANALYZE'])
I am trying to update a Google spreadsheet from a server that some target users can see on a daily basis. Here is what I tried:
Created a project in "console.developers.google.com" then selected "drive API" -> "credentials" -> "add credentials" -> "service accounts" -> "create Json file"
Now with this JSON file (project name-e4sdfsdsdf0c.json) I tried to access Spreadsheets.
import gdata.spreadsheet.service
import gdata.service
import urlparse
import httplib2
from oauth2client.file import Storage
from oauth2client.client import flow_from_clientsecrets
from oauth2client import tools
spreadsheet_key = '13jQtgSUXKBExMvZjECf6sdfsfgLfmRFVmZw6t7hYyX3g0'
storage = Storage("creds.dat")
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = tools.run_flow(flow_from_clientsecrets("project name-e4sdfsdsdf0c.json", scope=["https://spreadsheets.google.com/feeds"]), storage)
if credentials.access_token_expired:
credentials.refresh(httplib2.Http())
spr_client = gdata.spreadsheet.service.SpreadsheetsService(
additional_headers={'Authorization' : 'Bearer %s' % credentials.access_token})
worksheets = spr_client.GetSpreadsheetsFeed(spreadsheet_key)
print worksheets.title
But I am getting this error:
Invalid file format. See https://developers.google.com/api-client-library/python/guide/aaa_client_secrets Expected a JSON object with a single property for a "web" or "installed" application"
You created a service account but it looks like you're trying to access it using a client flow.
Have a look at the service account documentation here:
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
The first step would be to go back to the developer console and generate a p12 key. Then the basic flow for Python looks like this:
from oauth2client.client import SignedJwtAssertionCredentials
scope = 'https://www.googleapis.com/auth/drive.readonly https://spreadsheets.google.com/feeds'
client_email = '<your service account email address>'
with open("MyProject.p12") as f:
private_key = f.read()
credentials = SignedJwtAssertionCredentials(client_email, private_key, scope)
http_auth = credentials.authorize(Http())
Google Sheets API v4 appeared pretty straightforward here. After generating json file you may access spreadsheet with this code.
from oauth2client.service_account import ServiceAccountCredentials
from httplib2 import Http
from apiclient import discovery
scopes = ['https://www.googleapis.com/auth/spreadsheets']
credentials = ServiceAccountCredentials.from_json_keyfile_name(<path_to_your_client_secret.json>, scopes)
http_auth = credentials.authorize(Http())
discoveryUrl = ('https://sheets.googleapis.com/$discovery/rest?version=v4')
service = discovery.build('sheets', 'v4', http=http_auth, discoveryServiceUrl=discoveryUrl)
result = service.spreadsheets().values().update(...).execute()