We Keep Coding

Youtube API's subscriptions.list only shows very few channels

So I've been playing around with Youtube API V3 lately.
I tried to retreive all the subscriptions I have in my account and for that I used the code sample they had on their site, which is the following:
# -*- coding: utf-8 -*-
# Sample Python code for youtube.subscriptions.list
# See instructions for running these code samples locally:
# https://developers.google.com/explorer-help/code-samples#python
import os
import google_auth_oauthlib.flow
import googleapiclient.discovery
import googleapiclient.errors
scopes = ["https://www.googleapis.com/auth/youtube.readonly"]
def main():
# Disable OAuthlib's HTTPS verification when running locally.
# *DO NOT* leave this option enabled in production.
os.environ["OAUTHLIB_INSECURE_TRANSPORT"] = "1"
api_service_name = "youtube"
api_version = "v3"
client_secrets_file = "YOUR_CLIENT_SECRET_FILE.json"
# Get credentials and create an API client
flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(
client_secrets_file, scopes)
credentials = flow.run_console()
youtube = googleapiclient.discovery.build(
api_service_name, api_version, credentials=credentials)
request = youtube.subscriptions().list(
part="snippet,contentDetails",
mine=True
)
response = request.execute()
print(response)
if __name__ == "__main__":
main()
Although this works fine, It only retreives information of 5-6 channels, while I am subscribed to over 200 channells, I made sure I didn't exceed my quota either. Any ideas why?

As commenters said, you need to implement pagination. To receive more results per request, you can also change maxResults to 50 instead of the default 5 value (that's why you are talking about receiving only 5-6 results).

How to create a blob in an Azure Storage Container using Python & Azure Functions

I am having a lot of difficulty writing an API response as json to a blob within an Azure Storage Container. I have tried multiple solutions online but have not managed to see any through to success. I would like to share 2 attempts I have made and hopefully there is someone out there that can assist me in getting at least one methodology correct
Attempt/Method 1
I have tried to use a Service Principle to authenticate my BlobServiceClient from Azure-Storage-Blob. My service principal has been assigned the role of Storage Blob Data Contributor for the Container within which I am trying to create the blob. However on execution of the script I receive an error along the lines of "Unsupported Credential". Below is my script and the error:
My script and resulting error are:
import azure.functions as func
import requests
import json
import uuid
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
from msrestazure.azure_active_directory import ServicePrincipalCredentials
from azure.storage.common import TokenCredential
# Initialise parameters to obtain data from Rest API
url = "https://api.powerbi.com/v1.0/myorg/admin/groups?$top=1000&$expand=datasets,dataflows,reports,users,dashboards"
headers = {'Authorization': get_access_token()}
# Get response. I want to save the response output to a blob.
response = requests.get(url, headers=headers)
response = response.json()
# Initialise parameters for credentials
CLIENT = "bxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx7" # Azure App/Service Principal ID
KEY = "Gxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1" # Azure App/Service Principal Key
TENANT_ID = "cxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx7" # Tenant where Storage Account is which is different to the Tenant the App resides
RESOURCE = f"https://storageaccountxxxxxxxxx.blob.core.windows.net"
# Create credentials & token
credentials = ServicePrincipalCredentials(
client_id = CLIENT,
secret = KEY,
#tenant = TENANT_ID,
resource = RESOURCE
)
tokenCre = TokenCredential(credentials.token["access_token"])
# Initialise parameters for BlobServiceClient
ACCOUNT_URL = "https://storageaccountxxxxxxxxx.blob.core.windows.net/pbiactivity" # includes container name at end of url
#Create BlobServiceClient
blobService = BlobServiceClient(account_url = ACCOUNT_URL, token_credential=tokenCre)
#Create blobClient
blobClient = BlobClient(account_url = RESOURCE,container_name=CONTAINER_NAME, blob_name="response.json", credential = tokenCre )
#Upload response json as blob
blobClient.upload_blob(response, blob_type = "BlockBlob")
Click here for the error that comes after the upload_blob method call]1
Attempt/Method 2
In my second attempt I tried to create ,my BlobServiceClient using Azure-Storage-Blob using my storage account connection string. This method actually allows me to create containers, however when I try to upload a blob as in the script below, However I am unable to create blobs within a container as I get a 403 Forbidden response.
My script and resulting error are:
import requests
import json
import uuid
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
# Initialise parameters to obtain data from Rest API
url = "https://api.powerbi.com/v1.0/myorg/admin/groups?$top=1000&$expand=datasets,dataflows,reports,users,dashboards"
headers = {'Authorization': get_access_token()}
# Get response. I want to save the response output to a blob.
response = requests.get(url, headers=headers)
response = response.json()
# Initialise parameters
CONNECTION_STRING = "DefaultEndpointsProtocol=https;AccountName=storageaccountxxxxxxxxx;AccountKey=rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxQ==;EndpointSuffix=core.windows.net"
# Create blobServiceClient from connection string
blobServiceClient = BlobServiceClient.from_connection_string(conn_str=CONNECTION_STRING)
#Create blobClient
blobClient = blobServiceClient.get_blob_client(container = "pbiactivity", blob = "response.json")
#Upload response json to blob
blobClient.upload_blob(response, blob_type = "BlockBlob")
Click Here for the errors that come after the upload_blob method call]2

Here is one of the workaround that worked for me:-
import os
import logging
from azure.storage.blob import BlobServiceClient, BlobClient
#Initialise parameters
url = "<YourURL>"
headers = {'Authorization': get_access_token()}
#Get response
response = requests.get(url, headers=headers)
response = response.json()
connectionString= "<Your_Connection_String>"
containerName = "<Name_of_your_container>"
blobServiceClient = BlobServiceClient.from_connection_string(connectionString)
blobContainerClient = blobServiceClient.get_container_client(containerName)
#To create Container (If the container has already been created you can ignore this)
#blobContainerClient.create_container()
#Create blobClient
blobClient = blobServiceClient.get_blob_client(container = "<Name_of_your_container>", blob = "response.json")
with open("response", "rb") as blob_file:
blobClient.upload_blob(data=blob_file)
In my Storage Account:-

How to download a file from Google Drive using Python and the Drive API v3

I have tried downloading file from Google Drive to my local system using python script but facing a "forbidden" issue while running a Python script. The script is as follows:
import requests
url = "https://www.googleapis.com/drive/v3/files/1wPxpQwvEEOu9whmVVJA9PzGPM2XvZvhj?alt=media&export=download"
querystring = {"alt":"media","export":"download"}
headers = {
'Authorization': "Bearer TOKEN",
'Host': "www.googleapis.com",
'Accept-Encoding': "gzip, deflate",
'Connection': "keep-alive",
}
response = requests.request("GET", url, headers=headers, params=querystring)
print(response.url)
#
import wget
import os
from os.path import expanduser
myhome = expanduser("/home/sunarcgautam/Music")
### set working dir
os.chdir(myhome)
url = "https://www.googleapis.com/drive/v3/files/1wPxpQwvEEOu9whmVVJA9PzGPM2XvZvhj?alt=media&export=download"
print('downloading ...')
wget.download(response.url)
In this script, I have got forbidden issue. Am I doing anything wrong in the script?
I have also tried another script that I found on a Google Developer page, which is as follows:
import auth
import httplib2
SCOPES = "https://www.googleapis.com/auth/drive.scripts"
CLIENT_SECRET_FILE = "client_secret.json"
APPLICATION_NAME = "test_Download"
authInst = auth.auth(SCOPES, CLIENT_SECRET_FILE, APPLICATION_NAME)
credentials = authInst.getCredentials()
http = credentials.authorize(httplib2.Http())
drive_serivce = discovery.build('drive', 'v3', http=http)
file_id = '1Af6vN0uXj8_qgqac6f23QSAiKYCTu9cA'
request = drive_serivce.files().export_media(fileId=file_id,
mimeType='application/pdf')
fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
status, done = downloader.next_chunk()
print ("Download %d%%." % int(status.progress() * 100))
This script gives me a URL mismatch error.
So what should be given for redirect URL in Google console credentials? or any other solution for the issue? Do I have to authorise my Google console app from Google in both the script? If so, what will the process of authorising the app because I haven't found any document regarding that.

To make requests to Google APIs the work flow is in essence the following:
Go to developer console, log in if you haven't.
Create a Cloud Platform project.
Enable for your project, the APIs you are interested in using with you projects' apps (for example: Google Drive API).
Create and download OAuth 2.0 Client IDs credentials that will allow your app to gain authorization for using your enabled APIs.
Head over to OAuth consent screen, click on and add your scope using the button. (scope: https://www.googleapis.com/auth/drive.readonly for you). Choose Internal/External according to your needs, and for now ignore the warnings if any.
To get the valid token for making API request the app will go through the OAuth flow to receive the authorization token. (Since it needs consent)
During the OAuth flow the user will be redirected to your the OAuth consent screen, where it will be asked to approve or deny access to your app's requested scopes.
If consent is given, your app will receive an authorization token.
Pass the token in your request to your authorized API endpoints.[2]
Build a Drive Service to make API requests (You will need the valid token)[1]
NOTE:
The available methods for the Files resource for Drive API v3 are here.
When using the Python Google APIs Client, then you can use export_media() or get_media() as per Google APIs Client for Python documentation
IMPORTANT:
Also, check that the scope you are using, actually allows you to do what you want (Downloading Files from user's Drive) and set it accordingly. ATM you have an incorrect scope for your goal. See OAuth 2.0 API Scopes
Sample Code References:
Building a Drive Service:
import google_auth_oauthlib.flow
from google.auth.transport.requests import Request
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
class Auth:
def __init__(self, client_secret_filename, scopes):
self.client_secret = client_secret_filename
self.scopes = scopes
self.flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file(self.client_secret, self.scopes)
self.flow.redirect_uri = 'http://localhost:8080/'
self.creds = None
def get_credentials(self):
flow = InstalledAppFlow.from_client_secrets_file(self.client_secret, self.scopes)
self.creds = flow.run_local_server(port=8080)
return self.creds
# The scope you app will use.
# (NEEDS to be among the enabled in your OAuth consent screen)
SCOPES = "https://www.googleapis.com/auth/drive.readonly"
CLIENT_SECRET_FILE = "credentials.json"
credentials = Auth(client_secret_filename=CLIENT_SECRET_FILE, scopes=SCOPES).get_credentials()
drive_service = build('drive', 'v3', credentials=credentials)
Making the request to export or get a file
request = drive_service.files().export(fileId=file_id, mimeType='application/pdf')
fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
status, done = downloader.next_chunk()
print("Download %d%%" % int(status.progress() * 100))
# The file has been downloaded into RAM, now save it in a file
fh.seek(0)
with open('your_filename.pdf', 'wb') as f:
shutil.copyfileobj(fh, f, length=131072)

Downloading Youtube impressions without repeated authorization

I have this piece of code to extract some metrics about my YouTube channel and create a pandas dataframe from them.
import os
import google.oauth2.credentials
import google_auth_oauthlib.flow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
import json
SCOPES = ['https://www.googleapis.com/auth/yt-analytics.readonly']
API_SERVICE_NAME = 'youtubeAnalytics'
API_VERSION = 'v2'
CLIENT_SECRETS_FILE = 'client_secrets.json'
def get_service():
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
def execute_api_request(client_library_function, **kwargs):
response = client_library_function(
**kwargs
).execute()
with open('data.json', 'w') as fp:
json.dump(response, fp)
if __name__ == '__main__':
# Disable OAuthlib's HTTPs verification when running locally.
# *DO NOT* leave this option enabled when running in production.
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
youtubeAnalytics = get_service()
execute_api_request(
youtubeAnalytics.reports().query,
ids='channel==MINE',
startDate='2014-01-01',
endDate='2019-02-26',
metrics='averageViewDuration,views,likes,dislikes,subscribersGained,subscribersLost',
dimensions='day',
sort='day',
filters = 'country==US'
)
## Now, convert the json to dataframe
import json
import pandas as pd
with open('data.json') as json_data:
d = json.load(json_data)
colnames = [d['columnHeaders'][i]['name'] for i in range(0,len(d['columnHeaders']))]
Results = pd.DataFrame(d['rows'],columns = colnames)
Results.to_csv("Youtube_data.csv")
By running this code, a windows opens and asks me to login into youtube and then provide me the authorization code. Entering this authorization code finishes the running of above python program. However, you should repeat this authorization process each time you are running this program.
Is there anyway to bypass this repeated authorization such that this process can be automated?

You need to use the oauth2client.file.Storage class to store and retrieves the credentials object as (badly) explained here: https://developers.google.com/api-client-library/python/guide/aaa_oauth
You will need to modify your get_service function with something like this:
from oauth2client import client, file
def get_service():
flow = client.flow_from_clientsecrets(CLIENT_SECRETS_FILE, SCOPES)
storage = file.Storage(API_SERVICE_NAME + '.dat')
credentials = storage.get()
http = credentials.authorize(http=httplib2.Http())
service = build(API_SERVICE_NAME, API_VERSION, http=http)
return service
Hope this helps

People API returns no connections when authenticating via Service Account [duplicate]

I'm trying to programmatically access the list of contacts on my own personal Google Account using the Python Client Library
This is a script that will run on a server without user input, so I have it set up to use credentials from a Service Account I set up. My Google API console setup looks like this.
I'm using the following basic script, pulled from the examples provided in the API docs -
import json
from httplib2 import Http
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
# Only need read-only access
scopes = ['https://www.googleapis.com/auth/contacts.readonly']
# JSON file downloaded from Google API Console when creating the service account
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'keep-in-touch-5d3ebc885d4c.json', scopes)
# Build the API Service
service = build('people', 'v1', credentials=credentials)
# Query for the results
results = service.people().connections().list(resourceName='people/me').execute()
# The result set is a dictionary and should contain the key 'connections'
connections = results.get('connections', [])
print connections #=> [] - empty!
When I hit the API it returns a result set without any 'connections' key. Specifically it returns -
>>> results
{u'nextSyncToken': u'CNP66PXjKhIBMRj-EioECAAQAQ'}
Is there something pertaining to my setup or code that's incorrect? Is there a way to see the response HTTP status code or get any further detail about what it's trying to do?
Thanks!
Side note: When I try it using the "Try it!" feature in the API docs, it correctly returns my contacts. Although I doubt that uses the client library and instead relies on user authorization via OAuth

The personFields mask is required. Specify one or more valid paths. Valid paths are documented at https://developers.google.com/people/api/rest/v1/people.connections/list/.
Additionally, use fields mask to specify which fields are included in a partial response.
Instead of:
results = service.people().connections().list(resourceName='people/me').execute()
... try:
results = service.people().connections().list(resourceName='people/me',personFields='names,emailAddresses',fields='connections,totalItems,nextSyncToken').execute()

Here is a working demo. I just tested it right now. Python 3.5.2
google-api-python-client==1.6.4
httplib2==0.10.3
oauth2client==4.1.2
You can save it to demo.py and then just run it. I left the create_contact function in case you might want to use it and have one more example on the API usage.
CLIENT_ID and CLIENT_SECRET are environment variables so I don't accidentally share that in code.
"""Google API stuff."""
import httplib2
import json
import os
from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run_flow
CLIENT_ID = os.environ['CLIENT_ID']
CLIENT_SECRET = os.environ['CLIENT_SECRET']
SCOPE = 'https://www.googleapis.com/auth/contacts'
USER_AGENT = 'JugDemoStackOverflow/v0.1'
def make_flow():
"""Make flow."""
flow = OAuth2WebServerFlow(
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
scope=SCOPE,
user_agent=USER_AGENT,
)
return flow
def get_people():
"""Return a people_service."""
flow = make_flow()
storage = Storage('info.dat')
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = run_flow(flow, storage)
http = httplib2.Http()
http = credentials.authorize(http)
people_service = build(serviceName='people', version='v1', http=http)
return people_service
def create_contact(people, user):
"""Create a Google Contact."""
request = people.createContact(
body={
'names': [{'givenName': user.name}],
'phoneNumbers': [
{'canonicalForm': user.phone, 'value': user.phone}],
}
)
return request.execute()
def demo():
"""Demonstrate getting contacts from Google People."""
people_service = get_people()
people = people_service.people()
connections = people.connections().list(
resourceName='people/me',
personFields='names,emailAddresses,phoneNumbers',
pageSize=2000,
)
result = connections.execute()
s = json.dumps(result)
# with open('contacts.json', 'w') as f:
# f.write(s)
return s
if __name__ == '__main__':
print(demo())

With service account, in DwD - G Suite Domain-wide Delegation, is necessary impersonate or delegate user in this way
delegate = credentials.create_delegated('user#xxxx.xxx')

For fellow googlers: I have the same problem using the JS API.
I succeded on my personal gmail address, but not on my work one (g-suite) neither on my secondary gmail address.
Can't see the pattern. It's possible that the work one has contact listing deactivated.