We Keep Coding

Channel's uploads playlist id returning playlistNotFound

I am working on a project that gets the id of a channel's uploads playlist through their legacy username using the forUsername parameter with python3 and youtube-data-api. Here it searches for the uploads playlist string. From there, it gets the videos on their channel and the info I need.
When I try this for some channels, it works perfectly and returns exactly what I expect. With others however, it gives me a playlistNotFound error despite me printing the playlistId that I am recieving to ensure that it is correct.
The error (with playlist id UUNbg-_aV3kK4MEb2vgdBtsw from the channel ManchesterUnited):
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://youtube.googleapis.com/youtube/v3/playlistItems?part=snippet&playlistId=UUNbg-_aV3kK4MEb2vgdBtsw&maxResults=50&key=AIzaSy...&alt=json returned "The playlist identified with the request's <code>playlistId</code> parameter cannot be found.". Details: "[{'message': "The playlist identified with the request's <code>playlistId</code> parameter cannot be found.", 'domain': 'youtube.playlistItem', 'reason': 'playlistNotFound', 'location': 'playlistId', 'locationType': 'parameter'}]">
My code:
from scraper_api import ScraperAPIClient
import requests
import googleapiclient.discovery
import html
import gspread
from oauth2client.service_account import ServiceAccountCredentials
import json
username=input("what is the channel username? (go to page and then you will see in the url) ")
ytapi = requests.get("https://www.googleapis.com/youtube/v3/channels?part=contentDetails&forUsername="+username+"&key=AIzaSy...")
print(ytapi.text)
at=ytapi.text.split(' ')
c=1
b=0
viewnumbers = []
titles = []
vtypes = []
urls = []
for i in at:
c+=1
if 'uploads' in i:
for it in at[c-3:]:
if 'U' in it:
print(it)
idd=it[1:len(it)-2]
print(idd)
break
playlist_id = idd
youtube = googleapiclient.discovery.build("youtube", "v3", developerKey = "AIzaSy...")
request = youtube.playlistItems().list(
part = "snippet",
playlistId = playlist_id,
maxResults = 50
)
response = request.execute()

MS Graph API Outlook Attachments to s3 bucket

I'm attempting to pull email attachments from outlook and store them in an s3 bucket in aws. This is one of my first python projects, and its proving to be very difficult for me and is probably very messy code.
# see https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/dev/sample/device_flow_sample.py
# This authenticates for first time login.
# As long as you call acquire_token_silent before you invoke any graph APIs, the tokens will stay up to date.
# The refresh token is good for 90 days, and automatically updates. Once you login, the tokens will
# be updated and stored in the cache (and persisted to a file), and will stay alive more-or-less indefinitely
# (there are some things that can invalidate it on the server side).
from __future__ import with_statement
import io
import sys
import json
import logging
import os
import tarfile
import atexit
from wsgiref import headers
import requests
import msal
import boto3
import base64
from botocore.exceptions import ClientError
import codecs
print('Starting...')
# logging
logging.basicConfig(level=logging.DEBUG) # Enable DEBUG log for entire script
logging.getLogger("msal").setLevel(logging.INFO) # Optionally disable MSAL DEBUG logs
client=boto3.client('secretsmanager')
# config
config = dict(
authority = "https://login.microsoftonline.com/common",
client_id = '123456',
scope = ["Mail.ReadWrite"],
username = 'username',
cache_file = client.get_secret_value(SecretId="demo-ms-graph")['SecretBinary'],
endpoint = 'https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$expand=attachments&$search="hasAttachments:true"'
)
# cache
cache = msal.SerializableTokenCache()
if os.path.exists(config["cache_file"]):
with tarfile.open('token.cache.tar.gz', "w:gz") as tar:
tar.add(config["cache_file"])
bts = open('token.cache.tar.gz','rb').read()
print("Length before",len(bts))
#sec=client.update_secret(SecretId="demo-ms-graph", SecretBinary=bts)
sec=client.get_secret_value(SecretId="demo-ms-graph")['SecretBinary']
print("Length after",len(sec))
with tarfile.open(fileobj=io.BytesIO(sec), mode='r:gz') as t:
d=t.extractfile('token.cache')
#print file content
print("File content",str(d.read()))
with tarfile.open(fileobj=io.BytesIO(sec), mode='r:gz') as t:
d=t.extractfile('token.cache')
# app
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
token_cache=cache)
print('Connecting to app..')
# exists?
result = None
accounts = app.get_accounts()
if accounts:
logging.info("found accounts in the app")
for a in accounts:
print(a)
if a["username"] == config["username"]:
result = app.acquire_token_silent(config["scope"], account=a)
break
if result and "access_token" in result:
# Calling graph using the access token
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
#print("Graph API call result: %s" % json.dumps(graph_data, indent=2))
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
main = 'https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$expand=attachments&$search="hasAttachments:true"'
response = requests.get(main, headers={'Authorization': 'Bearer ' + result['access_token']})
if response.status_code != 200:
raise Exception(response.json())
response_json = response.json()
print('Starting upload...')
emails = response_json['value']
s3 = boto3.client('s3')
bucket ='demo-email-app'
for email in emails:
email_id = email['id']
subject = email['subject']
if email['hasAttachments']:
print(subject)
attachments = email['attachments']
for attachment in attachments:
name = attachment['name']
fileContent = json.dumps(email, indent=2)
s3.put_object(Bucket=bucket, Key=name.replace('.', '_') + '.json', Body=fileContent.encode('UTF-8'))
print('Upload Complete')
#download_email_attachments(email_id, headers)
print('All uploads complete')
My secret is stored as binary in secrets manager, and it seems to be able to pull the secret and prints it fine. I'm running into an error in the # exists? section with "AttributeError: 'NoneType' object has no attribute 'get'"
Am I approaching this in the wrong way? I have the token stored in secretsmanager, and am trying to retrieve it for msal to use and authenticate my user to the MS Graph API so I can pull attachments from the Outlook account and store those in an s3 bucket.

Twilio retrieve message history from recipient only via Python

Suppose I have this code which places all the message history between a twilio number and its recipients in an array. Is there a way to retrieve only the message history from all the recipients and not the sender (whom is I)
from twilio.rest import Client
account_sid = 'xxxx'
auth_token = 'xxxx'
client = Client(account_sid, auth_token)
text=[]
messages = client.messages.list()
for record in messages:
print(record.body)
text.append(record.body)
print(text)

There are some examples of filter criteria with the above command, you can filter on to: to your specific Twilio number to just see that side of the conversation (via body key).
# Download the helper library from https://www.twilio.com/docs/python/install
from datetime import datetime
from twilio.rest import Client
# Your Account Sid and Auth Token from twilio.com/console
# DANGER! This is insecure. See http://twil.io/secure
account_sid = 'ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'your_auth_token'
client = Client(account_sid, auth_token)
messages = client.messages.list(
to='+15558675310',
limit=20
)
for record in messages:
print(record.sid)

How do I call an API Gateway with Cognito credentials in Python

I've managed to setup an API Gateway secured with Cognito. The unauthenticated user role has an access policy that should grant it access to the gateway. I've also managed to use boto3 to retrieve an identity ID from the pool and obtain the associated open ID token, as well as the associated secret and access keys.
How do I now make a call to the gateway using these credentials? Is there a way to use boto3 to handle signing a request to a particular method on the API?

My code is based largely on the questioner's own answer, but I've tried to make it clearer where all the values come from.
import boto3
import requests
from requests_aws4auth import AWS4Auth
# Use 'pip install boto3 requests requests-aws4auth' to get these
region_name = 'ap-southeast-2' # or 'us-west-1' or whatever
# 12 decimal digits from your AWS login page
account_id = '123456789012'
# I've only found this in the sample code for other languages, e.g. JavaScript
# Services→Cognito→Manage Federated Identities→(your-id-pool)→Sample code
identity_pool_id = 'ap-southeast-2:fedcba98-7654-3210-1234-56789abcdef0'
# Create a new identity
boto3.setup_default_session(region_name = region_name)
identity_client = boto3.client('cognito-identity', region_name=region_name)
identity_response = identity_client.get_id(AccountId=account_id,
IdentityPoolId=identity_pool_id)
# We normally wouldn't log this, but to illustrate:
identity_id = identity_response['IdentityId']
print ('identity_id:', identity_id) # good idea not to log this
# Get the identity's credentials
credentials_response = identity_client.get_credentials_for_identity(IdentityId=identity_id)
credentials = credentials_response['Credentials']
access_key_id = credentials['AccessKeyId']
secret_key = credentials['SecretKey']
service = 'execute-api'
session_token = credentials['SessionToken']
expiration = credentials['Expiration']
# Again, we normally wouldn't log this:
print ('access_key_id', access_key_id)
print ('secret_key', secret_key)
print ('session_token', session_token)
print ('expiration', expiration)
# The access_key_id will look something like 'AKIABC123DE456FG7890', similar to
# Services→IAM→Users→(AWS_USER_NAME)→Security credentials→Access key ID
# Get the authorisation object
auth = AWS4Auth(access_key_id, secret_key, region_name, service,
session_token=session_token)
current_app['auth'] = auth
# Just an illustration again:
print ('auth: %(service)s(%(date)s) %(region)s:%(access_id)s' % auth.__dict__)
# We'll use that object to send a request to our app. This app doesn't
# exist in real life, though, so you'll need to edit the following quite
# heavily:
# Services→Cognito→Manage your User Pools→(your-user-pool)→Apps→App name
app_name = 'my-app-name'
api_path = 'dev/helloworld'
method = 'GET'
headers = {}
body = ''
url = 'https://%s.%s.%s.amazonaws.com/%s' % (app_name, service, region_name,
api_path)
response = requests.request(method, url, auth=auth, data=body, headers=headers)

The following code (and the requests-aws4auth library) did the job:
import boto3
import datetime
import json
from requests_aws4auth import AWS4Auth
import requests
boto3.setup_default_session(region_name='us-east-1')
identity = boto3.client('cognito-identity', region_name='us-east-1')
account_id='XXXXXXXXXXXXXXX'
identity_pool_id='us-east-1:YYY-YYYY-YYY-YY'
api_prefix='ZZZZZZZZZ'
response = identity.get_id(AccountId=account_id, IdentityPoolId=identity_pool_id)
identity_id = response['IdentityId']
print ("Identity ID: %s"%identity_id)
resp = identity.get_credentials_for_identity(IdentityId=identity_id)
secretKey = resp['Credentials']['SecretKey']
accessKey = resp['Credentials']['AccessKeyId']
sessionToken = resp['Credentials']['SessionToken']
expiration = resp['Credentials']['Expiration']
print ("\nSecret Key: %s"%(secretKey))
print ("\nAccess Key %s"%(accessKey))
print ("\nSession Token: %s"%(sessionToken))
print ("\nExpiration: %s"%(expiration))
method = 'GET'
headers = {}
body = ''
service = 'execute-api'
url = 'https://%s.execute-api.us-east-1.amazonaws.com/dev/helloworld' % api_prefix
region = 'us-east-1'
auth = AWS4Auth(accessKey, secretKey, region, service, session_token=sessionToken)
response = requests.request(method, url, auth=auth, data=body, headers=headers)
print(response.text)

Next code is working really well.
Hope to help:
from pprint import pprint
import requests
from pycognito import Cognito
USER_POOL_ID = 'eu-central-1_XXXXXXXXXXX'
CLIENT_ID = 'XXXXXXXXXXXX'
CLIENT_SECRET = 'XXXXXXXXXXX'
u = Cognito(USER_POOL_ID,CLIENT_ID, client_secret=CLIENT_SECRET, username='cognito user name')
u.authenticate('cognito user password')
id_token = u.id_token
headers = {'Authorization': 'Bearer ' + id_token}
api_url = 'https://XXXXXXXXXXX.execute-api.eu-central-1.amazonaws.com/stage/XXXXXXXXXXX'
r = requests.get(api_url, headers=headers)
pprint(dict(r.headers))
print(r.status_code)
print(r.text)

Here is an example from our public docs: http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
Cognito creds are no different than any other temporary creds, and the signing process is also the same. If you want to move back to Python the example above should be good, or I would guess that there are third-party libraries out there to do the signature for you.

identity_pool_id how to get
If you have not federated pool which could give you "identity_pool_id" ,
execution code below will give you identity_pool_id
import boto3
boto3.setup_default_session(
aws_access_key_id='AKIAJ7TBC72BPWNEWIDQ',
aws_secret_access_key='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',
region_name='ap-southeast-2')
client = boto3.client('cognito-identity')
response = client.list_identity_pools(MaxResults=3,)
print("IdentityPoolId-- ", response)

Oauth Python twitter auth

import oauth2 as oauth
key = "xxxxx"
secret = "xxxxx"
request_token_url = "https://api.twitter.com/oauth/request_token"
consumer = oauth.Consumer(key= key, secret= secret)
client = oauth.Client(consumer)
resp, content = client.request(request_token_url, "GET")
print content
Once i have executed this script it works fine but what would be the best way to deal with the output it gives me.
The output is something like "xxxxxxxx&yyyyyyyyyy&zzzzzzzzz"
And i need to use the varaibles xxxx and yyyy how can i parse these and put them into variables?
Thanks
William

output = content.split('&')
x = output[0]
y = output[1]
also check out Tweepy for Python interaction with Twitter.
https://github.com/tweepy/tweepy