We Keep Coding

MS Graph API Outlook Attachments to s3 bucket

I'm attempting to pull email attachments from outlook and store them in an s3 bucket in aws. This is one of my first python projects, and its proving to be very difficult for me and is probably very messy code.
# see https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/dev/sample/device_flow_sample.py
# This authenticates for first time login.
# As long as you call acquire_token_silent before you invoke any graph APIs, the tokens will stay up to date.
# The refresh token is good for 90 days, and automatically updates. Once you login, the tokens will
# be updated and stored in the cache (and persisted to a file), and will stay alive more-or-less indefinitely
# (there are some things that can invalidate it on the server side).
from __future__ import with_statement
import io
import sys
import json
import logging
import os
import tarfile
import atexit
from wsgiref import headers
import requests
import msal
import boto3
import base64
from botocore.exceptions import ClientError
import codecs
print('Starting...')
# logging
logging.basicConfig(level=logging.DEBUG) # Enable DEBUG log for entire script
logging.getLogger("msal").setLevel(logging.INFO) # Optionally disable MSAL DEBUG logs
client=boto3.client('secretsmanager')
# config
config = dict(
authority = "https://login.microsoftonline.com/common",
client_id = '123456',
scope = ["Mail.ReadWrite"],
username = 'username',
cache_file = client.get_secret_value(SecretId="demo-ms-graph")['SecretBinary'],
endpoint = 'https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$expand=attachments&$search="hasAttachments:true"'
)
# cache
cache = msal.SerializableTokenCache()
if os.path.exists(config["cache_file"]):
with tarfile.open('token.cache.tar.gz', "w:gz") as tar:
tar.add(config["cache_file"])
bts = open('token.cache.tar.gz','rb').read()
print("Length before",len(bts))
#sec=client.update_secret(SecretId="demo-ms-graph", SecretBinary=bts)
sec=client.get_secret_value(SecretId="demo-ms-graph")['SecretBinary']
print("Length after",len(sec))
with tarfile.open(fileobj=io.BytesIO(sec), mode='r:gz') as t:
d=t.extractfile('token.cache')
#print file content
print("File content",str(d.read()))
with tarfile.open(fileobj=io.BytesIO(sec), mode='r:gz') as t:
d=t.extractfile('token.cache')
# app
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
token_cache=cache)
print('Connecting to app..')
# exists?
result = None
accounts = app.get_accounts()
if accounts:
logging.info("found accounts in the app")
for a in accounts:
print(a)
if a["username"] == config["username"]:
result = app.acquire_token_silent(config["scope"], account=a)
break
if result and "access_token" in result:
# Calling graph using the access token
graph_data = requests.get( # Use token to call downstream service
config["endpoint"],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
#print("Graph API call result: %s" % json.dumps(graph_data, indent=2))
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
main = 'https://graph.microsoft.com/v1.0/me/mailFolders/inbox/messages?$expand=attachments&$search="hasAttachments:true"'
response = requests.get(main, headers={'Authorization': 'Bearer ' + result['access_token']})
if response.status_code != 200:
raise Exception(response.json())
response_json = response.json()
print('Starting upload...')
emails = response_json['value']
s3 = boto3.client('s3')
bucket ='demo-email-app'
for email in emails:
email_id = email['id']
subject = email['subject']
if email['hasAttachments']:
print(subject)
attachments = email['attachments']
for attachment in attachments:
name = attachment['name']
fileContent = json.dumps(email, indent=2)
s3.put_object(Bucket=bucket, Key=name.replace('.', '_') + '.json', Body=fileContent.encode('UTF-8'))
print('Upload Complete')
#download_email_attachments(email_id, headers)
print('All uploads complete')
My secret is stored as binary in secrets manager, and it seems to be able to pull the secret and prints it fine. I'm running into an error in the # exists? section with "AttributeError: 'NoneType' object has no attribute 'get'"
Am I approaching this in the wrong way? I have the token stored in secretsmanager, and am trying to retrieve it for msal to use and authenticate my user to the MS Graph API so I can pull attachments from the Outlook account and store those in an s3 bucket.

Google Ads API login via python

Is there any Python code sample available to get Google ads campaign data using Python via service account.
The information available on https://developers.google.com/google-ads/api/docs/oauth/service-accounts
is not sufficient enough to start. I have created service code and trying to use following code, but I am not even sure of this is depreciated on or the new one
from google.oauth2.service_account import Credentials
from google.ads.googleads.client import GoogleAdsClient
SCOPES = ['https://www.googleapis.com/auth/adwords']
PATH_TO_SERVICE_ACCOUNT_JSON = ''
CUSTOMER_ID = ''
DEVELOPER_TOKEN = ''
QUERY = ''
credentials = Credentials.from_service_account_file(PATH_TO_SERVICE_ACCOUNT_JSON, scopes=SCOPES, subject="<AN ACTUAL USER'S EMAIL HERE>")
googleads_client = GoogleAdsClient(credentials=credentials, developer_token=DEVELOPER_TOKEN, version="v7")
ga_service = googleads_client.get_service("GoogleAdsService")
response = ga_service.search(customer_id=CUSTOMER_ID, query=QUERY)

insert json object to data lake

i have few python api endpoints thet get data in request body . i want to insert/add this data to azure datalake every time api call any ideas?
example api endpoint
#main.route("/order/add", methods=["POST"])
def post_add_new_order():
data = request.json
for key in data:
if not typesModule.key_type_and_value_type_are_equal(key, data[key]):
return {"err": "One of the value types is incorrect"}
want to insert this data to azure data lake

If you want to add data to Azure Data Lake Storage Gen1 in python package, we can use the package azure-datalake-store to implement it.
For example
Create a service principal
az login
az ad sp create-for-rbac -n 'Myapp' --skip-assignment
Assign the service principal to the Azure Data Lake Storage Gen1 account file or folder access control.
The ACL for Azure data lake gen1 has three permissions. There are Read, Write, and Execute. Please configure it according to your need. For more details, please refer to here and here
Code
import json
import azure.datalake.store.lib as lib
from azure.datalake.store.core import AzureDLFileSystem
RESOURCE = 'https://datalake.azure.net/'
client_id = '42e0d***c4c522d988c4'
client_secret = 'Gbx2eK6****ClJDfQpIjoae:'
tenant = 'e4c9ab4e-bd27-40d5-8459-230ba2a757fb'
#main.route("/order/add", methods=["POST"])
def post_add_new_order():
data = request.get_json()
json_data = json.dumps(data).encode('utf-8')
adlCreds = lib.auth(tenant_id = tenant,
client_secret = client_secret,
client_id = client_id,
resource=RESOURCE)
adlsFileSystemClient = AzureDLFileSystem(adlCreds, store_name='testbowman')
# check if the file exist
if adlsFileSystemClient.access('/test/data.json'):
#append content
with adlsFileSystemClient.open(path='/test/data.json', mode='ab') as f:
f.write(json_data)
f.write(b'\r\n')
else:
#create file and write
with adlsFileSystemClient.open(path='/test/data.json', mode='wb') as f:
f.write(json_data)
f.write(b'\r\n')
return {'you sent' : data}

ADAL Python to Refresh PowerBI dataset

I found a piece of code on Azure documentation that allows getting credentials without MFA. But I'm wondering if is possible to use it to connect to PowerBI API.
The piece of code that I'm using is:
import adal
import requests
from msrestazure.azure_active_directory import AADTokenCredentials
def authenticate_client_key():
authority_host_uri = 'https://login.microsoftonline.com'
tenant = 'tenant'
authority_uri = authority_host_uri + '/' + tenant
resource_uri = 'https://management.core.windows.net/'
client_id = 'clientid'
client_secret = 'client-secret'
context = adal.AuthenticationContext(authority_uri, api_version=None)
mgmt_token = context.acquire_token_with_client_credentials(resource_uri, client_id, client_secret)
credentials = AADTokenCredentials(mgmt_token, client_id)
return credentials
source: https://azure.microsoft.com/en-us/resources/samples/data-lake-analytics-python-auth-options/
According to the code written on PowerShell, the aim is to insert the access_token into the header of the following POST request
POST https://api.powerbi.com/v1.0/myorg/groups/me/datasets/{dataset_id}/refreshes
Source:https://powerbi.microsoft.com/en-us/blog/announcing-data-refresh-apis-in-the-power-bi-service/
I have tried to use the credentials into the POST request, but seems is not working.
I have tried
url = 'https://api.powerbi.com/v1.0/myorg/groups/me/datasets/datasetid/refreshes'
requests.post(url,data=mgmt_token)
Is it possible to merge this two codes?
Regards,

You can use the pypowerbi package to refresh Power BI datasets or you can check how to do it yourself by inspecting the code. https://github.com/cmberryau/pypowerbi
pip install pypowerbi
import adal
from pypowerbi.client import PowerBIClient
# you might need to change these, but i doubt it
authority_url = 'https://login.windows.net/common'
resource_url = 'https://analysis.windows.net/powerbi/api'
api_url = 'https://api.powerbi.com'
# change these to your credentials
client_id = '00000000-0000-0000-0000-000000000000'
username = 'someone#somecompany.com'
password = 'averygoodpassword'
# first you need to authenticate using adal
context = adal.AuthenticationContext(authority=authority_url,
validate_authority=True,
api_version=None)
# get your authentication token
token = context.acquire_token_with_username_password(resource=resource_url,
client_id=client_id,
username=username,
password=password)
# create your powerbi api client
client = PowerBIClient(api_url, token)
# Refresh the desired dataset (dataset and group IDs can be taken from the browser URL)
client.datasets.refresh_dataset(dataset_id='data-set-id-goes-here',
notify_option='MailOnCompletion',
group_id='group-id-goes-here')

Your code for acquiring an access token looks ok, but to use it with Power BI REST API, you must change resource_uri to be https://analysis.windows.net/powerbi/api.
When making a request to Power BI REST API, you must add Authorization header with value Bearer {accessToken}, where {accessToken} is the token acquired. I can't write in python, but you should do something like this:
headers = {'Authorization': 'Bearer ' + accessToken, 'Content-Type': 'application/json'}
url = 'https://api.powerbi.com/v1.0/myorg/groups/me/datasets/datasetid/refreshes'
requests.post(url, headers=headers)
(of course, you need to replace datasetid with actual value in url).
For example, here is how it can be done in C#:
string redirectUri = "https://login.live.com/oauth20_desktop.srf";
string resourceUri = "https://analysis.windows.net/powerbi/api";
string authorityUri = "https://login.windows.net/common/oauth2/authorize";
string clientId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";
string powerBIApiUrl = $"https://api.powerbi.com/v1.0/myorg/datasets/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/refreshes";
AuthenticationContext authContext = new AuthenticationContext(authorityUri, new TokenCache());
var authenticationResult = await authContext.AcquireTokenAsync(resourceUri, clientId, new Uri(redirectUri), new PlatformParameters(PromptBehavior.Auto));
var accessToken = authenticationResult.AccessToken;
var request = WebRequest.Create(powerBIApiUrl) as HttpWebRequest;
request.KeepAlive = true;
request.Method = "POST";
request.ContentLength = 0;
request.Headers.Add("Authorization", String.Format("Bearer {0}", accessToken));
using (Stream writer = request.GetRequestStream())
{
var response = (HttpWebResponse)request.GetResponse();
}

How do I call an API Gateway with Cognito credentials in Python

I've managed to setup an API Gateway secured with Cognito. The unauthenticated user role has an access policy that should grant it access to the gateway. I've also managed to use boto3 to retrieve an identity ID from the pool and obtain the associated open ID token, as well as the associated secret and access keys.
How do I now make a call to the gateway using these credentials? Is there a way to use boto3 to handle signing a request to a particular method on the API?

My code is based largely on the questioner's own answer, but I've tried to make it clearer where all the values come from.
import boto3
import requests
from requests_aws4auth import AWS4Auth
# Use 'pip install boto3 requests requests-aws4auth' to get these
region_name = 'ap-southeast-2' # or 'us-west-1' or whatever
# 12 decimal digits from your AWS login page
account_id = '123456789012'
# I've only found this in the sample code for other languages, e.g. JavaScript
# Services→Cognito→Manage Federated Identities→(your-id-pool)→Sample code
identity_pool_id = 'ap-southeast-2:fedcba98-7654-3210-1234-56789abcdef0'
# Create a new identity
boto3.setup_default_session(region_name = region_name)
identity_client = boto3.client('cognito-identity', region_name=region_name)
identity_response = identity_client.get_id(AccountId=account_id,
IdentityPoolId=identity_pool_id)
# We normally wouldn't log this, but to illustrate:
identity_id = identity_response['IdentityId']
print ('identity_id:', identity_id) # good idea not to log this
# Get the identity's credentials
credentials_response = identity_client.get_credentials_for_identity(IdentityId=identity_id)
credentials = credentials_response['Credentials']
access_key_id = credentials['AccessKeyId']
secret_key = credentials['SecretKey']
service = 'execute-api'
session_token = credentials['SessionToken']
expiration = credentials['Expiration']
# Again, we normally wouldn't log this:
print ('access_key_id', access_key_id)
print ('secret_key', secret_key)
print ('session_token', session_token)
print ('expiration', expiration)
# The access_key_id will look something like 'AKIABC123DE456FG7890', similar to
# Services→IAM→Users→(AWS_USER_NAME)→Security credentials→Access key ID
# Get the authorisation object
auth = AWS4Auth(access_key_id, secret_key, region_name, service,
session_token=session_token)
current_app['auth'] = auth
# Just an illustration again:
print ('auth: %(service)s(%(date)s) %(region)s:%(access_id)s' % auth.__dict__)
# We'll use that object to send a request to our app. This app doesn't
# exist in real life, though, so you'll need to edit the following quite
# heavily:
# Services→Cognito→Manage your User Pools→(your-user-pool)→Apps→App name
app_name = 'my-app-name'
api_path = 'dev/helloworld'
method = 'GET'
headers = {}
body = ''
url = 'https://%s.%s.%s.amazonaws.com/%s' % (app_name, service, region_name,
api_path)
response = requests.request(method, url, auth=auth, data=body, headers=headers)

The following code (and the requests-aws4auth library) did the job:
import boto3
import datetime
import json
from requests_aws4auth import AWS4Auth
import requests
boto3.setup_default_session(region_name='us-east-1')
identity = boto3.client('cognito-identity', region_name='us-east-1')
account_id='XXXXXXXXXXXXXXX'
identity_pool_id='us-east-1:YYY-YYYY-YYY-YY'
api_prefix='ZZZZZZZZZ'
response = identity.get_id(AccountId=account_id, IdentityPoolId=identity_pool_id)
identity_id = response['IdentityId']
print ("Identity ID: %s"%identity_id)
resp = identity.get_credentials_for_identity(IdentityId=identity_id)
secretKey = resp['Credentials']['SecretKey']
accessKey = resp['Credentials']['AccessKeyId']
sessionToken = resp['Credentials']['SessionToken']
expiration = resp['Credentials']['Expiration']
print ("\nSecret Key: %s"%(secretKey))
print ("\nAccess Key %s"%(accessKey))
print ("\nSession Token: %s"%(sessionToken))
print ("\nExpiration: %s"%(expiration))
method = 'GET'
headers = {}
body = ''
service = 'execute-api'
url = 'https://%s.execute-api.us-east-1.amazonaws.com/dev/helloworld' % api_prefix
region = 'us-east-1'
auth = AWS4Auth(accessKey, secretKey, region, service, session_token=sessionToken)
response = requests.request(method, url, auth=auth, data=body, headers=headers)
print(response.text)

Next code is working really well.
Hope to help:
from pprint import pprint
import requests
from pycognito import Cognito
USER_POOL_ID = 'eu-central-1_XXXXXXXXXXX'
CLIENT_ID = 'XXXXXXXXXXXX'
CLIENT_SECRET = 'XXXXXXXXXXX'
u = Cognito(USER_POOL_ID,CLIENT_ID, client_secret=CLIENT_SECRET, username='cognito user name')
u.authenticate('cognito user password')
id_token = u.id_token
headers = {'Authorization': 'Bearer ' + id_token}
api_url = 'https://XXXXXXXXXXX.execute-api.eu-central-1.amazonaws.com/stage/XXXXXXXXXXX'
r = requests.get(api_url, headers=headers)
pprint(dict(r.headers))
print(r.status_code)
print(r.text)

Here is an example from our public docs: http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
Cognito creds are no different than any other temporary creds, and the signing process is also the same. If you want to move back to Python the example above should be good, or I would guess that there are third-party libraries out there to do the signature for you.

identity_pool_id how to get
If you have not federated pool which could give you "identity_pool_id" ,
execution code below will give you identity_pool_id
import boto3
boto3.setup_default_session(
aws_access_key_id='AKIAJ7TBC72BPWNEWIDQ',
aws_secret_access_key='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',
region_name='ap-southeast-2')
client = boto3.client('cognito-identity')
response = client.list_identity_pools(MaxResults=3,)
print("IdentityPoolId-- ", response)