This is a very weird problem that I have got stuck into, will really appreciate if someone could provide some direction. I am trying to access value for request_url from web_token.py module.
when I only try to run web_token.py separately over pycharm and print request_url it works fine and generates the url. I zip both these files and upload it to lambda function but when testing it I get an error "Unable to import module 'retrieve_accounts': No module named boto.sts". I even tried putting the code of web_token.py inside retrieve_accounts.py but getting the same error. I am sure I am missing something very basic it looks boto.sts is not being recognized while running the python script. Can somebody please give some guidance. Thank You!
retrieve_accounts.py
import boto3
import web_token
def get_account(event, context):
client = boto3.client('dynamodb')
NameID = "testname#org.com"
ManagerEmail = "test1#eorg.com"
response = client.scan(
TableName='Sandbox-Users',
ScanFilter={
'NameID': {
'AttributeValueList': [
{
'S': NameID,
},
],
'ComparisonOperator': 'EQ'
}
}
)
if response["Count"] > 0:
client = boto3.client('dynamodb')
response = client.get_item(
Key={
'NameID': {
'S': NameID,
},
'ManagerEmail': {
'S': ManagerEmail,
},
},
TableName='Sandbox-Users',
)
return web_token.request_url ----------->here
else:
response = client.put_item(
Item={
'NameID': {
'S': NameID,
},
'ManagerEmail': {
'S': ManagerEmail,
}
},
TableName='Sandbox-Users'
)
return "Create Account"
web_token.py
import httplib
import urllib, json
from boto.sts import STSConnection -------->Error here
sts_connection = STSConnection()
assumed_role_object = sts_connection.assume_role(
role_arn="arn:aws:iam::454084028794:role/AMPSandboxRole",
role_session_name="AssumeRoleSession"
)
# Step 3: Format resulting temporary credentials into JSON
json_string_with_temp_credentials = '{'
json_string_with_temp_credentials += '"sessionId":"' +
assumed_role_object.credentials.access_key + '",'
json_string_with_temp_credentials += '"sessionKey":"' +
assumed_role_object.credentials.secret_key + '",'
json_string_with_temp_credentials += '"sessionToken":"' +
assumed_role_object.credentials.session_token + '"'
json_string_with_temp_credentials += '}'
# Step 4. Make request to AWS federation endpoint to get sign-in token.
Construct the parameter string with the sign-in action request, a 12-hour session duration, and the JSON
document with temporary credentials as parameters.
request_parameters = "?Action=getSigninToken"
request_parameters += "&SessionDuration=43200"
request_parameters += "&Session=" +
urllib.quote_plus(json_string_with_temp_credentials)
request_url = "/federation" + request_parameters
conn = httplib.HTTPSConnection("signin.aws.amazon.com")
conn.request("GET", request_url)
r = conn.getresponse()
# Returns a JSON document with a single element named SigninToken.
signin_token = json.loads(r.read())
request_parameters = "?Action=login"
request_parameters += "&Issuer=sandbox.com"
request_parameters += "&Destination=" +
urllib.quote_plus("https://console.aws.amazon.com/")
request_parameters += "&SigninToken=" + signin_token["SigninToken"]
request_url = "https://signin.aws.amazon.com/federation" +
request_parameters
AWS Lambda Python environments include boto3 (and botocore). They don't include the older boto (a precursor to boto3), hence the import failure.
You could potentially include boto in your upload but it's not advisable to mix boto and boto3 if you can avoid it. Use one or the other, preferably boto3.
Related
When I make requests to the 'Inventory' or 'Sales' endpoints, I get successful responses. This confirms that my signature and IAM are correct. I am manually signing the requests because I am not sure how to use boto3 to create the signature.
I am not sure why, but when making POST requests, the response will state The Canonical String for this request should have been... and provides the payload_hash below. I am copy/pasting the string provides. Once I do this, I then get the InvalidInput response. However, when using GET requests, I am able to use payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest() as seen below and it works fine.
This is the same response as with GET_SALES_AND_TRAFFIC_REPORT and other reports.
CREATING CANONICAL STRING :
t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d')
canonical_querystring = '/reports/2021-06-30/reports'
signed_headers = 'host;user-agent;x-amz-access-token;x-amz-date'
payload_hash = 'deda182f2e780e6c5e6abb9b19a087a8db6c620c39e784bf4a3384e76d742278'
# payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()
canonical_request = method + '\n' + canonical_querystring + '\n' + '\n' + 'host:' + host + '\n' + 'user-agent:python-requests/2.27.1' + '\n' + 'x-amz-access-token:' + access + '\n' + 'x-amz-date:' + amzdate + '\n' + '\n' + signed_headers + '\n' + payload_hash
REQUEST:
headers = {
'x-amz-access-token': access,
'x-amz-date':amzdate,
'Authorization':authorization_header,
'Content-Type': 'application/json'
}
data = {
'marketplaceIds':'ATVPDKIKX0DER',
'reportType': 'GET_SALES_AND_TRAFFIC_REPORT',
'dataStartTime':'2022-03-10T20:11:24.000Z',
'dataEndTime':'2022-03-20T20:11:24.000Z',
'reportOptions':{
'dateGranularity':'WEEK',
'asinGranularity':'CHILD'
}
}
r = requests.post(
'https://sellingpartnerapi-na.amazon.com/reports/2021-06-30/reports',
headers = headers,
data = data
)
print(r)
print(r.text)
RESPONSE:
<Response [400]>
{
"errors": [
{
"code": "InvalidInput",
"message": "Invalid Input",
"details": ""
}
]
}
Took me awhile to get the signing stuff to work in postman, just looking at your request I believe MarketplaceIds needs to be an array, although you would get a specific error if that was the main problem:
"marketplaceIds": [
"ATVPDKIKX0DER"
]
I was just working on this. So, I used AWS4Auth for creating a canonical string just to be sure that I'm not messing with any of it.
A way to do that: (Assuming you have already generated LWA access token, which I can see in your header as 'access')
import json
import boto3
from requests_aws4auth.aws4auth import AWS4Auth
client = boto3.client('sts')
aws_account_id = 'YOUR_ACCOUNT_ID'
iamrole = 'arn:aws:iam::'+aws_account_id+':role/YOU_ROLE_NAME'
response = client.assume_role(
RoleArn= iamrole,
RoleSessionName='SPAPIRoleSession'
)
# Initializing AccessKey, SecretKey and SessionToken variables to be used in signature signing.
AccessKeyId = response['Credentials']['AccessKeyId']
SecretAccessKey = response['Credentials']['SecretAccessKey']
SessionToken = response['Credentials']['SessionToken']
#add your region accordingly: my region is us-east-1
auth = AWS4Auth(AccessKeyId, SecretAccessKey, 'us-east-1', 'execute-api', session_token=SessionToken)
# Create headers to add to SP API request. Headers should include: content_type and "x-amz-access-token"(b)
headers = {'content-type': 'application/json','Accept': 'application/json','x-amz-access-token':access}
#your params: I made a small change to 'dataEndTime', if you mentioned the granularity as a week, make sure the start and end dates are 7 days apart. Look at the AWS documentation for more details
data = {
'marketplaceIds':'ATVPDKIKX0DER',
'reportType': 'GET_SALES_AND_TRAFFIC_REPORT',
'dataStartTime':'2022-03-10T20:11:24.000Z',
'dataEndTime':'2022-03-17T20:11:24.000Z',
'reportOptions':{
'dateGranularity':'WEEK',
'asinGranularity':'CHILD'}
}
# Change data to json and add auth as additional parameter
reportId = requests.post("https://sellingpartnerapi-na.amazon.com/reports/2021-06-30/reports",
json=(data),
headers=headers,
auth=auth)
print(reportId)
print(reportId.json())
RESPONSE:
<Response [202]>
{'reportId': '54120019456'}
This should result in giving you the reportId required. Let me know if this helps. Thanks
I am trying to send an authentificated message over an API at iconomi.com.
I am used to sign message when dealing with other exchange API, but can't be authentificated with this specific one.
I read the official documentation for authentification:
You generate the ICN-SIGN header by creating a sha512 HMAC using the
base64-decoded secret key on the prehash string timestamp + method +
requestPath + body (where + represents string concatenation) and
base64-encode the output, where:
the timestamp value is the same as the ICN-TIMESTAMP header. the body
is the request body string or omitted if there is no request body
(typically for GET requests). method must always be in upper case
Example: base64_encode(HMAC_SHA512(secret_key, timestamp + upper_case(method) + requestPath + body))
I found also a java client example on the official github, please see bellow signature generation in java :
private String generateServerDigest(String method, String uri, long timestamp, String body) {
//return timestamp + request.getMethodValue() + uri + body;
String checkDigestString = timestamp + method + uri + body;// "GET+/v1/daa-list+123123123"; //timestamp in epoch milliseconds
// hash server composited digest with algorithm and apikeys secret
SecretKeySpec signingKey = new SecretKeySpec(apiSecret.getBytes(), "HmacSHA512");
Mac mac;
try {
mac = Mac.getInstance(signingKey.getAlgorithm());
mac.init(signingKey);
} catch (NoSuchAlgorithmException | InvalidKeyException e) {
log.warn("Could not ={}", signingKey.getAlgorithm());
return null;
}
return Base64.getEncoder().encodeToString(mac.doFinal(checkDigestString.getBytes()));
}
Please note that checkDigestString code timestamp + method + uri + body and comment GET+/v1/daa-list+123123123 are already different on the official doc.
And this is my python implementation attempt :
def sign(timestamp,method,requestPath,body):
global api_secret
base64_decoded_secret_key = base64.b64decode(api_secret)
content_to_hash = (str(timestamp) + method.upper() + requestPath + body).encode('utf-8')
sign_digest = hmac.new(base64_decoded_secret_key, content_to_hash , hashlib.sha512).digest()
return base64.b64encode(sign_digest).decode('utf-8')
When I try this signature method with requestPath = "/v1/user/balance" (which required to be authentificated), it fail without error...
Does any one used with both java and python may help me to convert this signature method to python ?
This code will work for GET:
import time,requests
import hashlib,hmac,base64
api_key = "my api key"
api_secret = "my api secret"
defaut_encoding = "utf8"
uri = "https://api.iconomi.com"
requestPath = "/v1/user/balance"
api_url_target = uri+requestPath # https://api.iconomi.com/v1/user/balance
method="GET"
body=""
icn_timestamp = int(1000.*time.time())
message = (str(icn_timestamp) + method.upper() + requestPath + body).encode(defaut_encoding)
signature_digest = hmac.new(api_secret.encode(defaut_encoding), message, hashlib.sha512).digest() #here digest is byte
b64_signature_digest= base64.b64encode(signature_digest).decode(defaut_encoding)
headers_sign= {
"ICN-API-KEY":api_key,
"ICN-SIGN":b64_signature_digest,
"ICN-TIMESTAMP":str(icn_timestamp)
}
s=requests.session()
res = s.get(api_url_target,headers=headers_sign,timeout=3, verify=True).content
print (res)
Update for #Karl comment, this code will work for POST:
import time,requests
import hashlib,hmac,base64,json
api_key = "my api key"
api_secret = "my api secret"
ticker = "my ticker strategy"
defaut_encoding = "utf8"
uri = "https://api.iconomi.com"
requestPath = "/v1/strategies/"+ticker+"/structure"
api_url_target = uri+requestPath # https://api.iconomi.com/v1/strategies/{my ticker strategy}/structure
method="POST"
body="{'ticker': ticker, 'values': [{'rebalancedWeight': 1., 'targetWeight':1., 'assetTicker': 'XMR', 'assetName': 'Monero', 'assetCategory': 'Privacy'}]}"
icn_timestamp = int(1000.*time.time())
message = (str(icn_timestamp) + method.upper() + requestPath + body).encode(defaut_encoding)
signature_digest = hmac.new(api_secret.encode(defaut_encoding), message, hashlib.sha512).digest() #here digest is byte
b64_signature_digest= base64.b64encode(signature_digest).decode(defaut_encoding)
headers_sign= {
"ICN-API-KEY":api_key,
"ICN-SIGN":b64_signature_digest,
"ICN-TIMESTAMP":str(icn_timestamp)
}
s=requests.session()
res = s.post(api_url_target,headers=headers_sign,json = json.loads(body), timeout=3, verify=True).content
print (res)
I'm trying to run simple search via Python SDK (Python 3.8.5, splunk-sdk 1.6.14). Examples that are presented on dev.splunk.com are clear but something goes wrong when I run search with my own parameters
The code is as simple as this
search_kwargs_params = {
"exec_mode": "blocking",
"earliest_time": "2020-09-04T06:57:00.000-00:00",
"latest_time": "2020-11-08T07:00:00.000-00:00",
}
search_query = 'search index=qwe1 trace=111-aaa-222 action=Event.OpenCase'
job = self.service.jobs.create(search_query, **search_kwargs_params)
for result in results.ResultsReader(job.results()):
print(result)
But search returns no results. When I run same query manually in Splunk web GUI it works fine.
I've also tried to put all parameters in 'search_kwargs_params' dictionary, widened search time period and got some search results but they seem to be inappropriate to what I got in GUI.
Can someone advise?
This worked for me. You may also try this:
import requests
import time
import json
scheme = 'https'
host = '<your host>'
username = '<your username>'
password = '<your password>'
unique_id = '2021-03-22T18-43-00' #You may give any unique identifier here
search_query = 'search <your splunk query>'
post_data = { 'id' : unique_id,
'search' : search_query,
'earliest_time' : '1',
'latest_time' : 'now',
}
#'earliest_time' : '1', 'latest_time' : 'now'
#This will run the search query for all time
splunk_search_base_url = scheme + '://' + host +
'/servicesNS/{}/search/search/jobs'.format(username)
resp = requests.post(splunk_search_base_url, data = post_data, verify = False, auth =
(username, password))
print(resp.text)
is_job_completed = ''
while(is_job_completed != 'DONE'):
time.sleep(5)
get_data = {'output_mode' : 'json'}
job_status_base_url = scheme + '://' + host +
'/servicesNS/{}/search/search/jobs/{}'.format(username, unique_id)
resp_job_status = requests.post(job_status_base_url, data = get_data, verify =
False, auth = (username, password))
resp_job_status_data = resp_job_status.json()
is_job_completed = resp_job_status_data['entry'][0]['content']['dispatchState']
print("Current job status is {}".format(is_job_completed))
splunk_summary_base_url = scheme + '://' + host +
'/servicesNS/{}/search/search/jobs/{}/results?count=0'.format(username, unique_id)
splunk_summary_results = requests.get(splunk_summary_base_url, data = get_data, verify
= False, auth = (username, password))
splunk_summary_data = splunk_summary_results.json()
#Print the results in python format (strings will be in single quotes)
for data in splunk_summary_data['results']:
print(data)
print('status code...')
print(splunk_summary_results.status_code)
print('raise for status...')
print(splunk_summary_results.raise_for_status())
print('Results as JSON : ')
#Print the results in valid JSON format (Strings will be in double quotes)
#To get complete json data:
print(json.dumps(splunk_summary_data))
#To get only the relevant json data:
print(json.dumps(splunk_summary_data['results']))
Cheers!
You may also like to have a look at this very handy tutorial. https://www.youtube.com/watch?v=mmTzzp2ldgU
I'm working on a project with Python(3.6) & Django(1.10) in which I need to create a function at Google cloud using API request.
How can upload code in the form of a zip archive while creating that function?
Here's what I have tried:
From views.py :
def post(self, request, *args, **kwargs):
if request.method == 'POST':
post_data = request.POST.copy()
post_data.update({'user': request.user.pk})
form = forms.SlsForm(post_data, request.FILES)
print('get post request')
if form.is_valid():
func_obj = form
func_obj.user = request.user
func_obj.project = form.cleaned_data['project']
func_obj.fname = form.cleaned_data['fname']
func_obj.fmemory = form.cleaned_data['fmemory']
func_obj.entryPoint = form.cleaned_data['entryPoint']
func_obj.sourceFile = form.cleaned_data['sourceFile']
func_obj.sc_github = form.cleaned_data['sc_github']
func_obj.sc_inline_index = form.cleaned_data['sc_inline_index']
func_obj.sc_inline_package = form.cleaned_data['sc_inline_package']
func_obj.bucket = form.cleaned_data['bucket']
func_obj.save()
service = discovery.build('cloudfunctions', 'v1', http=views.getauth(), cache_discovery=False)
requ = service.projects().locations().functions().generateUploadUrl(parent='projects/' + func_obj.project + '/locations/us-central1', body={})
resp = requ.execute()
print(resp)
try:
auth = views.getauth()
# Prepare Request Body
req_body = {
"CloudFunction": {
"name": func_obj.fname,
"entryPoint": func_obj.entryPoint,
"timeout": '60s',
"availableMemoryMb": func_obj.fmemory,
"sourceArchiveUrl": func_obj.sc_github,
},
"sourceUploadUrl": func_obj.bucket,
}
service = discovery.build('cloudfunctions', 'v1beta2', http=auth, cachce_dicovery=False)
func_req = service.projects().locations().functions().create(location='projects/' + func_obj.project
+ '/locations/-',
body=req_body)
func_res = func_req.execute()
print(func_res)
return HttpResponse('Submitted',)
except:
return HttpResponse(status=500)
return HttpResponse('Sent!')
Updated Code below:
if form.is_valid():
func_obj = form
func_obj.user = request.user
func_obj.project = form.cleaned_data['project']
func_obj.fname = form.cleaned_data['fname']
func_obj.fmemory = form.cleaned_data['fmemory']
func_obj.entryPoint = form.cleaned_data['entryPoint']
func_obj.sourceFile = form.cleaned_data['sourceFile']
func_obj.sc_github = form.cleaned_data['sc_github']
func_obj.sc_inline_index = form.cleaned_data['sc_inline_index']
func_obj.sc_inline_package = form.cleaned_data['sc_inline_package']
func_obj.bucket = form.cleaned_data['bucket']
func_obj.save()
#######################################################################
# FIRST APPROACH FOR FUNCTION CREATION USING STORAGE BUCKET
#######################################################################
file_name = os.path.join(IGui.settings.BASE_DIR, 'media/archives/', func_obj.sourceFile.name)
print(file_name)
service = discovery.build('cloudfunctions', 'v1')
func_api = service.projects().locations().functions()
url_svc_req = func_api.generateUploadUrl(parent='projects/'
+ func_obj.project
+ '/locations/us-central1',
body={})
url_svc_res = url_svc_req.execute()
print(url_svc_res)
upload_url = url_svc_res['uploadUrl']
print(upload_url)
headers = {
'content-type': 'application/zip',
'x-goog-content-length-range': '0,104857600'
}
print(requests.put(upload_url, headers=headers, data=func_obj.sourceFile.name))
auth = views.getauth()
# Prepare Request Body
name = "projects/{}/locations/us-central1/functions/{}".format(func_obj.project, func_obj.fname,)
print(name)
req_body = {
"name": name,
"entryPoint": func_obj.entryPoint,
"timeout": "3.5s",
"availableMemoryMb": func_obj.fmemory,
"sourceUploadUrl": upload_url,
"httpsTrigger": {},
}
service = discovery.build('cloudfunctions', 'v1')
func_api = service.projects().locations().functions()
response = func_api.create(location='projects/' + func_obj.project + '/locations/us-central1',
body=req_body).execute()
pprint.pprint(response)
Now the function has been created successfully, but it fails because the source code doesn't upload to storage bucket, that's maybe something wrong at:
upload_url = url_svc_res['uploadUrl']
print(upload_url)
headers = {
'content-type': 'application/zip',
'x-goog-content-length-range': '0,104857600'
}
print(requests.put(upload_url, headers=headers, data=func_obj.sourceFile.name))
In the request body you have a dictionary "CloudFunction" inside the request. The content of "CloudFunction" should be directly in request.
request_body = {
"name": parent + '/functions/' + name,
"entryPoint": entry_point,
"sourceUploadUrl": upload_url,
"httpsTrigger": {}
}
I recomend using "Try this API" to discover the structure of projects.locations.functions.create .
"sourceArchiveUrl" and "sourceUploadUrl" can't appear together. This is explained in Resorce Cloud Function:
// Union field source_code can be only one of the following:
"sourceArchiveUrl": string,
"sourceRepository": { object(SourceRepository) },
"sourceUploadUrl": string,
// End of list of possible types for union field source_code.
In the rest of the answer I assume that you want to use "sourceUploadUrl". It requires you to pass it a URL returned to you by .generateUploadUrl(...).execute(). See documentation:
sourceUploadUrl -> string
The Google Cloud Storage signed URL used for source uploading,
generated by [google.cloud.functions.v1.GenerateUploadUrl][]
But before passing it you need to upload a zip file to this URL:
curl -X PUT "${URL}" -H 'content-type:application/zip' -H 'x-goog-content-length-range: 0,104857600' -T test.zip
or in python:
headers = {
'content-type':'application/zip',
'x-goog-content-length-range':'0,104857600'
}
print(requests.put(upload_url, headers=headers, data=data))
This is the trickiest part:
the case matters and it should be lowercase. Because the signature is calculated from a hash (here)
you need 'content-type':'application/zip'. I deduced this one logically, because documentation doesn't mention it. (here)
x-goog-content-length-range: min,max is obligatory for all PUT requests for cloud storage and is assumed implicitly in this case. More on it here
104857600, the max in previous entry, is a magical number which I didn't found mentioned anywhere.
where data is a FileLikeObject.
I also assume that you want to use the httpsTrigger. For a cloud function you can only choose one trigger field. Here it's said that trigger is a Union field. For httpsTrigger however that you can just leave it to be an empty dictionary, as its content do not affect the outcome. As of now.
request_body = {
"name": parent + '/functions/' + name,
"entryPoint": entry_point,
"sourceUploadUrl": upload_url,
"httpsTrigger": {}
}
You can safely use 'v1' instead of 'v1beta2' for .create().
Here is a full working example. It would be to complicated if I presented it to you as part of your code, but you can easily integrate it.
import pprint
import zipfile
import requests
from tempfile import TemporaryFile
from googleapiclient import discovery
project_id = 'your_project_id'
region = 'us-central1'
parent = 'projects/{}/locations/{}'.format(project_id, region)
print(parent)
name = 'ExampleFunctionFibonacci'
entry_point = "fibonacci"
service = discovery.build('cloudfunctions', 'v1')
CloudFunctionsAPI = service.projects().locations().functions()
upload_url = CloudFunctionsAPI.generateUploadUrl(parent=parent, body={}).execute()['uploadUrl']
print(upload_url)
payload = """/**
* Responds to any HTTP request that can provide a "message" field in the body.
*
* #param {Object} req Cloud Function request context.
* #param {Object} res Cloud Function response context.
*/
exports.""" + entry_point + """= function """ + entry_point + """ (req, res) {
if (req.body.message === undefined) {
// This is an error case, as "message" is required
res.status(400).send('No message defined!');
} else {
// Everything is ok
console.log(req.body.message);
res.status(200).end();
}
};"""
with TemporaryFile() as data:
with zipfile.ZipFile(data, 'w', zipfile.ZIP_DEFLATED) as archive:
archive.writestr('function.js', payload)
data.seek(0)
headers = {
'content-type':'application/zip',
'x-goog-content-length-range':'0,104857600'
}
print(requests.put(upload_url, headers=headers, data=data))
# Prepare Request Body
# https://cloud.google.com/functions/docs/reference/rest/v1/projects.locations.functions#resource-cloudfunction
request_body = {
"name": parent + '/functions/' + name,
"entryPoint": entry_point,
"sourceUploadUrl": upload_url,
"httpsTrigger": {},
"runtime": 'nodejs8'
}
print('https://{}-{}.cloudfunctions.net/{}'.format(region,project_id,name))
response = CloudFunctionsAPI.create(location=parent, body=request_body).execute()
pprint.pprint(response)
Open and upload a zip file like following:
file_name = os.path.join(IGui.settings.BASE_DIR, 'media/archives/', func_obj.sourceFile.name)
headers = {
'content-type': 'application/zip',
'x-goog-content-length-range': '0,104857600'
}
with open(file_name, 'rb') as data:
print(requests.put(upload_url, headers=headers, data=data))
Is there a way to automate checking Google Page Speed scores?
So I figured out how to do this using the Google Page Speed API buried in the documentation.
The TL:DR explanation is you can use the following URL setup, replacing the bracketed values after enabling Google Pagespeed API in Cloud Console (and if in a browser you must also have Authenticated Google Cloud User Signed In).
https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url={YOUR_SITE_URL}/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key={YOUR_API_KEY}
As you can see from the link above you will need a Google Pagespeed API Key. These are from scratch setup instructions. If your project is already on Cloud Console you can skip the first few steps.
Go to Cloud https://console.developers.google.com
Sign up for an account if necessary.
Create a Project
Go to Menu > API Manager > Credentials
Create credentials button > API Key
Copy Key then hit close
Menu > Dashboard > Enable API
Use Search to find PageSpeed insights API and click on it
Use the “|> ENABLE” button near the title
Once you have an API Key you can replace the values in the URL, it should look something like this:
https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url=https://www.example.com/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key=FaKeApIKey29nS8U22mM
The parameter strategy=desktop in the url can be changed to strategy=mobile. For mobile you get a speed and a usability score. Here's what the start of the JSON looks like:
{
"kind": "pagespeedonline#result",
"id": "https://www.example.com/fake/page”,
"responseCode": 200,
"title": "Example Domain",
"ruleGroups": {
"SPEED": {
"score": 100
},
"USABILITY": {
"score": 100
}
},
....continued...
So I automated this using a Python & Python Unit Test.
import requests
import json
import unittest
from globes import *
api_key = '' # Add API key. Found here: https://console.developers.google.com/apis/credentials/key/
base = 'http://example.com'
locale_code = 'en_US'
def get_insights_json(self, page_url, local, device_type, api_key, speed_or_useability, expected_score):
url = 'https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url=' + page_url + '&filter_third_party_resources=true&locale=' + local + '&screenshot=false&strategy=' + device_type + '&key=' + api_key
# print "Getting :: " + url
r = requests.get(url)
return_code = r.status_code
try: self.assertEqual(return_code, 200)
except AssertionError, e: self.verificationErrors.append(str(page_url) + " did not return 200")
return_text = r.text
return_json = json.loads(return_text)
score = return_json['ruleGroups'][speed_or_useability]['score']
print 'Getting ' + speed_or_useability + ' for ' + page_url + ' and got a score of ' + str(score)
try: self.assertTrue(int(score) >= expected_score)
except AssertionError, e: self.verificationErrors.append(str(page_url) + ' expected ' + device_type + ' speed score to be greater than ' + str(expected_score) + ', instead got ' + str(score) )
class TestAllAPIs(unittest.TestCase):
def setUp(self):
self.verificationErrors = []
self.maxDiff = None
def tearDown(self):
self.assertEqual([], self.verificationErrors)
def test_desktop_speed(self):
current_page = base + '' # You could add to the url to test other pages, I tend to do this is a loop using a list I set up by base url.
device_type = 'desktop'
target = 'SPEED'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
def test_mobile_speed(self):
current_page = base + ''
device_type = 'mobile'
target = 'SPEED'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
def test_mobile_useability(self):
current_page = base + ''
device_type = 'mobile'
target = 'USABILITY'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
if __name__ == "__main__":
unittest.main()
There is one thing that's a little of a mystery to me is why browsers need to be authenticated with Google to get JSON from URL but Python requests does not.
Anybody using this guide (as of April 2022) will need to update to the following:
https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url={YOUR_SITE_URL}/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key={YOUR_API_KEY}
The difference is the "/v2/" needs to be replaced with "/v5/"