Is there a way to automate checking Google Page Speed scores?
So I figured out how to do this using the Google Page Speed API buried in the documentation.
The TL:DR explanation is you can use the following URL setup, replacing the bracketed values after enabling Google Pagespeed API in Cloud Console (and if in a browser you must also have Authenticated Google Cloud User Signed In).
https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url={YOUR_SITE_URL}/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key={YOUR_API_KEY}
As you can see from the link above you will need a Google Pagespeed API Key. These are from scratch setup instructions. If your project is already on Cloud Console you can skip the first few steps.
Go to Cloud https://console.developers.google.com
Sign up for an account if necessary.
Create a Project
Go to Menu > API Manager > Credentials
Create credentials button > API Key
Copy Key then hit close
Menu > Dashboard > Enable API
Use Search to find PageSpeed insights API and click on it
Use the “|> ENABLE” button near the title
Once you have an API Key you can replace the values in the URL, it should look something like this:
https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url=https://www.example.com/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key=FaKeApIKey29nS8U22mM
The parameter strategy=desktop in the url can be changed to strategy=mobile. For mobile you get a speed and a usability score. Here's what the start of the JSON looks like:
{
"kind": "pagespeedonline#result",
"id": "https://www.example.com/fake/page”,
"responseCode": 200,
"title": "Example Domain",
"ruleGroups": {
"SPEED": {
"score": 100
},
"USABILITY": {
"score": 100
}
},
....continued...
So I automated this using a Python & Python Unit Test.
import requests
import json
import unittest
from globes import *
api_key = '' # Add API key. Found here: https://console.developers.google.com/apis/credentials/key/
base = 'http://example.com'
locale_code = 'en_US'
def get_insights_json(self, page_url, local, device_type, api_key, speed_or_useability, expected_score):
url = 'https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url=' + page_url + '&filter_third_party_resources=true&locale=' + local + '&screenshot=false&strategy=' + device_type + '&key=' + api_key
# print "Getting :: " + url
r = requests.get(url)
return_code = r.status_code
try: self.assertEqual(return_code, 200)
except AssertionError, e: self.verificationErrors.append(str(page_url) + " did not return 200")
return_text = r.text
return_json = json.loads(return_text)
score = return_json['ruleGroups'][speed_or_useability]['score']
print 'Getting ' + speed_or_useability + ' for ' + page_url + ' and got a score of ' + str(score)
try: self.assertTrue(int(score) >= expected_score)
except AssertionError, e: self.verificationErrors.append(str(page_url) + ' expected ' + device_type + ' speed score to be greater than ' + str(expected_score) + ', instead got ' + str(score) )
class TestAllAPIs(unittest.TestCase):
def setUp(self):
self.verificationErrors = []
self.maxDiff = None
def tearDown(self):
self.assertEqual([], self.verificationErrors)
def test_desktop_speed(self):
current_page = base + '' # You could add to the url to test other pages, I tend to do this is a loop using a list I set up by base url.
device_type = 'desktop'
target = 'SPEED'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
def test_mobile_speed(self):
current_page = base + ''
device_type = 'mobile'
target = 'SPEED'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
def test_mobile_useability(self):
current_page = base + ''
device_type = 'mobile'
target = 'USABILITY'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
if __name__ == "__main__":
unittest.main()
There is one thing that's a little of a mystery to me is why browsers need to be authenticated with Google to get JSON from URL but Python requests does not.
Anybody using this guide (as of April 2022) will need to update to the following:
https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url={YOUR_SITE_URL}/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key={YOUR_API_KEY}
The difference is the "/v2/" needs to be replaced with "/v5/"
Related
I'm trying to run simple search via Python SDK (Python 3.8.5, splunk-sdk 1.6.14). Examples that are presented on dev.splunk.com are clear but something goes wrong when I run search with my own parameters
The code is as simple as this
search_kwargs_params = {
"exec_mode": "blocking",
"earliest_time": "2020-09-04T06:57:00.000-00:00",
"latest_time": "2020-11-08T07:00:00.000-00:00",
}
search_query = 'search index=qwe1 trace=111-aaa-222 action=Event.OpenCase'
job = self.service.jobs.create(search_query, **search_kwargs_params)
for result in results.ResultsReader(job.results()):
print(result)
But search returns no results. When I run same query manually in Splunk web GUI it works fine.
I've also tried to put all parameters in 'search_kwargs_params' dictionary, widened search time period and got some search results but they seem to be inappropriate to what I got in GUI.
Can someone advise?
This worked for me. You may also try this:
import requests
import time
import json
scheme = 'https'
host = '<your host>'
username = '<your username>'
password = '<your password>'
unique_id = '2021-03-22T18-43-00' #You may give any unique identifier here
search_query = 'search <your splunk query>'
post_data = { 'id' : unique_id,
'search' : search_query,
'earliest_time' : '1',
'latest_time' : 'now',
}
#'earliest_time' : '1', 'latest_time' : 'now'
#This will run the search query for all time
splunk_search_base_url = scheme + '://' + host +
'/servicesNS/{}/search/search/jobs'.format(username)
resp = requests.post(splunk_search_base_url, data = post_data, verify = False, auth =
(username, password))
print(resp.text)
is_job_completed = ''
while(is_job_completed != 'DONE'):
time.sleep(5)
get_data = {'output_mode' : 'json'}
job_status_base_url = scheme + '://' + host +
'/servicesNS/{}/search/search/jobs/{}'.format(username, unique_id)
resp_job_status = requests.post(job_status_base_url, data = get_data, verify =
False, auth = (username, password))
resp_job_status_data = resp_job_status.json()
is_job_completed = resp_job_status_data['entry'][0]['content']['dispatchState']
print("Current job status is {}".format(is_job_completed))
splunk_summary_base_url = scheme + '://' + host +
'/servicesNS/{}/search/search/jobs/{}/results?count=0'.format(username, unique_id)
splunk_summary_results = requests.get(splunk_summary_base_url, data = get_data, verify
= False, auth = (username, password))
splunk_summary_data = splunk_summary_results.json()
#Print the results in python format (strings will be in single quotes)
for data in splunk_summary_data['results']:
print(data)
print('status code...')
print(splunk_summary_results.status_code)
print('raise for status...')
print(splunk_summary_results.raise_for_status())
print('Results as JSON : ')
#Print the results in valid JSON format (Strings will be in double quotes)
#To get complete json data:
print(json.dumps(splunk_summary_data))
#To get only the relevant json data:
print(json.dumps(splunk_summary_data['results']))
Cheers!
You may also like to have a look at this very handy tutorial. https://www.youtube.com/watch?v=mmTzzp2ldgU
I want to get the output of a Job or its status using Python in Azure automation. I can do it in powershell but I want to do it in Python.Is there any equvalent SDK for -AzureRmAutomationJob in python??
Here is the sample code to create a sample Azure automation task and getting the output of the job.
"""
import time
import uuid
import requests
import automationassets
# Automation resource group and account to start runbook job in
_AUTOMATION_RESOURCE_GROUP = "contoso"
_AUTOMATION_ACCOUNT = "contosodev"
# Set up required body values for a runbook.
# Make sure you have a hello_world_python runbook published in the automation account
# with an argument of -n
body = {
"properties":
{
"runbook":
{
"name":"hello_world_python"
},
"parameters":
{
"[PARAMETER 1]":"-n",
"[PARAMETER 2]":"world"
}
}
}
# Return token based on Azure automation Runas connection
def get_automation_runas_token(runas_connection):
""" Returs a token that can be used to authenticate against Azure resources """
from OpenSSL import crypto
import adal
# Get the Azure Automation RunAs service principal certificate
cert = automationassets.get_automation_certificate("AzureRunAsCertificate")
sp_cert = crypto.load_pkcs12(cert)
pem_pkey = crypto.dump_privatekey(crypto.FILETYPE_PEM, sp_cert.get_privatekey())
# Get run as connection information for the Azure Automation service principal
application_id = runas_connection["ApplicationId"]
thumbprint = runas_connection["CertificateThumbprint"]
tenant_id = runas_connection["TenantId"]
# Authenticate with service principal certificate
resource = "https://management.core.windows.net/"
authority_url = ("https://login.microsoftonline.com/" + tenant_id)
context = adal.AuthenticationContext(authority_url)
azure_credential = context.acquire_token_with_client_certificate(
resource,
application_id,
pem_pkey,
thumbprint)
# Return the token
return azure_credential.get('accessToken')
# Authenticate to Azure using the Azure Automation RunAs service principal
automation_runas_connection = automationassets.get_automation_connection("AzureRunAsConnection")
access_token = get_automation_runas_token(automation_runas_connection)
# Set what resources to act against
subscription_id = str(automation_runas_connection["SubscriptionId"])
job_id = str(uuid.uuid4())
# Set up URI to create a new automation job
uri = ("https://management.azure.com/subscriptions/" + subscription_id
+ "/resourceGroups/" + _AUTOMATION_RESOURCE_GROUP
+ "/providers/Microsoft.Automation/automationAccounts/" + _AUTOMATION_ACCOUNT
+ "/jobs/(" + job_id + ")?api-version=2015-10-31")
# Make request to create new automation job
headers = {"Authorization": 'Bearer ' + access_token}
json_output = requests.put(uri, json=body, headers=headers).json()
# Get results of the automation job
_RETRY = 360 # stop after 60 minutes (360 * 10 sleep seconds / 60 seconds in a minute)
_SLEEP_SECONDS = 10
status_counter = 0
while status_counter < _RETRY:
status_counter = status_counter + 1
job = requests.get(uri, headers=headers).json()
status = job['properties']['status']
if status == 'Completed' or status == 'Failed' or status == 'Suspended' or status == 'Stopped':
break
time.sleep(_SLEEP_SECONDS)
# if job did not complete in an hour, throw an exception
if status_counter == 360:
raise StandardError("Job did not complete in 60 minutes.")
if job['properties']['status'] != 'Completed':
raise StandardError("Job did not complete successfully.")
# Get output streams from the job
uri = ("https://management.azure.com/subscriptions/" + subscription_id
+ "/resourceGroups/" + _AUTOMATION_RESOURCE_GROUP
+ "/providers/Microsoft.Automation/automationAccounts/" + _AUTOMATION_ACCOUNT
+ "/jobs/" + job_id
+ "/streams?$filter=properties/streamType%20eq%20'Output'&api-version=2015-10-31")
job_streams = requests.get(uri, headers=headers).json()
# For each stream id, print out the text
for stream in job_streams['value']:
uri = ("https://management.azure.com/subscriptions/" + subscription_id
+ "/resourceGroups/" + _AUTOMATION_RESOURCE_GROUP
+ "/providers/Microsoft.Automation/automationAccounts/" + _AUTOMATION_ACCOUNT
+ "/jobs/" + job_id
+ "/streams/" + stream['properties']['jobStreamId']
+ "?$filter=properties/streamType%20eq%20'Output'&api-version=2015-10-31")
output_stream = requests.get(uri, headers=headers).json()
print output_stream['properties']['streamText']
You can refer this url for further reference.Hope it helps.
This is a very weird problem that I have got stuck into, will really appreciate if someone could provide some direction. I am trying to access value for request_url from web_token.py module.
when I only try to run web_token.py separately over pycharm and print request_url it works fine and generates the url. I zip both these files and upload it to lambda function but when testing it I get an error "Unable to import module 'retrieve_accounts': No module named boto.sts". I even tried putting the code of web_token.py inside retrieve_accounts.py but getting the same error. I am sure I am missing something very basic it looks boto.sts is not being recognized while running the python script. Can somebody please give some guidance. Thank You!
retrieve_accounts.py
import boto3
import web_token
def get_account(event, context):
client = boto3.client('dynamodb')
NameID = "testname#org.com"
ManagerEmail = "test1#eorg.com"
response = client.scan(
TableName='Sandbox-Users',
ScanFilter={
'NameID': {
'AttributeValueList': [
{
'S': NameID,
},
],
'ComparisonOperator': 'EQ'
}
}
)
if response["Count"] > 0:
client = boto3.client('dynamodb')
response = client.get_item(
Key={
'NameID': {
'S': NameID,
},
'ManagerEmail': {
'S': ManagerEmail,
},
},
TableName='Sandbox-Users',
)
return web_token.request_url ----------->here
else:
response = client.put_item(
Item={
'NameID': {
'S': NameID,
},
'ManagerEmail': {
'S': ManagerEmail,
}
},
TableName='Sandbox-Users'
)
return "Create Account"
web_token.py
import httplib
import urllib, json
from boto.sts import STSConnection -------->Error here
sts_connection = STSConnection()
assumed_role_object = sts_connection.assume_role(
role_arn="arn:aws:iam::454084028794:role/AMPSandboxRole",
role_session_name="AssumeRoleSession"
)
# Step 3: Format resulting temporary credentials into JSON
json_string_with_temp_credentials = '{'
json_string_with_temp_credentials += '"sessionId":"' +
assumed_role_object.credentials.access_key + '",'
json_string_with_temp_credentials += '"sessionKey":"' +
assumed_role_object.credentials.secret_key + '",'
json_string_with_temp_credentials += '"sessionToken":"' +
assumed_role_object.credentials.session_token + '"'
json_string_with_temp_credentials += '}'
# Step 4. Make request to AWS federation endpoint to get sign-in token.
Construct the parameter string with the sign-in action request, a 12-hour session duration, and the JSON
document with temporary credentials as parameters.
request_parameters = "?Action=getSigninToken"
request_parameters += "&SessionDuration=43200"
request_parameters += "&Session=" +
urllib.quote_plus(json_string_with_temp_credentials)
request_url = "/federation" + request_parameters
conn = httplib.HTTPSConnection("signin.aws.amazon.com")
conn.request("GET", request_url)
r = conn.getresponse()
# Returns a JSON document with a single element named SigninToken.
signin_token = json.loads(r.read())
request_parameters = "?Action=login"
request_parameters += "&Issuer=sandbox.com"
request_parameters += "&Destination=" +
urllib.quote_plus("https://console.aws.amazon.com/")
request_parameters += "&SigninToken=" + signin_token["SigninToken"]
request_url = "https://signin.aws.amazon.com/federation" +
request_parameters
AWS Lambda Python environments include boto3 (and botocore). They don't include the older boto (a precursor to boto3), hence the import failure.
You could potentially include boto in your upload but it's not advisable to mix boto and boto3 if you can avoid it. Use one or the other, preferably boto3.
I am trying to use the TransactionSearch API to bring up merchant transaction history. I have been stuck with a "security header is not valid error" and wasn't able to find a solution in the other posts. My code works fine on the sandbox server and I generated live credentials+signature from PayPal. I assume that if they gave me credentials that I have permission to access the API.
My python code is as follows:
import requests
def getTransactionHistory(start_datetime, end_datetime):
headers = {'X-PAYPAL-SECURITY-USERID' : api_username, 'X-PAYPAL-SECURITY-PASSWORD' : api_password,
'X-PAYPAL-SECURITY-SIGNATURE' : api_sig}
data = 'USER=' + api_username + '&PWD=' + api_password + '&SIGNATURE=' + api_sig + '&METHOD=' + \
'TransactionSearch' + '&STARTDATE=' + start_datetime + '&ENDDATE=' + end_datetime + \
'&VERSION=94'
print data
req = requests.post(base+nvp_point, data=data)
return req.text
r = getTransactionHistory('2012-01-01T05:38:48Z', '2012-01-02T05:38:48Z')
The above code is correct. There was a '-' in my signature or password, which was not copy-pasted correctly. Solved.
Context
I am writing a python/django application which accesses Fitbit data.
Problem
To access a users data I must get their token for which can be used repeatably to access fitness data. The following goes through the current steps.
1. Firstly I present the user with a link:
def register_2(request):
if request.user.is_authenticated():
oauth = OAuth2Session(
auth_2_client_id,
redirect_uri = redirect_uri,
scope = fitbit_scope)
authorization_url, state = oauth.authorization_url(fitbit_url_authorise_2)
return render_to_response('register.html',{
"authorization_url" : authorization_url
},context_instance=RequestContext(request))
return HttpResponseRedirect("/")
2. User goes to Fitbit, logs into their account and authorises access.
3. The user is then returned to my site with a code that should allow me to get the token.
def callback_2(request):
if request.user.is_authenticated():
code = request.GET.get('code')
state = request.GET.get('state')
oauth = OAuth2Session(
client_id = auth_2_client_id,
redirect_uri = redirect_uri
)
token = oauth.fetch_token(
code = code,
token_url = fitbit_url_access_2,
authorization_response = request.build_absolute_uri()
)
Once callback_2 is called I get the error:
(missing_token) Missing access token parameter.
Resources:
Fitbit OAUTH2 API
OAuth2Session Docs
Found a way around this. Pretty simple after all the effort of research. The following is a custom method using requests.post() and base64.b64encode().
def callback_2(request):
if request.user.is_authenticated():
code = request.GET.get('code')
state = request.GET.get('state')
url = fitbit_url_access_2
data = "client_id=" + auth_2_client_id + "&" +\
"grant_type=" + "authorization_code" + "&" +\
"redirect_uri=" + redirect_uri_special + "&" +\
"code=" + code
headers = {
'Authorization': 'Basic ' + base64.b64encode(auth_2_client_id + ':' + consumer_secret),
'Content-Type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=data, headers=headers).json()