We Keep Coding

Cloudwatch Insight Query Lambda

I have a problem, as I trying to automate the cloudwatch Logs insight query. I found this code
def getInsightLogCounts():
print("starting cloudwatch insight queries")
msg = ''
#STEP 1 - create a dict with query message as key and query as value. This can be stored in the db as well
query_Dict = getQueryDict()
log_group = '/the/loggroup/you/want/to/'
print("starting cloudwatch insight queries for " + str(log_group))
for x in query_Dict.items():
query_key = x[0]
query = x[1]
print("query key : " + str(query_key) + " \n query : " + str(query))
#STEP 2 - Create a query response object using the start_query method of the logs. Here we are fetching data for the last 24 hours
start_query_response = logs_client.start_query(
logGroupName=log_group,
queryString=query,
startTime=int((datetime.today() - timedelta(hours=24)).timestamp()),
endTime=int(datetime.now().timestamp()),
limit=1)
query_id = start_query_response['queryId']
response = None
#STEP3 - run a while loop and waith for the query to complete.
while response == None or response['status'] == 'Running':
time.sleep(1)
response = logs_client.get_query_results(queryId=query_id)
#STEP4 - Extract the result and proceed to the next query
if response and response['results'] and response['results'][0] and response['results'][0][1]:
#response found update msg
msg = msg + str(query_key) + " : " + str(response['results'][0][1].get('value')) + "; \n"
print("query value returned for " + str(query_key) + " is : " + str(response['results'][0][1].get('value')))
else:
msg = msg + str(query_key) + " : 0" + "; \n"
print("no query data returned for " + str(query_key))
return msg
Now I need to fit my query into this code, but I'm new to the Lambda (Python) codes.
My query is (sample one)
fields #timestamp, #message
| filter #message like 'END RequestId'
| sort #timestamp desc
If anyone has some ideas I would be very grateful if you can help me with this or just give me some advice
this is a link for the original post I took the code from.
https://medium.com/xebia-engineering/accessing-cloudwatch-metrics-and-insights-from-aws-lambda-1119c40ff80b#5833
Sorry guys, for the not clear post before. I manage to fit my code here so hopefully, someone with a big heart can help me

Python - How do I get and use the data from firebase?

Here is my updated code with the revisions. I only added the user_num = 1 to try to avoid the unsupported operand error I was getting , but that didn't work either.
.py
class MetropolisApp(App):
user_num = 1
database_url = 'https://metropolis-58211.firebaseio.com/'
def build(self):
self.my_firebase = MyFirebase()
return ControlScreens()
def on_start(self):
user_num_req = requests.get("https://metropolis-58211.firebaseio.com/user_num.json")
self.user_num = user_num_req.json()
db = requests.get('https://metropolis-58211.firebaseio.com/' + str(self.user_num) + '.json')
#print(db.ok)
data = json.loads(db.content.decode())
#print(data)
def post(self, firstname, lastname, email, username, password):
self.user_num += 1
# Make a python dictionary
fname_data = {"fname": firstname}
lname_data = {"lname": lastname}
email_data = {"email": email}
user_id = {"userid": username}
user_pw = {"userpw": password}
# Send the python dictionary
requests.post(url=self.database_url + str(self.user_num) + '.json', json=fname_data)
requests.post(url=self.database_url + str(self.user_num) + '.json', json=lname_data)
requests.post(url=self.database_url + str(self.user_num) + '.json', json=email_data)
requests.post(url=self.database_url + str(self.user_num) + '.json', json=user_id)
requests.post(url=self.database_url + str(self.user_num) + '.json', json=user_pw)
def in_database(self, firstname, lastname, email, username, password):
fname_data = {"fname": firstname}
lname_data = {"lname": lastname}
email_data = {"email": email}
user_id = {"userid": username}
user_pw = {"userpw": password}
requests.patch(url=self.database_url + str(self.user_num) + '.json', json=fname_data)
requests.patch(url=self.database_url + str(self.user_num) + '.json', json=lname_data)
requests.patch(url=self.database_url + str(self.user_num) + '.json', json=email_data)
requests.patch(url=self.database_url + str(self.user_num) + '.json', json=user_id)
requests.patch(url=self.database_url + str(self.user_num) + '.json', json=user_pw)
check_request = requests.get(
'https://metropolis-58211.firebaseio.com/.json?orderBy="userid"&equalTo="%s"' % self.root.ids[
'user_login'].text)
print(check_request.json())
# print(self.root.ids)
Not sure if the .kv file is needed, but here is the main code in which I am having errors.
Edited Database

There are a couple concepts to understand here. The first is the rule system of Firebase's real-time database. The rule system dictates what kind of reads/writes to the database are allowed. An important part of the rule system is using indexes to tell Firebase that you want to be able to query your data by a certain parameter. For example, you need to query your database by the userid parameter. In your rules tab of your real-time database, you need to tell Firebase to let you 'indexOn' the 'userid' parameter, like so:
{
"rules": {
".read": true, // Note - this allows all reads from anyone anywhere. only use for testing
".write": true, // same comment
".indexOn": ["userid"] // You can allow multiple indexOns by doing ["userid","foo","bar"]
}
}
The ".indexOn" portion above is part of your top level rules (/ in your database), and you want to index on data that is inside some special identifier (/special_identifier/userid). The special identifier for your test user is 2 in this case.
Now that you are allowing your database to be queried by the userid parameter, you need to actually send a request using python to query it. The next concept here is sending some special arguments in the URL of your get request (read Firebase's documentation here). In this case, you need to try to get data from your top level (/) location, and orderBy userid and only return values equalTo self.root.ids['user_login'].text
Your URL in this case would be:
'https://metropolis-58211.firebaseio.com/.json?orderBy="userid"&equalTo="%s"'%self.root.ids['user_login'].text
Notice how you use the ? character to specify you're going to send some special parameters, and the & means another special parameter is coming. Also make sure your values that you are ordering-by or equaling-to are surrounded in " (assuming their strings in your database, which they are at the moment).
When you do a get request to that URL, either it will return the data for that user, or an empty dictionary.
check_request = requests.get('https://metropolis-58211.firebaseio.com/.json?orderBy="userid"&equalTo="%s"'%self.root.ids['user_login'].text)
print(check_request.json()) # If this has data, the user exists, if it's an empty dictionary, {}, the user doesn't exist

Check if Text Message contains a Word and if yes send a CustomText

I use a working python script to Check Crypto Coin Price drops, If a Coin drops -3% or more I get a message in Telegram and the same Message is sent to my Email.
Now I want to send a Custom Text for every Market like BTC/ETH/USDT and so on...
I want with python to Check the message and If Message Contains a price drop from BTC Market send text1
ETH Market send text2
USDT Market send text3
The Bot gets the CoinName like this Example: IOTA/BTC < In this case I need the bot to see the word BTC and Send text1. I tryed some things but can't figure it out.
Here the section where the text for the Message is,
`bot.send_message(parse_mode='HTML', chat_id=BINANCE_ID,
text = "<a href = 'https://www.binance.com/tradeDetail.html?symbol {C}_{M}'> {Coin} </a> ".format(C = CoinItself,
M = Market,
Coin = Coin)
+ "▼" + str(RoundPercent) + "% "
+ str(floatDictPrice) + " -> "
+ str(floatDataPrice) + " [Binance]")
binance_prices.update( { CurrentCoin: dataPrice })
send_message("<a href = 'https://www.binance.com/tradeDetail.html?symbol={C}_{M}'> {Coin} </a> ".format(C = CoinItself,
M = Market,
Coin = Coin)
+ "▼" + str(RoundPercent) + "% "
+ str(floatDictPrice) + " -> "
+ str(floatDataPrice) + " [Binance]")`
The bot.send_message section sends the message to telegram & the send_message section sends the same Message text to my Email.
`search_word = input("BTC")
if(search_word == send_message):
print("text1")
else:
print("word not found")`
I tried this one. Any Ideas how to get this working?

What AChampion means is the following:
if search_word in send_message:
print("text1")
else:
print("word not found")

How to automate Google PageSpeed Insights tests using Python

Is there a way to automate checking Google Page Speed scores?

So I figured out how to do this using the Google Page Speed API buried in the documentation.
The TL:DR explanation is you can use the following URL setup, replacing the bracketed values after enabling Google Pagespeed API in Cloud Console (and if in a browser you must also have Authenticated Google Cloud User Signed In).
https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url={YOUR_SITE_URL}/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key={YOUR_API_KEY}
As you can see from the link above you will need a Google Pagespeed API Key. These are from scratch setup instructions. If your project is already on Cloud Console you can skip the first few steps.
Go to Cloud https://console.developers.google.com
Sign up for an account if necessary.
Create a Project
Go to Menu > API Manager > Credentials
Create credentials button > API Key
Copy Key then hit close
Menu > Dashboard > Enable API
Use Search to find PageSpeed insights API and click on it
Use the “|> ENABLE” button near the title
Once you have an API Key you can replace the values in the URL, it should look something like this:
https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url=https://www.example.com/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key=FaKeApIKey29nS8U22mM
The parameter strategy=desktop in the url can be changed to strategy=mobile. For mobile you get a speed and a usability score. Here's what the start of the JSON looks like:
{
"kind": "pagespeedonline#result",
"id": "https://www.example.com/fake/page”,
"responseCode": 200,
"title": "Example Domain",
"ruleGroups": {
"SPEED": {
"score": 100
},
"USABILITY": {
"score": 100
}
},
....continued...
So I automated this using a Python & Python Unit Test.
import requests
import json
import unittest
from globes import *
api_key = '' # Add API key. Found here: https://console.developers.google.com/apis/credentials/key/
base = 'http://example.com'
locale_code = 'en_US'
def get_insights_json(self, page_url, local, device_type, api_key, speed_or_useability, expected_score):
url = 'https://www.googleapis.com/pagespeedonline/v2/runPagespeed?url=' + page_url + '&filter_third_party_resources=true&locale=' + local + '&screenshot=false&strategy=' + device_type + '&key=' + api_key
# print "Getting :: " + url
r = requests.get(url)
return_code = r.status_code
try: self.assertEqual(return_code, 200)
except AssertionError, e: self.verificationErrors.append(str(page_url) + " did not return 200")
return_text = r.text
return_json = json.loads(return_text)
score = return_json['ruleGroups'][speed_or_useability]['score']
print 'Getting ' + speed_or_useability + ' for ' + page_url + ' and got a score of ' + str(score)
try: self.assertTrue(int(score) >= expected_score)
except AssertionError, e: self.verificationErrors.append(str(page_url) + ' expected ' + device_type + ' speed score to be greater than ' + str(expected_score) + ', instead got ' + str(score) )
class TestAllAPIs(unittest.TestCase):
def setUp(self):
self.verificationErrors = []
self.maxDiff = None
def tearDown(self):
self.assertEqual([], self.verificationErrors)
def test_desktop_speed(self):
current_page = base + '' # You could add to the url to test other pages, I tend to do this is a loop using a list I set up by base url.
device_type = 'desktop'
target = 'SPEED'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
def test_mobile_speed(self):
current_page = base + ''
device_type = 'mobile'
target = 'SPEED'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
def test_mobile_useability(self):
current_page = base + ''
device_type = 'mobile'
target = 'USABILITY'
get_insights_json(self, current_page, locale_code, device_type, api_key, target, 80)
if __name__ == "__main__":
unittest.main()
There is one thing that's a little of a mystery to me is why browsers need to be authenticated with Google to get JSON from URL but Python requests does not.

Anybody using this guide (as of April 2022) will need to update to the following:
https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url={YOUR_SITE_URL}/&filter_third_party_resources=true&locale=en_US&screenshot=false&strategy=desktop&key={YOUR_API_KEY}
The difference is the "/v2/" needs to be replaced with "/v5/"

Security header is not valid (python)

I am trying to use the TransactionSearch API to bring up merchant transaction history. I have been stuck with a "security header is not valid error" and wasn't able to find a solution in the other posts. My code works fine on the sandbox server and I generated live credentials+signature from PayPal. I assume that if they gave me credentials that I have permission to access the API.
My python code is as follows:
import requests
def getTransactionHistory(start_datetime, end_datetime):
headers = {'X-PAYPAL-SECURITY-USERID' : api_username, 'X-PAYPAL-SECURITY-PASSWORD' : api_password,
'X-PAYPAL-SECURITY-SIGNATURE' : api_sig}
data = 'USER=' + api_username + '&PWD=' + api_password + '&SIGNATURE=' + api_sig + '&METHOD=' + \
'TransactionSearch' + '&STARTDATE=' + start_datetime + '&ENDDATE=' + end_datetime + \
'&VERSION=94'
print data
req = requests.post(base+nvp_point, data=data)
return req.text
r = getTransactionHistory('2012-01-01T05:38:48Z', '2012-01-02T05:38:48Z')

The above code is correct. There was a '-' in my signature or password, which was not copy-pasted correctly. Solved.