We Keep Coding

Create Lambda - DynamoDB Count Function

I am creating a SAM web app, with the backend being an API in front of a Python Lambda function with a DynamoDB table that maintains a count of the number of HTTP calls to the API. The API must also return this number. The yaml code itself loads normally. My problem is writing the Lambda function to iterate and return the count. Here is my code:
def lambda_handler(event, context):
dynamodb = boto3.resource("dynamodb")
ddbTableName = os.environ["databaseName"]
table = dynamodb.Table(ddbTableName)
# Update item in table or add if doesn't exist
ddbResponse = table.update_item(
Key={"id": "VisitorCount"},
UpdateExpression="SET count = count + :value",
ExpressionAttributeValues={":value": Decimal(context)},
ReturnValues="UPDATED_NEW",
)
# Format dynamodb response into variable
responseBody = json.dumps({"VisitorCount": ddbResponse["Attributes"]["count"]})
# Create api response object
apiResponse = {"isBase64Encoded": False, "statusCode": 200, "body": responseBody}
# Return api response object
return apiResponse
I can get VisitorCount to be a string, but not a number. I get this error: [ERROR] TypeError: lambda_handler() missing 1 required positional argument: 'cou    response = request_handler(event, lambda_context)le_event_request
What is going on?
[UPDATE] I found the original error, which was that the function was not properly received by the SAM app. Changing the name fixed this, and it is now being read. Now I have to troubleshoot the actual Python. New Code:
import json
import boto3
import os
dynamodb = boto3.resource("dynamodb")
ddbTableName = os.environ["databaseName"]
table = dynamodb.Table(ddbTableName)
Key = {"VisitorCount": { "N" : "0" }}
def handler(event, context):
# Update item in table or add if doesn't exist
ddbResponse = table.update_item(
UpdateExpression= "set VisitorCount = VisitorCount + :val",
ExpressionAttributeValues={":val": {"N":"1"}},
ReturnValues="UPDATED_NEW",
)
# Format dynamodb response into variable
responseBody = json.dumps({"VisitorCount": ddbResponse["Attributes"]["count"]})
# Create api response object
apiResponse = {"isBase64Encoded": False, "statusCode": 200,"body": responseBody}
# Return api response object
return apiResponse
I am getting a syntax error on Line 13, which is
UpdateExpression= "set VisitorCount = VisitorCount + :val",
But I can't tell where I am going wrong on this. It should update the DynamoDB table to increase the count by 1. Looking at the AWS guide it appears to be the correct syntax.

Not sure what the exact error is but ddbResponse will be like this:
ddbResponse = table.update_item(
Key={
'key1': aaa,
'key2': bbb
},
UpdateExpression= "set VisitorCount = VisitorCount + :val",
ExpressionAttributeValues={":val": Decimal(1)},
ReturnValues="UPDATED_NEW",
)
Specify item to be updated with Key (one item for one Lambda call)
Set Decimal(1) for ExpressionAttributeValues
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.Python.03.html#GettingStarted.Python.03.04

Gmail api: How do I get the recipient's email using the ID?

I'm trying to get the recipient's email using the ID that I have retrieved. Below is the code.
messages = service.users().threads().list(userId='me', q='to:').execute().get('threads', [])
for message in messages:
#print(message)
# print(dir(message))
# print(message.get())
if search in message['snippet']:
#print(message['id'])
message_id = message['id']
# print(message_id)
full_message = service.users().messages().get(userId='me', id=message_id, format="raw").execute()
#print(full_message)
msg_headers = full_message['payload']['headers']
#print(msg_headers)
msg_str = base64.urlsafe_b64decode(full_message['raw'].encode('ASCII'))
mime_msg = email.message_from_bytes(msg_str)
#print(mime_msg)
x = re.findall('id=(\w+)\'', str(mime_msg))
print(x)
if len(x)> 0:
return x[0]
if msg_headers[x]['name'] == "To":
return msg_headers[x[0]]['value']
else:
return None
I get the error below. How do I resolve this?
Traceback (most recent call last):
File "C:\test\test2.py", line 102, in <module>
test = get_email_with("Test string")
File "C:\test\test2.py", line 55, in get_email_with
msg_headers = full_message['payload']['headers']
KeyError: 'payload'

Issue:
When the format is set to RAW, the field payload is not populated in the response.
From the API official docs:
RAW: Returns the full email message data with body content in the raw field as a base64url encoded string; the payload field is not used.
Solution:
Use either FULL or METADATA as format when calling messages().get, instead of RAW (format is an option parameter, and if it is not specified, FULL is used as default).
So you should change this:
full_message = service.users().messages().get(userId='me', id=message_id, format="raw").execute()
To this:
full_message = service.users().messages().get(userId='me', id=message_id).execute()
Or, alternatively, use format="raw" anyway and decode the base-64 encoded string full_message["raw"]. Edit: In this case, though, you'd have to remove this line: msg_headers = full_message['payload']['headers'].
Edit:
So, if you want to retrieve the recipient, you can do the following:
full_message = service.users().messages().get(userId='me', id=message_id).execute()
msg_headers = full_message['payload']['headers']
for header in msg_headers:
if header["name"] == "To":
return header["value"]
Reference:
API reference > Format

The response from Gmail can be a little confusing. You are on the right track with
msg_headers = full_message['payload']['headers']
One of the headers is called has a name of "Delivered-To" the value will be the email address that the mail was sent to. So basiclly you need to search your msg_headers and find the one called "Delivered-To"
Unfortunately i am not a python dev i cant help you with the code. Python quickstart may give you some ideas on how to get the response out of the object.
response
"payload": {
"partId": "",
"mimeType": "multipart/alternative",
"filename": "",
"headers": [
{
"name": "Delivered-To",
"value": "xxxx#gmail.com"
},

Python, Graphql Mutation and Dango API

I am working on a project that takes signups from a Django form and transfers them to a website, the info is mainly the data ( Name, surname, email...) and some extra-information (tags).
One of the scripts give me the following error in the cronjob_logs;
Traceback (most recent call last): File
"/usr/local/lib/python2.7/dist-packages/django_cron/management/commands/runcrons.py",
line 71, in run_cron_with_cache_check manager.run(force) File
"/usr/local/lib/python2.7/dist-packages/django_cron/init.py", line
215, in run self.msg = self.cron_job.do() File
"/home/django/django_project/ogx/cron.py", line 31, in do ep_id =
get_ep_id(ep.email) File
"/home/django/django_project/ogx/graph_helper.py", line 75, in
get_ep_id ''', {"query": email}) File
"/usr/local/lib/python2.7/dist-packages/graphqlclient/client.py", line
11, in execute return self._send(query, variables) File
"/usr/local/lib/python2.7/dist-packages/graphqlclient/client.py", line
34, in _send raise e HTTPError: HTTP Error 500: Internal Server Error
The script was working normally some time ago, as for the graph_helper.py it is as follows;
def get_ep_id(email):
client = GraphQLClient(
'*this part I took off for confidentiality*')
result = client.execute('''
query Myquery($query: String!){
allPeople(q:$query)
{
data
{
id
full_name
}
}
}
''', {"query": email})
data = json.loads(result)
if len(data['data']['allPeople']['data']) > 0:
return data['data']['allPeople']['data'][0]['id']
else:
return None
The cron.py in question is the following;
class FetchEPsIDs(CronJobBase):
RUN_EVERY_MINS = 30
schedule = Schedule(run_every_mins=RUN_EVERY_MINS)
code = 'ogx.FetchEPsIDs' # a unique code
def do(self):
eps_query = mc_ogx_app.objects.filter(ep_id__isnull=True)
for ep in eps_query:
ep_id = get_ep_id(ep.email)
ep.ep_id = ep_id
ep.save()
As for the second script;
It is meant to update data called tags taken from the form and sent to the website through the API, now the script itself executes properly with no issues but it does not do what it is supposed to; Here you have the Cron...
class UpdateEpsTags(CronJobBase):
RUN_EVERY_MINS = 30
schedule = Schedule(run_every_mins=RUN_EVERY_MINS)
code = 'ogx.UpdateEpsTags' # a unique code
def do(self):
access_token = 'taken out for confidentiality'
eps_query = mc_ogx_app.objects.filter(ep_id__isnull=False, tags_uploaded=False)
for ep in eps_query:
if len(str(ep.ep_id)) >= 2:
tags_list = []
if ep.country_pref is not None:
tags_list.append(ep.country_pref.tag_id)
if ep.career_pref is not None:
tags_list.append(ep.career_pref.tag_id)
first_tags_list = return_user_tag_list(ep)
tags_list = tags_list + first_tags_list
if ep.product_ogv:
tags_list.append([7])
if ep.product_oge:
tags_list.append([9])
if ep.product_ogt:
tags_list.append([8])
try:
update_ep_tags(int(ep.ep_id), tags_list, access_token,ep.chosen_ref)
ep.tags_uploaded = True
ep.save()
except:
ep.save()
Now for the graphQl query in the script, it goes as follows;
def update_ep_tags(person_id, tags_list, token,referral):
client = GraphQLClient(
'taken out for confidentiality')
result = client.execute('''
mutation Mymutation($persons: [Int]!, $tags: [Int]!,$id: ID!, $referral: String!){
bulkTagUpdateForPeople(person_ids:$persons, tag_list_ids:$tags)
{
id
full_name
tag_lists
{
id
name
}
}
updatePerson(id:$id , person:
{
referral_type:$referral
})
{
full_name
referral_type
programmes
{
short_name
}
}
}
''', {"persons": [person_id], "tags": tags_list, "id": person_id, "referral": referral})
return result
Now executing the query on GraphQl I get Nullability mismatch on variable $id and argument id (ID / ID!).

Tried checking the datatypes and also running the query on Insomnia to check if it works, it does work just fine, I do not seem to grasp where the error is coming from

Solved:
What I am basically trying to do is reference using an email, the email is no longer supported by the API and therefor the query just gives a runtime error.

Python Error not reading config.cfg?

Traceback (most recent call last):
File "/Users/jondevereux/Desktop/Data reporting/kpex_code/1PD/api_python_publisher_1PD.py", line 40, in <module>
username = parser.get('api_samples', 'username')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ConfigParser.py", line 607, in get
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'api_samples'
The config file is in the correct directory (same as .py and has the appropriate section api_samples:
[api_samples]
authentication_url = https://crowdcontrol.lotame.com/auth/v1/tickets
api_url = https://api.lotame.com/2/
username = xxx
password = xxx
Script works on co-workers PC not on mine? I had to use pip to install requests - i'm wondering I i'm missing something else?
Code is as follows:
# Set up the libs we need
import requests
import sys
import csv
import json
from ConfigParser import SafeConfigParser # used to get information from a config file
reload(sys)
sys.setdefaultencoding('utf-8')
'''
Now let's get what we need from our config file, including the username and password
We are assuming we have a config file called config.config in the same directory
where this python script is run, where the config file looks like:
[api_samples]
authentication_url = https://crowdcontrol.lotame.com/auth/v1/tickets
api_url = https://api.lotame.com/2/
username = USERNAME_FOR_API_CALLS
password = PASSWORD
'''
# Set up our Parser and get the values - usernames and password should never be in code!
parser = SafeConfigParser()
parser.read('config.cfg')
username = parser.get('api_samples', 'username')
password = parser.get('api_samples', 'password')
authentication_url = parser.get('api_samples', 'authentication_url')
base_api_url = parser.get('api_samples', 'api_url')
# OK, all set with our parameters, let's get ready to make our call to get a Ticket Granting Ticket
# Add the username and password to the payload (requests encodes for us, no need to urlencode)
payload = {'username': username,
'password': password}
# We want to set some headers since we are going to post some url encoded params.
headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain", "User-Agent":"python" }
# Now, let's make our Ticket Granting Ticket request. We get the location from the response header
tg_ticket_location = requests.post(authentication_url, data=payload).headers['location']
# Let's take a look at what a Ticket Granting Ticket looks like:
# print ('Ticket Granting Ticket - %s \n') % (tg_ticket_location[tg_ticket_location.rfind('/') + 1:])
# Now we have our Ticket Granting Ticket, we can get a service ticket for the service we want to call
# The first service call will be to get information on behavior id 5990.
# service_call = base_api_url + 'behaviors/5990'
# Add the service call to the payload and get the ticket
#payload = {'service': service_call}
#service_ticket = requests.post( tg_ticket_location, data=payload ).text
# Let's take a look at the service ticket
#print ('Here is our Service Ticket - %s \n') % ( service_ticket )
'''
Now let's make our call to the service ... remember we need to be quick about it because
we only have 10 seconds to do it before the Service Ticket expires.
A couple of things to note:
JSON is the default response, and it is what we want, so we don't need to specify
like {'Accept':'application/json'}, but we will anyway because it is a good practice.
We don't need to pass any parameters to this call, so we just add the parameter
notation and then 'ticket=[The Service Ticet]'
'''
headers = {'Accept':'application/json'}
#behavior_info = requests.get( ('%s?ticket=%s') % (service_call, service_ticket), headers=headers)
# Let's print out our JSON to see what it looks like
# requests support JSON on it's own, so not other package needed for this
# print ('Behavior Information: \n %s \n') % (behavior_info.json() )
'''
Now let's get the names and IDs of some audiences
We can reuse our Ticket Granting Ticket for a 3 hour period ( we haven't passed that yet),
so let's use it to get a service ticket for the audiences service call.
Note that here we do have a parameter that is part of the call. That needs to be included
in the Service Ticket request.
We plan to make a call to the audience service to get the first 10 audiences in the system
ascending by audience id. We don't need to pass the sort order, because it defaults to ascending
'''
# Set up our call and get our new Service Ticket, we plan to sort by id
# Please insert audiences ID below:
audienceids = ['243733','243736','241134','242480','240678','242473','242483','241119','243732','242492','243784','242497','242485','243785','242486','242487','245166','245167','245168','245169','245170','245171','240860']
f = open("publisher_report_1PD.csv", 'w+')
title_str = ['1PD % Contribution','audienceId','publisherName','audienceName']
print >> f,(title_str)
for audience_id in audienceids:
service_call = base_api_url + 'reports/audiences/' + audience_id + '/publisher?stat_interval=LAST_MONTH&page_count=100&page_num=1&sort_attr=audienceName&inc_network=false&sort_order=ASC'
payload = {'service': service_call}
# Let's get the new Service Ticket, we can print it again to see it is a new ticket
service_ticket = requests.post( tg_ticket_location, data=payload ).text
#print ('Here is our new Service Ticket - %s \n') % ( service_ticket )
# Use the new ticket to query the service, remember we did have a parameter this time,
# so we need to & 'ticket=[The Service Ticket]' to the parameter list
audience_list = requests.get( ('%s&ticket=%s') % (service_call, service_ticket)).json()
#print audience_list
# create an array to hold the audiences, pull ou the details we want, and print it out
audiences = []
for ln in audience_list['stats']:
audiences.append({ 'audienceId': ln['audienceId'], 'audienceName': ln['audienceName'], 'publisherName': ln['publisherName'], '1PD % Contribution': ln['percentOfAudience']})
for ii in range( 0, len(audiences) ):
data = audiences[ii]
data_str = json.dumps(data)
result = data_str.replace("\"","")
result1 = result.replace("{1PD % Contribution:","")
result2 = result1.replace("publisherName: ","")
result3 = result2.replace("audienceName: ","")
result4 = result3.replace("audienceId: ","")
result5 = result4.replace("}","")
print >> f,(result5)
# Once we are done with the Ticket Granting Ticket we should clean it up'
remove_tgt = requests.delete( tg_ticket_location )
print ( 'Status for closing TGT - %s') % (remove_tgt.status_code)
i = input('YAY! Gotcha!!')

I see only one reason for your problem: you run script from different folder and then script is looking for config.cfg in different folder.
You can get full path to folder with script
import os
script_folder = os.path.dirname(os.path.realpath(__file__))
and create full path to config.cfg
parser.read( os.path.join(script_folder, 'config.cfg') )

Unexpected type-error python

I am trying to create a client used to mainly test out the responses of a server asynchronously. I have created a function that basically waits for the next response from the server, if a requestId is provided when this function is called it will look for the next response with the requestId provided. Here is the function:
def getNextResponse(self, requestId = None):
logger = logging.getLogger(__name__)
self.acknowledge += 1
logger.info("requestId ack for this response: {}".format(requestId))
while(not self.response):
pass
self.acknowledge -= 1
logger.info("requestId unset for this response: {}".format(requestId))
message = json.loads(self.messagesList[len(self.messagesList)-1])
if(requestId != None):
while(requestId != message['body']['requestId']):
self.acknowledge += 1
while(not self.response):
pass
self.acknowledge -= 1
message = self.messagesList[len(self.messagesList)-1]
self.startMonitor -= 1
return message['body']
I also have helper functions for each command which can be sent to the engine below is one of said helper function for a ping command:
def ping(self, sessionId = None, requestId = None, version="1.0"):
result = {
"method": "Ping"
}
if(None != version):
result['version'] = version
if(None != sessionId):
result['sessionId'] = sessionId
if(None != requestId):
result['requestId'] = requestId
logger = logging.getLogger(__name__)
logger.info("Message Sent: " + json.dumps(result, indent=4))
self.startMonitor += 1
self.ws.send(json.dumps(result))
message = self.getNextResponse(requestId = requestId)
return message
It basically sets up a json object which contains all the parameters that the server expects and then sends the entire json message to the server. After it has been sent i call getNextResponse to await a response from the server. The requestId is set to None by default, so if no requestId is provided, it will just look for the very next response returned by the server. Since this can be quite inconsistent because of other asynchronous commands, one can provided a unique requestId for the command so that the response from the server will also contain this requestId thus making each response unique.
In my test case I am generating a random requestId by using:
def genRequestId(self):
x = random.randint(10000000, 99999999)
print x
reqId = str(x)+"-97a2-11e6-9346-fde5d2234523"
return reqId
The problem that I encountered is that sometimes (seems to be random), when I call ping in one of my test cases, i get this error:
message = self.getNextResponse(requestId = requestId)
TypeError: string indices must be integers, not str
I am quite confused by this error, requestId that I am generating inside ping is supposed to be a string and I am not referencing inside it in any way. I have tried removing the reference to the parameter like so:
message = self.getNextResponse(requestId)
But I am still getting this error. The error doesn't go any deeper inside the getNextResponse function which leads me to believe that it is coming from inside the ping function when I try to call it. Any help would be greatly appreciated!
EDIT: Here is the error
Traceback (most recent call last):
File "ctetest./RegressionTest/WebsocketTest\test_18_RecTimer.py", line 385, in test009_recTimer_Start_withAudio
response = client.endSession(sessionId = sessionId, requestId = requestId_2)
File "ctetest./RegressionTest/WebsocketTest../.././CTESetupClass\WebsocketCl
ient.py", line 528, in endSession
message = self.getNextResponse(requestId)
File "ctetest./RegressionTest/WebsocketTest../.././CTESetupClass\WebsocketCl
ient.py", line 49, in wrapper
raise ret
TypeError: string indices must be integers, not str

you have two statements in your code that look very similar:
message = json.loads(self.messagesList[len(self.messagesList)-1])
and then further down:
message = self.messagesList[len(self.messagesList)-1]
The first will set message to a json object (probably dict) where the second one assigns message to a string, I'm assuming this is not intended and the cause for your error.