We Keep Coding

S3 PreSigned URL is cut when sent in an email

I have a script which generates an S3 PreSigned URL and sends it in an email.
The script works fine, but when the email is sent, it adds a new-line to the URL, which breaks it and makes it unusable in the email.
The only packages installed:
boto3
Jinja2
The script:
import boto3
from botocore.config import Config
from jinja2 import Environment, FileSystemLoader
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import smtplib
# AWS
my_config = Config(
region_name = 'eu-west-1'
)
s3 = boto3.client('s3', config=my_config)
bucket = "name of bucket"
# Jinja2
loader = FileSystemLoader('templates')
env = Environment(loader=loader)
email_template = env.get_template('test_template.html')
def create_presigned_url(bucket, object_name, expiration=259200):
response = s3.generate_presigned_url('get_object', Params={'Bucket': bucket, 'Key': object_name}, ExpiresIn=expiration)
return response
def sendEmail(download_link):
toEmail = 'me#email.com'
msg = MIMEMultipart('alternative')
title = 'Test email'
sender = 'me#email.com'
rendered_template = MIMEText(email_template.render({'download_link':download_link}), 'html')
msg['Subject'] = title
msg['From'] = sender
msg['To'] = toEmail
receivers = toEmail
msg.attach(rendered_template)
try:
smtpObj = smtplib.SMTP('mail.server.com', 25)
smtpObj.sendmail(sender, receivers, msg.as_string())
print ("Successfully sent email")
smtpObj.quit()
except Exception as e:
print (e, "Error: unable to send email")
if __name__ == "__main__":
selectedFile = 'file.txt'
# Download link
downloadURL = create_presigned_url(bucket, selectedFile)
# print(downloadURL)
# Send email
sendEmail(downloadURL)
Results
When I run the script, for some reason I get a newline somewhere inside of this long URL, which breaks the URL:
Here's the source from Outlook:
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">Download link: https://redacted/9%2FJOUw%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEH0aCWV1LXdlc3QtMSJHMEUCIEBMYHD9wKhOfrR00jTv7RIcsRe3NbOU31QniJpEdps8AiEAi377qRmvQQXb5dXhRGJcXulFhRunGTSd0GRyGXHR2kMqzAQIdhAEGgwzNTgyODkxNTgxMTIiDH3bqAZMxVyKH3f6xyqpBL40WhPQMShoV8x1epn85Ml6qQ8Y1xdHe16xyoMWKqylbLGrndMFtYyOgs6LAlDvGJPrcF9xymGf8BqGsGHwCuWGdEcisxvwR%2FUoigjHBXP55fHpF%2FXnVupCRYDIVA5N%2BVOKW5%2BcljVN9KC3RKKvEeUncTGnXIaW5UHNAPiFSrgbbj9X%2FyBptkFmj5f4x2Zblm8crQS0rMTveCuoki3E06NO%2FKiDNiJQpF1vVphb%2F0spIR3CUxSx9HJHjvRBWTWQn9bmmT8rhp0lx%2Bzx9RLlpmE6hRRF6KBpNW%2B86y3EB%2BtMxVBuEhC5M1rCyjou6efK%2FA96wuwBN%2FmjD663vyZipiOGrj4yOFIMPklBu4L1SnfkxhZN8%2BNWXzwc%2B%2B5%2FNfL%2BVzyFpWS7TbGIM4A9TEvL3bPlgafvIl%2Fi24MOrN47UshpdpHGAjG20PBr0cbi75G7D%2B3UoSn%2Bzp0hZkAEACnwWtWtzEpVWbwatx%2FL1T8XF43o8OiKWqCfVxBjoZSc1itxRDOUqonYbCGY2Y0NlkXpvpHBZMcg7530dIFRBBxhTZo4RVXqkymTM4hEvDUw74R%2BDovr%2F%2BG5ji52Wpcng
954ESTpzMjOtuBXKcPtmEWTqx4au99ZP8lxbqKjq3BO%2FJLqrzTCPSEs6CTv7YbtzUqQ0r%2BkFyAU2RnpTTcYPJ5SD8ytlb4qUHb5RhEcn3bbJ5fsIRx%2B6q3LrhWkorDNKp5jh6oth1roRxXQM0swgN%2BzmwY6qQGnjWLAgUSUB9yf3heEdiFZo4DnC7ipW6BgsnkoeZJcPz5Ysx5PG4kzelCP89AsXQGD%2BtFqweusgWJVLo3dfyK3iLJ5myohn7mjSf1YVE%2B5CGlajc2HZl%2BoUOhI5gMMxpFXtpIL6jgTyY5r6ZwCKZ9g1afHO1kUF4VYir2M2BWYHTcB%2Bu8TANzoc15RJih8XmE%2FAWd%2FMQM7SQOQxsbmCiRSv5AeYMuok%2FSw&Expires=1668345190
I tried:
using | safe inside of my Jinja2 template.
I tried using the href HTML tag, no dice.
I don't know what else I can check and have no idea why it's happening. People mentioned this might be the cause: https://www.rfc-editor.org/rfc/rfc2822#section-2.1.1

It looks like the problem is related to max line length defined in the "Internet Message Format" RFC document 5322
2.1.1. Line Length Limits
There are two limits that this standard places on the number of characters in a line. Each line of characters MUST be no more than 998 characters, and SHOULD be no more than 78 characters, excluding the CRLF.
...
The more conservative 78 character recommendation is to accommodate the many implementations of user interfaces that display these messages which may truncate, or disastrously wrap, the display of more than 78 characters per line, in spite of the fact that such implementations are non-conformant to the intent of this specification (and that of [RFC2821] if they actually cause information to be lost). Again, even though this limitation is put on messages, it is encumbant upon implementations which display messages
Your line (with redacted information) has 961 characters, so adding those redacted information I assume you go over that limit of 998 characters.
Now, my thinking is that your local SMTP server is not splitting the line but the one on Amazon server may do that.
I assume that reducing the line length is not really an option :)
I would try with changing the markup, because now Outlook is looking for links and highlighting them in your message. I would try to add html markup for that with new line character inside, which hopefully will be ignored by the email client.
<a href="https://redacted/9%2FJOUw%3D&x-amz......
........">Download link</a>

try to change your jinja template to get something like
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
Download link:
Download link
as you didn't add <a href= your link is interpretated as long text => and in long text editor can add line breaks

Python Lambda to send files uploaded to s3 as email attachments

We have an online form that gives people the option to upload multiple files. The form is built by a third party, so I don't have any involvement with them. When someone uploads files using the form it dumps the files into a new folder within an s3 bucket. I want to be able to do the following:
Get the files triggered by form filler's upload
Attach the files to an email
Send the email to specific people.
I have done quite a lot of research, but I'm still new to coding and am trying to use Python, which I have decided to focus on as my first proper language.
The code I have got so far is borrowed from other examples I have seen and adjusted. But so far it has only sent me emails with the files when I have uploaded single files into the root of the bucket. Not multiple files into a folder in the bucket. This appears to be because the / in the path of the files in folders breaks the sendrawemail functionality.
This lambda is set up to be triggered by a PUT notification on the creation of new files on the s3 bucket.
UPDATE: I have now used different code and have got a solution. Please see the bottom for this.
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.application import MIMEApplication
import boto3
def lambda_handler(event, context):
file_obj = event["Records"][0]
bucket_name = str(file_obj['s3']['bucket']['name'])
key = str(file_obj['s3']['object']['key'])
msg = MIMEMultipart()
new_body = "test body"
text_part = MIMEText(new_body, _subtype="html")
msg.attach(text_part)
filename = str(event["Records"][0]['s3']['object']['key'])
msg["To"] = "test#test.com"
msg["From"] = "test#test.com"
s3_object = boto3.client('s3')
s3_object = s3_object.get_object(Bucket=str(bucket_name), Key=str(key))
body = s3_object['Body'].read()
part = MIMEApplication(body, filename)
part.add_header("Content-Disposition", 'attachment', filename=filename)
msg.attach(part)
ses_aws_client = boto3.client('ses', 'eu-west-1')
ses_aws_client.send_raw_email(RawMessage={"Data" : msg.as_bytes()})
What I hoped was that when a folder, or multiple files were uploaded, I would be sent an email containing all of the files as attachments (I realise there's a 10mb limit to messages). However, when there are multiple files it appears to be sending multiple emails with one file per email. And if the file is in a folder, so has a key value with a slash in it, the send_raw_email shows the following error:
[ERROR] ClientError: An error occurred (InvalidParameterValue) when calling the SendRawEmail operation: Expected ';', got "/"
I presume I need to encode the path in some way? Is there any way of collating all the files newly uploaded into one email?
Any help would be massively welcome.
Edit 1: As per Jeril's response, I'm sharing the full log entry from the failure. Also I missed a line in my original code block so I will update that, too.
12:14:59
[ERROR] ClientError: An error occurred (InvalidParameterValue) when calling the SendRawEmail operation: Expected ';', got "/" Traceback (most recent call last): File "/var/task/lambda_function.py", line 26, in lambda_handler ses_aws_client.send_raw_email(RawMessage={"Data" : msg.as_bytes()}) File "/var/runtime/botocore/client.py", line 320, in _api_call return self._make_api_call(opera
[ERROR] ClientError: An error occurred (InvalidParameterValue) when calling the SendRawEmail operation: Expected ';', got "/"
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 26, in lambda_handler
ses_aws_client.send_raw_email(RawMessage={"Data" : msg.as_bytes()})
File "/var/runtime/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
Update:
I have now managed to get the essential functionality of this working:
import os.path
import boto3
import email
from botocore.exceptions import ClientError
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication
s3 = boto3.client("s3")
def lambda_handler(event, context):
# Replace sender#example.com with your "From" address.
# This address must be verified with Amazon SES.
SENDER = "Test Test <test#test.com>"
# Replace recipient#example.com with a "To" address. If your account
# is still in the sandbox, this address must be verified.
RECIPIENT = "Test Test <test#test.com>"
# Specify a configuration set. If you do not want to use a configuration
# set, comment the following variable, and the
# ConfigurationSetName=CONFIGURATION_SET argument below.
# CONFIGURATION_SET = "ConfigSet"
AWS_REGION = "eu-west-1"
SUBJECT = "Test Send Mesage with Attachment"
# This is the start of the process to pull the files we need from the S3 bucket into the email.
# Get the records for the triggered event
FILEOBJ = event["Records"][0]
# Extract the bucket name from the records for the triggered event
BUCKET_NAME = str(FILEOBJ['s3']['bucket']['name'])
# Extract the object key (basicaly the file name/path - note that in S3 there are
# no folders, the path is part of the name) from the records for the triggered event
KEY = str(FILEOBJ['s3']['object']['key'])
# extract just the last portion of the file name from the file. This is what the file
# would have been called prior to being uploaded to the S3 bucket
FILE_NAME = os.path.basename(KEY)
# Using the file name, create a new file location for the lambda. This has to
# be in the tmp dir because that's the only place lambdas let you store up to
# 500mb of stuff, hence the '/tmp/'+ prefix
TMP_FILE_NAME = '/tmp/' +FILE_NAME
# Download the file/s from the event (extracted above) to the tmp location
s3.download_file(BUCKET_NAME, KEY, TMP_FILE_NAME)
# Make explicit that the attachment will have the tmp file path/name. You could just
# use the TMP_FILE_NAME in the statments below if you'd like.
ATTACHMENT = TMP_FILE_NAME
# The email body for recipients with non-HTML email clients.
BODY_TEXT = "Hello,\r\nPlease see the attached file related to recent submission."
# The HTML body of the email.
BODY_HTML = """\
<html>
<head></head>
<body>
<h1>Hello!</h1>
<p>Please see the attached file related to recent submission.</p>
</body>
</html>
"""
# The character encoding for the email.
CHARSET = "utf-8"
# Create a new SES resource and specify a region.
client = boto3.client('ses',region_name=AWS_REGION)
# Create a multipart/mixed parent container.
msg = MIMEMultipart('mixed')
# Add subject, from and to lines.
msg['Subject'] = SUBJECT
msg['From'] = SENDER
msg['To'] = RECIPIENT
# Create a multipart/alternative child container.
msg_body = MIMEMultipart('alternative')
# Encode the text and HTML content and set the character encoding. This step is
# necessary if you're sending a message with characters outside the ASCII range.
textpart = MIMEText(BODY_TEXT.encode(CHARSET), 'plain', CHARSET)
htmlpart = MIMEText(BODY_HTML.encode(CHARSET), 'html', CHARSET)
# Add the text and HTML parts to the child container.
msg_body.attach(textpart)
msg_body.attach(htmlpart)
# Define the attachment part and encode it using MIMEApplication.
att = MIMEApplication(open(ATTACHMENT, 'rb').read())
# Add a header to tell the email client to treat this part as an attachment,
# and to give the attachment a name.
att.add_header('Content-Disposition','attachment',filename=os.path.basename(ATTACHMENT))
# Attach the multipart/alternative child container to the multipart/mixed
# parent container.
msg.attach(msg_body)
# Add the attachment to the parent container.
msg.attach(att)
print(msg)
try:
#Provide the contents of the email.
response = client.send_raw_email(
Source=SENDER,
Destinations=[
RECIPIENT
],
RawMessage={
'Data':msg.as_string(),
},
# ConfigurationSetName=CONFIGURATION_SET
)
# Display an error if something goes wrong.
except ClientError as e:
print(e.response['Error']['Message'])
else:
print("Email sent! Message ID:"),
print(response['MessageId'])
The only issue now is that if I upload multiple files, I get sent an email per file. Is there any way of collating them all into one email?

Adding Content-Disposition header in Python - email isn't sent

Following the directions in Python's email examples and in several Stack Overflow questions, I wrote the following function (sending through the Gmail SMTP server):
def send_email(emailaddr, message, attachmentfile = None, subject = None):
try:
smtpconn = smtplib.SMTP(mainconf["SMTPHOST"], mainconf["SMTPPORT"])
smtpconn.set_debuglevel(1)
smtpconn.ehlo()
smtpconn.starttls()
smtpconn.login(mainconf["SMTPUSER"], mainconf["SMTPPASS"])
if not attachmentfile:
message = '\n' + message
smtpconn.sendmail(mainconf["EMAILFROM"], emailaddr, message)
else:
multipart = MIMEMultipart()
multipart['Subject'] = subject if subject else "Attachment"
multipart['From'] = mainconf["EMAILFROM"]
multipart['To'] = emailaddr
with open(attachmentfile, 'rb') as fp:
filepart = MIMEApplication(fp.read())
multipart.attach(filepart)
multipart.attach(message)
smtpconn.sendmail(mainconf["EMAILFROM"], emailaddr, multipart.as_string())
generallog.info("Sent an email to {0}".format(emailaddr))
except:
generallog.warn("Email sending to {0} failed with error message {1}".format(emailaddr, traceback.format_exception_only(sys.exc_info()[0], sys.exc_info()[1])))
This works fine for sending an email with an attachment, but it results in the famous noname problem (that's just one of several SO questions on the subject). So I add in this:
filepart.add_header('Content-Disposition','attachment',filename=os.basename(attachmentfile))
Then Gmail simply refuses to send the email. When I run this code in the Python shell, I get the standard message accepted message, but the message is never delivered.
Why might this be happening?

How to send multicast message (multiple users) using xmpp and Python xmpppy (XEP-0033: Extended Stanza Addressing)

I am using xmpppy http://xmpppy.sourceforge.net/ to send Jabber notifications, it is working well for single destinations using the following code:
# pip install https://github.com/rochacbruno/xmpppy/tarball/master
import xmpp
JABBER_SETTINGS = {"USERNAME": None, "PASSWORD": None, "DOMAIN": None, "RESOURCE": None}
def get_jabber_client():
client = xmpp.Client(JABBER_SETTINGS.get('DOMAIN'))
client.connect(server=(JABBER_SETTINGS.get('DOMAIN'), '5222'))
client.auth(
JABBER_SETTINGS.get('USERNAME'),
JABBER_SETTINGS.get('PASSWORD'),
JABBER_SETTINGS.get('RESOURCE')
)
client.sendInitPresence()
return client
def send_message(to, message):
client = get_jabber_client()
xmpp_message = xmpp.Message(to, message)
client.send(xmpp_message)
client.disconnect()
send_message("single.destination#domain.com", "Hello World!")
But now I need to send the message to multiple destinations, for now I am doing.
for users in list_of_users:
send_message(user, "Hello World!")
Which works fine, but every time I call it starts the process of authentication and takes a lot of time.
I've tried to create a single client and use the same client to send the message.
def send_message(to, message):
if isinstance(to, basestring):
to = [to]
assert isinstance(to, (list, tuple))
client = get_jabber_client()
for destination in to:
xmpp_message = xmpp.Message(destination, message)
client.send(xmpp_message)
client.disconnect()
send_message(['user1...', 'user2...'], "Hello World!")
The code above works, but only the first user in the list gets the message well formatted, the other users receives the message in pure XML.
I saw this code (in .net), mentioning XEP-0033: Extended Stanza Addressing http://forum.ag-software.net/thread/1482-Send-Message-To-all-users-in-contact-list
var addresses = new Addresses();
addresses.AddAddress(new Address
{
Type = Type.to,
Jid = "hildjj#jabber.org/Work",
Description = "Joe Hildebrand"
});
addresses.AddAddress(new Address
{
Type = Type.cc,
Jid = "jer#jabber.org/Home",
Description = "Jeremie Miller"
});
var msg = new Matrix.Xmpp.Client.Message();
msg.Add(addresses);
msg.To = "multicast.jabber.org";
msg.Body = "Hello, world!";
builds the following Xml:
<message to='multicast.jabber.org'>
<addresses xmlns='http://jabber.org/protocol/address'>
<address type='to' jid='hildjj#jabber.org/Work' desc='Joe Hildebrand'/>
<address type='cc' jid='jer#jabber.org/Home' desc='Jeremie Miller'/>
</addresses>
<body>Hello, world!</body>
</message>
But I did not found the way to do the same in Python using xmpppy, any idea on how to build the multicast stanza and send the message to multiple users using Python?
Thanks.

You could probably get this done without XEP-0033. In this loop, you overwrite the value of the variable message the first time, and subsequent messages will get garbled contents:
for destination in to:
message = xmpp.Message(destination, message)
client.send(message)
Try this instead:
for destination in to:
xmpp_message = xmpp.Message(destination, message)
client.send(xmpp_message)

How to receive mail using python

I would like to receive email using python. So far I have been able to get the subject but not the body. Here is the code I have been using:
import poplib
from email import parser
pop_conn = poplib.POP3_SSL('pop.gmail.com')
pop_conn.user('myusername')
pop_conn.pass_('mypassword')
#Get messages from server:
messages = [pop_conn.retr(i) for i in range(1, len(pop_conn.list()[1]) + 1)]
# Concat message pieces:
messages = ["\n".join(mssg[1]) for mssg in messages]
#Parse message intom an email object:
messages = [parser.Parser().parsestr(mssg) for mssg in messages]
for message in messages:
print message['subject']
print message['body']
pop_conn.quit()
My issue is that when I run this code it properly returns the Subject but not the body. So if I send an email with the subject "Tester" and the body "This is a test message" it looks like this in IDLE.
>>>>Tester >>>>None
So it appears to be accurately assessing the subject but not the body, I think it is in the parsing method right? The issue is that I don't know enough about these libraries to figure out how to change it so that it returns both a subject and a body.

The object message does not have a body, you will need to parse the multiple parts, like this:
for part in message.walk():
if part.get_content_type():
body = part.get_payload(decode=True)
The walk() function iterates depth-first through the parts of the email, and you are looking for the parts that have a content-type. The content types can be either text/plain or text/html, and sometimes one e-mail can contain both (if the message content_type is set to multipart/alternative).

The email parser returns an email.message.Message object, which does not contain a body key, as you'll see if you run
print message.keys()
What you want is the get_payload() method:
for message in messages:
print message['subject']
print message.get_payload()
pop_conn.quit()
But this gets complicated when it comes to multi-part messages; get_payload() returns a list of parts, each of which is a Message object. You can get a particular part of the multipart message by using get_payload(i), which returns the ith part, raises an IndexError if i is out of range, or raises a TypeError if the message is not multipart.
As Gustavo Costa De Oliveir points out, you can use the walk() method to get the parts in order -- it does a depth-first traversal of the parts and subparts of the message.
There's more about the email.parser module at http://docs.python.org/library/email.message.html#email.message.Message.

it also good return data in correct encoding in message contains some multilingual content
charset = part.get_content_charset()
content = part.get_payload(decode=True)
content = content.decode(charset).encode('utf-8')

Here is how I solved the problem using python 3 new capabilities:
import imaplib
import email
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(username, password)
mail.select(readonly=True) # refresh inbox
status, message_ids = mail.search(None, 'ALL') # get all emails
for message_id in message_ids[0].split(): # returns all message ids
# for every id get the actual email
status, message_data = mail.fetch(message_id, '(RFC822)')
actual_message = email.message_from_bytes(message_data[0][1])
# extract the needed fields
email_date = actual_message["Date"]
subject = actual_message["Subject"]
message_body = get_message_body(actual_message)
Now get_message_body is actually pretty tricky due to MIME format. I used the function suggested in this answer.
This particular example works with Gmail, but IMAP is a standard protocol, so it should work for other email providers as well, possibly with minor changes.

if u want to use IMAP4. Use outlook python library, download here : https://github.com/awangga/outlook
to retrieve unread email from your inbox :
import outlook
mail = outlook.Outlook()
mail.login('emailaccount#live.com','yourpassword')
mail.inbox()
print mail.unread()
to retrive email element :
print mail.mailbody()
print mail.mailsubject()
print mail.mailfrom()
print mail.mailto()