Im not sure if this is google or email in general, but I'm seeing a some encoding that im not sure how to handle. Here is a snip that is the Washington post mailer form my google acct.
the subject
b'=?UTF-8?Q?The_Morning:_Peru=E2=80=99s_deadly_protests?='
actually reads
The Morning: Peru’s deadly protests
part of the body.
<!DOCTYPE html><html xmlns=3D"http://www.w3.org/1999/xhtml" xmlns:v=3D"urn:=
schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-microsoft-com:office:offi=
ce"><head> <title>The Morning: Peru=E2=80=99s deadly protests</title> <!--[=
if !mso]><!-- --> <meta http-equiv=3D"X-UA-Compatible" content=3D"IE=3Dedge=
"> <!--<![endif]--> <meta http-equiv=3D"Content-Type" content=3D"text/html;=
charset=3DUTF-8"> <meta name=3D"viewport" content=3D"width=3Ddevice-width,=
initial-scale=3D1"> <style type=3D"text/css">#outlook a{padding:0}body{marg=
in:0;padding:0;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}tabl=
e,td{border-collapse:collapse;mso-table-lspace:0;mso-table-rspace:0}img{bor=
der:0;height:auto;line-height:100%;outline:0;text-decoration:none;-ms-inter=
polation-mode:bicubic}p{display:block;margin:13px 0}p,ul{margin-top:0}#medi=
a (max-width:600px){body{padding:0 15px!important}}</style> <!--[if mso]>=
=0A <xml>=0A <o:OfficeDocumentSettings>=0A
Everything is line wrapped with a '=\f\n' which is no problem to remove with
bodytext = ''.join( bodytext.split('=\r\n') )
But there is other stuff in there like =3D =0D=0A and yeh they are ascii encodings but what decoder library do we use to decode this?
For reference or to try yourself: here is the python code.
import email, datetime
from imapclient import IMAPClient
with IMAPClient('imap.gmail.com', port=None, use_uid=True, ssl=True, stream=False, ssl_context=None, timeout=None) as client:
client.login("######gmail.com", "######")
client.select_folder('INBOX')
SEARCH_SINCE = (datetime.datetime.now() - datetime.timedelta( 4 )).date()
search_for = "Morning"
seq_nums = client.search([u'SUBJECT', f'{search_for}', u'SINCE', SEARCH_SINCE ])
print('seq nums', seq_nums)
for seqid, objs in client.fetch( seq_nums, [b'ENVELOPE', b'BODY[TEXT]']).items():
msg_body = email.message_from_bytes( objs[b'BODY[TEXT]'] )
envelope = objs[b'ENVELOPE']
print('subject', envelope.subject)
print('body', msg_body.get_payload()[:1000])
I also use red box and imap_tools to do this stuff but they are in order of magnitude slower than this IMAPClient method.
Google provides an API for accessing Gmail data in Python. You can use the Gmail API to read, send, delete, and modify emails.(https://developers.google.com/gmail/api/quickstart/python)
To get started, you'll need to install the Google Client Library for Python. You can do this using pip:
pip install --upgrade google-api-python-client
Once you have the library installed, you can use the Gmail API to access your emails. You'll need to create a project in the Google Developer Console and enable the Gmail API. Then, you'll need to create credentials for your project.
Once you have your credentials, you can use the Gmail API to access your emails. You can use the Gmail API to read emails, send emails, delete emails, and modify emails.
For example, you can use the Gmail API to read emails from a specific sender:
from googleapiclient.discovery import build
service = build('gmail', 'v1', credentials=credentials)
# Get emails from a specific sender
results = service.users().messages().list(userId='me', q='from:example#example.com').execute()
You can also use the Gmail API to send emails:
from googleapiclient.discovery import build
service = build('gmail', 'v1', credentials=credentials)
# Send an email
message = {
'raw': base64.urlsafe_b64encode(message.as_string())
}
service.users().messages().send(userId='me', body=message).execute()
You can find more information about using the Gmail API in Python in the official documentation.
Ok thank you Robert for the 2 links that described the encoding. No less than 2 different parsing methods. The working code follows:
import email, datetime
from imapclient import IMAPClient
from email.header import decode_header, make_header
import quopri
with IMAPClient('imap.gmail.com', port=None, use_uid=True, ssl=True, stream=False, ssl_context=None, timeout=None) as client:
client.login("######gmail.com", "######")
client.select_folder('INBOX')
SEARCH_SINCE = (datetime.datetime.now() - datetime.timedelta( 4 )).date()
search_for = "Morning"
seq_nums = client.search([u'SUBJECT', f'{search_for}', u'SINCE', SEARCH_SINCE ])
print('seq nums', seq_nums)
for seqid, objs in client.fetch( seq_nums, [b'ENVELOPE', b'BODY[TEXT]']).items():
msg_body = email.message_from_bytes( objs[b'BODY[TEXT]'] )
envelope = objs[b'ENVELOPE']
print('subject', make_header(decode_header(envelope.subject.decode('utf-8'))))
x = quopri.decodestring(objs[b'BODY[TEXT]'])
print(x.decode('utf-8')[:1000])
Related
I have a script which generates an S3 PreSigned URL and sends it in an email.
The script works fine, but when the email is sent, it adds a new-line to the URL, which breaks it and makes it unusable in the email.
The only packages installed:
boto3
Jinja2
The script:
import boto3
from botocore.config import Config
from jinja2 import Environment, FileSystemLoader
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import smtplib
# AWS
my_config = Config(
region_name = 'eu-west-1'
)
s3 = boto3.client('s3', config=my_config)
bucket = "name of bucket"
# Jinja2
loader = FileSystemLoader('templates')
env = Environment(loader=loader)
email_template = env.get_template('test_template.html')
def create_presigned_url(bucket, object_name, expiration=259200):
response = s3.generate_presigned_url('get_object', Params={'Bucket': bucket, 'Key': object_name}, ExpiresIn=expiration)
return response
def sendEmail(download_link):
toEmail = 'me#email.com'
msg = MIMEMultipart('alternative')
title = 'Test email'
sender = 'me#email.com'
rendered_template = MIMEText(email_template.render({'download_link':download_link}), 'html')
msg['Subject'] = title
msg['From'] = sender
msg['To'] = toEmail
receivers = toEmail
msg.attach(rendered_template)
try:
smtpObj = smtplib.SMTP('mail.server.com', 25)
smtpObj.sendmail(sender, receivers, msg.as_string())
print ("Successfully sent email")
smtpObj.quit()
except Exception as e:
print (e, "Error: unable to send email")
if __name__ == "__main__":
selectedFile = 'file.txt'
# Download link
downloadURL = create_presigned_url(bucket, selectedFile)
# print(downloadURL)
# Send email
sendEmail(downloadURL)
Results
When I run the script, for some reason I get a newline somewhere inside of this long URL, which breaks the URL:
Here's the source from Outlook:
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">Download link: https://redacted/9%2FJOUw%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEH0aCWV1LXdlc3QtMSJHMEUCIEBMYHD9wKhOfrR00jTv7RIcsRe3NbOU31QniJpEdps8AiEAi377qRmvQQXb5dXhRGJcXulFhRunGTSd0GRyGXHR2kMqzAQIdhAEGgwzNTgyODkxNTgxMTIiDH3bqAZMxVyKH3f6xyqpBL40WhPQMShoV8x1epn85Ml6qQ8Y1xdHe16xyoMWKqylbLGrndMFtYyOgs6LAlDvGJPrcF9xymGf8BqGsGHwCuWGdEcisxvwR%2FUoigjHBXP55fHpF%2FXnVupCRYDIVA5N%2BVOKW5%2BcljVN9KC3RKKvEeUncTGnXIaW5UHNAPiFSrgbbj9X%2FyBptkFmj5f4x2Zblm8crQS0rMTveCuoki3E06NO%2FKiDNiJQpF1vVphb%2F0spIR3CUxSx9HJHjvRBWTWQn9bmmT8rhp0lx%2Bzx9RLlpmE6hRRF6KBpNW%2B86y3EB%2BtMxVBuEhC5M1rCyjou6efK%2FA96wuwBN%2FmjD663vyZipiOGrj4yOFIMPklBu4L1SnfkxhZN8%2BNWXzwc%2B%2B5%2FNfL%2BVzyFpWS7TbGIM4A9TEvL3bPlgafvIl%2Fi24MOrN47UshpdpHGAjG20PBr0cbi75G7D%2B3UoSn%2Bzp0hZkAEACnwWtWtzEpVWbwatx%2FL1T8XF43o8OiKWqCfVxBjoZSc1itxRDOUqonYbCGY2Y0NlkXpvpHBZMcg7530dIFRBBxhTZo4RVXqkymTM4hEvDUw74R%2BDovr%2F%2BG5ji52Wpcng
954ESTpzMjOtuBXKcPtmEWTqx4au99ZP8lxbqKjq3BO%2FJLqrzTCPSEs6CTv7YbtzUqQ0r%2BkFyAU2RnpTTcYPJ5SD8ytlb4qUHb5RhEcn3bbJ5fsIRx%2B6q3LrhWkorDNKp5jh6oth1roRxXQM0swgN%2BzmwY6qQGnjWLAgUSUB9yf3heEdiFZo4DnC7ipW6BgsnkoeZJcPz5Ysx5PG4kzelCP89AsXQGD%2BtFqweusgWJVLo3dfyK3iLJ5myohn7mjSf1YVE%2B5CGlajc2HZl%2BoUOhI5gMMxpFXtpIL6jgTyY5r6ZwCKZ9g1afHO1kUF4VYir2M2BWYHTcB%2Bu8TANzoc15RJih8XmE%2FAWd%2FMQM7SQOQxsbmCiRSv5AeYMuok%2FSw&Expires=1668345190
I tried:
using | safe inside of my Jinja2 template.
I tried using the href HTML tag, no dice.
I don't know what else I can check and have no idea why it's happening. People mentioned this might be the cause: https://www.rfc-editor.org/rfc/rfc2822#section-2.1.1
It looks like the problem is related to max line length defined in the "Internet Message Format" RFC document 5322
2.1.1. Line Length Limits
There are two limits that this standard places on the number of characters in a line. Each line of characters MUST be no more than 998 characters, and SHOULD be no more than 78 characters, excluding the CRLF.
...
The more conservative 78 character recommendation is to accommodate the many implementations of user interfaces that display these messages which may truncate, or disastrously wrap, the display of more than 78 characters per line, in spite of the fact that such implementations are non-conformant to the intent of this specification (and that of [RFC2821] if they actually cause information to be lost). Again, even though this limitation is put on messages, it is encumbant upon implementations which display messages
Your line (with redacted information) has 961 characters, so adding those redacted information I assume you go over that limit of 998 characters.
Now, my thinking is that your local SMTP server is not splitting the line but the one on Amazon server may do that.
I assume that reducing the line length is not really an option :)
I would try with changing the markup, because now Outlook is looking for links and highlighting them in your message. I would try to add html markup for that with new line character inside, which hopefully will be ignored by the email client.
<a href="https://redacted/9%2FJOUw%3D&x-amz......
........">Download link</a>
try to change your jinja template to get something like
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
Download link:
Download link
as you didn't add <a href= your link is interpretated as long text => and in long text editor can add line breaks
Looking to create and send messages with multiple files attached. Per the online gmail api documentation, there is a function for building messages with an attachment but no documentation for howto use it to create a message with multiple attachments.
Can I use the gmail API to send messages with multiple attachments programmatically? How might one do this?
With this function, you can send to one or multiple recipient emails, and also you can attach zero, one or more files. Coding improvement recommendations are welcome, however the way it is now, it works.
Python v3.7
smtplib from https://github.com/python/cpython/blob/3.7/Lib/smtplib.py (download the code and create the smtplib.py on your project folder)
def send_email(se_from, se_pwd, se_to, se_subject, se_plain_text='', se_html_text='', se_attachments=[]):
""" Send an email with the specifications in parameters
The following youtube channel helped me a lot to build this function:
https://www.youtube.com/watch?v=JRCJ6RtE3xU
How to Send Emails Using Python - Plain Text, Adding Attachments, HTML Emails, and More
Corey Schafer youtube channel
Input:
se_from : email address that will send the email
se_pwd : password for authentication (uses SMTP.SSL for authentication)
se_to : destination email. For various emails, use ['email1#example.com', 'email2#example.com']
se_subject : email subject line
se_plain_text : body text in plain format, in case html is not supported
se_html_text : body text in html format
se_attachments : list of attachments. For various attachments, use ['path1\file1.ext1', 'path2\file2.ext2', 'path3\file3.ext3']. Follow your OS guidelines for directory paths. Empty list ([]) if no attachments
Returns
-------
se_error_code : returns True if email was successful (still need to incorporate exception handling routines)
"""
import smtplib
from email.message import EmailMessage
# Join email parts following smtp structure
msg = EmailMessage()
msg['From'] = se_from
msg['To'] = se_to
msg['Subject'] = se_subject
msg.set_content(se_plain_text)
# Adds the html text only if there is one
if se_html_text != '':
msg.add_alternative("""{}""".format(se_html_text), subtype='html')
# Checks if there are files to be sent in the email
if len(se_attachments) > 0:
# Goes through every file in files list
for file in se_attachments:
with open(file, 'rb') as f:
file_data = f.read()
file_name = f.name
# Attaches the file to the message. Leaves google to detect the application to open it
msg.add_attachment(file_data, maintype='application', subtype='octet-stream', filename=file_name)
# Sends the email that has been built
with smtplib.SMTP_SSL('smtp.gmail.com', 465) as smtp:
smtp.login(se_from, se_pwd)
smtp.send_message(msg)
return True
Don't forget to activate less secure apps on your google account (https://myaccount.google.com/lesssecureapps) for this code to work.
Hope this helps
What is the way to pull out all emails from Gmail?
I did full_sync, but that didn't return all of my email - only about 3000 emails, while I know I have more. In the documentation they did not mention about this.
My code snippet:
history = service.users().history().list(
userId='me',
startHistoryId=start_history_id,
maxResults=500,
labelId='INBOX'
).execute()
if "history" in history:
try:
for message in history["history"]:
batch.add(
service.users().messages().get(userId='me', id=message["messages"][0]["id"]),
callback="somecallbak",
request_id=request_id
)
batch.execute()
while 'nextPageToken' in history:
If you are doing a full sync, you should refer to this documentation, that recommends two steps:
listing all the messages with the users.messages.list method
for each of the entry get the required information using the users.messages.get method
So you don't need use the users.history.list as you will have an hard time finding the startHistoryId from which to start.
You can achieve this with a snipped similar to the one below (tested and working on my python 3.x console). As suggested by others I used the python client pagination and batch request functionalities.
from httplib2 import Http
from googleapiclient.discovery import build
from oauth2client import client, tools, file
# callback for the batch request (see below)
def print_gmail_message(request_id, response, exception):
if exception is not None:
print('messages.get failed for message id {}: {}'.format(request_id, exception))
else:
print(response)
# Scopes
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly', ]
# where do we store our credentials?
creds_store = file.Storage('gmail-list.json')
start_creds = creds_store.get()
# standard oauth2 authentication flow
if not start_creds or start_creds.invalid:
# client_id.json is exported from your gcp project
start_flow = client.flow_from_clientsecrets('client_id.json', SCOPES)
start_creds = tools.run_flow(start_flow, creds_store)
# Gmail SDK
http = Http()
gmail_sdk = build('gmail', 'v1', http=start_creds.authorize(http))
# messages.list parameters
msg_list_params = {
'userId': 'me'
}
# messages.list API
message_list_api = gmail_sdk.users().messages()
# first request
message_list_req = message_list_api.list(**msg_list_params)
while message_list_req is not None:
gmail_msg_list = message_list_req.execute()
# we build the batch request
batch = gmail_sdk.new_batch_http_request(callback=print_gmail_message)
for gmail_message in gmail_msg_list['messages']:
msg_get_params = {
'userId': 'me',
'id': gmail_message['id'],
'format': 'full',
}
batch.add(gmail_sdk.users().messages().get(**msg_get_params), request_id=gmail_message['id'])
batch.execute(http=http)
# pagination handling
message_list_req = message_list_api.list_next(message_list_req, gmail_msg_list)
As suggested in this link, you may use batch requests.
Use batch and request 100 messages at a time. You will need to make 1000 requests but the good news is that's quite fine and it'll be easier for everyone (no downloading 1GB response in a single request!).
Also based from this thread, you could save the next page token on every request and use it in your next request. If there is no next page token in the response, you know that you have gotten all messages.
I am trying to send a simple mail with python
import smtplib
server = smtplib.SMTP('smtp.gmail.com', 587)
server.ehlo()
server.starttls()
server.login("mymail#gmail.com", "mypassword")
msg = "Hello world"
server.sendmail("mymail#gmail.com", "mymail#gmail.com", msg)
server.quit()
But I get this err:
server.login("user#gmail.com", "psw")
File "C:\Python\lib\smtplib.py", line 652, in login
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (534, b'5.7.14 <https://accounts.google.com/ContinueSignIn?sarp=1&scc=1&plt=AKgnsbuxb\n5.7.14 4i2u8qU8V3jgf6uGv8da1RAGPJyctRvIFy_kjai6aKVx_B6qVhoz_dzFpvfPC18H-jeM6K\n5.7.14 cnm2HVuq-wr-uw59hD31ms-cxMmnZuq6Z3_liDaDmu8_UqaiUwR4FUiuX2i5pPdQjJzFvv\n5.7.14 4VrEF5XT4ol2iN17gnB_jITpwzsjH9Ox3NCNcfl7SriHr5m7esc15PWI0CG_2CTlyh7RxW\n5.7.14 XhoJPajs8GMd-khOQWUqucywfrfo> Please log in via your web browser and\n5.7.14 then try again.\n5.7.14 Learn more at\n5.7.14 https://support.google.com/mail/bin/answer.py?answer=78754 ef10sm13614207wjd.49 - gsmtp')
What should I do?
Thanks
It seems as if you require something that Google calls an app password.
Basically, you generate a 16 digit password, which is unique to your app. You enter this specific password in the python program, instead of the password you regularly use to log into your Google account.
This allows you to still enjoy the benefits of 2-step verification while also being able to use third party applications, such as your own python program.
Here are the instructions from Google on how to generate such an app password:
https://support.google.com/accounts/answer/185833?hl=en
you can use this code:
import smtplib
session = smtplib.SMTP('smtp.gmail.com', 587)
session.ehlo()
session.starttls()
session.login('youremail#gmail.com',' password')
headers = "\r\n".join(["from: " + 'youremail#gmail.com',
"subject: " + "test",
"to: " + 'contactemail#gmail.com',
"mime-version: 1.0",
"content-type: text/html"])
# body_of_email can be plaintext or html!
content = headers + "\r\n\r\n" + "body_of_email"
session.sendmail('youremail#gmail.com', 'contactemail#gmail.com', content)
just remember if your email is gmail after first run you get an error. after that you should login to your email account and approve access to your account from another app ( you will receive a messege after login)
You could use a free mail API such as mailgun:
import requests
def send_simple_message(target):
return requests.post(
"https://api.mailgun.net/v3/samples.mailgun.org/messages",
auth=("api", "key-3ax6xnjp29jd6fds4gc373sgvjxteol0"),
data={"from": "Excited User <excited#samples.mailgun.org>",
"to": [target],
"subject": "Hello",
"text": "Testing some Mailgun awesomeness!"})
send_simple_message('target#email.com')
Using an API like this avoids the issue of individual account authentication all together.
See also: This question for info on using smtplib
Yea, like the answer posted, it was a matter of authentication :)
I'd like to further help you with sending emails by advising the yagmail package (I'm the maintainer, sorry for the advertising, but I feel it can really help!). Note that I'm also maintaining a list of common errors there, such as the authentication error.
The whole code for you would be:
import yagmail
yag = yagmail.SMTP('user', 'pw')
yag.send(contents = msg)
Note that I provide defaults for all arguments, for example if you want to send to yourself, you can omit "to = myemail#gmail.com", if you don't want a subject, you can omit it also.
Furthermore, the goal is also to make it really easy to attach html code or images (and other files).
Where you put contents you can do something like:
contents = ['Body text, and here is an embedded image:', 'http://somedomain/image.png',
'You can also find an audio file attached.', '/local/path/song.mp3']
Wow, how easy it is to send attachments! This would take like 20 lines without yagmail ;)
Also, if you set it up once, you'll never have to enter the password again (and have it safely stored). In your case you can do something like:
import yagmail
yagmail.SMTP().send(contents = contents)
which is much more concise!
I'd invite you to have a look at the github or install it directly with pip install yagmail.
A use Mailgun to send b a email, after b receive the email and reply to a.If I want to track the email coming from b, How I can get the email?
Here is the code:
1.sendmail.py
from smtplib import SMTP
import requests
login_name = "postmaster#zzb.mailgun.org"
password = "********"
def send_message_via_smtp():
smtp = SMTP("smtp.mailgun.org", 587)
smtp.login(login_name, password)
smtp.sendmail("zebozhuang#163.com","348284770#qq.com", "Subject:mailgun test \n\n just for test.\n\n")
smtp.quit()
if __name__=="__main__":
send_message_via_smtp()
2.create_route.py
import requests
from werkzeug.datastructures import MultiDict
def create_route():
return requests.post(
"https://api.mailgun.net/v2/routes",
auth=("api", "key-9c4-t2q6fouilngjummvtv1rge7t00f2"),
data=MultiDict([("priority", 1),
("description", "Sample route"),
("expression", "match_recipient('.*#qq.com')"),
("action", "forward('qiyazhuang#gmail.com')"),
("action", "stop()")])
)
I create the route and I run the script sendmail.py.After someone who use email 348284770#qq.com reply to the other who use email zebozhuang#163.com, the Gmail
can not receive the message by using the Mailgun method 'forward'.
Could anyone tell me why?
Your messages are likely being delivered. Check the "Logs" tab of the Mailgun Control Panel.
Do you see any entries that look like this:
Routed: .*#qq.com -> qiyazhuang#gmail.com 'SUBJECT HERE'
The "Routed" prefix means that the message triggered a Route. If you're seeing this, and the next log entry is prefixed with "Delivered", the message is likely being delivered to Gmail without issue. Check your Gmail spam folder if you still don't see the messages in the inbox folder.
Disclaimer: I work for Mailgun Support. :)