Python sending email too slow - python

I have the script below which I'm using to send say 10 messages myself<->myself. However, I've noticed that Python really takes a while to do that. Last year I needed a system to send about 200 emails with attachments and text and I implemented it with msmtp + bash. As far as I remember it was much faster.
Moving the while loop inside (around the smtp_serv.sendmail(sender, recepient, msg) function yields similar results).
Am I doing something wrong? Surely it can't be slower than bash + msmtp (and I'm only sending a 'hi' message, no attachments).
#! /usr/bin/python3.1
def sendmail(recepient, msg):
import smtplib
# Parameters
sender = 'login#gmail.com'
password = 'password'
smtpStr = 'smtp.gmail.com'
smtpPort = 587
# /Parameters
smtp_serv = smtplib.SMTP(smtpStr, smtpPort)
smtp_serv.ehlo_or_helo_if_needed()
smtp_serv.starttls()
smtp_serv.ehlo()
recepientExists = smtp_serv.verify(recepient)
if recepientExists[0] == 250:
smtp_serv.login(sender, password)
try:
smtp_serv.sendmail(sender, recepient, msg)
except smtplib.SMTPException:
print(recepientExists[1])
else:
print('Error', recepientExists[0], ':', recepientExists[1])
smtp_serv.quit()
for in in range(10):
sendmail('receiver#gmail.com', 'hi')

In this script it takes five times more time to setup SMTP connection (5 seconds) than to send a e-mail (1 second) so it could make sense to setup a single connection and send several e-mails instead of creating the connection each time:
#!/usr/bin/env python3
import smtplib
from contextlib import contextmanager
from datetime import datetime
from email.mime.text import MIMEText
from netrc import netrc
from timeit import default_timer as timer
#contextmanager
def logined(sender, password, smtp_host='smtp.gmail.com', smtp_port=587):
start = timer(); smtp_serv = smtplib.SMTP(smtp_host, smtp_port, timeout=10)
try: # make smtp server and login
smtp_serv.ehlo_or_helo_if_needed()
smtp_serv.starttls()
smtp_serv.ehlo()
print('smtp setup took (%.2f seconds passed)' % (timer()-start,))
start = timer(); smtp_serv.login(sender, password)
print('login took %.2f seconds' % (timer()-start,))
start = timer(); yield smtp_serv
finally:
print('Operations with smtp_serv took %.2f seconds' % (timer()-start,))
start = timer(); smtp_serv.quit()
print('Quiting took %.2f seconds' % (timer()-start,))
smtp_host = 'smtp.gmail.com'
login, _, password = netrc().authenticators(smtp_host)
with logined(login, password, smtp_host) as smtp_serv:
for i in range(10):
msg = MIMEText('#%d timestamp %s' % (i, datetime.utcnow()))
msg['Subject'] = 'test #%d' % i
msg['From'] = login
msg['To'] = login
smtp_serv.send_message(msg)
Output
smtp setup took (5.43 seconds passed)
login took 0.40 seconds
Operations with smtp_serv took 9.84 seconds
Quiting took 0.05 seconds
If your Python version doesn't have .send_message() then you could use:
smtp_serv.sendmail(from, to, msg.as_string())

You are opening the connection to the SMTP server and then closing it for each email. It would be more efficient to keep the connection open while sending all of the emails.

Maybe this comes very late, but I think it is relevant for the matter.
I had the same issue recently and realized, by searching around, that the call to connect the SMTP server may be very time consuming due to issues with domain name resolution, since the SMTP server performs a reverse lookup to verify the connecting client.
In my case this call was taking around 1 minute!:
s = smtplib.SMTP(smtp_server)
Solution was to fix the domain name resolution on the Linux box. After that, connection became very quick.
Hope this may be of help.

The real answer here is "profile that code!". Time how long different parts of the code take so you know where most of the time is spent. That way you'll have a real answer without guesswork.
Still, my guess would be that it is the calls to smtp_serv.verify(recipient) may be the slow ones. Reasons might be that the server sometimes needs to ask other SMTP servers for info, or that it does throttling on these operations to avoid having spammers use them massively to gather email addresses.
Also, try pinging the SMTP server. If the ping-pong takes significant time, I would expect sending each email would take at least that long.

Related

Send email at specific time with millisecond precision

I would like to send email at given time preferably using gmail. The rationale behind this is that the school I am applying is ordering candidates based on when they receive the participation email after given time.
I could use gmail schedule send feature but there is X delay between sending email from gmail server to school email and it potentially should be achieveable to cut it down slightly so I thought about python script to do it. I think I can get around sending it at a given time but struggle to actually send the message.
There are threads in stackoverflow suggesting python solution e.g.: How do I schedule an email to send at a certain time using cron and smtp, in python? but unfortunately it looks like gmail disabled the option to send mails from non-authorized apps.
Other threads suggest that enabling less secure apps is needed. Unfortunately, this setting has been closed by Google. What could be the way around it?
My sample code:
`
def send_mail():
try:
server_ssl = smtplib.SMTP_SSL('smtp.gmail.com', 465)
server_ssl.ehlo() # optional
print('Server initialized')
sent_from = gmail_user
to = ['xxxxxxxx#gmail.com']
subject = 'my subject'
body = 'my body'
email_text = """\
From: %s
To: %s
Subject: %s
%s
""" % (sent_from, ", ".join(to), subject, body)
server_ssl.sendmail(sent_from, to, email_text)
server_ssl.close()
except Exception as inst:
print('Something went wrong')
print(type(inst)) # the exception instance
print(inst.args) # arguments stored in .args
print(inst)
`
is returning an error:
<class 'smtplib.SMTPSenderRefused'> (530, b'5.7.0 Authentication Required. Learn more at\n5.7.0 https://support.google.com/mail/?p=WantAuthError
You need to supply the login to the account, as well as an apps password. You cant just send an email without being authenticated to the mail server.
with smtplib.SMTP_SSL('smtp.gmail.com', 465, context=context) as server:
print( 'waiting to login...')
server.login(sender_email, password)
print( 'waiting to send...')
server.sendmail(sender_email, receiver_email, text)

How to Handle python script errors and sending them by email

I am relatively new to Python.
I have a Python script that grabs some information from the internet and then inserts in a database. I want to run it with cron daily and have the script send me an email each day after it is done.
I already have the email part figured out and it is working for sending the elapsed time and the total records inserted in the database, like this:
at the start of the script:
records_inserted = 0
t1_start = time.perf_counter()
and at the end of the script:
t1_stop = time.perf_counter()
msg = EmailMessage()
msg.set_content("Total elapsed time: %.1f [min]" % ((t1_stop-t1_start)/60) +
"\n\nThe total number of records inserted in the database was: " +
str(records_inserted))
email_from = “xxxxxxx#gmail.com"
email_to = “xxxxx#xxxx.com"
gmail_password = “xxxxxx”
msg['From'] = email_from
msg['To'] = email_to
msg['Subject'] = “Script executed"
server = smtplib.SMTP('smtp.gmail.com', 587)
server.starttls()
server.login(email_from, gmail_password)
server.send_message(msg)
server.quit()
Now, if something in the webpages where I'm getting the content from breaks or changes or there is any error for any reason, I would like to see the errors also in that daily email.
What would be the best way to handle and store(maybe an array) in case there would be one or several errors during execution, and have the error number and description written as part of the email body so I am aware as soon as possible and can take a look and correct as necessary?
Thanks!
The solution proposed by #fernandezcuesta in the comments is working nicely for me.
I wrappped all the code that 'does the stuff' into a try: except Exception as _ex: and attach repr(_ex) to my mail

Accessing Gmail SMTP via Python - AUTH issues

I'm writing this currently for an assignment but years back I had the same problem and just gave up. Further searches had me try a variety of things and I cannot get past the login portion.
from socket import *
from ssl import *
msg = "\r\n I love computer networks!"
endmsg = "\r\n.\r\n"
clientSocket = socket(AF_INET, SOCK_STREAM)
mailserver = getaddrinfo('smtp.gmail.com',465, AF_INET, SOCK_STREAM, IPPROTO_TCP)[0][4]
clientSocket = wrap_socket(clientSocket)
clientSocket.connect(mailserver)
recv = clientSocket.recv(1024)
print recv
if recv[:3] != '220':
print '220 reply not received from server.'
clientSocket.send('EHLO Nolan\r\n')
recv = clientSocket.recv(1024)
print recv
if recv1[:3] != '250':
print '250 reply not received from server.'
clientSocket.send('AUTH LOGIN ' + 'my Google email'.encode('base64','strict'))
recv = clientSocket.recv(1024)
print recv
print recv.split()[1].decode('base64')
clientSocket.send("my password".encode('base64','strict'))
recv = clientSocket.recv(1024)
print recv
# And onto other tasks
I'm under the impression I don't need STARTTLS as I start the connection with SSL. If I change AUTH LOGIN to AUTH PLAIN it doesn't know how to decode it. If I use PLAIN but don't encode it my program just hangs. If I terminate it with "\n\r" (not encoded) I get a broken pipe. Using the code above I get the password request.
Then sending the password I get 535-5.7.8 Username and Password not excepted.It's the same password I sign in with?!? I already setup my account to allow less secure devices. 2-step verification... started the process seems to be orientated to mobile phones, not my python app.
I've changed EHLO to use my IP, as I believe RFC 5321 says that's how it should be. Didn't matter.
I've looked through RFC 4954...
I dug into smtplib to uncover "AUTH PLAIN " + encode_base64("\0%s\0%s" % (user,pass),eol='') Just hangs... waiting...
I have no clue.
Update
I changed the smtplib to output every string submitted. This confuses me even more:
ehlo [127.0.1.1]
AUTH PLAIN [almost forgot one can decode this base64]==
mail FROM:<[gmail account]>
rcpt TO:<[hotmail account]>
data
quit
My AUTH string is exactly the same but it hangs. And I am receiving the email at my hotmail account (using smtplib, not my code).
If you want a basic, non-SSL connection to Gmail, you can use this:
import smtplib
gmail = smtplib.SMTP("smtp.gmail.com",587)
gmail.ehlo() #open the connection
gmail.starttls()
gmail.login("gmaillogin","gmailpassword") #login*
gmail.quit() #close the connection
*Note: You will need to generate a gmail application password from your google account.
Edit:
If you want to use SSL encryption, use this (thanks to furas):
import smtplib
gmail = smtp.SMTP_SSL("smtp.gmail.com",465)
gmail.ehlo()
gmail.login("gmaillogin","gmailpassword")
gmail.quit()
As is typically my fashion, it's the simple things. I just spent hours looking over one program and Googling only to have one error pop out at me: the parameter order in my definition was not how I used it in my code.
Here... I changed my code to use AUTH PLAIN and the issue... I had been putting in '\n\r' not '\r\n' (if I had ended the command with anything, only omitted because '\n\r' produced errors on some commands and went through okay on others). Thank you for classical music to study by on your favorite streaming video site. It raised my intelligence briefly. :)
clientSocket.send('AUTH PLAIN ' + encode_base64("\0%s\0%s" % ([email],[password]), eol="") + '\r\n')

pySerial buffering gpsd coordinates on Raspberry Pi

I am having quite a frustrating issue, and I'm quite new to Python/Raspberry pi
I have a script that is to match gps coordinates with a SSID filtered iwlist scan and then email the resulting file out whenever there is a wifi connection.
My issue is, the results are continuously buffered, so the gps coordinates could be significantly far away from the scanned SSIDs.
I have the serial line being flushed at the beginning of the while loop, but it doesn't seem to work. I also have python running un-buffered (-u, but that may only be for the text files..) I added Output.flush(), but I have not been able to test it yet. Figured I'd ask first.
So, my question is, is there a way to turn off the serial line buffer so every iteration is getting the GPS coordinate at the time the while loop is executing? All of my text files have file.flush() after they have been written. Does that need to be before the files are written, or would that not affect the serial buffer? What am I missing?
Any help would be appreciated
#!/usr/bin/python -u
import os
import gps
import time
import serial
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email import Encoders
username = 'gmail account'
password = 'account password'
port = serial.Serial("/dev/ttyAMA0", baudrate=9600)
#Setting GPS session, listen on port 2947 (gpsd) of localhost
session = gps.gps("localhost", "2947")
session.stream(gps.WATCH_ENABLE | gps.WATCH_NEWSTYLE)
#Begin GPS and WiFi Scan loop..
while True:
try:
port.flushInput()
#port.flushOutput() commented out because I haven't tested with this yet.
time.sleep(.5)
report = session.next()
#Uncomment below for report data
#print report
if report['class'] == 'TPV':
if hasattr(report, 'lat' and 'lon'):
#setting GPS variables and strings
latitude = report.lat
latString = "%f" %(latitude)
longitude = report.lon
lonString = "%f" %(longitude)
#WiFi scan and parse script. I don't think it is necessary to post,
#but if it is, I will. All text files are flushed before being closed
#Email when connected
ping = os.system('sudo ping -c4 8.8.8.8')
try:
if ping == 0:
msg = MIMEMultipart()
msg['Subject'] = "GPS/WiFi data from GPS PILOT"
msg['From'] = username
msg['To'] = username
body = "GPS/WiFi data attached.."
msg.attach(MIMEText(body, 'plain'))
part = MIMEBase('application', "octet-stream")
part.set_payload(open("/home/pi/gpsMaster/dataLog.csv", "rb").read())
Encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="GPSWiFi_Scan.csv"')
msg.attach(part)
server = smtplib.SMTP('smtp.gmail.com:587')
server.ehlo()
server.starttls()
server.login(username, password)
server.sendmail(username, username, msg.as_string())
server.quit()
os.system("sudo rm /home/pi/gpsMaster/dataLog.csv")
else:
pass
except smtplib.SMTPException:
os.system('sudo reboot now')
#Adding loop delay
time.sleep(10)
#Loop ending exceptions
except KeyError:
pass
except IOError:
print ("ERROR")
#os.system('sudo reboot now')
except KeyboardInterrupt:
quit()
except StopIteration:
session = None
os.system("sudo reboot now")
I was having a similar issue. I fixed by flushing input and output.
port.flushInput()
port.flushOutput()
I appreciate that this is an old question, but it is still relevant as I hit the same problem. The issue is due to:
time.sleep(10)
in the main loop. The gpsd thread is adding received data to a queue which is accessed via:
report = session.next()
This mean that when you get to pull the next sample, over time it will drift further and further from reality (you are using GPS coordinates that lag behind your current location). We have a big queue, the consumer is pulling data much more slowly than it is being added due to the 10s sleep. I had much fun solving this and only by cutting the system to its bare essentials did I finally happen across the cause. The solution is to pull the data as soon as it is available, but discard as many data points as necessary until your delay time has been achieved. This stops the gpsd queue getting out of sync as the gpsd Python client is consuming them too slowly which causes coordinate drift over time.
Pseudo code (it can be polished, an exercise left for the reader):
import time
start = time.time()
while True:
# Consume the report from the gpsd
report = session.next()
# What time is it?
now = time.time()
# Magic number of 10s
if now > (start + 10):
# use the data contained in report
# Update with the latest time ready for the next iteration.
start = now

Error smtplib in python

i've a python script, run on a raspeberry pi that monitoring my server room and periodically send me an e-mail and make other stuff.
First time script is running it send a notification and works fine, then wait for a time (5 or 30 minutues) depending if an alarm is triggered, so i've write a for loop to wait. when the loop is finish, script make a call of the function for sending e-mail and python return an error.
Could someone help me ?
i test it also on debian 7.7.0 box and i've the same error.
Below a sample script and the error.
Thanks in advance.
Dom.
File "/usr/lib/python2.7/smtplib.py", line 361, in getreply
raise SMTPServerDisconnected("Connection unexpectedly closed")
smtplib.SMTPServerDisconnected: Connection unexpectedly closed
# Import
from time import sleep
import time
import smtplib
import sys
import string
smtp_server = "10.0.0.4" # mail server address
mail_from = "server#domain.com"
mail_to = "user#gmail.com"
mail_subj = "Server room notify"
server = smtplib.SMTP(smtp_server)
valore = 24 # value for the funtion
def SendAlertEmail(t_value):
log_t = time.strftime("%d/%m/%A %H:%M:%S ")
Text = string.join("Server room notify \n " + str(log_t) + str(t_value))
Body_email = string.join((
"From: %s" % mail_from,
"To: %s" % mail_to,
"Subject: %s" % mail_subj,
"",
Text
), "\r\n")
server.sendmail(mail_from,mail_to,Body_email)
return
SendAlertEmail(valore) # calling from here works fine
for i in range(0,500):
sleep(1)
print(str(i))
SendAlertEmail(valore) # after 500 secs goes in error. Test on debian 7.7.0 python 2.7.3 gcc 4.7.2
smtplib opens a TCP connection at the creation of the SMTP object. You try to reuse it after 500 seconds of inactivity when it is likely that a timeout has already closed the connection.
You should keep only the server name, an create a new SMTP object for each connection in your use case, because the delay between sending messages is too high.
As Serge Ballesta mentions, you should open and close the connection to the server for each one the sent e-mails.
Some months ago I programmed a class to send e-mails in Python, which you can find here: https://github.com/kmundnic/Habitus/blob/master/data_sender.py You create a DataSender object and it handles the connections to the server. Also, it calls a JSON file where the accounts and password is stored, so you don't save it in your code.
Feel free to use it. It was adapted from http://kutuma.blogspot.com/2007/08/sending-emails-via-gmail-with-python.html, as it's mentioned in the comments at the beginning of the source code.

Categories