Disclaimer: I hesitated on the title, due to the broad nature of this question (see below), other options included:
How to send a mail from localhost, using Python code only?
How to send email from Python code, without usage of external SMTP server?
Is it possible to send an email DIRECTLY to it's destination, using localhost and Python only?
First, a little bit of context:
For the sake of learning I am building a website with user registration feature. The idea is that after registration user will receive an email with activation link. I would like to compose & send email from Python code, and that is the part where I would like to ask for some clarifications.
My understanding, before I began (obviously naive =) can be illustrated like this (given that there is a legitimate email address user#example.com):
After searching for examples, I bumped into some questions & answers on stackoverflow (1, 2, 3, 4). From these I've distilled the following snippet, to compose and send an email from Python code:
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Message content here')
message['Subject'] = 'Your subject here'
message['From'] = 'me#example.com'
message['To'] = 'user#example.com'
smtp_server = smtplib.SMTP('smtp.server.address:587')
smtp_server.send_message(message)
smtp_server.quit()
Next (obvious) question was what to pass to smtplib.SMTP() instead of 'smtp.server.address:587'. From the comments to this answer, I discovered that local SMTP server (just for testing purposes though) could be started via python3 -m smtpd -c DebuggingServer -n localhost:1025, then smtp_server = smtplib.SMTP('smtp.server.address:587') could be changed to smtp_server = smtplib.SMTP('localhost:1025') and all the sent emails will be displayed in the console (from where python3 -m smtpd -c DebuggingServer -n localhost:1025 command was executed), being enough for testing — it was not what I wanted (my aim was — the ability to send a mail to 'real-world' email address from local machine, using Python code only).
So, the next step would be to setup a local SMTP server, capable of sending an email to external 'real-world' email-address (as I wanted to do it all from Python code, so the server itself would better be implemented in Python too). I recalled reading in some magazine (in early 2000), that spammers use local servers for sending mails (that particular article was talking about Sambar, development for which have ended in 2007, and which was not written in Python :-) I thought there should be some present-day solution with similar functionality. So I started searching, my hope was to find (on stackoverflow or elsewhere) a reasonably short code snippet, which will do what I wanted. I haven't found such a code snippet, but I came across a snippet titled (Python) Send Email without Mail Server (which uses chilkat API), though all I needed (supposedly) was right there, in the comments to code, the first line clearly stated:
Is it really possible to send email without connecting to a mail server? Not really.
and a few lines below:
Here's what happens inside those other components that claim to not need a mail server: The component does a DNS MX lookup using the intended recipient's email address to find the mail server (i.e. SMTP server) for that domain. It then connects to that server and delivers the email. You're still connecting to an SMTP server — just not YOUR server.
Reading that, made me understand — I, clearly, was lacking some details in my understanding (reflected on picture above) of the process. To correct this I have read the whole RFC on SMTP.
After reading the RFC, my improved understanding of the process, might be pictured like this:
From this understanding, came the actual questions I'd like to clarify:
Can my "improved understanding" be considered correct, as a general picture?
What addresses, exactly, are returned by MX lookup?
using host -t mx gmail.com command (suggested by this answer), I was able to retrieve the following:
gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
- but none of these are mentioned in the [official docs][13] (ones that are there: `smtp-relay.gmail.com`, `smtp.gmail.com`, `aspmx.l.google.com`)
Is authentication always needed to pass an email to SMTP-server of an established mail service (say gmail)?
I understand that to use, say smtp.gmail.com for mail submission, you'll need, regardless if the recipient has a #gmail address or not (as it stated in docs):
Your full Gmail or G Suite email address is required for authentication.
But, if an email to user#gmail.com is submitted to SMTP-server not owned by gmail, then it'll be redirected to one of the gmail servers (directly or via gateway/relay). In this case (I assume) sender of an email will only need to authenticate on mail submission, so after that gmail server will accept the mail without authentication?
If yes, what is preventing me from "pretending" to be such a gateway/relay and hand over emails directly to their designated SMTPs? Then it, also should be pretty easy to write a "proxy-SMTP", which will just search for an appropriate server via MX lookup, and hand an email to it, sort of, directly.
Documentation on gmail SMTP, also mentions aspmx.l.google.com server, which does not require authentication, though:
Mail can only be sent to Gmail or G Suite users.
With that being said, I assume the following snippet should work, for submitting a mail to ExistingUser#gmail.com mailbox:
<!-- language: lang-python -->
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Message test content')
message['Subject'] = 'Test mail!'
message['From'] = 'me#whatever.com'
message['To'] = 'ExistingUser#gmail.com'
smtp_server = smtplib.SMTP('aspmx.l.google.com:25')
smtp_server.send_message(message)
smtp_server.quit()
When ran, the code above (with ExistingUser#gmail.com replaced by the valid mail) throws OSError: [Errno 65] No route to host. All I want to confirm here is that the communication to aspmx.l.google.com is handled correctly in code.
Your understanding of how mail works is roughly correct. Some additional notes that may clear things up:
SMTP is used for two distinct purposes. You seem to be confusing these two.:
The first use, typically called "submission", is to send a mail from an MUA (Mail User Agent, your mail program, Outlook, Thunderbird, ...) to an MTA (Mail Transfer Agent, typically called "mail server"). MTAs are run by your ISP, or by mail-providers such as GMail. Typically, their use is restricted by either IP address (only customers of said ISP can use it), or username/password.
The second use is to send mail from one MTA to another MTA. This part is, usually, wide open, since you are probably willing to accept inbound mail from anyone. This is also the location where anti-spam measures are taken.
In order to send a mail, you need, at least, the second part of SMTP: the ability to talk to another MTA to deliver the mail.
The typical way to send mails is to compose the mail in your application, then send it off to an MTA mail server for delivery. Depending on your setup, that MTA can be either installed on the same machine as your Python code is running on (localhost), or can be a more "central" mail server (possibly requiring authentication).
"Your" MTA will take care of all the nasty details of delivering mail such as:
Doing DNS lookups to find out the MTA's to contact to relay the mail. This includes MX-lookup, but also other fallback mechanisms such as A-records.
Retrying delivery, if the first attempt fails temporarily
Generating a bounce message, if the message fails permanently
Make multiple copies of the message, in case of multiple recipients on different domains
Signing the message with DKIM to reduce the chance of it being marked as SPAM.
...
You could, of course, re-implement all these features within your own Python code, and effectively combine an MTA with your application, but I strongly advise against it. Mail is surprisingly hard to get right...
Bottom line: Try to send the mail via SMTP to the mail server of your provider or another mail service. If that is not possible: think really hard if you want to run your own mail server. Being marked as a spammer happens easily; getting removed from spam-lists is much harder. Don't re-implement SMTP-code in your application.
Thanks to these answers, to my additional questions: 1, 2, 3, as well as these two questions (and answers) of other people: one, two — I think I am now ready to answer the questions I have posted, on my own.
I will address the questions one by one:
Yes, as a general picture, sending of an email can be portrayed like this:
MX lookup returns address(es) of server(s) which receive email destined to the specified domain.
As to "Why smtp-relay.gmail.com, smtp.gmail.com, aspmx.l.google.com are not returned by host -t mx gmail.com command?". This point is, pretty much, covered in another answer to this question. The main points to grasp here are:
servers returned by MX lookup are responsible for receiving of emails for the domain (gmail, in this particular case)
servers listed in gmail docs are meant for the mail sending (i.e. mails that gmail user wants to send, to other gmail user or otherwise, are submitted to those servers)
Authentication is not needed, for servers receiving emails (i.e. the ones returned by MX lookup).
There are a couple things that prevent such servers from being abused:
many ISPs block outbound connections to port 25 (which is default port for mail receiving servers), to prevent such "direct" mail sending
there are numerous measures taken on the side of receiving servers, which are, mainly, intended to prevent spamming, but as a result will, likely, prevent such "direct" mail sending, as well (some examples are: DNSBL — list of blocked IPs, DKIM — is an email authentication method designed to detect forged sender addresses in emails (if you do not have your own, legitimate, mail server, you will use someone other's domain for From field, that is where you might be hit by DKIM)
Code snippet is OK. The error is produced, in all probability, due to the blocking on the ISP's side.
With all that being said, code snippet:
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Message content here')
message['Subject'] = 'Your subject here'
message['From'] = 'me#example.com'
message['To'] = 'user#example.com'
smtp_server = smtplib.SMTP('smtp.server.address:25')
smtp_server.send_message(message)
smtp_server.quit()
will actually send an email (see this question, for real-world, working example) given that smtp.server.address:25 is legitimate server and there are no blockings on ISP's and/or smtp.server.address side.
I have a pretty basic networking assignment where I have to make a manual SMTP request (with the HELO, MAIL FROM:, and the like) as the client to the SMTP server in order to send an email.
It doesn't seem like you can do this anymore with Gmail, as it requires an encrypted SMTP request for obvious reasons.
I know I can do this with OpenSSL, but this is to be build with Python and it seems like an external library would have to be used for this, which isn't allowed on the assignment.
So if possible I'd like to be able to submit this unencrypted for simplicity's sake. Are there any test servers to send non-encrypted SMTP over now a days?
Inbox.py is here to save you (made by the author of python-requests: Kenneth Reitz), it's a very simple SMTP server "made for Humans".
Useful for your test purpose, I think.
I'm building a application that needs to send and receive emails.
However I do not want to have a separate email server (or use IMAP and POP3), since I need to create/delete/manage inboxes on the fly, with no email inbox passwords, etc.
I have an email storage database in place, and I can receive emails by using a custom smtpd server, replacing postfix. However, that way I'm not able to send emails via postfix (using smtplib, connecting through port 25 to postfix and sending emails)
Any solution to this problem? How to send emails with a custom smtp server? Can I configure postfix to relay all incoming emails to a custom smtp server running in another port, and still use postfix on port 25 to send emails?
Thanks for your time
By using a custom SMTP server, you run the risk of inadvertently creating security holes or violating the SMTP protocol in some way. With so many great SMTP servers out there (Postfix, exim, sendmail...), that doesn't sound like a good option to me.
The easiest way I can think of to solve that issue is to use Postfix to relay inbound and outbound e-mail.
Inbound e-mail can be configured to be piped to an application, and outbound e-mail can be configured to be delivered by Postfix, either directly or relayed through a different server.
This way, you can use, instead of a custom SMTP server, an application that is able to parse RFC822-compliant message files. This is better than doing exactly the same thing, but with the overhead of having to implement the SMTP protocol.
This approach probably won't scale well should you need to receive a high volume of messages - every message will fork+exec a new process. If that is a requirement, a good approach would be to keep a custom SMTP server to do that job, but let Postfix relay it the messages - you will then benefit from Postfix's architecture in front of your parser.
Assuming you follow the approach of piping the messages to an application, all you need to do in Postfix is to
Configure Postfix's alias_maps' parameter to look for such a map:
alias_maps = hash:/etc/aliases, hash:/etc/postfix/app-aliases
Then, configure the map to pipe messages sent to each address into an application:
test: "|/usr/local/bin/your-app"
As usual, don't forget to $ postalias app-aliases.
This will make a message sent to test#yourdomain be piped into /usr/local/bin/your-app, which acts as an e-mail gateway to your application.
buildin an smtp client in python . which can send mail , and also show that mail has been received through any mail service for example gmail !!
Create mail messages (possibly with multipart attachments) with email.
The email package is a library for managing email messages, including MIME and other RFC 2822-based message documents.
Send mail using smtplib
The smtplib module defines an SMTP client session object that can be used to send mail to any Internet machine with an SMTP or ESMTP listener daemon.
If you are interested in browsing a remote mailbox (for example, to see if the message you sent have arrived), you need a mail service accessible via a known protocol. An popular example is the imaplib module, implementing the IMAP4 protocol. IMAP is supported by gmail.
This (imaplib) module defines three classes, IMAP4, IMAP4_SSL and IMAP4_stream, which encapsulate a connection to an IMAP4 server and implement a large subset of the IMAP4rev1 client protocol as defined in RFC 2060. It is backward compatible with IMAP4 (RFC 1730) servers, but note that the STATUS command is not supported in IMAP4.
If you want the Python standard library to do the work for you (recommended!), use smtplib. To see whether sending the mail worked, just open your inbox ;)
If you want to implement the protocol yourself (is this homework?), then read up on the SMTP protocol and use e.g. the socket module.
Depends what you mean by "received". It's possible to verify "delivery" of a message to a server but there is no 100% reliable guarantee it actually ended up in a mailbox. smtplib will throw an exception on certain conditions (like the remote end reporting user not found) but just as often the remote end will accept the mail and then either filter it or send a bounce notice at a later time.
I am writing a web application that requires user interaction via email. I'm curious if there is a best practice or recommended source for learning about processing email. I am writing my application in Python, but I'm not sure what mail server to use or how to format the message or subject line to account for automated processing. I'm also looking for guidance on processing bouncebacks.
There are some pretty serious concerns here for how to send email automatically, and here are a few:
Use an email library. Python includes one called 'email'. This is your friend, it will stop you from doing anything tragically wrong. Read an example from the Python Manual.
Some points that will stop you from getting blocked by spam filters:
Always send from a valid email address. You must be able to send email to this address and have it received (it can go into /dev/null after it's received, but it must be possible to /deliver/ there). This will stop spam filters that do Sender Address Verification from blocking your mail.
The email address you send from on the server.sendmail(fromaddr, [toaddr]) line will be where bounces go. The From: line in the email is a totally different address, and that's where mail will go when the user hits 'Reply:'. Use this to your advantage, bounces can go to one place, while reply goes to another.
Send email to a local mail server, I recommend postfix. This local server will receive your mail and be responsible for sending it to your upstream server. Once it has been delivered to the local server, treat it as 'sent' from a programmatic point of view.
If you have a site that is on a static ip in a datacenter of good reputation, don't be afraid to simply relay the mail directly to the internet. If you're in a datacenter full of script kiddies and spammers, you will need to relay this mail via a public MTA of good reputation, hopefully you will be able to work this out without a hassle.
Don't send an email in only HTML. Always send it in Plain and HTML, or just Plain. Be nice, I use a text only email client, and you don't want to annoy me.
Verify that you're not running SPF on your email domain, or get it configured to allow your server to send the mail. Do this by doing a TXT lookup on your domain.
$ dig google.com txt
...snip...
;; ANSWER SECTION:
google.com. 300 IN TXT "v=spf1 include:_netblocks.google.com ~all"
As you can see from that result, there's an SPF record there. If you don't have SPF, there won't be a TXT record. Read more about SPF on wikipedia.
Hope that helps.
Some general information with regards to automated mail processing...
First, the mail server "brand" itself isn't that important for broadcasting or receiving emails. All of them support the standard smtp / pop3 communications protocol. Most even have IMAP support and have some level of spam filtering. That said, try to use a current generation email server.
Second, be aware that in an effort to reduce spam a lot of the receiving mail servers out there will simply throw a message away instead of responding back that a mail account doesn't exist. Which means you may not receive those.
Bear in mind that getting past spam filters is an art. A number of isp's watch for duplicate messages, messages that look like spam based on keywords or other content, etc. This is sometimes independent of the quantity of messages sent; I've seen messages with as few as 50 copies get blocked by AOL even though they were legitimate emails. So, testing is your friend and look into this article on wikipedia on anti-spam techniques. Then make sure your not doing that crap.
**
As far as processing the messages, just remember it's a queued system. Connect to the server via POP3 to retrieve messages, open it, do some action, delete the message or archive it, and move on.
With regards to bouncebacks, let the mail server do most of the work. You should be able to configure it to notify a certain email account on the server in the event that it is unable to deliver a message. You can check that account periodically and process the Non Delivery Reports as necessary.