Most IMAP email clients have the ability to receive emails automatically in real time, without refreshing on a fixed interval. This is because the email client keeps a connection open to the server, and if there isn't any email data exchanged to keep the connection alive, the email client will send a NOOP packet on a fixed interval (just like other TCP protocols have their own keepalive packets). In Thunderbird, this behavior can be controlled by the 'Allow immediate server notifications when new messages arrive' option.
I'm writing a Python program that needs to know instantly when emails come in. The standard procedure for receiving emails is (after connection and login) select('inbox'), search(None, 'ALL') (or some specific search term), and fetch(). I can't figure out how to listen for new emails in real time. Every guide I've read suggests running the search() and fetch() functions in a loop. I've tried running the read() function while sending myself emails, but I have never seen read() output any data. Perhaps the server needs to know to push emails to the client? I haven't been able to confirm or refute this theory.
How can I instantly receive new emails with imaplib, or a similar Python library?
While I did not know at the time of originally posting this question, I was looking for a solution that implements the IMAP IDLE command, as defined in RFC2177.
Since you want to fetch emails asynchronously you should use the below library
https://github.com/bamthomas/aioimaplib
I used aioimaplib in my code, but imaplib2 also supposedly supports IDLE; see: https://web.archive.org/web/20090814230913/http://blog.hokkertjes.nl/2009/03/11/python-imap-idle-with-imaplib2/
Given an email address is it possible to determine what the smtp address is using python. I am building an email application using python smtplib.
Thanks
The Python smtplib module does not perform mail routing, so it leaves that to a smart host.
You could use dnspython to look up MX records for the destination domain (and A and AAAA records if there is no MX record). The resulting address you can then pass to smtplib.
However, this is only a small fraction of the work that is needed to implement proper mail routing: If the target mail server is unreachable or times out, you are supposed to try the next server. Similarly if it responds with a 4xx error code. If all servers are unavailable, you need to temporarily store the message locally, so you need to implement your own mail queue. Furthermore, your external IP address may be blacklisted for mail delivery.
Therefore, it is usually easier to install a local mail server such as Exim or Postfix and use that to inject mail, possibly routing it via a smart host with good email reputation to avoid blacklisting.
I hope this is not too opinionated because I'm looking for a finite answer.
I'm developing a UDP server in Python 3.x that is utilizing asyncio. The server is going to work with a game engine to process pretty much every interaction a player makes in the game. Therefore, I have to authenticate the game client with the game server in some way as well ensure replay attacks do not happen on top of everything else that could harm or spoof the game server.
When it comes to authenticating with UDP, I'm at a loss. My plan is to have both the game client and game server authenticate per user and game session. That means having something like a public key on the client end and a private key on the server end where the server can authenticate the client is approved.
During that authentication, I am going to generate a symmetric key that the game server makes and passes it down to the game client so every packet sent can be verified with that symmetric key using HMAC. If not, all packets are dropped.
My Question
Is this the best approach? Creating some type of public key where something like a token is generated per session to ensure packets coming to the UDP server are from authenticated clients? My worry here is the keys are still stored in a Windows EXE file and still likely can be cracked and extracted or am I just too paranoid?
Going to answer my own question!
No It's Not The Best Approach
The reason this is not the best approach is because even with this process, my use of public/private key was incorrect. I was not referring to encrypting data between the client and server, I was referring to creating a nonce with having some type of password on the client that matches the password on the server.
This means the moment I send credentials from the client to the server, it's being sent without encryption on the line or in the buffer to the server. Once this authentication is accepted by the server, it will hash the password, store it and then send the nonce back to the client, which will also be without encryption.
The username, the password, and the nonce will be vulnerable to Man in the Middle attacks from both a third-party and the client itself. Which means they can sniff the packets, snag the transmission and crack the user or attempt replays being they have the nonce.
Solution
My game client does not have SSL/TSL/DTSL options. This is why I actually posted the question, but did not clarify this. My server, as it's custom within Python, does have that ability though. But, I found a plugin for the game engine last night that allows for AES encryption of the data being written to the buffer before said buffer is sent as a packet to the UDP server, which means I can encrypt the passing of data in those packets in a secure way that hopefully the server can decrypt with a legit private key.
Doing this will be a good option because it likely will be my only option. Now when a malicious user sniffs the traffic, they wont be able to see credentials or the nonce. The only issue after that is cracking the EXE, which I'm fine with for now.
Disclaimer: I hesitated on the title, due to the broad nature of this question (see below), other options included:
How to send a mail from localhost, using Python code only?
How to send email from Python code, without usage of external SMTP server?
Is it possible to send an email DIRECTLY to it's destination, using localhost and Python only?
First, a little bit of context:
For the sake of learning I am building a website with user registration feature. The idea is that after registration user will receive an email with activation link. I would like to compose & send email from Python code, and that is the part where I would like to ask for some clarifications.
My understanding, before I began (obviously naive =) can be illustrated like this (given that there is a legitimate email address user#example.com):
After searching for examples, I bumped into some questions & answers on stackoverflow (1, 2, 3, 4). From these I've distilled the following snippet, to compose and send an email from Python code:
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Message content here')
message['Subject'] = 'Your subject here'
message['From'] = 'me#example.com'
message['To'] = 'user#example.com'
smtp_server = smtplib.SMTP('smtp.server.address:587')
smtp_server.send_message(message)
smtp_server.quit()
Next (obvious) question was what to pass to smtplib.SMTP() instead of 'smtp.server.address:587'. From the comments to this answer, I discovered that local SMTP server (just for testing purposes though) could be started via python3 -m smtpd -c DebuggingServer -n localhost:1025, then smtp_server = smtplib.SMTP('smtp.server.address:587') could be changed to smtp_server = smtplib.SMTP('localhost:1025') and all the sent emails will be displayed in the console (from where python3 -m smtpd -c DebuggingServer -n localhost:1025 command was executed), being enough for testing — it was not what I wanted (my aim was — the ability to send a mail to 'real-world' email address from local machine, using Python code only).
So, the next step would be to setup a local SMTP server, capable of sending an email to external 'real-world' email-address (as I wanted to do it all from Python code, so the server itself would better be implemented in Python too). I recalled reading in some magazine (in early 2000), that spammers use local servers for sending mails (that particular article was talking about Sambar, development for which have ended in 2007, and which was not written in Python :-) I thought there should be some present-day solution with similar functionality. So I started searching, my hope was to find (on stackoverflow or elsewhere) a reasonably short code snippet, which will do what I wanted. I haven't found such a code snippet, but I came across a snippet titled (Python) Send Email without Mail Server (which uses chilkat API), though all I needed (supposedly) was right there, in the comments to code, the first line clearly stated:
Is it really possible to send email without connecting to a mail server? Not really.
and a few lines below:
Here's what happens inside those other components that claim to not need a mail server: The component does a DNS MX lookup using the intended recipient's email address to find the mail server (i.e. SMTP server) for that domain. It then connects to that server and delivers the email. You're still connecting to an SMTP server — just not YOUR server.
Reading that, made me understand — I, clearly, was lacking some details in my understanding (reflected on picture above) of the process. To correct this I have read the whole RFC on SMTP.
After reading the RFC, my improved understanding of the process, might be pictured like this:
From this understanding, came the actual questions I'd like to clarify:
Can my "improved understanding" be considered correct, as a general picture?
What addresses, exactly, are returned by MX lookup?
using host -t mx gmail.com command (suggested by this answer), I was able to retrieve the following:
gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com.
gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
- but none of these are mentioned in the [official docs][13] (ones that are there: `smtp-relay.gmail.com`, `smtp.gmail.com`, `aspmx.l.google.com`)
Is authentication always needed to pass an email to SMTP-server of an established mail service (say gmail)?
I understand that to use, say smtp.gmail.com for mail submission, you'll need, regardless if the recipient has a #gmail address or not (as it stated in docs):
Your full Gmail or G Suite email address is required for authentication.
But, if an email to user#gmail.com is submitted to SMTP-server not owned by gmail, then it'll be redirected to one of the gmail servers (directly or via gateway/relay). In this case (I assume) sender of an email will only need to authenticate on mail submission, so after that gmail server will accept the mail without authentication?
If yes, what is preventing me from "pretending" to be such a gateway/relay and hand over emails directly to their designated SMTPs? Then it, also should be pretty easy to write a "proxy-SMTP", which will just search for an appropriate server via MX lookup, and hand an email to it, sort of, directly.
Documentation on gmail SMTP, also mentions aspmx.l.google.com server, which does not require authentication, though:
Mail can only be sent to Gmail or G Suite users.
With that being said, I assume the following snippet should work, for submitting a mail to ExistingUser#gmail.com mailbox:
<!-- language: lang-python -->
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Message test content')
message['Subject'] = 'Test mail!'
message['From'] = 'me#whatever.com'
message['To'] = 'ExistingUser#gmail.com'
smtp_server = smtplib.SMTP('aspmx.l.google.com:25')
smtp_server.send_message(message)
smtp_server.quit()
When ran, the code above (with ExistingUser#gmail.com replaced by the valid mail) throws OSError: [Errno 65] No route to host. All I want to confirm here is that the communication to aspmx.l.google.com is handled correctly in code.
Your understanding of how mail works is roughly correct. Some additional notes that may clear things up:
SMTP is used for two distinct purposes. You seem to be confusing these two.:
The first use, typically called "submission", is to send a mail from an MUA (Mail User Agent, your mail program, Outlook, Thunderbird, ...) to an MTA (Mail Transfer Agent, typically called "mail server"). MTAs are run by your ISP, or by mail-providers such as GMail. Typically, their use is restricted by either IP address (only customers of said ISP can use it), or username/password.
The second use is to send mail from one MTA to another MTA. This part is, usually, wide open, since you are probably willing to accept inbound mail from anyone. This is also the location where anti-spam measures are taken.
In order to send a mail, you need, at least, the second part of SMTP: the ability to talk to another MTA to deliver the mail.
The typical way to send mails is to compose the mail in your application, then send it off to an MTA mail server for delivery. Depending on your setup, that MTA can be either installed on the same machine as your Python code is running on (localhost), or can be a more "central" mail server (possibly requiring authentication).
"Your" MTA will take care of all the nasty details of delivering mail such as:
Doing DNS lookups to find out the MTA's to contact to relay the mail. This includes MX-lookup, but also other fallback mechanisms such as A-records.
Retrying delivery, if the first attempt fails temporarily
Generating a bounce message, if the message fails permanently
Make multiple copies of the message, in case of multiple recipients on different domains
Signing the message with DKIM to reduce the chance of it being marked as SPAM.
...
You could, of course, re-implement all these features within your own Python code, and effectively combine an MTA with your application, but I strongly advise against it. Mail is surprisingly hard to get right...
Bottom line: Try to send the mail via SMTP to the mail server of your provider or another mail service. If that is not possible: think really hard if you want to run your own mail server. Being marked as a spammer happens easily; getting removed from spam-lists is much harder. Don't re-implement SMTP-code in your application.
Thanks to these answers, to my additional questions: 1, 2, 3, as well as these two questions (and answers) of other people: one, two — I think I am now ready to answer the questions I have posted, on my own.
I will address the questions one by one:
Yes, as a general picture, sending of an email can be portrayed like this:
MX lookup returns address(es) of server(s) which receive email destined to the specified domain.
As to "Why smtp-relay.gmail.com, smtp.gmail.com, aspmx.l.google.com are not returned by host -t mx gmail.com command?". This point is, pretty much, covered in another answer to this question. The main points to grasp here are:
servers returned by MX lookup are responsible for receiving of emails for the domain (gmail, in this particular case)
servers listed in gmail docs are meant for the mail sending (i.e. mails that gmail user wants to send, to other gmail user or otherwise, are submitted to those servers)
Authentication is not needed, for servers receiving emails (i.e. the ones returned by MX lookup).
There are a couple things that prevent such servers from being abused:
many ISPs block outbound connections to port 25 (which is default port for mail receiving servers), to prevent such "direct" mail sending
there are numerous measures taken on the side of receiving servers, which are, mainly, intended to prevent spamming, but as a result will, likely, prevent such "direct" mail sending, as well (some examples are: DNSBL — list of blocked IPs, DKIM — is an email authentication method designed to detect forged sender addresses in emails (if you do not have your own, legitimate, mail server, you will use someone other's domain for From field, that is where you might be hit by DKIM)
Code snippet is OK. The error is produced, in all probability, due to the blocking on the ISP's side.
With all that being said, code snippet:
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Message content here')
message['Subject'] = 'Your subject here'
message['From'] = 'me#example.com'
message['To'] = 'user#example.com'
smtp_server = smtplib.SMTP('smtp.server.address:25')
smtp_server.send_message(message)
smtp_server.quit()
will actually send an email (see this question, for real-world, working example) given that smtp.server.address:25 is legitimate server and there are no blockings on ISP's and/or smtp.server.address side.
I'm trying to code a web chat using the tornado.
The client (user) sends a long post request, during which I send him messages in response. But I am having problems with checking whether the user is online.
When a user sign out or simply close the tab / browser - everything is simple, executed on_connection_close() and I understand that it is disconnected, but if the client lost the Internet connection, then on_connection_close does not work.
How can I check, whether the user is online?
You might want to take look at tornadio2 which is Tornado+Socket.io, and implement multiplex connection, one to handle to push messages, the other is to ping the server so that you can check if the client is still connected or not.
The multiplex does not open multiple connections but uses the single connection to virtually connect to different Handlers. Look at multiplexed.py line 66.
class RouterConnection(SocketConnection):
__endpoints__ = {'/chat': ChatConnection,
'/ping': PingConnection}
The multiplex example is also an sample chat application