I have an application that emails individuals on different occurrences. The entire application is on a single server. I am currently sending emails through SendGrid. At what volume of emails would it make sense to use a system like RabbitMQ to send out emails?
Maximum rate = 1 email per minute? 1 email per second? 10 emails per second? How would I evaluate when the switch makes sense?
Why are you considering RabbitMQ ? it is better to consider using a MTA/Mail relay like Postfix where you submit your emails and it handles them for you in a queue.
You can configure it to dispatch the queue on different mail relays, set the email throughput, how much retry shall be made on a failed sending ...
Having Rabbitmq is good option when your are considering scaling in future, I mean in terms of new smpt send mail workers or new email server, as of now if you have single server and not going to more in it then rabbitmq will load your server even more and will be issue to maintain, but if your going to have more then 100 mails per second then it makes sense to have rabbitmq its goal to make your calling function free as soon as possible by offloading all the load from function to rabbitmq queue and then save it till worker or consumer doesn't pick them, this will help in fail cases too as your having your mails saved in rabbitmq and if the consumer fails you still have your mails, when it starts (smtp send worker) then rabbitmq will provide the rest of the mails to it. I hope this makes sense, please feel free to ask other stuff about it, I used rabbitmq for sending mail but in my case we are having one server running only rabbitmq only so there it makes sense.
It is normally a good idea to look at Postfix or another email server but if you're sending out email to customer on email servers you don't manage (Gmail, Rackspace, MS Live, another company), you might want to look at something that offloads that entirely. I've had luck using SendGrid. They handle a lot of that email related junk for you and they are pretty cheap. Definitely worth the money, outbound SMTP servers are a pain.
EDIT:
It occurred to me that it might not be clear why it would be good to use a service like this if you have external delivery of email. SendGrid (and its ilk) spend a lot of time studying deliverability and they will help make sure your email doesn't end up in the SPAM folder.
Related
Most IMAP email clients have the ability to receive emails automatically in real time, without refreshing on a fixed interval. This is because the email client keeps a connection open to the server, and if there isn't any email data exchanged to keep the connection alive, the email client will send a NOOP packet on a fixed interval (just like other TCP protocols have their own keepalive packets). In Thunderbird, this behavior can be controlled by the 'Allow immediate server notifications when new messages arrive' option.
I'm writing a Python program that needs to know instantly when emails come in. The standard procedure for receiving emails is (after connection and login) select('inbox'), search(None, 'ALL') (or some specific search term), and fetch(). I can't figure out how to listen for new emails in real time. Every guide I've read suggests running the search() and fetch() functions in a loop. I've tried running the read() function while sending myself emails, but I have never seen read() output any data. Perhaps the server needs to know to push emails to the client? I haven't been able to confirm or refute this theory.
How can I instantly receive new emails with imaplib, or a similar Python library?
While I did not know at the time of originally posting this question, I was looking for a solution that implements the IMAP IDLE command, as defined in RFC2177.
Since you want to fetch emails asynchronously you should use the below library
https://github.com/bamthomas/aioimaplib
I used aioimaplib in my code, but imaplib2 also supposedly supports IDLE; see: https://web.archive.org/web/20090814230913/http://blog.hokkertjes.nl/2009/03/11/python-imap-idle-with-imaplib2/
This might be a confusing question...
But take for example the game Battlefield2.
Is it in any way possible to send chat messages to the game through Python? The reason I am asking this is because I've seen messages appear from no where on various games, and want to know if it's possible. Now, the way I understand a packet is like this...
It's a small collection of data with a sender IP address and a recipient IP address. Contained within the packet is the data. So, in theory... If the 'packet name' for the chat message is like:
SRV_CHAT|<Sender_Info>|<Message Text>
(For example): SRV_CHAT|10.1.1.5,Player1|hello, how's the game?
Does that mean that I can create a Python script to send my own message to a LAN game and have it appear? If it is possible, how can I go about dissecting the actual packet information in order to discover any parameters? What I mean by this, is finding that data contained in there...
It's just a thought and a question I've had for a long time.
Thanks!
Yes, it is technically possible. But not very feasible.
What you said about packets is essentially correct, and you can read more about that here.
However, beyond packets, there are entire networks of protocols that determine where they go, and who receives them, you can read about that here.
This means that if you were to manage to emulate the connection to the games server, you could possibly send data to that server, but most likely the server does not support random connections simply sending messages, and the server expects to receive connections which are made by other game clients, and unless you can emulate the clients responses and requests correctly, it will probably not work.
This means that your idea to use Python to directly connect to a server and send messages in the format similar to what you suggested would not work.
However. The server most definitely supports server messages, messages that are sent FROM the server itself into the game, usually if someone hosts a game for instance, they can send messages as the game server (host) of the game. Or, if it is a dedicated server, they might not be in the game itself and send messages to the players in the game through a management console.
It is most likely that many 'plugins' or methods to hook into this control are available, meaning that you could send a message to the application running the game server, telling it that it should send a message into the game.
This would be less emulation and more implementation of the games management system.
It is important to note, that many games are different, and their operation is different, meaning this may or may not work, depending on what options are available. It is my experience that games like Battlefield2 (which I have not played) usually have these tools builtin.
Specifically with BattleField2, there are some links I found that you might like.
http://www.gamefront.com/files/listing/pub2/Battlefield_2/Official_Server_Files/Dedicated_Server/
http://www.bf2cc.com/
I'm looking for a way to take gads of inbound SMTP messages and drop them onto an AMQP broker for further routing and processing. The messages won't actually end up in a mailbox, but instead SMTP is used as a message gateway.
I've written a Postfix After-Queue Content Filter in Python that drops the inbound SMTP message onto a RabbitMQ broker. That works well - I get the raw message over a queue and it gets picked up nicely by a consumer. The issue is that the AMQP connection is created and torn down with each message... the Content Filter script gets re-executed from scratch each time. I imagine that will end up being a performance issue.
If I could leverage something re-entrant I could reuse the connection. Or maybe I'm just approaching the whole thing incorrectly...
Making an AMQP connection over plain TCP is pretty quick. Perhaps if you're using SSL then it's another story but you sure that enqueueing the raw message onto the AMQP exchange is going to be the bottleneck? My guess would be that actually delivering the message via SMTP is going to be much slower so how fast you can queue things up isn't going to affect the throughput of the system.
If this piece does turn out to be a bottleneck I rather like creating little web servers using Sinatra, or Rack but it sounds like you might prefer a Python based solution. Have the postfix content filter perform a HTTP POST using curl to a webserver, which maintains a persistent connection to the AMQP server.
Of course now you have an extra moving part for which you need to think about monitoring, error handling and security.
Use SwiftMQ. It has a JavaMail bridge which receives your eMails from an IMAP or POP3 account, converts it into a JMS message which then can be consumed by an AMQP 0.9.1 and/or AMQP 1.0 and of course JMS client.
You can make Postfix deliver all or any of your emails to an external program, where you can throw them anywhere you want. Some examples can be found here.
Important note:
I've asked this question already on ServerFault: https://serverfault.com/questions/349065/clustering-tcp-servers-so-can-send-data-to-all-clients, but I'd also like a programmers perspective on the problem.
I'm developing a real-time mobile app by setting up a TCP connection between the app and server backend. Each user can send messages to all other users.
(I'm making the TCP server in Python with Twisted, am creating my own 'protocol' for communication between the app/backend and hosting it on Amazon Web Services.)
Currently I'm trying to make the backend scalable (and reliable). As far as I can tell, the system could cope with more users by upgrading to a bigger server (which could become rather limiting), or by adding new servers in a cluster configuration - i.e. having several servers sitting behind a load balancer, probably with 1 database they all access.
I have sketched out the rough architecture of this:
However what if the Red user sends a message to all other connected users? Red's server has a TCP connection with Red, but not with Green.
I can think of a one way to deal with this problem:
Each server could have an open TCP (or SSL) connection with each other server. When one server wants to send a message to all users it simply passes this along it's connection to the other servers. A record could be kept in the database of which servers are online (and their IP address), and one of the servers could be a boss - i.e. decides if others are up and running, if not it could remove them from the database (if a server was up and lost it's connection to the boss it could check the database and see if it had been removed, and restart if it had - else it could assume the boss was down.)
Clearly this needs refinement but shows the general principle.
Alternatively I'm not sure if this is possible (- definitely seems like wishful thinking on my part):
Perhaps users could just connect to a box or router, and all servers could message all users through it?
If you know how to cluster TCP servers effectively, or a design pattern that provides a solution, or have any comments at all, then I would be very grateful. Thank you :-)
You need to decide (or if you already did this - to share these decisions with us) reliability requirements for your system: should all messages be sent to all users in any case (e.g. one or more servers crashed), can you tolerate sending the same message twice to the same user on server crash? Your system complexity depends directly on these decisions.
The simplest version is when a message is not delivered to all users on server crash. All your servers keep TCP connection to each other. One of them receives a message from a user and sends it to all other connected users (to this server) and to all other connected servers. Other servers send this message to all their users. To scale the system you just run additional server which connects to all existing servers.
Have a look how it is handled with IRC servers. They essentially can do this already. Everbody can send to everybody else, on all servers. Or just to single users, also on another server. And to groups, called "channels". It works best by routing amongst the servers.
It's not that hard, if you can make sure the servers know each other and can talk to each other.
On a side note: At 9/11, the most reliable internet news source was the IRC network. All the www sites were down because of bandwidth; it took them ages to even get a plain-text web page back up. During this time, IRC networks were able to provide near real-time, moderated news channels across the atlantic. You maybe could no longer log into a server on the other side, but at least the servers were able to keep up a server-to-server connection across.
An obvious choice is to use the DB as a clearinghouse for messages. You have to store incoming messages somewhere anyway, lest they be lost if a server suddenly crashes. Put incoming messages into the central database and have notification processes on the TCP servers grab the messages and send them to the correct users.
TCP server cannot be clustered, the snapshot you put here is a classic HTTP server example.
Since the device will send TCP connection to server, say, pure socket, there will be noway of establishing a load-balancing server.
I am writing a web application that requires user interaction via email. I'm curious if there is a best practice or recommended source for learning about processing email. I am writing my application in Python, but I'm not sure what mail server to use or how to format the message or subject line to account for automated processing. I'm also looking for guidance on processing bouncebacks.
There are some pretty serious concerns here for how to send email automatically, and here are a few:
Use an email library. Python includes one called 'email'. This is your friend, it will stop you from doing anything tragically wrong. Read an example from the Python Manual.
Some points that will stop you from getting blocked by spam filters:
Always send from a valid email address. You must be able to send email to this address and have it received (it can go into /dev/null after it's received, but it must be possible to /deliver/ there). This will stop spam filters that do Sender Address Verification from blocking your mail.
The email address you send from on the server.sendmail(fromaddr, [toaddr]) line will be where bounces go. The From: line in the email is a totally different address, and that's where mail will go when the user hits 'Reply:'. Use this to your advantage, bounces can go to one place, while reply goes to another.
Send email to a local mail server, I recommend postfix. This local server will receive your mail and be responsible for sending it to your upstream server. Once it has been delivered to the local server, treat it as 'sent' from a programmatic point of view.
If you have a site that is on a static ip in a datacenter of good reputation, don't be afraid to simply relay the mail directly to the internet. If you're in a datacenter full of script kiddies and spammers, you will need to relay this mail via a public MTA of good reputation, hopefully you will be able to work this out without a hassle.
Don't send an email in only HTML. Always send it in Plain and HTML, or just Plain. Be nice, I use a text only email client, and you don't want to annoy me.
Verify that you're not running SPF on your email domain, or get it configured to allow your server to send the mail. Do this by doing a TXT lookup on your domain.
$ dig google.com txt
...snip...
;; ANSWER SECTION:
google.com. 300 IN TXT "v=spf1 include:_netblocks.google.com ~all"
As you can see from that result, there's an SPF record there. If you don't have SPF, there won't be a TXT record. Read more about SPF on wikipedia.
Hope that helps.
Some general information with regards to automated mail processing...
First, the mail server "brand" itself isn't that important for broadcasting or receiving emails. All of them support the standard smtp / pop3 communications protocol. Most even have IMAP support and have some level of spam filtering. That said, try to use a current generation email server.
Second, be aware that in an effort to reduce spam a lot of the receiving mail servers out there will simply throw a message away instead of responding back that a mail account doesn't exist. Which means you may not receive those.
Bear in mind that getting past spam filters is an art. A number of isp's watch for duplicate messages, messages that look like spam based on keywords or other content, etc. This is sometimes independent of the quantity of messages sent; I've seen messages with as few as 50 copies get blocked by AOL even though they were legitimate emails. So, testing is your friend and look into this article on wikipedia on anti-spam techniques. Then make sure your not doing that crap.
**
As far as processing the messages, just remember it's a queued system. Connect to the server via POP3 to retrieve messages, open it, do some action, delete the message or archive it, and move on.
With regards to bouncebacks, let the mail server do most of the work. You should be able to configure it to notify a certain email account on the server in the event that it is unable to deliver a message. You can check that account periodically and process the Non Delivery Reports as necessary.