If there any way to know if a client expects server to enable SSL?
I am building a small SMTP server and have implemented SSL on 465 but some clients do not expect SSL so obviously connection fails.
Is it possible to tell this in any way?
There is no clean way for a server to detect if a client expects to use SSL/TLS at the start of the connection. In fact, if the server is expected to send data first (as is the case with SMTP: the server sends a banner before the client sends any data), there is no way at all to do that.
This is the reasons why SSL/TLS is generally used in one of these two ways:
A new port number is designated for the SSL/TLS version of the protocol. For example, HTTP (port 443 instead of port 80), IMAP (port 993 instead of port 143), SMTP (port 465 instead of 25 or 587). The server knows to use SSL/TLS right away if it accepts the connection on the new port.
STARTTLS: The server and client start by talking the non-SSL/TLS version of the protocol, but the server indicates STARTTLS in its service capabilities announcement. The client accept the offer and requests it. Both server and client now restart the protocol using SSL/TLS.
STARTTLS is a bit less efficient because of the non-SSL/TLS conversation between the server and client that happens first (uses several network round trips) and it is not available for use with all protocols (HTTP doesn't support it), but it's generally preferred if available because it makes it easier for things like automatic configuration of email settings (no need to probe a bunch of possible ports and pick the best one).
Port 465 is an example of the first solution: pick a new port and run SSL/TLS on it. That means servers and clients are both supposed to use SSL/TLS right away for communications on that port.
If you are seeing clients trying to talk plaintext SMTP on port 465, those clients are BROKEN. There really isn't anything you can do to work around them. The clients have serious bugs which should be fixed...
Moreover, for SMTP, you really need to be using STARTTLS, not SMTP over SSL/TLS on port 465.
Related
I understand the upgrade handshake and then the creation of the WebSocket channel on a totally different socket, but I'm puzzled as to why this is not a problem when firewalls may block all traffic except that which is bound for 80 (or 443). It seems that the WebSocket traffic hosted at its own (non-80, non-443) port would get blocked -- but clearly WebSockets are mature and effective so I must be missing something. How does the WebSocket traffic on the non-80/non-443 port traverse firewalls? Is this related (no pun intended) to routing rules ESTABLISHED/RELATED? The following seems to come close to a solid answer:
https://stackoverflow.com/a/2291861/6100445
...that is, the WebSocket traffic is established in an outbound sense from the browser over HTTP and then established in an outbound sense from the WebSocket server over the new WebSocket port?
WebSockets are "designed to work over HTTP ports 443 and 80":
https://datatracker.ietf.org/doc/html/rfc6455
(also https://en.wikipedia.org/wiki/WebSocket)
But many tutorials launch separate WebSocket servers on totally different port numbers (e.g. 8001 is used commonly). A good example is the websockets package on PyPI, which uses port 8001 and flatly states WebSockets and HTTP servers should run on separate ports:
https://websockets.readthedocs.io/en/stable/intro/tutorial1.html
https://websockets.readthedocs.io/en/stable/faq/server.html#how-do-i-run-http-and-websocket-servers-on-the-same-port
A lot of material on the Web is hand-waving and glosses over some detail with statements like WebSockets "use the same port as HTTP and therefore get through firewalls" (I assume they mean the upgrade/handshake portion) but many other sources indicate that a WebSocket server (on the same computer as the HTTP server) should be established on a different port (I assume because the HTTP server is already bound to 80 (or 443), and this different non-80/non-443 port therefore carries the upgraded WebSocket traffic). The separate ports make sense from a TCP/IP socket binding perspective. What am I missing about how WebSockets use 80 (or 443) for the upgrade/handshake, a separate port for the WebSocket established traffic, yet still work through firewalls where the only traffic allowed is that which is destined for port 80 (or 443)?
I am on a journey of understanding what is the proper way to send an email from Python code. I have somewhat progressed in understanding of MX lookup, though: "the larger the island of knowledge, the longer the shoreline of wonder".
Thanks to this answer, I was able to send an email (to a disposable mailbox though), with this code-snippet:
import smtplib
from email.message import EmailMessage
message = EmailMessage()
message.set_content('Content of the message here.')
message['Subject'] = 'Mail sent from code'
message['From'] = 'whoever#whatever.com'
message['To'] = 'aloun36zmzazyxd3tyop#3mail.rocks'
smtplib.SMTP('mail.3mail.rocks:2525')
smtp_server.send_message(message)
smtp_server.quit()
Here is how I come up with SMTP address and port (mail.3mail.rocks:2525):
Done MX lookup for 3mail.rocks domain:
host -t mx 3mail.rocks
3mail.rocks mail is handled by 10 mail.3mail.rocks.
Then I just started checking ports used by default, with telnet mail.3mail.rocks xxx, this gave me the following results:
telnet mail.3mail.rocks 25
Trying 89.38.99.80...
telnet: connect to address 89.38.99.80: Connection refused
telnet: Unable to connect to remote host
telnet mail.3mail.rocks 465
Trying 89.38.99.80...
telnet: connect to address 89.38.99.80: Operation timed out
telnet: Unable to connect to remote host
telnet mail.3mail.rocks 587
Trying 89.38.99.80...
telnet: connect to address 89.38.99.80: Operation timed out
telnet: Unable to connect to remote host
telnet mail.3mail.rocks 2525
Trying 89.38.99.80...
Connected to mail.3mail.rocks.
Escape character is '^]'.
220 node1 ESMTP Haraka 2.8.16 ready
So, that is how I figured out the needed port (by brute-force, essentially).
I went on to test my snippet on another disposable mail service (mailforspam.com), following the same steps — MX lookup (host -t mx mailforspam.com) returned:
mailforspam.com mail is handled by 10 mail2.mailforspam.com.
mailforspam.com mail is handled by 10 mail1.mailforspam.com.
Though I was not able to connect via telnet (I have tried both servers mail2.mailforspam.com and mail1.mailforspam.com) to any of the default ports: port 25 — Connection refused, ports 2525, 587, 465 — Operation timed out.
Questions are:
How do I figure out the proper ports for the server accepting mails on behalf of a particular domain (one that is returned by MX lookup)? My understanding here is that "default" ports are just conventions, and, in fact, servers can use any free port they choose.
I assume that when email is sent from one email provider to another, the SMTP server it is submitted to (one belonging to the user that is sending email) does something similar (i.e. MX lookup => connection to mail accepting server => submitting an email). How do such "real-world" servers figure out the proper port (or they just brute-forcing through the default ones)?
How do I figure out the proper ports for the server accepting mails on behalf of a particular domain (one that is returned by MX lookup)?
What you've shown in your question is more or less correct. You may want to try the ports in a different order. Also, port 2525 is not an official port from any standard I'm aware of but seems to be a convention for bypassing firewalls that block the submission port 587.
One thing to note is that "accepting mail" is not actually one thing. There are "mail user agents" that do "submission" and "mail transfer agents" that do "transfer". "Submission" and "transfer" often live on different ports which explains some of the diversity you've seen. Figure out whether you're doing submission or transfer and select the appropriate group of ports.
My understanding here is that "default" ports are just conventions, and, in fact, servers can use any free port they choose.
This isn't really true, at least not if the servers want anyone to be able to find them, because ...
How do such "real-world" servers figure out the proper port (or they just brute-forcing through the default ones)?
Mail servers that actually want to be able to receive mail must run on a standard port number. For MTAs this means port 25 with maybe a fallback to 465 (though this isn't standardized either). For MUAs this means port 587 with maybe a fallback to 2525 (also not standardized but apparently in common use as a workaround to MUAs being blocked).
In particular, MX records carry no port information, nor does any other DNS record type related to SMTP.
The MX you get needs to support port 25, that's part of the SMTP definition. If you are unable to connect, chances are the block is in the firewall on your own side — port 25 outbound is aggressively blocked from consumer-grade networks, in an attempt to curb direct-injection spam.
Port 25 between authorized servers is not blocked, there is no reason or need for the server to figure out a different port number.
I am using a library (ShareDB) for operational transformation, and the server and client side use a websocket-json-stream to communicate. However this ShareDB is being run on nodejs as a service (I'm using zerorpc to control my node processes), as my main web framework is Tornado (python). I understand from this thread that with a stateful protocol such as TCP, the connections are differentiated by the client port (so only one server port is required). And according to this response regarding how websockets handle multiple incoming requests, there is no difference in the underlying transport channel between tcp and websockets.
So my question is, if I create a websocket from the client to the python server, and then also from the client to my nodejs code (the ShareDB service) how can the server differentiate which socket goes with which? Is it the servers responsibility to only have a single socket 'listening' for a connection a given time (i.e. to first establish communication with the Python server and then to start listening for the second websocket?)
The simplest way to run two server processes on the same physical server box is to have each of them listen on a different port and then the client connects to the appropriate port on that server to indicate which server it is trying to connect to.
If you can only have one incoming port due to your server environment, then you can use something like a proxy. You still have your two servers listening on different ports, but neither one is listening on the port that is open to the outside world. The proxy listens on the one incoming port that is open to the outside world and then based on some characteristics of the incoming connection, the proxy directs that incoming connection to the appropriate server process.
The proxy can be configured to identify which process you are trying to connect to either via the URL or the DNS hostname.
I've made a server (python, twisted) for my online game. Started with TCP, then later added constant updates with UDP (saw a big speed improvement). But now, I need to connect each UDP socket client with each TCP client.
I'm doing this by having each client first connect to the TCP server, and getting a unique ID. Then the client sends this ID to the UDP server, connecting it also. I then have a main list of TCP clients (ordered by the unique ID).
My goal is to be able to send messages to the same client over both TCP and UDP.
What is the best way to link a UDP and TCP socket to the same client?
Can I just take the IP address of a new TCP client, and send them data over UDP to that IP? Or is it necessary for the client to connect twice, once for TCP and once for UDP (by sending a 'connect' message)?
Finally, if anyone with knowledge of TCP/UDP could tell me (i'm new!), will the same client have the same IP address when connecting over UDP vs TCP (from the same machine)? (I need to know this, to secure my server, but I don't want to accidentally block some fair users)
Answering your last question: no. Because:
If client is behind NAT, and the gateway (with NAT) has more than one IP, every connection can be seen by you as connection from different IP.
Another problem is when few different clients that are behind the same NAT will connect with your server, you will have more than one pair of TCP-UDP clients. And it will be impossible to join correct pairs.
Your method seems to be good solution for the problem.
1- Can I just take the IP address of a new TCP client, and send them data over UDP to that IP? NO in the general case, but ...
2- is it necessary for the client to connect twice, once for TCP and once for UDP ? NO, definitively
3- will the same client have the same IP address when connecting over UDP vs TCP (from the same machine)? YES except in special cases
You really need some basic knowledge of the TCP, UDP and IP protocol to go further, and idealy, on the OSI model.
Basics (but you should read articles on wikipedia to have a deeper understanding) :
TCP and UDP are 2 protocol over IP
IP is a routable protocol : it can pass through routers
TCP is a connected protocol : it can pass through gateways or proxies (firewalls and NATs)
UDP in a not connected protocol : it cannot pass through gateways
a single machine may have more than one network interface (hardware slot) : each will have different IP address
a single interface may have more than one IP address
in the general case, client machines have only one network interface and one IP address - anyway you can require that a client presents same address to TCP and UDP when connecting to your server
Network Address Translation is when there is a gateway between a local network and the wild internet that always presents its own IP address and keep track of TCP connections to send back packets to the correct client
In fact the most serious problem is if there is a gateway between the client and your server. While the client and the server are two (virtual) machines for which you have direct keyboard access, no problem, but corporate networks are generally protected by a firewall acting as a NAT, and many domestic ADSL routers also include a firewall and a NAT. In that case just forget UDP. It is possible to instruct a domestic router to pass all UDP traffic to a single local IP, but it is not necessarily an easy job. In addition, that means that if a user of yours has more than one machine at home, he will be allowed to use only one at a time and will have to reconfigure his router to switch to another one !
First of all when you send data with TCP or UDP you have to give the port.
If your client connect with TCP and after your server send a response with UDP the packet will be reject by the client.
Why? Because you have to register a port for connection and you can not be sure the port is correctly open on the client.
So when you begin a connection in TCP the client open a port to send data and receive the response. You have to make the same with UDP. When client begin all communication with server you can be sure all the necessary port are open.
Don't forget to send data on the port which the connection was open.
Can I just take the IP address of a new TCP client, and send them data over UDP to that IP? Or is it necessary for the client to connect twice, once for TCP and once for UDP (by sending a 'connect' message)?
Why you don't want create 2 connections?
You have to use UDP for movement for example. because if you create an FPS you can send the player's position every 50ms so it's really important to use UDP.
It's not just a question of better connection. If you want to have a really good connection between client and server you need to use Async connection and use STREAM. But if you use stream you'r TCP socket do not signal the end of a socket but you have a better transmition. So you have to write something to show the packet end (for example <EOF>).
But you have a problem with this. Every socket you receive you have to analyze the data and split over the <EOF>. It can take a lot a processor.
With UDP the packet always have a end signal. But you need to implement a security check.
I am little stumped: I have a simple messenger client program (pure python, sockets), and I wanted to add proxy support (http/s, socks), however I am a little confused on how to go about it. I am assuming that the connection on the socket level will be done to the proxy server, at which point the headers should contain a CONNECT + destination IP (of the chat server) and authentication, (if proxy requires so), however the rest is a little beyond me. How is the subsequent connection handled, specifically the reading/writing, etc...
Are there any guides on proxy support implementation for socket based (tcp) programming in Python?
Thank you
Maybe use something like SocksiPy which does all the protocol details for you and would let you connect through a SOCKS proxy as you would without it?
It is pretty simple - after you send the HTTP request: CONNECT example.com:1234 HTTP/1.0\r\nHost: example.com:1234\r\n<additional headers incl. authentication>\r\n\r\n, the server responds with HTTP/1.0 200 Connection established\r\n\r\n and then (after the double line ends) you can communicate just as you would communicate with example.com port 1234 without the proxy (as I understand you already have the client-server communication part done).
Have a look at stunnel.
Stunnel can allow you to secure
non-SSL aware daemons and protocols
(like POP, IMAP, LDAP, etc) by having
Stunnel provide the encryption,
requiring no changes to the daemon's
code