I am trying to learn python sockets, but am becoming very confused by the results of the example code from the website (found here).
The only modification I have made is replacing socket.gethostname() in the server with the local IP of my server, to allow me to run this on two computers.
When I connect, attempting to connect on port 12345 as in the example, I get this output:
Got connection from ('10.0.1.10', 37492)
This leads me to believe that it is connecting on port 37492. I would like it to connect on the port I tell it to, so I can port forward. Am I misunderstanding, or is there an extra command to specify it.
Edit: I am uploading my code:
Client.py
#!/usr/bin/python # This is client.py file
import socket # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 12345 # Reserve a port for your service.
s.connect(("10.0.1.42", port))
print s.recv(1024)
s.close # Close the socket when done
Server.py
import socket
s = socket.socket() # Create a socket object
host = "10.0.1.42" # Get local machine name
port = 12345 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
c.send('Thank you for connecting')
c.close() # Close the connection
You have reached that point in your networking life where you need to understand protocol multiplexing. Good for you.
Think of the TCP/IP stack. An application communicates with a remote application by passing application-layer data to the transport (end-to-end) layer, which passes it to the network layer (internetwork layer) which tries, without guarantees, to have packets reach the IP destination host over a sequence of hops determined by cooperating routers that dynamically update their routing tables by talking to connected routers. Each router conversation goes over a physical transport of some kind (ISDN, Ethernet, PPP - in TCP/IP the task of creating packets and transmitting the appropriate bit stream is regarded as a single "subnetwork" layer, but this is ultimately split into two when differentiation is required between the OSI physical layer (Layer 1) and the data link layer (layer 2) for protocols like DHCP.
When TCP and UDP were designed, the designers imagined that each server would listen on a specific port. This typically has the inherent limitation that the port can only handle one version of your service protocol (though protocols like HTTP take care to be backwards-compatible so that old servers/clients can generally interoperate with newer ones). There is often a service called portmapper running on port 111 that allows servers to register the port number they are running on, and clients to query the registered servers by service (program) number and protocol version. This is a part of the Sun-designed RPC protocols, intended to expand the range of listening ports beyond just those that were pre-allocated by standards. Since the preallocated ports were numbered from 1 to 1023, and since those ports typically (on a sensible operating system) require a high level of privilege, RPC also enabled non-privileged server processes as well as allowing a server to be responsive to multiple versions of network application protocols such as NFS.
However the server side works, the fact remains that there has to be some way for the network layer to decide which TCP connection (or UDP listener) to deliver a specific packet to. Similarly for the transport layer (I'll just consider TCP here since it's connection-oriented - UDP is similar, but doesn't mind losing packets). Suppose I'm a server and I get two connections from two different client processes on the same machine. The destination (IP address, port number) will be the same if the clients are using the same version of the same protocol, or if the service only listens on a single port.
The server's network layer looks at the incoming IP datagram and sees that it's addressed to a specific server port. So it hands it off to that port in the transport layer (the layer above the network layer). The server, being a popular destination, may have several connections from different different client processes on the same machine. This is where the magic of ephemeral ports appears.
When the client requests a port to use to connect to a service, the TCP layer guarantees that no other process on that machine (technically, that interface, since different interfaces have unique IP addresses, but that's a detail) will be allocated the same port number while the client process continues to use it.
So protocol multiplexing and demultiplexing relies on five pieces of information:
(sender IP, sender port, protocol, receiver IP, receiver port)
The protocol is a field in the IP header as are the source and destination IP addresses. The sending and receiving port numbers are in the transport layer segment header.
When an incoming packet arrives, the guaranteed uniqueness of different ephemeral ports from the same client (endpoint) allows the transport layer to differentiate between different connections to the same server IP address from the same client IP address and port (the worst case for demultiplexing) by their source IP address and port. The (transport) protocol is included to ensure that TCP and UDP traffic don't get mixed up. The TCP/UDP constraints on uniqueness of ephemeral ports guarantee that any server can only receive one connection from a specific combination of (IP address, port number) and it's that that allows connections from the same machine to be demultiplexed into separate streams corresponding to the different origins.
In Python when you connect a socket to a remote endpoint the socket.accept() call returns the (IP address, port number) pair for the remote endpoint. You can use that to discover who is communicating with you, but if you just want to talk back you can simply write() the socket.
The key word is "from." That's the port that the client is connecting from, 12345 is the one your server is listening on and the client is connecting to.
The message that appears comes from the server. It just gives you information that connection was established from the client's port 37492.
This is what happens:
Your server (server.py) is listening on port 12345. Your client (client.py) connects to the port 12345 of the server. The TCP connection is always established between two ports - source and destination.
So, looking from your client app perspective 12345 is the destination port and 37492 is the source port. In other words client establishes a connection from its local port 37492 to the remote servers port 12345.
If you want to set up port forwarding you may still do it as the port on which server listens is well known (12345) and the source port of the client doesn't really matter in this situation.
The port that you get in your output is the source port. Your client program sends to the server on the port you choose (in this case, 12345), but it also needs a port to receive data sent by the server, so it randomly chooses a source port and tells it to the server.
I suggest you read some more about TCP and ports in general.
Related
I'm using this code for the client:
import socket
HOST = "Server IP"
PORT = 5555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall(b'Hello, world')
data = s.recv(1024)
print('Received', repr(data))
s.close()
And this code for the server:
import socket
HOST = "Client IP"
PORT = 5555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen()
conn, addr = s.accept()
print('Connected by', addr)
while True:
data = conn.recv(1024)
if not data:
break
conn.sendall(data)
s.close()
and what I'd like to do is simply send some data from a device to another. The problem is that I could make it work just by sending data on devices on the same network (that is, send data from a CMD window to another, or send data from the PC to the phone, where I have Termux with Python installed, if the PC and the phone are connected to the same network).
However, how do I send data between two different networks? I suppose that in the HOST variable on the client side I should put the public IP address of the server and in the HOST variable on server side I should put the public IP address of the client right?
But then suppose that in the client network there are many devices connected, where do I specify the private IP address of the client to send the data to a specific device?
The short answer is that you can't -- most routers connected to the Internet these days include a firewall and/or NAT, both of which are explicitly designed to prevent Internet packets from the outside world from getting to devices on the local LAN, for security reasons. In particular, the private addresses assigned to devices on the LAN (192.168.1.5, 10.0.0.10, and so on) are non-routable, so they are not meaningful outside the context of the local network and cannot be used to route traffic across the Internet.
Obviously that's not the full answer, though, since various peer-to-peer programs in use today (e.g. Skype) do manage to communicate in the way you want to do; however, they all have to work around the above problem one way or another. Some ways they do it include:
Have the user of the LAN that is to receive the incoming TCP connection configure port forwarding on his router. This tells the router that any TCP connections received on a particular port should be automatically forwarded to the specified private IP address on the LAN. Downsides of this approach are that it opens a hole in the user's firewall that might be later exploited by bad actors, and that many users aren't willing (or even able) to reconfigure their routers in this way.
Run a server with an Internet-facing IP address, and have your program always connect to it. This server can then work as a proxy, forwarding TCP data from one connection to the other as necessary. This avoids the problem, at the cost of having to maintain your own 24/7 public-facing server and route all your traffic through it.
Use TCP hole punching to convince the router(s) to allow your incoming TCP connections by exploiting the rules/logic they (sometimes) use to decide when to allow incoming traffic. This gets you the direct connections you want, but it's fairly complicated to set up, doesn't work on all routers, and generally requires the implementation of step (2) (above) as a prerequisite, since the connection-bootstrapping process itself requires some communication that must be facilitated by a third-party.
to work, change s.listen() to s.listen(5)
I was able to connect from my network 192.168.1.x to other 192.168.2.x
Now, depending of your requirements you will need some configuration to work in the Internet.
the clients should have the IP depending the localization of your server, if server application is:
Connected direct to the Internet, client should have gateway IP address
Connected to a router, you can configure a DMZ (not recommended, not secure, just for tests) or configure ports (secure and recommended), the clients should have the gateway IP address of the router
For a connection is to be made, a bounded socket should be listening for clients. Client needs to know both ip address and port. For bounding a socket why we need an ip address of the server itself when the program(which listens for clients) itself is running on the server?
Simply because a server has multiple addresses, at least the loopback one at 127.0.0.1 (IP v4) and one per physical network interfaces. For example a corporate proxy has commonly two interfaces, one on the internal network and one on the public one. Most have a third one for the DMZ. Being member of different networks, those interfaces must have different addresses. And it make sense to open some services on only one interface.
But you also can use the ANY address (0.0.0.0 in IPv4) that means to accept connections on any interface.
Its been a few days since I'm playing around Hole Punching in order to have some kind of reliable behaviour, but I'm now at a dead end.
UDP Hole punching works great: simply first send a packet to the remote, and get the remote to send a packet the otherway as it will land through the source NAT. Its rather reliable from what I tried.
But now comes TCP... I don't get it.
Right now, I can establish a connection through NATs but only with connecting sockets:
A.connect(B) -> Crash agains't B's NAT, but open a hole in A's NAT.
B.connect(A) -> Get in A's NAT hole, reach A's connecting socket.
But now, the two sockets that sended the SYN packets for connection are connected.
You would think that I would have done it, got a connection through 2 NATs, hooray.
But the problem is that this is not a normal behaviour, and given to this paper: http://www.brynosaurus.com/pub/net/p2pnat/, I should be able to have a listening socket in parallel to the connecting socket.
So I did bind a listening socket, which would accept inbound connections.
But inbound connections are always caught by the connecting socket and not the by the listening one...
e.g:
#!/usr/bin/env python3
from socket import *
from threading import Thread
Socket = socket
# The used endpoints:
LOCAL = '0.0.0.0', 7000
REMOTE = 'remote', 7000
# Create the listening socket, bind it and make it listen:
Listening = Socket(AF_INET, SOCK_STREAM)
Listening.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
Listening.bind(LOCAL)
Listening.listen(5)
# Just start in another thread some kind of debug:
# Print the addr of any connecting client:
def handle():
while not Listening._closed:
client, addr = Listening.accept()
print('ACCEPTED', addr)
Thread(target=handle).start()
# Now creating the connecting socket:
Connecting = Socket(AF_INET, SOCK_STREAM)
Connecting.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
Connecting.bind(LOCAL)
# Now we can attempt a connection:
try:
Connecting.connect(REMOTE)
print('CONNECTED', Connecting.getpeername())
except Exception as e:
print('TRIED', type(e), e)
Now with this script, just agree on a port with a friend or whatever, and execute it on one end's, the Connecting.connect(...) should run for a bit (waiting for timeout, because SYN packet crashed into the distant NAT, but fortunately opened a hole in his own NAT), meanwhile execute the script on the other end, now the Connecting.connect(...) will return because it will have connected.
The weirdest part being: The Listening socket was never triggerred.
Why ? How to get the listening socket to catch inbound connections over the connecting socket ?
Note: Closing the connecting socket does send something on the network which immediately close the hole, at least it does on my network.
2nd-Note: I'm on windows.
Edit: The main problem is that in any circumstance, this script outputs CONNECTED [...] rather than CLIENT [...], which given some lecture should not happen.
So, after more tests and readings, here's what I came to:
In fact, it is possible to bind a listen socket and a socket making outbound connections on the same address (ip, port).
But the behaviour of the sockets heavily depends on the system / TCP stack implementation, as mentionned in http://www.brynosaurus.com/pub/net/p2pnat/ at §4.3:
What the client applications observe to happen with their sockets during TCP hole punching depends on the timing and the TCP implementations involved. Suppose that A's first outbound SYN packet to B's public endpoint is dropped by NAT B, but B's first subsequent SYN packet to A's public endpoint gets through to A before A's TCP retransmits its SYN. Depending on the operating system involved, one of two things may happen:
A's TCP implementation notices that the session endpoints for the incoming SYN match those of an outbound session A was attempting to initiate. A's TCP stack therefore associates this new session with the socket that the local application on A was using to connect() to B's public endpoint. The application's asynchronous connect() call succeeds, and nothing happens with the application's listen socket.
Since the received SYN packet did not include an ACK for A's previous outbound SYN, A's TCP replies to B's public endpoint with a SYN-ACK packet, the SYN part being merely a replay of A's original outbound SYN, using the same sequence number. Once B's TCP receives A's SYN-ACK, it responds with its own ACK for A's SYN, and the TCP session enters the connected state on both ends.
Alternatively, A's TCP implementation might instead notice that A has an active listen socket on that port waiting for incoming connection attempts. Since B's SYN looks like an incoming connection attempt, A's TCP creates a new stream socket with which to associate the new TCP session, and hands this new socket to the application via the application's next accept() call on its listen socket. A's TCP then responds to B with a SYN-ACK as above, and TCP connection setup proceeds as usual for client/server-style connections.
Since A's prior outbound connect() attempt to B used a combination of source and destination endpoints that is now in use by another socket, namely the one just returned to the application via accept(), A's asynchronous connect() attempt must fail at some point, typically with an “address in use” error. The application nevertheless has the working peer-to-peer stream socket it needs to communicate with B, so it ignores this failure.
The first behavior above appears to be usual for BSD-based operating systems, whereas the second behavior appears more common under Linux and Windows.
So I'm actually in the first case. On my Windows 10.
This implies that in order to make a reliable method for TCP Hole Punching, I need to bind a listening socket at the same time as a connecting socket, but I later need to detect which one triggered (listening or connecting) and pass it down the flow of the application.
Why listen socket is not triggered
I think the answer is somewhere here. TCP connection is defined by tuple of four elements:
Local address
Local port
Remote address
Remote port
When you establish TCP connection you create binding from this tuple to the connecting socket on local host.
When SYN is sent via NAT it creates binding:
- Local address/port -> Public address/port
When remote side sends its SYN to the Public address/port this addresses are translated to Local address/port and delivered to local machine. On this machine this connection is not distinguishable from initial connection and successfully established (with SYN/ACK).
This means that no INITIAL SYN is received on local side.
How to get the listening socket to catch inbound connections over the connecting socket?
It is impossible with source NAT. To accept new connection behind NAT you need destination NAT that maps some public IP/Port to you private IP/port
I've made a server (python, twisted) for my online game. Started with TCP, then later added constant updates with UDP (saw a big speed improvement). But now, I need to connect each UDP socket client with each TCP client.
I'm doing this by having each client first connect to the TCP server, and getting a unique ID. Then the client sends this ID to the UDP server, connecting it also. I then have a main list of TCP clients (ordered by the unique ID).
My goal is to be able to send messages to the same client over both TCP and UDP.
What is the best way to link a UDP and TCP socket to the same client?
Can I just take the IP address of a new TCP client, and send them data over UDP to that IP? Or is it necessary for the client to connect twice, once for TCP and once for UDP (by sending a 'connect' message)?
Finally, if anyone with knowledge of TCP/UDP could tell me (i'm new!), will the same client have the same IP address when connecting over UDP vs TCP (from the same machine)? (I need to know this, to secure my server, but I don't want to accidentally block some fair users)
Answering your last question: no. Because:
If client is behind NAT, and the gateway (with NAT) has more than one IP, every connection can be seen by you as connection from different IP.
Another problem is when few different clients that are behind the same NAT will connect with your server, you will have more than one pair of TCP-UDP clients. And it will be impossible to join correct pairs.
Your method seems to be good solution for the problem.
1- Can I just take the IP address of a new TCP client, and send them data over UDP to that IP? NO in the general case, but ...
2- is it necessary for the client to connect twice, once for TCP and once for UDP ? NO, definitively
3- will the same client have the same IP address when connecting over UDP vs TCP (from the same machine)? YES except in special cases
You really need some basic knowledge of the TCP, UDP and IP protocol to go further, and idealy, on the OSI model.
Basics (but you should read articles on wikipedia to have a deeper understanding) :
TCP and UDP are 2 protocol over IP
IP is a routable protocol : it can pass through routers
TCP is a connected protocol : it can pass through gateways or proxies (firewalls and NATs)
UDP in a not connected protocol : it cannot pass through gateways
a single machine may have more than one network interface (hardware slot) : each will have different IP address
a single interface may have more than one IP address
in the general case, client machines have only one network interface and one IP address - anyway you can require that a client presents same address to TCP and UDP when connecting to your server
Network Address Translation is when there is a gateway between a local network and the wild internet that always presents its own IP address and keep track of TCP connections to send back packets to the correct client
In fact the most serious problem is if there is a gateway between the client and your server. While the client and the server are two (virtual) machines for which you have direct keyboard access, no problem, but corporate networks are generally protected by a firewall acting as a NAT, and many domestic ADSL routers also include a firewall and a NAT. In that case just forget UDP. It is possible to instruct a domestic router to pass all UDP traffic to a single local IP, but it is not necessarily an easy job. In addition, that means that if a user of yours has more than one machine at home, he will be allowed to use only one at a time and will have to reconfigure his router to switch to another one !
First of all when you send data with TCP or UDP you have to give the port.
If your client connect with TCP and after your server send a response with UDP the packet will be reject by the client.
Why? Because you have to register a port for connection and you can not be sure the port is correctly open on the client.
So when you begin a connection in TCP the client open a port to send data and receive the response. You have to make the same with UDP. When client begin all communication with server you can be sure all the necessary port are open.
Don't forget to send data on the port which the connection was open.
Can I just take the IP address of a new TCP client, and send them data over UDP to that IP? Or is it necessary for the client to connect twice, once for TCP and once for UDP (by sending a 'connect' message)?
Why you don't want create 2 connections?
You have to use UDP for movement for example. because if you create an FPS you can send the player's position every 50ms so it's really important to use UDP.
It's not just a question of better connection. If you want to have a really good connection between client and server you need to use Async connection and use STREAM. But if you use stream you'r TCP socket do not signal the end of a socket but you have a better transmition. So you have to write something to show the packet end (for example <EOF>).
But you have a problem with this. Every socket you receive you have to analyze the data and split over the <EOF>. It can take a lot a processor.
With UDP the packet always have a end signal. But you need to implement a security check.
I have a python socket that is supposed to be listening to all incoming sockets on port 2022 and I have port forwarding enabled as well as a dynamic DNS service. When I am connected to the same network and I try to connect to my pc using the pc's IP address it works like a charm but when I try to use the public IP of my network it won't work although I have done proper port forwarding. Here is my code, it stucks on the last line and doesn't accept any connections outside the network:
IPC= '0.0.0.0'
PORTC =2022
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.bind((IPC,PORTC))
client_socket.listen(5)
connection =client_socket.accept()[0].makefile('wb')
If you can connect from a different PC inside your private network, then there's nothing wrong with your Python code. It's almost certain that there's something wrong with your router or the port forwarding config.
Make sure that you've re-reviewed your port forwarding settings and perhaps try restarting the router (be careful to verify your internally assigned IP addresses haven't changed if you go this route).