RPC over TCP with multiple clients on same machine - python

I'm building an RPC Server in golang that uses msgpack.
The client is built in python using the mprpc library (msgpack over TCP with gevent).
My issue is, being an absolute noob in networking, I discovered that I can't use the same address/port with multiple clients running at once on the same computer (socket already bound i guess, it just stalls and timeouts).
I have looked around quite a bit but I'm not sure what I should be doing to be able to have multiple clients on the same machine talk to a server (msgpack back and forth). Is this a case where I need to use ZeroMQ ? Or requests over HTTP ?
Thanks !

TCP is a connection-oriented protocol. This means that only the server needs to have a fixed, known port. The client can use any port it wants, because nobody is making a connection to the client.
So, how does the server know how to talk to the client? Whenever it accepts a connection, it's told who the connection is from. But usually, you don't even need that, because the socket keeps track of who the connection is from. Just recv and send on that socket, and you're talking to the right client.
You should probably read the Socket Programming HOWTO in the Python docs, or some other tutorial, but briefly…
A server starts like this:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', 12345))
sock.listen(5)
while True:
csock, addr = sock.accept()
It binds a port and listens and loops around accepting connections and doing something with them.
A client, on the other hand, just does this:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('localhost', 12345))
… or, equivalently:
sock = socket.create_connection(('localhost', 12345))
It doesn't call bind, it just creates a connection, letting the sockets library pick an arbitrary port on the appropriate interface for that connection. Unless you've got thousands of sockets already open, it should always be able to find a free port for you.

If you want to have two way connection, then HTTP is not suitable for this. Because HTTP is designed in a way that the server only responds to a request, which prevents server to issue a request itself. There are other solutions that provide two way connection(server to client and client to server in same time).
WebSocket is the first thing that comes to my mind. Of course ZeroMQ also can do this.

Related

Does using sockets over a LAN introduce any security issues?

I'm having some trouble understanding the security of sockets. I have some code set up in Python, which looks a bit like this:
server.py:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((socket.gethostname(), 1234))
s.listen(5)
client.py:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((socket.gethostname(), 1234)
I set myself a project to make a VOIP over LAN, or as close as I can get. I'll be using strictly private IP addresses in this program. I read that opening ports can be dangerous, so I'm a little nervous.
As you can see, I am using the port 1234. Could this introduce a vulnerability from any external threats?
When people say "opening ports can be dangerous" what they mean is that making an application available to connections from an unknown source (such as the Internet) is risky. There may be bugs in the application that allow a remote attacker to execute commands, for example.
As long as you don't accept connections directly from the Internet you're probably fine. If an attacker has access to your LAN, you have more serious things to worry about.

How send data between two devices using sockets in Python?

I'm using this code for the client:
import socket
HOST = "Server IP"
PORT = 5555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.sendall(b'Hello, world')
data = s.recv(1024)
print('Received', repr(data))
s.close()
And this code for the server:
import socket
HOST = "Client IP"
PORT = 5555
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen()
conn, addr = s.accept()
print('Connected by', addr)
while True:
data = conn.recv(1024)
if not data:
break
conn.sendall(data)
s.close()
and what I'd like to do is simply send some data from a device to another. The problem is that I could make it work just by sending data on devices on the same network (that is, send data from a CMD window to another, or send data from the PC to the phone, where I have Termux with Python installed, if the PC and the phone are connected to the same network).
However, how do I send data between two different networks? I suppose that in the HOST variable on the client side I should put the public IP address of the server and in the HOST variable on server side I should put the public IP address of the client right?
But then suppose that in the client network there are many devices connected, where do I specify the private IP address of the client to send the data to a specific device?
The short answer is that you can't -- most routers connected to the Internet these days include a firewall and/or NAT, both of which are explicitly designed to prevent Internet packets from the outside world from getting to devices on the local LAN, for security reasons. In particular, the private addresses assigned to devices on the LAN (192.168.1.5, 10.0.0.10, and so on) are non-routable, so they are not meaningful outside the context of the local network and cannot be used to route traffic across the Internet.
Obviously that's not the full answer, though, since various peer-to-peer programs in use today (e.g. Skype) do manage to communicate in the way you want to do; however, they all have to work around the above problem one way or another. Some ways they do it include:
Have the user of the LAN that is to receive the incoming TCP connection configure port forwarding on his router. This tells the router that any TCP connections received on a particular port should be automatically forwarded to the specified private IP address on the LAN. Downsides of this approach are that it opens a hole in the user's firewall that might be later exploited by bad actors, and that many users aren't willing (or even able) to reconfigure their routers in this way.
Run a server with an Internet-facing IP address, and have your program always connect to it. This server can then work as a proxy, forwarding TCP data from one connection to the other as necessary. This avoids the problem, at the cost of having to maintain your own 24/7 public-facing server and route all your traffic through it.
Use TCP hole punching to convince the router(s) to allow your incoming TCP connections by exploiting the rules/logic they (sometimes) use to decide when to allow incoming traffic. This gets you the direct connections you want, but it's fairly complicated to set up, doesn't work on all routers, and generally requires the implementation of step (2) (above) as a prerequisite, since the connection-bootstrapping process itself requires some communication that must be facilitated by a third-party.
to work, change s.listen() to s.listen(5)
I was able to connect from my network 192.168.1.x to other 192.168.2.x
Now, depending of your requirements you will need some configuration to work in the Internet.
the clients should have the IP depending the localization of your server, if server application is:
Connected direct to the Internet, client should have gateway IP address
Connected to a router, you can configure a DMZ (not recommended, not secure, just for tests) or configure ports (secure and recommended), the clients should have the gateway IP address of the router

socket programming with python on linux - s.connect() time-out

I'm using a TCP socket to connect to a certain web page, as it will be running in a loop trying to create socket, connect, send data, and receive then close the socket. the problem is s.connect() times out at random iterations and specific web site. i read about TCP time_wait so
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
was added with no luck. then one second of sleep after each iteration, but still no luck.
i should note that it works fine in another slower network, but not in the fast one i'm trying to run the code.
i think following pseudo code can clarify it more:
while(1)
{time.sleep(1)
s = socket.socket() #TCP socket creation
s.settimeout(2) # 2 seconds for time-out which is more than needed
s.connect()
s.send(message) #which is a HTTP get request
s.recv() # which is a HTTP response
s.close()}
my code routine follows these steps, each of socket methods are implemented correctly, but i get time-outs on connect. i am sure about server functioning correctly. i suspected it might have something to do with TCP time_wait , but comments don't seem to agree
A connect times out if the TCP handshake with the server is not finished within a specific time, usually because the server does not respond in time. There might be various reasons for is, like the server being down, getting the wrong IP address for the server during DNS lookup, server being overloaded etc.
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Adding this on the client side on the connection has no use.
Adding this on the server side will help if the problem was caused by a crashed server since it will allow the server to bind to the listener address again without waiting and thus the server is faster available again. But, I have no idea if this is really the problem you are facing.

Python, Listening on two sockets at the same time

I am making a python server that should handle connection from remote hosts on port 9090 and connection only from localhost (a GUI sending data) on port 9091.
I think the problem is in socket.listen(backlog), because this blocks everything.
So there is a way to make socket1.listen(backlog) and socket2.listen(backlog) running at the same time?
Also, is it a good, and above all safe, idea to make a socket for receiving data from a GUI (written in Java)?
This can be done in C: Listen to multiple ports from one server
and C++: Create multiple listening sockets
so you should be able to do this in Python.
The only thing that would make this not safe is if your server performs a dangerous activity in response to the GUI, but you can probably add code to check and protect against hacks.
I am using python2.7.3, and as stated in my comment I couldn't reproduce the listen blocking issue.
Notes:
for the simplicity's sake I am only using IPv4 sockets
I included Python's prompter in the code to show that it didn't block
regarding eventual security concerns: the socket (s_local) listening on 9091, only accepts connections from localhost (from other hosts will fail)
Here's a snippet from a CentOS5 machine:
>>> import socket
>>> s_global = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s_global.bind(("", 9090))
>>> backlog = 0xFF
>>> s_global.listen(backlog)
>>> s_local = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s_local.bind(("127.0.0.1", 9091))
>>> s_local.listen(backlog)
>>>
and here's some output on the same machine:
netstat -an | grep 909
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:9091 0.0.0.0:* LISTEN
As seen, both server sockets are listening for connections.
Now, if something is wrong in my environments (some reading revealed that by default sockets are blocking by default), after creating each socket, you could add (as specified in one comment) the following line(s):
for s_global
s_global.setblocking(0)
for s_local
s_local.setblocking(0)
Now in order to receive incoming connections from both sockets, you should use Python's select module (used for async IO) with no need for other threads; your server app should constantly listen for connections, something like:
TIMEOUT = 1 #seconds
terminate = False # variable to be modified externally to end the loop
listening_sockets = [s_global, s_local]
while (not terminate):
notified_socket_list = select.select(listening_sockets, [], [], TIMEOUT)[0]
for notified_socket in notified_socket_list:
incoming_socket, addr = notified_socket.accept()
handle(incoming_socket, addr)
Note: worth being mentioned: epoll(select.epoll) is performance-wise preferred to select(select.select) but i haven't tried it yet.
Now, regarding the connections handling function(handle): if your server is designed only for educative purposes, you can leave it as it is, but if your server is designed to accept lots of simultaneous connections and lots of data exchange from/to each client, you should handle each connection in a separate thread(this is a general advice that doesn't apply to Python because of its GIL, sometimes threads only slow things down)/process as done in SocketServer.py - part of Python's standard library.
#Edit1 - reply to 1st comment:
It's simple, all you have to do is modify a little bit the inner for loop:
for notified_socket in notified_socket_list:
incoming_socket, addr = notified_socket.accept()
if notified_socket == s_global:
handle_global(incoming_socket, addr)
else:
handle_local(incoming_socket, addr)
or you can use my original solution and do the differentiate the connections in handle: the addr argument contains the (local) port that the socket connected to; you can test it and depending on its value (9090 or 9091) handle the connection differently.

Python client server how UDP is supposed to work?

I have a client-server "snake" game working really well with TCP connections, and I would like to try it the UDP way.
I wonder how it is supposed to be used ? I know how UDP works, how to make a simple ECHO example, but I wonder how to do the following :
For instance with TCP, every TICK (1/15 second) server sends to the client the new Snake head position.
With UDP, am I supposed to do something like this :
Client SIDE :
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
serverAddr = (('localhost', PORT))
while 1:
client.sendto('askForNewHead', serverAddr)
msg, addrServer = client.recvfrom(1024)
game.addPosition(msg)
Server SIDE :
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((HOST, PORT))
while 1:
data, addr = server.recvfrom(1024)
server.sendto(headPosition, addr)
So here Client has to ask server to get the new head position, and then server sends the answer. I managed to make it work this way, but I can't figure out if it is a good way of doing.
Seems weird that client has to ask udp for an update while with my TCP connection, client has just to wait untill he receives a message.
There are differences between TCP and UDP but not the way you describe. Like with TCP the client can recvfrom to get messages from the server without asking each time for new data. The differences are:
With TCP the initial connect includes a packet exchange between client and server. Unless the client socket was already bound to an IP and port it will be bound to the clients IP and a free port will be allocated. Because of the handshake between client and server the server knows where to contact the client and thus can send data to the packet without getting data from the client before.
With UDP there is no initial handshake. Unless already bound, the socket will be bound to clients IP and a free port when sending the first packet to the server. Only when receiving this packet the server knows the IP and port of the client and can send data back.
Which means, that you don't need to 'askForNewHead' all the time. Instead the client has to send only a single packet to the server so that the server knows where to send all future packets.
But there are other important differences between TCP and UDP:
With UDP packets may be lost or could arrive in a different order. With TCP you have a guaranteed delivery.
With UDP there is no real connection, only an exchange of packets between two peers. With TCP you have the start and end of a connection. This is relevant for packet filters in firewalls or router, which often need to maintain the state of a connection. Because UDP has no end-of-connection the packet filters will just use a simple timeout, often as low as 30 seconds. Thus, if the client is inside a home network and waits passively for data from server, it might wait forever if the packet filter closed the state because of the timeout. To work around this data have to be transmitted in regular intervals so that the state does not time out.
One often finds the argument, that UDP is faster then TCP. This is plain wrong. But you might see latency problems if packets get lost because TCP will notice packet loss and send the packet again and also reduce wire speed to loose less packets. With UDP instead you have to deal with the packet loss and other congestion problems yourself. There are situations like real time audio, where it is ok to loose some packets but low latency is important. These are situations where UDP is good, but in most other situations TCP is better.
UDP is different to TCP, and I believe with python the client does have to ask for an update from the server.
Although it is fun to learn and use a different way of communicating over the internet, for python I would really recommend sticking with TCP.
You don't have to ask the server for a update. But since UDP is connection-less the server can send head-positions without being asked. But the client should send i'm-alive-packets to the server, but this could happen every 10 seconds or so.

Categories