We Keep Coding

Python TCP Socket Is Merging Data?

so I basically have just a Client & Server application going on here and the server sends out data in a loop that looks like this {"Message": "Status Check From Server"}
For some reason after the server sends this out around 3 times, the Client then stops receiving data for about 10 seconds then prints out what looks like multiple merged messages of the same data, it looks something like this {"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Serv I literally have no clue what this is, I've tried so hard to debug this and just can't figure it out, I don't know if it's my code or if this is simply how TCP works. I'll put some of the code down below.
SERVER SIDE | HAD TO CHANGE SOME OF THE CODE IN HERE TO MAKE SENSE
# Binding Of The Socket
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.bind(('0.0.0.0', 777))
SOCK.listen()
# This Is The Function I Have Written Up That Sends That Status Check Message To The Clients
def DeviceChecker():
global DEVICE_LIST
while True:
for DEVICE in DEVICE_LIST:
try:
DEVICE.send(json.dumps({'Message': 'Status Check From Server'}).encode())
DATA = DEVICE.recv(4096, socket.MSG_PEEK)
if len(DATA) == 0:
raise BrokenPipeError
except BrokenPipeError:
DEVICE_LIST.remove(DEVICE)
CLIENT SIDE
# This Is Where The Client Connects To The Server
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.connect(('0.0.0.0', 777))
# This Is Where The Client Is Receiving The Data
def DataListener(SOCK):
while True:
try:
DATA = SOCK.recv(4096).strip()
print(DATA)
except BrokenPipeError:
return

Because of the way TCP sockets work, individual write() calls do not result in individual "messages". TCP is designed to break large data streams into packets for transmission over the internet, and rejoin them on the receiving end. The stream of data from one socket to another, in TCP streams at least, is continuous, and individual write()'s are lumped into the same buffer. This is the same issue as in this answered question which describes creating a file-like object using socket.makefile(), then using write() and then flush() on that object to ensure all the data is sent. If you need to make sure your messages are received one at a time, you can have your client send an acknowlegement after each message it receives, and have your server wait for that acknowlegement before sending the next message. As a side note, the socket.send() method is not guaranteed send all the data you give it, and will return, as an int, the number of bytes it actually sent. If you need to make sure all the data is sent in one function call, use socket.sendall(). That will call send() as many times as it needs to.

Python socket programming sequence queries

I am trying to send queries coming from a client to a base server that forwards then to 2 more servers that the base server is connected to.
(Those 2 servers find a match with the query in their database, if it matches, they send the query back, if no match, they send nothing)
First I send the query from the base server to the 1st server called ts1, If I don't get a reply, I send it to 2nd server called ts2. If I don't get a reply from ts2 as well, I send a error host message to the client.
I've tried these while and inner while loops to accomplish this task but my program hangs after sending the first query. Because the ts1 server sends nothing on the first query.
How can I correct my code so it wont hang and the entire code executes? maybe its non-blocking or blocking of the sockets ? I don't know how that is implemented.
while True:
# receive a message from the client
data_from_client = csockid.recv(1024)
word = data_from_client.decode('utf-8').lower()
if word == '':
break
print('[Client]: ' + word)
print("[ls Server]: Sending " + word + " to ts1 & ts2 for Query....")
ts1.send(word.encode('utf-8'))
while True:
search_from_ts1 = ts1.recv(1024)
ts1_server_answer = search_from_ts1.decode('utf-8')
print("[ls Server]: sending reply from ts1 server " + ts1_server_answer + " to Client")
csockid.send(ts1_server_answer.encode('utf-8'))
if not search_from_ts1:
ts2.send(word.encode('utf-8'))
while True:
search_from_ts2 = ts2.recv(1024)
ts2_server_answer = search_from_ts2.decode('utf-8')
print("[ls Server]: sending reply from ts2 server " + ts2_server_answer + " to Client")
csockid.send(ts2_server_answer.encode('utf-8'))
if not search_from_ts2:
print("[ls Server]: No match found in both ts1 & ts2")
error = " - Error:HOST NOT FOUND"
csockid.send(error.encode('utf-8'))
break
break
if not data_from_client:
break

You might want to set a timeout, then try the other server, but that might make the code unacceptably slow. At this point you might want to explore async functions which give you the instruments for doing this properly (sending the request to both servers "simultaneously" and then take whatever is available but have a timeout on both request replies).

Have a look at the non-blocking sockets section in https://docs.python.org/2/howto/sockets.html for reference. It tells you how to set up non-blocking sockets and the use of select to poll more than one socket.

Can't make a Python socket blocking

I'm trying to write a fairly simple client-server Python application using socket and SocketServer. To allow for two-way communication between client and server, the client maintains one connected socket with the server so it can listen for messages in a separate thread, while the main thread creates one-time-use sockets to send messages to the server. I want my "listening" socket to be blocking, as it is running in a separate thread whose only purpose is to wait for data without blocking the main program. Here is the function where I create this socket:
def connect(self, alias, serverIP):
if not alias or not isinstance(alias, str):
print "ERROR: Must specify an alias"
return
self.serverIP = serverIP
self.downConnection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.downConnection.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.downConnection.setblocking(1)
self.downConnection.connect((self.serverIP, 11100))
self.downConnection.send("SENDSERVER CONNECT %s" % alias)
Here is the loop where the persistent socket listens for messages from the server (with some debugging code thrown in):
i = 0
while True:
print "LOOP", i,
if self.closed:
break
try:
data = self.downConnection.recv(1024)
except socket.timeout, e:
print "Timeout"
pass
else:
print "Received %d" % len(data)
if data:
self.received(data)
i += 1
I would expect to see "Received ##" messages only when the server sends data, and maybe periodic "Timeout" messages otherwise. Instead, the output grows very rapidly and looks entirely like this:
LOOP 33858 Received 0
LOOP 33859 Received 0
LOOP 33860 Received 0
LOOP 33861 Received 0
LOOP 33862 Received 0
LOOP 33863 Received 0
LOOP 33864 Received 0
LOOP 33865 Received 0
So it seems that self.downConnection.recv() is immediately returning an empty string each time it is called, rather than blocking until it receives substantive data like it's supposed to. This is puzzling, as I'm explicitly setting the socket to be blocking (which I think is also the default setting). Constantly executing this loop instead of the thread spending most of its time waiting for data is wasting a good deal of CPU time. What am I doing wrong in setting up the blocking socket?
Here is the full server code. The Comms class is also the superclass of the client class, to allow for some basic common functionality.
Something does seem to be wrong with the connection from the server's end. The server can receive data from the client, but trying to send data to the client gives a socket.error: [Errno 9] Bad file descriptor exception.

Server disconnecting client when receiving 1 packet string, but not another

I've run in to a strange problem in a multiplayer online game I'm developing.
When the user clicks "Accept Quest" on the client, it performs the following action:
packet = "A:io-QS#"
tcpClient.send(packet.encode('utf-8'))
On the server, there is a thread created for each client that handles sending/receiving data:
while (client[self.id].authenticated == True):
try:
data = (self.connection.recv(1024)).decode('utf-8')
client[self.id].lastPacketTime = time.time()
client[self.id].processData(data)
except:
print("Client disconnected due to data receive error")
client[self.id].saveDataToDatabase()
client[self.id].authenticated = False
client[self.id].loggedIn = False
If the server receives the packet "A:io-QS#", it throws an exception and disconnects the client. I modified the client code mentioned aboe to send the packet "M:w#" (directional movement packet) and it doesn't throw an exception,
Only when the packet is "A:io-QS#".
The packet size isn't a concern (a much larger packet containing login credentials passes through this server loop just fine).
I tried commenting out the "client[self.id].processData(data)" line and the exception still occurs (but only with the packet 'A:io-QS#').
The server throws an exception after receiving the data but before acting upon it, so it's not a logic error.
I'm at a bit of a loss, does anybody see anything I'm missing or have any recommendations on how I could test this issue further?
Thanks!

I suggest using sys.exc_info() in the except block in the server to find more about the exception.

Python TCP Payload Duplication - Passing through data to multiple endpoints concurrently

this is my first post here!
My goal is to duplicate the payload of a unidirectional TCP stream and send this payload to multiple endpoints concurrently. I have a working prototype written in Python, however I am new to Python, and to Socket programming. Ideally the solution is capable of running in both Windows and *nix environments.
This prototype works, however it creates a new send TCP connection for each Buffer length (currently set to 4096 bytes). The main problem with this is I will eventually run out of local ports to send from, and ideally I would like the data to pass from each single incoming TCP stream to one single TCP stream out (for each endpoint). The incoming data can vary from less than 1024 bytes to hundreds of megabytes.
At the moment a new outgoing TCP stream is initiated for every 4096 bytes. I am not sure if the problem is in my implementation of threading, or if I have missed something else really obvious.
In my research I have found that select() could help, however I am not sure if it would be appropriate because I may need to process some of the incoming data and respond to the sending client for certain cases in the future.
Here is the code I have so far (some of the code variations I have tried are commented out):
#!/usr/bin/python
#One way TCP payload duplication
import sys
import threading
from socket import *
bufsize = 4096
host= ''
# Methods:
#handles sending the data to the endpoints
def send(endpoint,port,data):
sendSocket = socket(AF_INET, SOCK_STREAM)
#sendSocket.setblocking(1)
sendSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
#sendport = sendSocket.getsockname
#print sendport
try:
sendSocket.connect((endpoint, port))
sendSocket.sendall(data)
except IOError as msg:
print "Send Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
sys.exit()
#handles threading for sending data to endpoints
def forward(service, ENDPOINT_LIST, port, data):
#for each endpoint in the endpoint list start a new send thread
for endpoint in ENDPOINT_LIST:
print "Forwarding data for %s from %s:%s to %s:%s" % (service,host,port,endpoint,port)
#send(endpoint,port,data)
ethread = threading.Thread(target=send, args=(endpoint,port,data))
ethread.start()
#handles threading for incoming clients
def clientthread(conn,service,ENDPOINT_LIST,port):
while True:
#receive data form client
data = conn.recv(bufsize)
if not data:
break
cthread = threading.Thread(target=forward, args=(service, ENDPOINT_LIST, port, data))
cthread.start()
#no data? then close the connection
conn.close()
#handles listening to sockets for incoming connections
def listen(service, ENDPOINT_LIST, port):
#create the socket
listenSocket = socket(AF_INET, SOCK_STREAM)
#Allow reusing addresses - I think this is important to stop local ports getting eaten up by never-ending tcp streams that don't close
listenSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
#try to bind the socket to host and port
try:
listenSocket.bind((host, port))
#display an error message if you can't
except IOError as msg:
print "Bind Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
sys.exit()
#start listening on the socket
listenSocket.listen(10)
print "Service %s on port %s is listening" %(service,port)
while True:
#wait to accept a connection
conn, addr = listenSocket.accept()
print 'Connected to ' + addr[0] + ':' + str(addr[1]) + ' on port ' + str(port)
#start new thread for each connection
lthread = threading.Thread(target=clientthread , args=(conn,service,ENDPOINT_LIST,port))
lthread.start()
#If no data close the connection
listenSocket.close()
service = "Dumb-one-way-tcp-service-name1"
ENDPOINT_LIST = ["192.168.1.100","192.168.1.200"]
port = 55551
listen(service,ENDPOINT_LIST,port)
I have looked into other libraries to try to achieve my goal, including using:
Twisted
Asyncore
Scapy
However I found them quite complicated for my modest needs and programming skill level.
If anyone has any suggestions on how I could refine the approach I have, or any other ways this goal could be achieved, please let me know!

In short, your question is there are not enough ports, right? It seems you didn't close socket after sending. Try this in send():
...
except IOError as msg:
print "Send Failed. Error Code: " + str(msg[0]) + ' Message: ' + msg[1]
sys.exit()
finally:
sendSocket.close()

There are two ways, if you don't want to learn some more advanced framework like Twisted.
The closest to what you're doing: use threads, but you need to have one thread per outgoing connection --- and not per outgoing packet. Create 3 Queue.Queue objects, and create 3 threads, passing to each one one of the Queue objects and one of the destinations. Each thread opens a socket, and then in a loop, it gets the next string from its own Queue and sends it to the socket. The clientthread (which can be just the main thread, a priori) receives data as strings, and puts each of these strings into all the Queues. This way, the packets sent don't get out of order, as they could if you create one thread per packet.
The alternative is to avoid threads completely, and use select(). It's a bit more mind-bending. Basically you have only one big loop that starts with select(). It needs careful management to pass the correct list of sockets to select(): you want the call to select() to wake up either when there is incoming data from the inbound socket, or if an outbound socket is both ready to send more and there is something more to send. In this model you'd have 3 lists of strings; when you read incoming data you append it to all three lists; the select() call is passed the list of outbound sockets that have a non-empty list (so, more to send); and when sending, you must not use sendall() in this model but send(), and if less than the full string was sent, you must re-add the remainder to the beginning of the corresponding list.