I've got a client-server app I'm making and I'm having a bit of trouble when the server wait for data from the client.
After my the client connects to the server socket, the server open him new thread and get data from the client (in JSON format).
So far, my code works for a single message. When I added while loop, that always accept messages I got a problem. After some tests I found that the recv() function not waiting for new data and continues to the next line, and this is what creates the problem.
I will be happy if you can help me fix the problem.
my receive data loop (The first iteration of the loop works but Receive data in the second iteration not wait for data and make problem because the next line not get any data)-
while True:
data = self.client.recv(self.size) # receive data
message = self.JSON_parser(data) # parser the data (data in json format)
process_message = processing.Processing(message[0]['key'],message[0]['user'],message[0]['data']) # send the receive data to the initialize process
process_return = process_message.action() # call to the action function
self.client.send(process_return) # send to the client message back
If recv() is returning an empty string, it means that the connection has been closed.
The socket may be closed either by the client, or by the server. In this case, looking at the server code you posted, I'm almost sure that the client is closing the connection, perhaps because it's exiting.
In general, your server code should look like this:
while True:
data = self.client.recv(self.size)
if not data:
# client closed the connection
break
message = self.JSON_parser(data)
...
Bonus tip: a long JSON message may require more than one call to recv(). Similarly, more than one JSON message may be returned by recv().
Be sure to implement buffering appropriately. Consider wrapping your socket into a file-like object and reading your messages line-by-line (assuming that your messages are delimited by newline characters).
Related
I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()
You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.
I am having a tcp server and a client written in python. The aim of the programs is that the server will be sending some numbers one after the other to the client and the client should process each one of the number in a separate thread.
The server loops over the number list and sends each one of them to client.
as:
for num in nums:
client_sock.send(str(num))
and the client loop as:
while True:
data = tcpClientSock.recv(BUFSIZE)
thread.start_new_thread( startFunction, (data, ) )
The problem is even though the server sends the program in separate send() call the client receives it all at once.
How can I avoid it? Should I use UDP instead of TCP in this situation?
you'll have to flush the socket on the sending end - add a CR/NL to do so (since you're sending a string)
TCP is a stream based protocol and not message based. This means there are no message boundaries for each time the server calls send(). In fact, each time send() is called, the bytes of data are just added to the stream.
On the receiving end, you'll receive bytes of the stream as they arrive. Since there are no message boundaries, you may receive part of a message or whole messages or whole + part of the next message.
In order to send message over a TCP stream, your protocol needs to establish message boundaries. This allows the receiver to interpret whether it has received a partial, full, or multiple messages.
In your example, the server is sending strings. The string termination servers as the message boundary. On the receiving side, you should be parsing out the strings and have handling for receiving partial strings
In my code I wrote something like this:
try:
s.sendall(data)
except Exception as e:
print e
Now, can I assume that if there wasn't any exception thrown by sendall that the other side of the socket (its kernel) did receive 'data'? If not then that means I need to send an application ack which seems unreasonable to me.
If I can assume that the other side's kernel did receive 'data' then that means that 'sendall' returns only when it sees tcp ack for all the bytes I have put in 'data' but I couldn't see any documentation for this, on the contrary, from searching the web I got the feeling that I cannot assume an ack was received.
can I assume that if there wasn't any exception thrown by sendall that the other side of the socket (its kernel) did receive 'data'?
No, you can't. All it tells you that the system successfully sent the data. It will not wait for the peer to ACK the data (i.e. data received at the OS kernel) or even wait until the data got processed by the peer application. This behavior is not specific to python.
And usually it does not matter much if the peer systems kernel received the data and put it into the applications socket buffer. All what really counts is if it received and processed the data inside the application, which might involve complex things like inserting the data into a database and waiting for a successful commit or even forwarding the data to yet another system. And since it is up to the application to decide when the data are really processed you have to make your application specific ACK to signal successful processing.
Yes you can :)
According to the socket.sendall docs:
socket.sendall(string[, flags]) Send data to the socket. The socket
must be connected to a remote socket. The optional flags argument has
the same meaning as for recv() above. Unlike send(), this method
continues to send data from string until either all data has been sent
or an error occurs. None is returned on success. On error, an
exception is raised, and there is no way to determine how much data,
if any, was successfully sent.
Specifically:
socket.sendall() will continue to send all data until it has completed or an error has occurred.
Update: To answer your comment about what's going on under the hook:
Looking at the socketmodule.c source code it looks like it repeatedly tries to "send all data" until there is no more data left to send. You can see this on L3611 } while (len > 0);. Hopefully this answers your question.
I am working on a simple web server that sends push notification to my iPhone app.
I've tried using existing Python packages, but they all suffer from this problem:
When an invalid token is sent, Apple will send back a error message and close the socket. I need to send a large number of push notifications, but if one of them has an invalid token, the rest won't be delivered to Apple.
So currently, my code looks something like this:
for m in messages:
socket.send(m)
But when the first of the messages has invalid token, so Apple closes the socket as soon as it receives the first message, socket still sends all the messages without any error.
I can't afford to do a recv after each message with a timeout, since that will take too long. What can I do to know when the socket is closed and open a new one?
If the socket had been closed on the other end a recv on it should return 0 or result in an error.
You could check the number of bytes sent by send.
Update: You might like to read here on another possibilty to monitor if a socket connection is still alive.
Update-1: Another approach might by to run a constantly reading thread on the socket in question in parallel to the write operation ... as you mention: time is your limit.
I learned that there is no way at TCP level that I can find out what packets were sent. I decided to just send all the messages and then wait (with timeout) for any error messages from Apple. If there is any error, repeat sending the ones that failed.
I decided to go with this, since getting an invalid token should be a rare event, and most of the time all the notifications will go through.
Hello I have written some client server code and write now I noticed I had a bug in how I am handling receiving a command
These are my commands
#Server Commands
CMD_MSG, CMD_MULTI, CMD_IP, CMD_AUDIO, CMD_AUDIO_MULTI, CMD_FILE = range(6)
I send a command like this
self.client(chr(CMD_AUDIO), data)
and receive like this
msg = conn.recv(2024)
if msg:
cmd, msg = ord(msg[0]),msg[1:]
if cmd == CMD_MSG:
#do something
The first command seems to work but if I call any other it seems to loop through them all. Its really bizarre
I can post more code if needed.
But any ideas on how to handle the commands being sent to my server would be great
*cheers
Assuming you're using a stream (TCP) socket, the first rule of stream sockets is that you will not receive data in the same groups it is sent. If you send three messages of 10 bytes each, you may receive at the other end one block of 30 bytes, 30 blocks of one byte each, or anything in between.
You must structure your protocol so that the receiver knows how long each message within the stream is (either by adding a length field, or by having fixed length message formats), and you must save the unused portion of any recv() that crosses a message boundary to use in the next message.
The alternative to stream/TCP sockets is datagram/UDP sockets. These preserve message boundaries, but do not guarantee delivery or ordering of the messages. Depending on what you're doing, this may be acceptable, but probably not.