I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.
How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.
Related
so I basically have just a Client & Server application going on here and the server sends out data in a loop that looks like this {"Message": "Status Check From Server"}
For some reason after the server sends this out around 3 times, the Client then stops receiving data for about 10 seconds then prints out what looks like multiple merged messages of the same data, it looks something like this {"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Serv I literally have no clue what this is, I've tried so hard to debug this and just can't figure it out, I don't know if it's my code or if this is simply how TCP works. I'll put some of the code down below.
SERVER SIDE | HAD TO CHANGE SOME OF THE CODE IN HERE TO MAKE SENSE
# Binding Of The Socket
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.bind(('0.0.0.0', 777))
SOCK.listen()
# This Is The Function I Have Written Up That Sends That Status Check Message To The Clients
def DeviceChecker():
global DEVICE_LIST
while True:
for DEVICE in DEVICE_LIST:
try:
DEVICE.send(json.dumps({'Message': 'Status Check From Server'}).encode())
DATA = DEVICE.recv(4096, socket.MSG_PEEK)
if len(DATA) == 0:
raise BrokenPipeError
except BrokenPipeError:
DEVICE_LIST.remove(DEVICE)
CLIENT SIDE
# This Is Where The Client Connects To The Server
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.connect(('0.0.0.0', 777))
# This Is Where The Client Is Receiving The Data
def DataListener(SOCK):
while True:
try:
DATA = SOCK.recv(4096).strip()
print(DATA)
except BrokenPipeError:
return
Because of the way TCP sockets work, individual write() calls do not result in individual "messages". TCP is designed to break large data streams into packets for transmission over the internet, and rejoin them on the receiving end. The stream of data from one socket to another, in TCP streams at least, is continuous, and individual write()'s are lumped into the same buffer. This is the same issue as in this answered question which describes creating a file-like object using socket.makefile(), then using write() and then flush() on that object to ensure all the data is sent. If you need to make sure your messages are received one at a time, you can have your client send an acknowlegement after each message it receives, and have your server wait for that acknowlegement before sending the next message. As a side note, the socket.send() method is not guaranteed send all the data you give it, and will return, as an int, the number of bytes it actually sent. If you need to make sure all the data is sent in one function call, use socket.sendall(). That will call send() as many times as it needs to.
I am experimenting with python sockets to try to understand the whole concept better, but I have run into a problem. I have a simple server and a client, where the client sends a list to the server, and then waits for the server to send a string signaling the process is complete.
This is the client file:
import socket
import json
host = '192.168.1.102'
port = 14314
def request():
print 'Connecting'
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect((host, port))
print 'Sending request'
clientsocket.sendall(json.dumps([1, 2, 3, 4, 5, 6, 7, 8, 9]))
print 'Receiving data'
data = clientsocket.recv(512)
print 'Received: {}'.format(data)
request()
and here is the server file:
import socket
import json
host = '192.168.1.102'
port = 14314
def run():
print 'Binding socket'
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serversocket.bind((host, port))
print 'Waiting for client'
serversocket.listen(1)
clientsocket, addr = serversocket.accept()
print 'Receiving data'
raw_data = ''
while True:
tmp = clientsocket.recv(1024)
if not tmp:
break
raw_data += tmp
data = json.loads(raw_data)
print 'Received: {}'.format(data)
print 'Sending data'
clientsocket.sendall('done')
run()
The problem is that while the client is done sending the list, the server is stuck in the recv loop, waiting for nothing. The whole data has been received in the first iteration and in the second iteration there is nothing to be received because the client has moved on to the receiving part.
The weird part is that if I comment out the receive part from the client and the send part from the server, the process completes successfully. So, what am I doing wrong? Why is this not working?
Thanks.
The Docs for socket.recv talk about additional flags being able to be passed in to the recv function described in the unix documentation. So turning to that documentation, I found the following message:
If no messages are available at the socket, the receive calls wait for
a message to arrive, unless the socket is nonblocking (see fcntl(2)),
in which case the value -1 is returned
So once again, we're directed to another page. The documentation for fcntl says
Performs one of the operations described below on the open file
descriptor
So, normally the socket.recv function is blocking (it will wait indefinitely for new data), unless we use a file descriptor. How do we do that? Well there is a socket.makefile function that gives us a file descriptor attached to the socket. Cool. This SO question gives us an example of how we can read and write to a socket, using a file descriptor.
Well what if we don't want to use a file descriptor. Reading further into the unix documentation for the recv function, I see that I can use the MSG_DONTWAIT flag. This doesn't work in Windows, but I did find out that we can use socket.setbocking(False) to permamently change the socket to non-blocking mode. You would then need to ignore any "A non-blocking socket operation could not be completed immediately" errors. Those are normal and non-fatal(error #10035 of this page mentions it is non-fatal).
Another possible implementation would be to multi-thread your program, you can implement a receiving and a sending thread for your socket. This might give you the best performance, but it would be a lot of work to setup.
Python is awesome. I just found some libraries Python has that does asynchronous sockets too. There's asyncore, asynchat which have both been deprecated in favor of asyncio if that is available in the version of Python you are using.
Sorry for throwing so much out there. I don't know a whole lot about sockets. I used them once with the Paramiko library, and that was it. But it looks like there are a lot of ways of implementing them.
I'm trying to understand how send and receive are working.
I was trying to send continuously data to a server and i noticed that the server would receive mixed bytes because i was sending to much data at a time. See my code:
Server:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("",1996))
server.listen(0)
c,d = server.accept()
while True:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
Client:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.connect(("192.168.1.4",1996))
while True:
data = 1
server.send( struct.pack("i", data) )
Then i change the while loops to this:
Server:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
c.send( str.encode("Server received your message. You now can continue
sending more data") )
Client:
data = 1
server.send( struct.pack("i", data) )
#Wait to secure the send.
server.recv(1024)
This is working. I'm making sure that the client won't send data before the
server already receive the previous send.
But what if i want to do the same for the server too? How can i make sure that the server will send bytes to the client in a safe way?
I already tried this and i notice that i created an infinity loop because(I used multi-threading in order to send and receive at the same time on the server):
client was sending some data and then waiting to get a signal from the server
that he can send again.
the server was getting some data then sending the signal and after that waiting for a signal from the user that he can send again.
But because the client was actually sending data again, the whole thing was going on again and this caused me an infinity talk-reply loop.
So what can i do to make a continuously conversation between two sockets without mixing the bytes together?
Your problem is caused by Nagle algorithm which works by combining a number of small outgoing messages, and sending them all at once as TCP is a stream protocol. You can enable TCP_NODELAY socket option by calling sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) to sent data as soon as possible, even if there is only a small amount of data. And on the receiver side, it isn't going to get one packet at a time either, you must implement message boundaries itself if you want "continuous conversation between two sockets without mixing the bytes together".
I've run in to a strange problem in a multiplayer online game I'm developing.
When the user clicks "Accept Quest" on the client, it performs the following action:
packet = "A:io-QS#"
tcpClient.send(packet.encode('utf-8'))
On the server, there is a thread created for each client that handles sending/receiving data:
while (client[self.id].authenticated == True):
try:
data = (self.connection.recv(1024)).decode('utf-8')
client[self.id].lastPacketTime = time.time()
client[self.id].processData(data)
except:
print("Client disconnected due to data receive error")
client[self.id].saveDataToDatabase()
client[self.id].authenticated = False
client[self.id].loggedIn = False
If the server receives the packet "A:io-QS#", it throws an exception and disconnects the client. I modified the client code mentioned aboe to send the packet "M:w#" (directional movement packet) and it doesn't throw an exception,
Only when the packet is "A:io-QS#".
The packet size isn't a concern (a much larger packet containing login credentials passes through this server loop just fine).
I tried commenting out the "client[self.id].processData(data)" line and the exception still occurs (but only with the packet 'A:io-QS#').
The server throws an exception after receiving the data but before acting upon it, so it's not a logic error.
I'm at a bit of a loss, does anybody see anything I'm missing or have any recommendations on how I could test this issue further?
Thanks!
I suggest using sys.exc_info() in the except block in the server to find more about the exception.
I have a socket server running on an Arduino board and am trying to control it via a Python script. Using the basic example socket documentation, I have this:
import socket
import sys
TCP_IP = '192.168.254.100'
TCP_PORT = 5012
BUFFER_SIZE = 1024
MESSAGE = "Status"
# Create a TCP/IP socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MESSAGE)
# data = s.recv(BUFFER_SIZE)
# print (data)
s.close()
sys.exit()
My script works fine when I comment out the lines to receive the response of the server. However, if I attempt to read the response, my server and python script hangs.
On the server side, here is a snippet of what the code looks like:
void loop() {
// listen for incoming clients
client = server.available();
if (client){
Serial.println("Client connected");
while (client.connected()){
// Read the incoming TCP command
String command = ReadTCPCommand(&client);
// Debugging echo command to serial
command.trim();
Serial.println(command);
// Debugging echo command back to client
client.println(command);
// Phrase the command
PhraseTCPCommand(&client, &command);
}
// Stop the client
client.stop();
Serial.println("Client disconnected");
}
}
The library I am utilising for the server is the Arduino WiFi library.
The function PhraseTCPCommand, takes the command and triggers external events with the GPIO pins of the board. This action is performed fine by the Python script when the recv() is commented out. The response string sent from the server is terminated with a newline and carriage return. Could that be causing issues?
Additionally, I am able to connect and receive responses from the server with no issues using either telnet, netcat or PuTTY, which leads me to believe it's something to do with the way my Python script attempts to read the response from the server.
The response string sent from the server is terminated with a newline
and carriage return. Could that be causing issues?
No, what is causing the issue is possibly that the command MESSAGE is not terminated with a newline and the function ReadTCPCommand() expects one. Change to:
MESSAGE = "Status\n"
The issue here could be that your message has not been fully sent, and reflects a common misunderstanding of socket programming. Low level calls such as send do not promise to send the full message. Instead the caller must check the number of bytes actually sent (from the return value) and continue to iteratively call the function until the full message is sent.
In general, a bare send call without such iteration is a mistake.
Python does provides a higher level function socket.sendall however that will perform this job for you.