We Keep Coding

Python TCP Socket Is Merging Data?

so I basically have just a Client & Server application going on here and the server sends out data in a loop that looks like this {"Message": "Status Check From Server"}
For some reason after the server sends this out around 3 times, the Client then stops receiving data for about 10 seconds then prints out what looks like multiple merged messages of the same data, it looks something like this {"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Serv I literally have no clue what this is, I've tried so hard to debug this and just can't figure it out, I don't know if it's my code or if this is simply how TCP works. I'll put some of the code down below.
SERVER SIDE | HAD TO CHANGE SOME OF THE CODE IN HERE TO MAKE SENSE
# Binding Of The Socket
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.bind(('0.0.0.0', 777))
SOCK.listen()
# This Is The Function I Have Written Up That Sends That Status Check Message To The Clients
def DeviceChecker():
global DEVICE_LIST
while True:
for DEVICE in DEVICE_LIST:
try:
DEVICE.send(json.dumps({'Message': 'Status Check From Server'}).encode())
DATA = DEVICE.recv(4096, socket.MSG_PEEK)
if len(DATA) == 0:
raise BrokenPipeError
except BrokenPipeError:
DEVICE_LIST.remove(DEVICE)
CLIENT SIDE
# This Is Where The Client Connects To The Server
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.connect(('0.0.0.0', 777))
# This Is Where The Client Is Receiving The Data
def DataListener(SOCK):
while True:
try:
DATA = SOCK.recv(4096).strip()
print(DATA)
except BrokenPipeError:
return

Because of the way TCP sockets work, individual write() calls do not result in individual "messages". TCP is designed to break large data streams into packets for transmission over the internet, and rejoin them on the receiving end. The stream of data from one socket to another, in TCP streams at least, is continuous, and individual write()'s are lumped into the same buffer. This is the same issue as in this answered question which describes creating a file-like object using socket.makefile(), then using write() and then flush() on that object to ensure all the data is sent. If you need to make sure your messages are received one at a time, you can have your client send an acknowlegement after each message it receives, and have your server wait for that acknowlegement before sending the next message. As a side note, the socket.send() method is not guaranteed send all the data you give it, and will return, as an int, the number of bytes it actually sent. If you need to make sure all the data is sent in one function call, use socket.sendall(). That will call send() as many times as it needs to.

Continous receiving & sending on a socket

I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.

How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.

.recv function Socket programming TCP Server in Python

Im having trouble getting my very basic and simple TCP Server to properly work with http requests. This is what I have so far
from socket import *
import sys
serverSocket = socket(AF_INET, SOCK_STREAM)
serverSocket.bind(('', 4567))
serverSocket.listen(1)
while True:
print('Ready to serve...')
connectionSocket, addr = serverSocket.accept()
print("connected from: ", addr)
try:
message = connectionSocket.recv(1024)
filename = message.split()[1]
f = open(filename[1:])
outputdata = f.read()
connectionSocket.send("HTTP/1.1 200 OK\r\n")
for i in range(0, len(outputdata)):
connectionSocket.send(outputdata[i].encode())
connectionSocket.send("\r\n".encode())
connectionSocket.close()
except IOError:
connectionSocket.send("file not found")
serverSocket.close()
sys.exit()
The error comes from the open statement. I dont fully understand how this line of code's return value is organized.
message = connectionSocket.recv(1024)
I know that the return value is in bytes but when I try to use a fuction to turn it into a string like decode() i get errors as well
I have the .py file and the html file sitting in the same directory on my local machine and the way I test this is I just run this and open up a browser and type in
http://127.0.0.1:4567/helloworld.html
My code then promptly crashes after receiving the HTTP request.
Any and all help will be greatly appreciated!

There are numerous problems with your code and since you don't state what specific issues you are concerned about, here is what I see:
connectionSocket.send(outputdata[i].encode())
connectionSocket.send("\r\n".encode())
That appears to send a newline after every character you send back to the client.
Also, it doesn't deal with the client disconnecting because you're sending back invalid data.
Even if what you were trying to do didn't have these errors in it, you don't appear to be attempting to send back a valid http response.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html

Sending data multiple times over a single socket in python

I have a python program where I use a server socket to send data. There is a class which has some Threading methods. Each method checks a queue and if the queue is not empty, it sends the data over the sever socket. Queues are being filled with what clients send to server(server is listening for input requests). Sending is accomplished with a method call:
def send(self, data):
self.sqn += 1
try:
self.clisock.send(data)
except Exception, e:
print 'Send packet failed with error: ' + e.message
When the program starts, sending rate is around 500, but after a while it decreases instantly to 30 with this exception:
Send packet failed with error: <class 'socket.error'>>>[Errno 32] Broken pipe
I don't know what causes the rate to increase! Any idea?

That error is from your send function trying to write to a socket closed on the other side. If that is intended then catch the exception using
import errno, socket
try:
self.clisock.send(data)
except socket.error, err:
if err[0] == errno.EPIPE:
# do something
else:
pass # do something else
If this isn't intended behavior on the part of the client then you'll have to update your post with the corresponding client code.

Retrieve all the response in socket at one time

I am writing a client-sever program based on Python socket.
The client sends a command to the server and the server responds.
But now, some client can broadcast a message to other clients, so the client can receive more than one response at the same time.
data = s.recv(1024)
the line of code above will retrieve only one response from the server.
but if I use a while loop like this
while True:
data = s.recv(1024)
if not data: break
actually, data=s.recv(1024) will block the program when there is no data left.
I don't want to block the program and want to retrieve all the responses available in the connection at one time. Can anyone find a solution? Thank you.

You can use the select module to wait until the socket is readable or until a timeout has elapsed; you can then perform other processing. For example:
while True:
# If data can be received without blocking (timeout=0), read it now
ready = select.select([s], [], [], 0)
if s in ready[0]:
data = s.recv(1024)
# Process data
else:
# No data is available, perform other tasks

You could make the socket (s) non-blocking. This way, it will retrieve all the received responses and when there is none, it will return back. Of course, with non-blocking, you will have to periodically retry.
You could make the socket (s) non-blocking using the setblocking() method:
s.setblocking(0)
The other option is to use another thread to handle the receive part. This way, your main thread can continue doing its main task and act upon the message only if it receives one.

You can use socket.setblocking or socket.settimeout:
import socket
import sys
HOST = 'www.google.com'
PORT = 80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.setblocking(0)
s.sendall('Hello, world')
try:
data = s.recv(1024)
except:
print 'Oh noes! %s' % sys.exc_info()[0]
s.close()
socket.recv takes two parameters, the second is a set of flags. If you're on a Linux system, you can do man recv for a list of flags you can supply, and their corresponding errors.
Lastly, in general, you can't really know that the other side is done with sending you data (unless you're controlling both sides), even if you're both following a protocol. I believe the right way to go about it is to use timeouts, and quit after sending a reset (how you do this will depend upon what protocol you're using).