We Keep Coding

Python TCP Socket Is Merging Data?

so I basically have just a Client & Server application going on here and the server sends out data in a loop that looks like this {"Message": "Status Check From Server"}
For some reason after the server sends this out around 3 times, the Client then stops receiving data for about 10 seconds then prints out what looks like multiple merged messages of the same data, it looks something like this {"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Server"}{"Message": "Status Check From Serv I literally have no clue what this is, I've tried so hard to debug this and just can't figure it out, I don't know if it's my code or if this is simply how TCP works. I'll put some of the code down below.
SERVER SIDE | HAD TO CHANGE SOME OF THE CODE IN HERE TO MAKE SENSE
# Binding Of The Socket
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.bind(('0.0.0.0', 777))
SOCK.listen()
# This Is The Function I Have Written Up That Sends That Status Check Message To The Clients
def DeviceChecker():
global DEVICE_LIST
while True:
for DEVICE in DEVICE_LIST:
try:
DEVICE.send(json.dumps({'Message': 'Status Check From Server'}).encode())
DATA = DEVICE.recv(4096, socket.MSG_PEEK)
if len(DATA) == 0:
raise BrokenPipeError
except BrokenPipeError:
DEVICE_LIST.remove(DEVICE)
CLIENT SIDE
# This Is Where The Client Connects To The Server
SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
SOCK.connect(('0.0.0.0', 777))
# This Is Where The Client Is Receiving The Data
def DataListener(SOCK):
while True:
try:
DATA = SOCK.recv(4096).strip()
print(DATA)
except BrokenPipeError:
return

Because of the way TCP sockets work, individual write() calls do not result in individual "messages". TCP is designed to break large data streams into packets for transmission over the internet, and rejoin them on the receiving end. The stream of data from one socket to another, in TCP streams at least, is continuous, and individual write()'s are lumped into the same buffer. This is the same issue as in this answered question which describes creating a file-like object using socket.makefile(), then using write() and then flush() on that object to ensure all the data is sent. If you need to make sure your messages are received one at a time, you can have your client send an acknowlegement after each message it receives, and have your server wait for that acknowlegement before sending the next message. As a side note, the socket.send() method is not guaranteed send all the data you give it, and will return, as an int, the number of bytes it actually sent. If you need to make sure all the data is sent in one function call, use socket.sendall(). That will call send() as many times as it needs to.

Continous receiving & sending on a socket

I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.

How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.

Infinite wait for reply from server using sockets in Python

I'm trying to send a message to the API of a server in order to get a reply. I am using the following code:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('h68.p.ctrader.com',5211)
sock.connect(server_address)
message = "8=FIX.4.4|9=87|35=0|49=theBroker.12345|56=cServer|57=QUOTE|50=BVN's Message|34=1|52=20180322-21:26:01|10=101"
sock.send(bytes(message,'utf-8'))
data = sock.recv(3)
print(data)
sock.close()
However, when executing it, the message is sent to the server, but at the moment of receiving the reply from the server [data = sock.recv (3)], the program does not go on. It keeps the cursor blinking, like in an infinite loop. What is the cause of this problem? Was it the script? The message sent to the server? The server itself? How to solve the problem?
Note: This message is in a format required by the server API, which consists of "tag" = "value" | "tag" = "value" | "tag" = "value" ...

A quick look at the documentation for this API you're using turns up a couple of problems:
You're missing a | at the very end of the message.
You shouldn't actually be sending |s at all, that's just a human-readable translation of the actual message. All of them need to be replaced with \x01 or \u0001 (depending on whether you're creating a byte string or a Unicode string), the actual tag separator character.

Prevent print() statement from overwriting text from input() statement

So I'm working on a simple chat server in python.
The server is running fine and on the client I have one thread for receiving incoming data and one for sending messages.
I can send a message from client_1 to the server which passes it to all other clients which will then print the message.
Even though everything technically works fine, there is still one thing that is VERY annoying whenever it happens:
Say client_1 is typing text into the console.
At the same time client_2 sends a message to the server, the server sends it to client_1 and client_1 prints the message.
Now the text client_1 was originally typing into the console is no longer in the line it was supposed to be in.
This is what the consoles looked like before client_2 sent the string "test test": https://ibb.co/hFdeo7
and this is what they looked like after sending: https://ibb.co/mEWAvn
NOTE: If I were to press Enter on client_1 the message "TEST TEST TEST" would still be sent correctly. The problem only lies in the conflict between the text that is being printed and the text in the input() statement.
My code looks like This:
Server.py
connections = []
while True:
readable, writeable, exception = select.select(connections, [], [], 0)
for sock in readable:
if sock == server:
conn, addr = server.accept()
connections.append(conn)
else:
data = str(sock.recv(1024), 'utf8')
if data:
for s in connections:
if s != server_socket and s != sock:
s.send(bytes(data, 'utf8'))
else:
connections.remove(sock)
Client.py
def receive():
while True:
readable, writeable, exception = select.select([0, client], [], [])
for sock in readable:
if sock == client:
data = str(sock.recv(1024), 'utf8')
if data:
print(data)
def send():
while True:
readable, writeable, exception = select.select([0, client], [], [])
for sock in readable:
if sock == client:
pass
else:
msg = input()
client.send(bytes(msg, 'utf8'))
Thread(target=receive).start()
Thread(target=send).start()
Is there any way to solve this problem without running the send() and receive() functions in separate scripts, or using a GUI module like Tkinter?
EDIT: I would like to print the incoming message as soon as it is received but then display the input() prompt and typed text again afterwards.

Here's one approach to showing the incoming message and then redisplaying the input prompt and partial input string (as specified in a comment). It uses readline.get_line_buffer to read the currently-input string and redisplay it. A warning, though: reading and writing to the same stream from different threads without locking is going to be prone to glitches.
It just requires a small modification to the receive function:
if data:
print('\n' + data)
sys.stdout.write(readline.get_line_buffer())
sys.stdout.flush()
The '\n' in the print is so the incoming message doesn't land right on top of whatever's being typed. The flush is necessary so that you can write the current input without a newline but without it getting buffered. I might suggest adding a prompt (like '> ') to the input and the sys.stdout.write, just to make it clearer to the user what's happening.
Lastly, running this may mess up your terminal output. You might need to run reset afterwards. There's probably a way to clean up that prevents that but I don't know offhand what it is.

PYTHON - UDP listening without knowing the server

using python2.X (for Linux (++) and Windows (+))
I'm trying to create a python client, to listen to an adress and a port.
The messages are sent by an app on a specific adress and a specific port, but I don't know how it is written. I'm just sure it is functionnal and it uses UDP protocole.
So, I've written this code to receive the messages :
import socket
#---socket creation
connexion = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#---Bind
try:
connexion.bind(('', 10015))
except socket.error:
print "connexion failed"
connexion.close()
sys.exit()
#---testing
while 1:
data, addr = connexion.recvfrom(1024)
print "messages : ", data
At this point, there is no error running it, but the code stops at this line :
data, addr = connexion.recvfrom(1024)
and nothing happens... I think I'm not connected and can't receive messages but I don't know why.
I tried to change the adresse by : '192.168.X.X', but it's the same.
If anyone could help me, that would be great. I'm not very comfortable why python...
PS : sorry for my english.

I found my problem. I changed the IP adress (in the documentation, it was 192.168. 00 8. 0 15 ; I tried whithout the zeros like a normal adress (192.168.8.15) and I received my messages
I changed the structure of my code too : (don't know if it has impact) I put my "try: [...] except:[...]" on the line
data, addr = connexion.recvfrom(1024)
instead of the line :
connexion.bind(('', 10015))
Thanks for helpping ;)

Your code is all well and fine. You can verify that it works using the following Python snippet:
import socket
connexion = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
connexion.sendto("sending data via udp", ("localhost", 10015))
If a message send with this snippet is not displayed, check your firewall. To investigate whether (or what) your other application is actually producing try using tcpdump or wireshark.