We Keep Coding

Ruby client and Python server not receiving the full message while exchanging

I am trying to establish a client-server communication. The client is written in Ruby whereas the server is written in Python.
client.rb
require 'socket'
hostname = 'localhost'
port = 7778
s = TCPSocket.open(hostname, port)
s.write("2020-06-25T11:11:00+00:00 5 127.0.0.1 printer: event")
while line = s.gets
puts line.chop
end
s.close()
The ruby client sends a log to the Python server and tries to receive it back.
server.py
import socket
#Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#bind the socket to the port - tuple
server_address = ('localhost', 7778)
print('starting up on %s port %s' %server_address)
sock.bind(server_address)
#Listen for incoming connections
sock.listen(1)
while True:
print('waiting for a connection')
connection, client_address = sock.accept()
while True:
data = connection.recv(1024)
print('received "%s"' % data)
if data:
print('sending data back to the client')
connection.send(data)
else:
print('no more data from', client_address)
break
connection.close()
The log is sent to the python server and when the python server sends it back. When the ruby client receives it, it doesn't receive the full log.
example:
2020-06-25T11:11:00+00:00 5 127.0.0.1 printer: eve
I think this comes from the fact that TCP is a streaming protocol and we never know if we can get the full message each time.
Could you propose me a solution for both the client and the server so I can be sure they always receive the full message between each other? I would really appreciate it if anyone would help.

So the issue is that you're assuming the data received has a new line character - however the data you're sending is not terminated by a new line.
s.write("2020-06-25T11:11:00+00:00 5 127.0.0.1 printer: event") will not write the string with a new line character - you should use puts IO#puts
s.gets will return the data because the socket is closed by the python server after it has sent the data. So even getssays it will read the next line from the socket, in reality its just reading what remained in the buffer after the socket was closed.
line.chop will remove the last character, and you're using it here to strip a newline character (assuming that it has one from gets). However since there is no newline character it will remove the last character instead.
So the fix would be to replace in the ruby client s.write with s.puts.

Socket Fragmented Received Data

I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1

As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.

One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.

You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()

Python socket loop does not break

I created socket for two PC, one is Raspberry Pi and the other one is my laptop. I just connected two then I send string to test the connection. If I send a character "q" from the RPi, my PC should break out of the loop and close the connection but it does not. The part print("Listening") is still running. Why? See code below.
import socket
import time
# IP address of this PC.
TCP_IP = '192.168.137.1'
# Port.
TCP_PORT = 5005
# Size of buffer.
BUFFER_SIZE = 1024
# Create a socket, connect and listen to it.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print('Connection address:', addr)
while 1:
print("Listening")
data = conn.recv(BUFFER_SIZE)
data = data.decode()
if data=='q':
break
if data:
print ("Received data:", data)
# Echo back.
conn.send(data.encode())
time.sleep(1)
print("It breaks.")
conn.close()
s.close()

TCP is a stream oriented protocol. So data transmitted is a stream not a sequence of messages. So when you expect data to be q it actually is some_data_sent_before_q_and_finally_q.
The simplest way to repair the code is to use if data.endswith('q') instead of if data=='q'. May work and may not depending on how you actually use the connection. For example, this approach may fail with some_data_sent_before_q pretty long pause more_data_and_q and with some_data_sent_before_q_and_finally_q_plus_something_else_why_not.
Little bit more advanced way to solve the problem is to divide the stream into messages with separators - message_1<separator>message_2<separator>q<separator>. This method will allow you to treat every message separately.

Python UDP Server send lines of a text file

I need to simulate a UDP server, which sends content of a text file line by line in an endless loop. I have written the code below but at the other end I do not receive anything (The other side is Qt code and I am sure it works):
import socket
import time
# setup UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sa = ('localhost', 9763)
sock.bind(sa)
filename = '/home/saeid/Documents/data.txt' # file to read
numlines = sum(1 for line in open(filename)) # get number of lines in file
# get line of a file
def get_line(line):
with open(filename) as fp:
for i, l in enumerate(fp):
if i == line:
return l
# main loop
while True:
currline = 0
while currline < numlines:
sock.sendto(get_line(currline), sa)
currline += 1
time.sleep(1)
I am no python pro and cant figure out the problem :(

For starters this is messed up:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sa = ('localhost', 9763)
sock.bind(sa)
...
sock.sendto(get_line(currline), sa)
By binding you basically say "I want to listen on that host/port". But then you send the data to the same host/port. I assume that there's some other destination address, e.g. sock.sendto(get_line(currline), ('my_host', 1234)). By the way, why are you binding to the address anyway? sock.bind(sa) line is unnecessary, remove it.
The other thing is that your file reading code is extremely inefficient and difficult to read (it took me a while to understand what's going on). Try something like this:
with open(filename, 'r') as fo:
while True:
for line in fo:
sock.sendto(line, DEST_ADDRESS)
time.sleep(1)
fo.seek(0) # go back to the begining of the file and repeat
and get rid of get_line function.
That's at least what I've came up with after reading your description. If you don't want to inifinitely send the same file then you can get rid of while True: loop and fo.seek(0) call.

Uncle google came up with User Datagram Client and Server
User Datagram Client and Server
The user datagram protocol (UDP) works differently from TCP/IP. Where TCP is a stream oriented protocol, ensuring that all of the data is transmitted in the right order, UDP is a message oriented protocol. UDP does not require a long-lived connection, so setting up a UDP socket is a little simpler. On the other hand, UDP messages must fit within a single packet (for IPv4, that means they can only hold 65,507 bytes because the 65,535 byte packet also includes header information) and delivery is not guaranteed as it is with TCP.
Echo Server
Since there is no connection, per se, the server does not need to listen for and accept connections. It only needs to use bind() to associate its socket with a port, and then wait for individual messages.
import socket
import sys
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Bind the socket to the port
server_address = ('localhost', 10000)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
Messages are read from the socket using recvfrom(), which returns the data as well as the address of the client from which it was sent.
while True:
print >>sys.stderr, '\nwaiting to receive message'
data, address = sock.recvfrom(4096)
print >>sys.stderr, 'received %s bytes from %s' % (len(data), address)
print >>sys.stderr, data
if data:
sent = sock.sendto(data, address)
print >>sys.stderr, 'sent %s bytes back to %s' % (sent, address)
Echo Client
The UDP echo client is similar the server, but does not use bind() to attach its socket to an address. It uses sendto() to deliver its message directly to the server, and recvfrom() to receive the response.
import socket
import sys
# Create a UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_address = ('localhost', 10000)
message = 'This is the message. It will be repeated.'
try:
# Send data
print >>sys.stderr, 'sending "%s"' % message
sent = sock.sendto(message, server_address)
# Receive response
print >>sys.stderr, 'waiting to receive'
data, server = sock.recvfrom(4096)
print >>sys.stderr, 'received "%s"' % data
finally:
print >>sys.stderr, 'closing socket'
sock.close()
Client and Server Together
Running the server produces:
$ python ./socket_echo_server_dgram.py
starting up on localhost port 10000
waiting to receive message
received 42 bytes from ('127.0.0.1', 50139)
This is the message. It will be repeated.
sent 42 bytes back to ('127.0.0.1', 50139)
waiting to receive message
and the client output is:
$ python ./socket_echo_client_dgram.py
sending "This is the message. It will be repeated."
waiting to receive
received "This is the message. It will be repeated."
closing socket
$

You need to change the destination address or port in your call to sendto:
peer_address = ('localhost', peer_port)
sock.sendto(get_line(currline), peer_address)
To which address and port does the Qt application bind?

socket connection receive bytes python

I'm trying to send a stream of data via socket in Python. So far I manage to create a dummy_data_gen.py which sends a line containing 4 floats to the server.py. However, I'm still having issues in the stability of the all setup.
server.py:
import sys
import time
import socket
import numpy as np
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Bind the socket to the port
server_address = ('localhost', 5002)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
# Listen for incoming connections
sock.listen(1)
# Create a list to store the incoming data
data = []
while True:
# Wait for a connection
print >>sys.stderr, 'waiting for a connection'
connection, client_address = sock.accept()
try:
print >>sys.stderr, 'connection from', client_address
while True:
incoming_data = connection.recv(48).split(',')
print incoming_data
event = float(incoming_data[0]), float(incoming_data[1]), float(incoming_data[2]), float(incoming_data[3])
data += [event]
time.sleep(0.01)
finally:
# Clean up the connection
connection.close()
dummy_data_gen.py
import sys
import time
import socket
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect the socket to the port where the server is listening
server_address = ('localhost', 5002)
sock.connect(server_address)
file = '../data/myfile.txt'
# Simulating a real-time data stream at 100 Hz
try:
with open(file) as f:
for line in f:
sock.sendall(line)
time.sleep(0.01)
finally:
print >>sys.stderr, 'closing socket'
sock.close()
My problem is that sometimes the communication is working properly, however, I have situations where I receive more data per line than I should. In the following output example the first 7 lines are correct, however the following lines are incorrect and therefore problematic:
['391910745379', '24.134277', '-1.9487305', '-117.373535', '\n']
['391920745379', '24.434082', '-1.3491211', '-117.373535', '\n']
['391930745379', '23.68457', '-0.5996094', '-116.62402', '\n']
['391940745379', '24.434082', '-1.0493164', '-115.57471', '\n']
['391950745379', '24.134277', '-1.0493164', '-116.47412', '\n']
['391960745379', '23.234863', '-1.0493164', '-116.47412', '\n']
['391970745379', '24.583984', '-0.89941406', '-116.92383', '\n']
['391980745379', '23.384766', '-0.2998047', '-116.62402', '\n39']
['1990745379', '23.68457', '-0.5996094', '-115.72461', '\n39200']
['0745379', '23.834473', '-0.44970703', '-115.87451', '\n392010']
['745379', '23.534668', '-1.0493164', '-114.9751', '\n392020745']
['379', '23.384766', '-1.7988281', '-115.72461', '\n39203074537']
['9', '22.935059', '-0.44970703', '-114.9751', '\n392040745379', '']
I tried to play with the connection.recv bytes but I'm still facing this issue.
EDIT1: Following some suggestions I modified the server.py as follows:
del_message = '\n'
del_stream = ','
while True:
_buffer += connection.recv(1)
if del_message in _buffer:
incoming_data = _buffer.split(del_stream)
event = float(incoming_data[0]), \
float(incoming_data[1]), \
float(incoming_data[2]), \
float(incoming_data[3])
This approach seems to solve my issue, however the performance is extremely slow. My files contains approximately 6300 lines that were actually sent in 70 seconds (time interval at which the socket was closed on my dummy data generator). However, I took almost 10 minutes to receive all of the 6300 lines. It seems also that I receive more samples per second on the beginning rather that on the end of the stream.

If you have a message protocol that terminates messages with a newline, you need to write some code to implement that protocol. It won't work by magic.
You need a "receive a message" function, where "message" is defined as "a sequence of bytes delimited by a newline". You've never written any such function, so you're not receiving messages but just the chunks of bytes you're sending.