We Keep Coding

Socket Fragmented Received Data

I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1

As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.

One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.

You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()

Can socket receive infinite data stream python?

While trying to attempt to go for python based socket, I have 2 questions which I am not able to resolve. Kindly help me. Here is my sample socket code:
import socket
import threading
import chardet
bind_ip = '0.0.0.0'
bind_port = 9999
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(1) # max backlog of connections
print (('Listening on {}:{}').format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(4096 )
result = chardet.detect(request)
print(result)
print (request.decode(result['encoding']))
client_socket.send('ACK!'.encode(result['encoding']))
client_socket.close()
while True:
client_sock, address = server.accept()
print (('Accepted connection from {}:{}').format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
The above one is server and the below is client:
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 9999))
client.send(str('test data').encode("utf-16"))
response = client.recv(4096)
print(response.decode("utf-16"))
Now the questions:
1) What is the meaning of the number in this statement: client.recv(4096)? What is 4096, is it bit or byte of kilobyte for data receiving?
2) Can I receive infinite data stream through the socket? Meaning, as this statement client.recv(4096), whatever is the menaing of 4096, may be byte, then the socket will receive the 4096 bytes of data only. I do not have a control on the size of data received through the socket, hence, can I generalize it to accept any size of data through socket?
Please help me get the answers to above queries. I tried the python documentation for socket but didn't found much. I guess I miss something. Please help me get through it.

According to documentation
"If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from."
found here
"https://linux.die.net/man/2/recv"
which was from python 3.6 docs
"he maximum amount of data to be received at once is specified by bufsize. See the Unix manual page recv(2)"
found here
"https://docs.python.org/3.6/library/socket.html"
so it is a bytes object and it may actually truncate the message depending on the protocol. So message integrity is handled by the next layer up. So presumably you would loose part of a UDP packet, and you would get either a retry or an additional packet for TCP if the message was too large to fit in the buffer.

Using python sockets to connect to server, receive data and resend it to server

I have this sort of a catch-the-flag assignment where I need to connect to a server and a port, receive data, extract the number that was given in this data and resend it. I'm guessing I need to continue doing that until some flag will arrive.
I've tried doing that with python sockets (which I barely understand), and came up with this code:
import socket
import re
#AF_INET for IPv4, SOCK_STREAM for TCP
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Tell the socket what IP and port number to connect to
clientsocket.connect(('35.157.111.68', 10172))
while 1:
# Recieve 1024 bytes of data.
data = clientsocket.recv(1024)
if not data: break
# Get only numbers from string
data = re.sub('\D',"", data)
# Send our result to the server.
clientsocket.send(str(data))
It's establishing a connection and receiving the data, but when it sends the number back it doesn't accept it properly and it does it for only one round (doesn't loop).

TypeError: str, bytes or bytearray expected, not int [duplicate]

I'm trying to create a simple server to client based chat program and the issue is that when I try to execute c.sendto(data,client) this error appears saying that Client is an int but it's a tuple containing the port number and the address. I'm I supposed to convert the tuple to bytes so I can send the message to the clients?
Server Script
import socket
clients = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1",7999))
s.listen()
print("Waiting for connection")
c, addr = s.accept()
while True:
data , addr = c.recvfrom(1024)
print(data)
if addr not in clients:
clients.append(addr)
print(clients[0])
if data:
for client in clients:
print(client)
c.sendto(data,client)
s.close()
Client Script
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
addr = ("127.0.0.1",7999)
s.connect(addr)
while True:
string = input(">>")
s.send(string.encode("utf-8"))
data =s.recv(1024)
s.close()
Server Output

The problem is that you're using sendto() with a connection-mode socket. I think you want c.send(data) instead.
Details:
The Python docs for sendto say "The socket should not be connected to a remote socket, since the destination socket is specified by address." Also the man page for sendto says "If sendto() is used on a connection-mode (SOCK_STREAM, SOCK_SEQPACKET) socket, the arguments dest_addr and addrlen are ignored (and the error EISCONN may be returned when they are not NULL and 0)." I somewhat suspect that this is happening and Python is misreporting the error in a confusing way.
The sockets interface and networking in general can be pretty confusing but basically sendto() is reserved for SOCK_DGRAM which is UDP/IP type internet traffic, which you can think of as sending letters or postcards to a recipient. Each one goes out with a recipient address on it and there's no guarantee on order of receipt. On the other hand, connection-mode sockets like SOCK_STREAM use TCP/IP which is a bit more like a phone call in that you make a connection for a certain duration and and each thing you send is delivered in order at each end.
Since your code seems to be designed for communication over a connection I think you just want c.send(data) and not sendto.

You must first convert the string that contains the IP address into a byte or a string of bytes and then start communicating.
According to the code below, your error will be resolved.
Make sure your code is working correctly overall.
string = '192.168.1.102'
new_string = bytearray(string,"ascii")
ip_receiver = new_string
s.sendto(text.encode(), (ip_receiver, 5252))

Python: Socket.error Connection Refused in Unix [Errno 111]

I am trying to send UDP video packets using sockets in Python.
The Server IP address is :192.168.67.14
The Client IP address is 192.168.67.42
The Client and Server can ping each other. Below is the code used for establishing the socket:
Server Side:
import urllib, time, os, m3u8
from socket import *
# Socket initialization
s = socket(AF_INET, SOCK_DGRAM)
host = "192.168.67.42"
port = 5000
buf = 1024
addr = (host, port)
s.connect((host, port))
ts_filenames = []
while True:
playlist = "https://sevenwestmedia01-i.akamaihd.net/hls/live/224853/TEST1/master_lowl.m3u8"
m3u8_obj = m3u8.load(playlist)
ts_segments = m3u8_obj.__dict__.values()[6]
ts_segments_str = str(m3u8_obj.segments)
for line in ts_segments_str.splitlines():
if "https://" in line:
ts_id = line[-20:]
if ts_id not in ts_filenames:
print ts_id
ts_filenames.append(ts_id)
try:
ts_segment = urllib.URLopener()
ts_segment.retrieve(line, ts_id)
except:
pass
f = open(ts_id, "rb")
data = f.read(buf)
while (data):
if (s.sendto(data, addr)):
print "sending ..."
data = f.read(buf)
Client Side
import socket
s = socket.socket()
host = '192.168.67.14'
port = 5000
s.connect((host,port))
print s.recv(1024)
s.close
Exception I get:
Traceback (most recent call last): File "client.py", line 7, in
s.connect((host,port)) File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args) socket.error: [Errno 111] Connection refused
I spent some time looking into this discussion but I still not sure what to modify. Any suggestions please ?

You have multiple problems here. First, by using connect on the server end, you're telling the operating system that you will only be communicating with IP address "192.168.67.42" port 5000. That is probably not what you intended. (A server usually talks to whatever client wants to talk to it.)
Second, by not specifying SOCK_DGRAM in your client, you're getting the default socket type, which is SOCK_STREAM. That means your client is trying to connect to your server on TCP port 80 -- not UDP port 80 (the two namespaces are totally separate).
For a UDP "session", both sides need an IP address and a port number. If you do not bind a port specifically, the operating system will choose one for you quasi-randomly. In order to link up client and server, they must agree on at least one of those.
So a typical UDP server will bind to a well-known port (presumably you intended 5000 for that purpose). Then the client can connect to the server at that port. The code would look something like this (sans error handling):
Server side:
# Create socket
s = socket(AF_INET, SOCK_DGRAM)
# Bind to our well known port (leave address unspecified
# allowing us to receive on any local network address)
s.bind(('', 5000))
# Receive from client (so we know the client's address/port)
buffer, client_addr = s.recvfrom(1024)
# Now we can send to the client
s.sendto(some_buffer, client_addr)
The client is close to what you have, but you should send some data from the client to the server first so that the server knows your address:
s = socket(AF_INET, SOCK_DGRAM)
# Create connection to server (the OS will automatically
# bind a port for the client)
s.connect((host, port))
# Send dummy data to server so it knows our address/port
s.send(b'foo')
buffer = s.recv(1024)
Note that because you have used connect on the client side, you've permanently specified your peer's address and don't need to use recvfrom and sendto.

On the client side, this is wrong:
s = socket.socket()
for receiving UDP packets, you need to create a UDP socket, same as you did on the server side:
s = socket(AF_INET, SOCK_DGRAM)
Also, if you want your client to be able to receive UDP packets you will need to bind() it to port 5000 (connect() is neither necessary nor sufficient for that).