We Keep Coding

Cannot read or decode UDP packet in Python

I'm trying to read UDP Broadcast messages sent by an device in my network on port 2009. I can see Python receives data from the device, but it's in Bytes. I have tried to decode it with .decode() or .hex(), also with ignore errors, but it's still not "readable".
All I get from the machine is:
b' \x08\x19\x96VDLG\x03\xac\x11P\x00\x00\x00\x00\x01#0\x00\x0bZ'
The code is (very basic):
import socket
import codecs
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
client.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
client.bind(("", 2009))
while True:
data, addr = client.recvfrom(8192)
print(data)
Maybe someone with a little bit more experience reading broadcast can shed a light for me, because all I try fails.
It should read something like this:
Thanks!

loading image though socket in bytestring takes forever in python

I am wondering how i can send an image through a python socket.
I have already tried it with pickle and sending the byte string piece by piece. However that takes forever.
Here is what I have tried.
my server code:
import socket
from PIL import Image
import pickle
host=""
port=80
server=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", port))
server.listen()
path=("path")
image=Image.open(path)
def acc():
while True:
conn, addr=server.accept()
print("connected to %s" %(conn))
conn.send(pickle.dumps(image))
acc()
my client code:
import socket
import pickle
host="192.168.1.11"
port=80
c=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.connect((host, port))
while True:
while True:
data=[]
packet=c.recv(100000)
if not packet: break
data.append(packet)
data_arr=b"".join(data)
print(pickle.loads(data_arr))
If the answer is not with pickle or PIL it is fine. I just need a way how this works. I'm looking forward for answers!

import socket, time
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("10.0.0.9", 2000))
server.listen()
def acc(image_Path):
with open(image_Path, "rb") as image:
data = image.read() # Read the bytes from the path
image.close()
while True:
conn, addr = server.accept()
print("connected to %s" %(conn))
conn.sendall(data) # Send the bytes
acc("The path of the image to transform")
import socket
import pickle
host = "10.0.0.9"
port = 2000
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.connect((host, port))
while True:
data = c.recv(100000000) # We don't know the size of the image, so 100000000 just in case
with open("The path of the location of the image that recived", "wb") as newImage:
newImage.write(data) # Write the bytes in new path to create the image
newImage.close()
print("Got the image")

The answer is neither PIL nor Pickle!
If you have a 31kB JPEG of a 1920x1080 image and you open it with PIL, it will get JPEG-decompressed and expanded out into a 1920x1080x3 RGB image in memory - which will require a minimum of 6,220,800 bytes, i.e. you have increased its size by 200 times. Moral of the story is to send the (compressed) JPEG itself.
If you then pickle that 6MB monster, it will a) take longer and b) get bigger and it is entirely unnecessary because you can send binary data down a socket anyway.
The easiest way is to:
simply read() the JPEG/PNG file using Python read() rather than PIL Image.read()
do not pickle it
send a 4-byte header in network order (see htonl()) with the image size before your image and, on the receiving end, read 4 bytes and unpack them, then read the correct number of bytes in the image
put the received bytes into a BytesIO structure and then use PIL Image.open(BYTESIO_THING)

Socket Fragmented Received Data

I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1

As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.

One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.

You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()

Is there any way I can send large amounts of data (around 10,000 bytes) on UDP relatively fast using sockets?

Is there any way to send large amounts of data (around 10,000 bytes) relatively fast? I'm currently using socket.recvfrom and it takes around a minute to recieve each message.
Edit: This is how I recover the message, sock is a UDP socket, and max_buffer starts at 1024
while True:
try:
data, addr = sock.recvfrom(max_buffer) # buffer
print(data)
break
except OSError:
max_buffer *= 2
data = json.loads(data)
Im not going to paste the code for getting the message because its pretty long, but its a json dictionary which in total takes up around 10000 bytes. This is how I send it:
sock.sendto(json.dumps(data).encode(), address)
I'm sending messages to the server repeatedly.

Is there any way to send large amounts of data (around 10,000 bytes) relatively fast?
Yes, use socket.sendto(). Here is an example client:
import socket
import datetime
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
data = bytes(i%256 for i in range(10000))
print(datetime.datetime.now())
s.sendto(data, ('localhost', 5000))
print(datetime.datetime.now())
Notice that it creates a bytes object of 10,000 bytes and passes it in a single call to s.sendto().
Using this server:
import socket
import datetime
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('localhost', 5000))
data, addr = s.recvfrom(100*1024)
print(datetime.datetime.now())
I got this result from the client:
2018-06-19 13:36:13.443418
2018-06-19 13:36:13.443636
I got this result from the server:
2018-06-19 13:36:13.443666
So to the sendto() call took about 2/10000 of a second and the data arrived less than 1/10000 of a second later.

Sending a file over a socket with Python only working after force-closing socket

I'm trying to send a file over a socket in Python 2.7.x . My client and server code is below. It's working, but only after the client connection kills the socket. So I added raw_input('finished') to the end of both for debugging.
So if I start the server, then run the client... It looks like all but the last bit of the file sends, until I forcefully kill the client and then it's all there. So the problem is definitely in the server loop... I just don't know how to fix it. if not data: break isn't being triggered. But, if I do something like if len(data) < 1024: break it won't work for bigger files.
Any help is appreciated!
# client.py
import socket
conn = socket.socket()
conn.connect(('localhost', 1337))
f = open('test.jpg', 'rb')
data = f.read(1024)
while data:
conn.send(data)
data = f.read(1024)
f.close()
raw_input('finished')
# server.py
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('localhost', 1337))
s.listen(5)
conn, addr = s.accept()
f = open('test.jpg', 'wb')
while True:
data = conn.recv(1024)
if not data:
break
f.write(data)
f.close()
raw_input('finished')

From your posted code:
while data:
conn.send(data)
data = f.read(1024)
From the Python socket documentation:
socket.send(string[, flags])
[...]
Returns the number of bytes sent. Applications are responsible for checking
that all data has been sent; if only some of the data was transmitted, the
application needs to attempt delivery of the remaining data.
That should tell you what the problem is, but just to be explicit about it: send() may or may not accept all of the bytes you asked it to send before returning, and it's up to you to handle it correctly in the case where it only accepts the first (n) bytes rather than the entire buffer. If you don't check send()'s return value, then you will sometimes drop some of the bytes of your file without knowing it. You need to check send()'s return value and if it is less than len(data), call send() again (as many times as necessary) with the remaining bytes. Alternatively you could call conn.sendall() instead of conn.send(), since sendall() will perform that logic for you.