We Keep Coding

How to send big image (2MB) multiple files from server to client with UDP python socket

I want to transfer multiple images over UDP. I know it is possible using TCP, but I want it over UDP. Below is the code fragment I used. Is it possible to transfer bigger files over UDP? I manage to transfer small file sizes, but failed for large files. I appreciate any help or alternative way to do using UDP socket.
Client code receiving an image.
BUF_SIZE = 1024
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('localhost', port))
sock.settimeout(30)
def rcv_data(sock, num_packet_to_recv, file_size):
bytes_rcvd = bytearray()
f = open(download_dir , 'wb')
print('Receiving packets will start')
while num_packet_to_recv > 0:
try:
client_data, server_addr = sock.recvfrom(BUF_SIZE + 8)
seq_num = client_b_data[-8:]
img_data = client_data[:-8]
num_packet_to_recv = num_packet_to_recv - 1
#store img_data and seq_num to bytes_rcvd for later
#sorting by sequence number
f.write(sorted_bytes_rcvd)
except Exception as e:
print('rcv error {}'.format(e))
f.close()
Server side sending data
def send_img(host, port, file_name, num_pkt):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setblocking(0)
sock.settimeout(60)
client_addr = (host, port)
# send the file name
while num_pkt > 0:
img_part = requested_file.read(BUF_SIZE + 8)
sock.sendto(img_part + seq_num)
num_pkt -= 1

Your posted code seems to assume that no packets will be dropped en route from the sender to the receiver -- that assumption doesn't hold up in real life (not even when sender and receiver are both located on the same machine!), which is the most likely reason your transfers don't work except on very small files (where you can sort-of-rely on luck to ensure that all the packets make it through on the first try).
To implement a more robust mechanism, your receiver program will need some way to (a) detect when a packet has been dropped, and (b) react to that knowledge by sending a message back to the sender requesting the sender to retransmit the data from the "lost" packet. (And of course the retransmission-request-packet can also get dropped, so you'll need a way to handle that as well!)
Your sender program, meanwhile, will need to not only send the data (as it currently does) but also receive any incoming retransmit-request-packets from the receiver, and react to them by retransmitting the requested data.
Depending on the exact design of your protocol, both sender and receiver may also need to take action after a certain amount of time has passed with no data sent or received, to avoid having the file-transfer process stall out if all the packets whose reception would have advanced it get dropped.
Therefore your current approach of just calling blocking sendto() or recvfrom() in a loop won't be sufficient; instead both sender and receiver will need to implement some kind of state-machine that allows them to both send the appropriate packet(s) at the appropriate time and quickly receive and handle any packets that come in. This would typically be done either with separate sender and receiver threads, or alternatively by setting the socket to non-blocking mode and writing an event-loop around a blocking call to select() or poll(). (I prefer the latter, because IMO while state machines are tricky to get right, multithreading is even trickier)
Placing a sequence number into each packet is a good start; that allows the receiver to know how to order the data, and allows it to detect when there is a "hole" in the data it has received. Once it has detected a hole (i.e. one or more missing sequence numbers) it can send a packet back to the sender asking for those packets to be re-sent, and it will be up to the sender to react by re-sending those packets. Repeat as necessary until the receiver has received a packet with every possible sequence number (you'll also need to communicate to the receiver somehow how many sequence numbers to expect).

pySerial Capturing a long response

Hi guys I'm working a on script that will get data from a host using the Data Communications Standard (Developed by: Data Communication Standard Committee Lens Processing Division of The Vision Council), by serial port and pass the data into ModBus Protocol for the device to perform it's operations.
Since I don't fiscally have access to the host machine I'm trying to develop a secondary script to emulate the host. I am currently on the stage where I need to read a lot of information from the serial port and I get only part of the data. I was hoping to get the whole string sent on the send_job() function on my host emulator script.
Guys also can any of you tell me if this would be a good approach? the only thing the machine is supposed to do is grab 2 values from the host response and assign them to two modbus holding registers.
NOTE: the initialization function is hard coded because it will always be the same and the actual response data will not matter except for status. Also the job request is hard coded i only pass the job # that i get from a modbus holding register, the exact logic on how the host resolved this should not matter i only need to send the job number scanned from the device in this format.
main script:
def request_job_modbus(job):
data = F'[06][1c]req=33[0d][0a]job={job}[0d][0a][1e][1d]'.encode('ascii')
writer(data)
def get_job_from_serial():
response = serial_client.read_all()
resp = response.decode()
return resp
# TODO : SEND INIT SEQUENCE ONCE AND VERIFY IF REQUEST status=0
initiation_request()
init_response_status = get_init_status()
print('init method being active')
print(get_init_status())
while True:
# TODO: get job request data
job_serial = get_job_from_serial()
print(job_serial)
host emulation script:
def send_job():
job_response = '''[06][1c]ans=33[0d]job=30925[0d]status=0;"ok"[0d]do=l[0d]add=;2.50[0d]ar=1[0d]
bcerin=;3.93[0d]bcerup=;-2.97[0d]crib=;64.00[0d]do=l[0d]ellh=;64.00[0d]engmask=;613l[0d]
erdrin=;0.00[0d]erdrup=;10.00[0d]ernrin=;2.00[0d]ernrup=;-8.00[0d]ersgin=;0.00[0d]
ersgup=;4.00[0d]gax=;0.00[0d]gbasex=;-5.30[0d]gcrosx=;-7.96[0d]kprva=;275[0d]kprvm=;0.55[0d]
ldpath=\\uscqx-tcpmain-at\lds\iot\do\800468.sdf[0d]lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]'''.encode('ascii')
writer(job_response)
def get_init_request():
req = p.readline()
print(req)
request = req.decode()[4:11]
# print(request)
if request == 'req=ini':
print('request == req=ini??? <<<<<<< cumple condicion y enviala respuesta')
send_init_response()
send_job()
while True:
# print(get_init_request())
get_init_request()
what I get in screen: main script
init method being active
bce
erd
condition was met init status=0
outside loop
ers
condition was met init status=0
inside while loop
trigger reset <<<--------------------
5782
`:lmatid=;151[0d]lmatname=;f50[0d]
lnam=;vsp_basic_fh15[0d]sgerin=;0.00[0d]sgerup=;0.00[0d]sval=;5.18[0d]text_11=;[0d]
text_12=;[0d]tind=;1.53[0d][1e][1d]
outside loop
condition was met init status=0
outside loop
what I get in screen: host emulation script
b'[1c]req=ini[0d][0a][1e][1d]'
request == req=ini??? <<<<<<< cumple condicion y enviala respuesta
b''
b'[06][1c]req=33[0d][0a]job=5782[0d][0a][1e][1d]'
b''
b''
b''
b''
b''
b''

I'm suspect you're trying to write too much at once to a hardware buffer that is fairly small. Especially when dealing with low power hardware, assuming you can stuff an entire message into a buffer is not often correct. Even full modern PC's sometimes have very small buffers for legacy hardware like serial ports. You may find when you switch from development to actual hardware, that the RTS and DTR lines need to be used to determine when to send or receive data. This will be up to whoever designed the hardware unfortunately, as they are often also ignored.
I would try chunking your data transfer into smaller bits as a test to see if the whole message gets through. This is a quick and dirty first attempt that may have bugs, but it should get you down the right path:
def get_job_from_serial():
response = b'' #buffer for response
while True:
try:
response += serial_client.read() #read any available data or wait for timeout
#this technically could only be reading 1 char at a time, but any
#remotely modern pc should easily keep up with 9600 baud
except serial.SerialTimeoutException: #timeout probably means end of data
#you could also presumably check the length of the buffer if it's always
#a fixed length to determine if the entire message has been sent yet.
break
return response
def writer(command):
written = 0 #how many bytes have we actually written
chunksize = 128 #the smaller you go, the less likely to overflow
# a buffer, but the slower you go.
while written < len(command):
#you presumably might have to wait for p.dtr() == True or similar
#though it's just as likely to not have been implemented.
written += p.write(command[written:written+chunksize])
p.flush() #probably don't actually need this
P.S. I had to go to the source code for p.read_all (for some reason I couldn't find it online), and it does not do what I think you expect it does. The exact code for it is:
def read_all(self):
"""\
Read all bytes currently available in the buffer of the OS.
"""
return self.read(self.in_waiting)
There is no concept of waiting for a complete message, it just a shorthand for grab everything currently available.

python socket programming for transferring a photo

I'm new to socket programming in python. Here is an example of opening a TCP socket in a Mininet host and sending a photo from one host to another. In fact I changed the code that I had used to send a simple message to another host (writing the received data to a text file) in order to meet my requirements. Although when I implement this revised code, there is no error and it seems to transfer correctly, I am not sure whether this is a correct way to do this transmission or not. Since I'm running both hosts on the same machine, I thought it may have an influence on the result. I wanted to ask you to check whether this is a correct way to transfer or I should add or remove something.
mininetSocketTest.py
#!/usr/bin/python
from mininet.topo import Topo, SingleSwitchTopo
from mininet.net import Mininet
from mininet.log import lg, info
from mininet.cli import CLI
def main():
lg.setLogLevel('info')
net = Mininet(SingleSwitchTopo(k=2))
net.start()
h1 = net.get('h1')
p1 = h1.popen('python myClient2.py')
h2 = net.get('h2')
h2.cmd('python myServer2.py')
CLI( net )
#p1.terminate()
net.stop()
if __name__ == '__main__':
main()
myServer2.py
import socket
import sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('10.0.0.1', 12345))
buf = 1024
f = open("2.jpg",'wb')
s.listen(1)
conn , addr = s.accept()
while 1:
data = conn.recv(buf)
print(data[:10])
#print "PACKAGE RECEIVED..."
f.write(data)
if not data: break
#conn.send(data)
conn.close()
s.close()
myClient2.py:
import socket
import sys
f=open ("1.jpg", "rb")
print sys.getsizeof(f)
buf = 1024
data = f.read(buf)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('10.0.0.1',12345))
while (data):
if(s.sendall(data)):
#print "sending ..."
data = f.read(buf)
print(f.tell(), data[:10])
else:
s.close()
s.close()

This loop in client2 is wrong:
while (data):
if(s.send(data)):
print "sending ..."
data = f.read(buf)
As the send
docs say:
Returns the number of bytes sent. Applications are responsible for checking that all data has been sent; if only some of the data was transmitted, the application needs to attempt delivery of the remaining data. For further information on this topic, consult the Socket Programming HOWTO.
You're not even attempting to do this. So, while it probably works on localhost, on a lightly-loaded machine, with smallish files, it's going to break as soon as you try to use it for real.
As the help says, you need to do something to deliver the rest of the buffer. Since there's probably no good reason you can't just block until it's all sent, the simplest thing to do is to call sendall:
Unlike send(), this method continues to send data from bytes until either all data has been sent or an error occurs. None is returned on success. On error, an exception is raised…
And this brings up the next problem: You're not doing any exception handling anywhere. Maybe that's OK, but usually it isn't. For example, if one of your sockets goes down, but the other one is still up, do you want to abort the whole program and hard-drop your connection, or do you maybe want to finish sending whatever you have first?
You should at least probably use a with clause of a finally, to make sure you close your sockets cleanly, so the other side will get a nice EOF instead of an exception.
Also, your server code just serves a single client and then quits. Is that actually what you wanted? Usually, even if you don't need concurrent clients, you at least want to loop around accepting and servicing them one by one.
Finally, a server almost always wants to do this:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Without this, if you try to run the server again within a few seconds after it finished (a platform-specific number of seconds, which may even depend whether it finished with an exception instead of a clean shutdown), the bind will fail, in the same way as if you tried to bind a socket that's actually in use by another program.

First of all, you should use TCP and not UDP. TCP will ensure that your client/server has received the whole photo properly. UDP is more used for content streaming.
Absolutely not your use case.

Python Sockets, requesting file from server then waiting to receive it

I am attempting to send a string to my server from my client with a specific filename and then send that file to the client. For some reason it hangs even after it's received all of the file. It hangs on the:
m = s.recv(1024)
client.py
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.1.2", 54321))
s.send(b"File:test.txt")
f = open("newfile.txt", "wb")
data = None
while True:
m = s.recv(1024)
data = m
if m:
while m:
m = s.recv(1024)
data += m
else:
break
f.write(data)
f.close()
print("Done receiving")
server.py
import socket
import os
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("", 54321))
while True:
client_input = c.recv(1024)
command = client_input.split(":")[0]
if command == "File":
command_parameter = client_input.split(":")[1]
f = open(command_parameter, "rb")
l = os.path.getsize(command_parameter)
m = f.read(l)
c.sendall(m)
f.close()

TLDR
The reason recv blocks is because the socket connection is not shutdown after the file data was sent. The implementation currently has no way to know when the communication is over, which results in a deadlock between the two, remote processes. To avoid this, close the socket connection in the server, which will generate an end-of-file event in the client (i.e. recv returns a zero-length string).
More insight
Whenever you design any software where two processes communicate with each other, you have to define a protocol that disambiguates the communication such that both peers know exactly which state they are in at all times. Typically this involves using the syntax of the communication to help guide the interpretation of the data.
Currently, there are some problems with your implementation: it doesn't define an adequate protocol to resolve potential ambiguity. This becomes apparent when you consider the fact that each call to send in one peer doesn't necessarily correspond to exactly one call to recv in the other. That is, the calls to send and recv are not necessarily one-to-one. Consider sending the file name to the server on a heavily congested network: perhaps only half of the file name makes it to the server when the first call to recv returns. The server has no way (currently) to know if it has finished receiving the file name. The same is true in the client: how does the client know when the file has finished?
To work around this, we can introduce some syntax into the protocol and some logic into the server to ensure we get the complete file name before continuing. A simple solution would be to use an EOL character, i.e. \n to denote the end of the client's message. Now, 99.99% of the time in your testing this will take a single call to recv to read in. However you have to anticipate the cases in which it might take more than one call to recv. This can be implemented using a loop, obviously.
The client end is simpler for this demo. If the communication is over after the sending of the file, then that event can be used to denote the end of the data stream. This happens when the server closes the connection on its end.
If we were to expand the implementation to, say, allow for requests for multiple, back-to-back files, then we'd have to introduce some mechanism in the protocol for distinguishing the end of one file and the beginning of the next. Note that this also means the server would need to potentially buffer extra bytes that it reads in on previous iterations in case there is overlap. A stream implementation is generally useful for these sorts of things.

python socketserver occasionally stops sending (and receiving?) messages

I've been experiencing a problem with a socketserver I wrote where the socketserver will seem to stop sending and receiving data on one of the ports it uses (while the other port continues to handle data just fine). Interestingly, after waiting a minute (or up to an hour or so), the socketserver will start sending and receiving messages again without any observable intervention.
I am using the Eventlet socketing framework, python 2.7, everything running on an ubuntu aws instance with external apps opening persistent connections to the socketserver.
From some reading I've been doing, it looks like I may not be implementing my socket server correctly.
According to http://docs.python.org/howto/sockets.html:
fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited > > (shrug), or indicate how long they are (much better), or end by shutting down the connection.
I am not entirely sure that I am using a fix length message here (or am I?)
This is how I am receiving my data:
def socket_handler(sock, socket_type):
logg(1,"socket_handler:initializing")
while True:
recv = sock.recv(1024)
if not recv:
logg(1,"didn't recieve anything")
break
if len(recv) > 5:
logg(1,"socket handler: %s" % recv )
plug_id, phone_sid, recv_json = parse_json(recv)
send = 1
if "success" in recv_json and recv_json["success"] == "true" and socket_type == "plug":
send = 0
if send == 1:
send_wrapper(sock, message_relayer(recv, socket_type))
else:
logg(2, 'socket_handler:Ignoring received input: ' + str(recv) )
logg(1, 'Closing socket handle: [%s]' % str(sock))
sock.shutdown(socket.SHUT_RDWR)
sock.close()
"sock" is a socket object returned by the listener.accept() function.
The socket_handler function is called like so:
new_connection, address = listener.accept()
...<code omitted>...
pool.spawn_n(socket_handler, new_connection, socket_type)
Does my implementation look incorrect to anyone? Am I basically implementing a fixed length conversation protocol? What can I do to help investigate the issue or make my code more robust?
Thanks in advance,
T

You might be having buffering related problems if you're requesting to receive more bytes at the server (1024) than you're actually sending from the client.
To fix the problem, what's is usually done is encode the length of the message first and then the message itself. This way, the receiver can get the length field (which is of known size) and then read the rest of the message based on the decoded length.
Note: The length field is usually as many bytes long as you need in your protocol. Some protocols are 4-byte aligned and use a 32 bit field for this, but if you find that you've got enough with 1 or 2 bytes, then you can use that. The point here is that both client and server know the size of this field.