I'm using a remote linux server and I want to send an array via sockets from client with python so I used this code :
message = pickle.dumps(faceBlob)
message_header = bytes(f"{len(message):<{HEADER_LENGTH}}", "utf-8")
client_socket.send(message_header + message)
to receive it in the server I used a while loop to catch all the message since it is > 4096 :
def receive_blop(client_socket):
# Receive our "header" containing message length, it's size is defined and constant
message_header = client_socket.recv(HEADER_LENGTH)
# If we received no data, client gracefully closed a connection, for example using socket.close() or socket.shutdown(socket.SHUT_RDWR)
if not len(message_header):
return False
# Convert header to int value
message_length = int(message_header.decode('utf-8').strip())
fragments = []
print(message_length)
while True:
# this loop is infinite
print("I arrived her")
chunk = client_socket.recv(4096)
if not chunk:
break
fragments.append(chunk)
data_arr = b"".join(fragments)
# Return an object of message header and message data
return {'header': message_header, 'data': data_arr}
the server still printing the 'I arrived here' but receive the message until the connection is ended from the client
Your loop will continue until the client closes the connection. If they don't close until they get a response from you, you've got a deadlock.
Since you know the message length, you can stop the loop when you've received that many bytes.
received_length = 0
while received_len < message_length:
print("I arrived her")
chunk = client_socket.recv(message_length - received_length)
if not chunk:
break
fragments.append(chunk)
received_length += len(chunk)
Related
I'm trying to send a large amount of JSON (3000 lines or so) to a server from a client.
Sometimes, the server recieves no bytes at all, or it gets some of the bytes and not all.
It works find for small strings, less than a thousand lines of JSON.
Client
def rpiSendTest(request):
host = request.data["IP"]
port = 5098
timeout = 5
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
context.load_verify_locations(cafile='certs/cert.pem')
ssock = context.wrap_socket(sock, server_hostname=host)
try:
ssock.connect((host, port))
message = json.dumps(request.data)
success = ssock.sendall(message.encode())
return response.Response(status=status.HTTP_200_OK, data={"status": "testing"})
except Exception as e:
return response.Response(status=status.HTTP_400_BAD_REQUEST)
Server
while True:
# Once you grab a connection, start recieving data
try:
conn.settimeout(20)
data = conn.recv(90000)
if not data:
print("No more data")
# If there's no more data chunks
break
# If the test suite is big, it'll be sent in chunks of bytes, so we take 10000 sized chunks at a time and append to array
print("appending")
print(data)
fragments.append(data)
except Exception as e:
print(e)
data = b''.join(fragments)
# The array is in bytes, so we append to a big byte string
print("loading {}".format(data))
data = json.loads(data)
It would error on json.loads(data) as not all the bytes were sent. Or it wouldn't recieve any data at all.
I'm on a company network, would there be any reasons why this could be causing issues.
I am trying to implement UDP stop-and-wait protocol using python socket programming. I have been trying out my code with different retransmission times. And sometimes I get all the files correctly received at the receiver, but sometimes some packets get lost. For example when I ran it with 40ms five times, twice the file was received correctly and three times incorrectly. Why is this variability happening?
Here is my code for the sender and receiver:
senderSocket = socket(AF_INET, SOCK_DGRAM) # Create UDP socket for sender
r = 0 # Number of retransmissions
for i in range(0, len(messages)):
senderSocket.settimeout(retryTimeout) # After the message is sent, set retransmission timeout to listen for acknowledgement
while True:
senderSocket.sendto(messages[i], (hostName, portNumber))
acknowledged = False
while not acknowledged:
try:
ack, receiverAddress = senderSocket.recvfrom(2) # Receive ACK
acknowledged = True
except: # Socket timeout exception occurs when timeout expires but no ACK received
senderSocket.sendto(messages[i], (hostName, portNumber)) # Retransmit the message
r = r + 1 # Increment the number of retransmissions
break # On to the next message
senderSocket.close()
while True:
message, senderAddress = receiverSocket.recvfrom(1027) # Read from UDP socket into message, getting sender's address (sender IP and port)
header = message[:3] # Header is the first 3 bytes (index 0, 1, 2)
data = message[3:] # Rest is the data
first_byte = '{0:08b}'.format(header[0])
second_byte = '{0:08b}'.format(header[1])
seq_num = int(first_byte + second_byte, 2) # Convert bytes to decimal
if seq_num not in seq_nums: # Detect duplicates
seq_nums.append(seq_num)
file_content.extend(data)
ack = header[:2] # ACK is the receipt of the received message (sequence number)
receiverSocket.sendto(ack, senderAddress) # Send ACK
if header[2] == 1: # Sent multiple ACKs at lat message to make sure it receives and the sender closes
receiverSocket.sendto(ack, senderAddress)
receiverSocket.sendto(ack, senderAddress)
receiverSocket.sendto(ack, senderAddress)
break
receiverSocket.close()
I'm working on a program that receives a string from an Android app sent through WiFi, the program was originally written for Python 2.7, but after adding some additional functionalities I changed it to Python 3.7. However, after making that change, my data had an extra letter at the front and for the life of me I can't figure out why that is.
Here's a snippet of my code, it's a really simple if statement to see which command was sent from the Android app and controls Raspberry Pi (4) cam (v.2) with the command.
This part sets up the connections and wait to see which command I send.
isoCmd = ['auto','100','200','300','400','500','640','800']
HOST = ''
PORT = 21567
BUFSIZE = 1024
ADDR = (HOST,PORT)
brightness = 50
timelapse = 0
tcpSerSock = socket(AF_INET, SOCK_STREAM)
tcpSerSock.bind(ADDR)
tcpSerSock.listen(5)
while True:
print ('Waiting for connection')
tcpCliSock,addr = tcpSerSock.accept()
try:
while True:
data = ''
brightness = ' '
data = tcpCliSock.recv(BUFSIZE)
dataStr = str(data[1:])
print ("Here's data ",dataStr)
if not data:
break
if data in isoCmd:
if data == "auto":
camera.iso = 0
print ('ISO: Auto')
else:
camera.iso = int(data)
print ('ISO: '), data
When I start the program this is what I see:
Waiting for connection
#If I send command '300'
Here's data b'300'
Here's data b''
Waiting for connection
I'm not sure why there's the extra b'' is coming from. I have tested the code by just adding the "b" at the beginning of each items in the array which worked for any commands that I defined, not for any commands to control the Pi camera since well, there's no extra b at the beginning. (Did that make sense?) My point is, I know I'm able to send commands no problem, just not sure how to get rid of the extra letter. If anyone could give me some advice that would be great. Thanks for helping.
Byte strings are represented by the b-prefix.
Although you can see the string in output on printing, inherently they are bytes.
To get a normal string out of it, decode function can help.
dataStr.decode("utf-8")
b'data' simply means the data inside quotes has been received in bytes form, as mentioned in other answers also, you have to decode that with decode('utf-8') to get it in string form.
I have updated your program below, to be compatible for v3.7+
from socket import *
isoCmd = ['auto','100','200','300','400','500','640','800']
HOST = ''
PORT = 21567
BUFSIZE = 1024
ADDR = (HOST,PORT)
brightness = 50
timelapse = 0
tcpSerSock = socket(AF_INET, SOCK_STREAM)
tcpSerSock.bind(ADDR)
tcpSerSock.listen(5)
while True:
print ('Waiting for connection')
tcpCliSock,addr = tcpSerSock.accept()
try:
while True:
data = ''
brightness = ' '
data = tcpCliSock.recv(BUFSIZE).decode('utf-8')
print ("Here's data "+data)
if not data:
break
if data in isoCmd:
if data == "auto":
camera.iso = 0
print ('ISO: Auto')
else:
camera.iso = int(data)
print ('ISO: '+ data)
except Exception as e:
print(e)
The problem is: print() in client() only output one line, while I expect it to print several lines because the while loop it resides obviously run more than one time.
The problem occurs when I am testing the sample code given in book Foundations of Python Network Programming, 3rd Edition. This sample code basically create a simple TCP server/client that process/send simple text capitalisation request. Following is the code:
#!/usr/bin/env python3
# Foundations of Python Network Programming, Third Edition
# https://github.com/brandon-rhodes/fopnp/blob/m/py3/chapter03/tcp_deadlock.py
# TCP client and server that leave too much data waiting
import argparse, socket, sys
def server(host, port, bytecount):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
sock.listen(1)
print('Listening at', sock.getsockname())
while True:
sc, sockname = sock.accept()
print('Processing up to 1024 bytes at a time from', sockname)
n = 0
while True:
data = sc.recv(1024)
if not data:
break
output = data.decode('ascii').upper().encode('ascii')
sc.sendall(output) # send it back uppercase
n += len(data)
print('\r %d bytes processed so far' % (n,), end=' ')
sys.stdout.flush()
print()
sc.close()
print(' Socket closed')
def client(host, port, bytecount):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
bytecount = (bytecount + 15) // 15 * 16 # round up to a multiple of 16
message = b'CAP!' # 16-byte message to repeat over and over
print('Sending', bytecount, 'bytes of data, in chunks of 16 bytes')
sock.connect((host, port))
sent = 0
while sent < bytecount:
sock.sendall(message)
sent += len(message)
# print('test')
print('\r %d bytes sent' %(sent,), end=' ') # <-- THIS IS THE PROBLEMATIC PRINT()
sys.stdout.flush()
print()
sock.shutdown(socket.SHUT_WR)
print('Receiving all the data the server sends back')
received = 0
while True:
data = sock.recv(42)
if not received:
print(' The first data received says', repr(data))
if not data:
break
received += len(data)
print('\r %d bytes received' %(received,), end=' ')
print()
sock.close()
if __name__ == '__main__':
choices = {'client': client, 'server': server}
parser = argparse.ArgumentParser(description='Get deadlocked over TCP')
parser.add_argument('role', choices=choices, help='which role to play')
parser.add_argument('host', help='interface the server listens at;'
' host the client sends to')
parser.add_argument('bytecount', type=int, nargs='?', default=16,
help='number of bytes for client to send (default 16)')
parser.add_argument('-p', metavar='PORT', type=int, default=1060,
help='TCP port (default 1060)')
args = parser.parse_args()
function = choices[args.role]
function(args.host, args.p, args.bytecount)
After I have simply created server locally, I started my client with
python listing3-2.py client 0.0.0.0
This is the output I get:
Sending 32 bytes of data, in chunks of 16 bytes
32 bytes sent
Receiving all the data the server sends back The first data received
says b'CAP!CAP!CAP!CAP!CAP!CAP!CAP!CAP!' 32 bytes received
From this output we know that the while() loop has run 8 times(because of 8 'CAP!'s) but print('\r %d bytes sent' %(sent,), end=' ') has run only once. What is more strange to me is that sys.stdout.flush() doesn't work, despite the effort of the author who might have also noticed the problem.
Something even more weird happens when I try to add one line of print('test') just before the problematic print(), see what happens?
python listing3-2.py client 0.0.0.0
Sending 32 bytes of data, in chunks of 16 bytes test
4 bytes sent test
8 bytes sent test
12 bytes sent test
16 bytes sent test
20 bytes sent test
24 bytes sent test
28 bytes sent test
32 bytes sent
Receiving all the data the server sends back The first data received says
b'CAP!CAP!CAP!CAP!CAP!CAP!CAP!CAP!' 32 bytes received
With an additional print() just before, print('\r %d bytes sent' %(sent,), end=' ') run 8 times, though the structure of loop is absolutely unchanged. I am totally confused by the fact that adding a print() could bring such consequence. And I am also confused by the thought that print() method may have some strange characteristics if used in this situation.
It is the \r combined with end=' ' in the print message. That is a carriage return without linefeed, so all the lines print over each other. The extra print adds a linefeed and they stop printing over each other. Change it to \n to fix, or more simply:
print(sent,'bytes sent')
P.S. There is also a math error:
bytecount = (bytecount + 15) // 15 * 16
should be:
bytecount = (bytecount + 15) // 16 * 16
I'm writing a code used to send order to an avr. I send several information but between each write, I have to wait for an answer (I have to wait for the robot to reach a point on the coordinate system). As I read in the documentation, readline() should at least read until the timeout but as soon as I send the first coordinate, the readline() automatically return :
SerialException: device reports readiness to read but returned no data (device disconnected?)
When I put a sleep() between each write() in the for loop, everything works fine. I tried to use inWaiting() but it still does not work. Here is an example of how I used it:
for i in chemin_python:
self.serieInstance.ecrire("goto\n" + str(float(i.x)) + '\n' + str(float(-i.y)) + '\n')
while self.serieInstance.inWaiting():
pass
lu = self.serieInstance.readline()
lu = lu.split("\r\n")[0]
reponse = self.serieInstance.file_attente.get(lu)
if reponse != "FIN_GOTO":
log.logger.debug("Erreur asservissement (goto) : " + reponse)
Here an snipet how to use serial in python
s.write(command);
st = ''
initTime = time.time()
while True:
st += s.readline()
if timeout and (time.time() - initTime > t) : return TIMEOUT
if st != ERROR: return OK
else: return ERROR
This method allows you to separately control the timeout for gathering all the data for each line, and a different timeout for waiting on additional lines.
def serial_com(self, cmd):
'''Serial communications: send a command; get a response'''
# open serial port
try:
serial_port = serial.Serial(com_port, baudrate=115200, timeout=1)
except serial.SerialException as e:
print("could not open serial port '{}': {}".format(com_port, e))
# write to serial port
cmd += '\r'
serial_port.write(cmd.encode('utf-8'))
# read response from serial port
lines = []
while True:
line = serial_port.readline()
lines.append(line.decode('utf-8').rstrip())
# wait for new data after each line
timeout = time.time() + 0.1
while not serial_port.inWaiting() and timeout > time.time():
pass
if not serial_port.inWaiting():
break
#close the serial port
serial_port.close()
return lines