The problem is: print() in client() only output one line, while I expect it to print several lines because the while loop it resides obviously run more than one time.
The problem occurs when I am testing the sample code given in book Foundations of Python Network Programming, 3rd Edition. This sample code basically create a simple TCP server/client that process/send simple text capitalisation request. Following is the code:
#!/usr/bin/env python3
# Foundations of Python Network Programming, Third Edition
# https://github.com/brandon-rhodes/fopnp/blob/m/py3/chapter03/tcp_deadlock.py
# TCP client and server that leave too much data waiting
import argparse, socket, sys
def server(host, port, bytecount):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
sock.listen(1)
print('Listening at', sock.getsockname())
while True:
sc, sockname = sock.accept()
print('Processing up to 1024 bytes at a time from', sockname)
n = 0
while True:
data = sc.recv(1024)
if not data:
break
output = data.decode('ascii').upper().encode('ascii')
sc.sendall(output) # send it back uppercase
n += len(data)
print('\r %d bytes processed so far' % (n,), end=' ')
sys.stdout.flush()
print()
sc.close()
print(' Socket closed')
def client(host, port, bytecount):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
bytecount = (bytecount + 15) // 15 * 16 # round up to a multiple of 16
message = b'CAP!' # 16-byte message to repeat over and over
print('Sending', bytecount, 'bytes of data, in chunks of 16 bytes')
sock.connect((host, port))
sent = 0
while sent < bytecount:
sock.sendall(message)
sent += len(message)
# print('test')
print('\r %d bytes sent' %(sent,), end=' ') # <-- THIS IS THE PROBLEMATIC PRINT()
sys.stdout.flush()
print()
sock.shutdown(socket.SHUT_WR)
print('Receiving all the data the server sends back')
received = 0
while True:
data = sock.recv(42)
if not received:
print(' The first data received says', repr(data))
if not data:
break
received += len(data)
print('\r %d bytes received' %(received,), end=' ')
print()
sock.close()
if __name__ == '__main__':
choices = {'client': client, 'server': server}
parser = argparse.ArgumentParser(description='Get deadlocked over TCP')
parser.add_argument('role', choices=choices, help='which role to play')
parser.add_argument('host', help='interface the server listens at;'
' host the client sends to')
parser.add_argument('bytecount', type=int, nargs='?', default=16,
help='number of bytes for client to send (default 16)')
parser.add_argument('-p', metavar='PORT', type=int, default=1060,
help='TCP port (default 1060)')
args = parser.parse_args()
function = choices[args.role]
function(args.host, args.p, args.bytecount)
After I have simply created server locally, I started my client with
python listing3-2.py client 0.0.0.0
This is the output I get:
Sending 32 bytes of data, in chunks of 16 bytes
32 bytes sent
Receiving all the data the server sends back The first data received
says b'CAP!CAP!CAP!CAP!CAP!CAP!CAP!CAP!' 32 bytes received
From this output we know that the while() loop has run 8 times(because of 8 'CAP!'s) but print('\r %d bytes sent' %(sent,), end=' ') has run only once. What is more strange to me is that sys.stdout.flush() doesn't work, despite the effort of the author who might have also noticed the problem.
Something even more weird happens when I try to add one line of print('test') just before the problematic print(), see what happens?
python listing3-2.py client 0.0.0.0
Sending 32 bytes of data, in chunks of 16 bytes test
4 bytes sent test
8 bytes sent test
12 bytes sent test
16 bytes sent test
20 bytes sent test
24 bytes sent test
28 bytes sent test
32 bytes sent
Receiving all the data the server sends back The first data received says
b'CAP!CAP!CAP!CAP!CAP!CAP!CAP!CAP!' 32 bytes received
With an additional print() just before, print('\r %d bytes sent' %(sent,), end=' ') run 8 times, though the structure of loop is absolutely unchanged. I am totally confused by the fact that adding a print() could bring such consequence. And I am also confused by the thought that print() method may have some strange characteristics if used in this situation.
It is the \r combined with end=' ' in the print message. That is a carriage return without linefeed, so all the lines print over each other. The extra print adds a linefeed and they stop printing over each other. Change it to \n to fix, or more simply:
print(sent,'bytes sent')
P.S. There is also a math error:
bytecount = (bytecount + 15) // 15 * 16
should be:
bytecount = (bytecount + 15) // 16 * 16
Related
I am trying to implement UDP stop-and-wait protocol using python socket programming. I have been trying out my code with different retransmission times. And sometimes I get all the files correctly received at the receiver, but sometimes some packets get lost. For example when I ran it with 40ms five times, twice the file was received correctly and three times incorrectly. Why is this variability happening?
Here is my code for the sender and receiver:
senderSocket = socket(AF_INET, SOCK_DGRAM) # Create UDP socket for sender
r = 0 # Number of retransmissions
for i in range(0, len(messages)):
senderSocket.settimeout(retryTimeout) # After the message is sent, set retransmission timeout to listen for acknowledgement
while True:
senderSocket.sendto(messages[i], (hostName, portNumber))
acknowledged = False
while not acknowledged:
try:
ack, receiverAddress = senderSocket.recvfrom(2) # Receive ACK
acknowledged = True
except: # Socket timeout exception occurs when timeout expires but no ACK received
senderSocket.sendto(messages[i], (hostName, portNumber)) # Retransmit the message
r = r + 1 # Increment the number of retransmissions
break # On to the next message
senderSocket.close()
while True:
message, senderAddress = receiverSocket.recvfrom(1027) # Read from UDP socket into message, getting sender's address (sender IP and port)
header = message[:3] # Header is the first 3 bytes (index 0, 1, 2)
data = message[3:] # Rest is the data
first_byte = '{0:08b}'.format(header[0])
second_byte = '{0:08b}'.format(header[1])
seq_num = int(first_byte + second_byte, 2) # Convert bytes to decimal
if seq_num not in seq_nums: # Detect duplicates
seq_nums.append(seq_num)
file_content.extend(data)
ack = header[:2] # ACK is the receipt of the received message (sequence number)
receiverSocket.sendto(ack, senderAddress) # Send ACK
if header[2] == 1: # Sent multiple ACKs at lat message to make sure it receives and the sender closes
receiverSocket.sendto(ack, senderAddress)
receiverSocket.sendto(ack, senderAddress)
receiverSocket.sendto(ack, senderAddress)
break
receiverSocket.close()
Backstory: what I have done
{Codes at the bottom} I've already coded the multithreaded client and server programs using python socket, with the help of the following sites:
I. Echo Client and Server
II. Socket Server with Multiple Clients | Multithreading | Python
III. Python Socket Receive Large Amount of Data
Regarding Encryption & Decryption
(1) Exactly at what places in my codes should I encrypt/decrypt my message? Do
I encrypt the messages themselves after the user inputs or do I encrypt the byte streams after the input messages have been encoded?
(2) And how am I supposed to encrypt/decrypt the communication properly and efficiently? (It'd be nice to see code solutions with explanation, many thanks)
My Codes Currently
_server.py
import socket
import os
from _thread import *
import struct # Here to convert Python data types into byte streams (in string) and back
# ---- To Avoid Message Boundary Problem on top of TCP protocol ----
def send_msg(sock: socket, msg): # ---- Use this to send
# Prefix each message with a 4-byte length (network byte order)
msg = struct.pack('>I', len(msg)) + msg
sock.sendall(msg)
def recv_msg(sock: socket): # ---- Use this to receive
# Read message length and unpack it into an integer
raw_msglen = recvall(sock, 4)
if not raw_msglen:
return None
msglen = struct.unpack('>I', raw_msglen)[0]
# Read the message data
return recvall(sock, msglen)
def recvall(sock: socket, n: int):
# Helper function to receive n bytes or return None if EOF is hit
data = bytearray()
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data.extend(packet)
return data
# ---- Server Communication Setup
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT = 65432 # Port to listen on (non-privileged ports are > 1023)
ThreadCount = 0
try: # create socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print ("Socket successfully created")
except socket.error as err:
print ("socket creation failed with error %s" %(err))
try: # bind socket to an address
s.bind((HOST, PORT))
except socket.error as e:
print(str(e))
print('Waitiing for a Connection..')
s.listen(3)
def threaded_client(conn: socket):
conn.send(str.encode('Welcome to the Server'))
while True:
# data = conn.recv(2048) # receive message from client
data = recv_msg(conn)
reply = 'Server Says: ' + data.decode('utf-8')
if not data:
break
# conn.sendall(str.encode(reply))
send_msg(conn, str.encode(reply))
conn.close()
while True:
Client, addr = s.accept()
print('Connected to: ' + addr[0] + ':' + str(addr[1]))
start_new_thread(threaded_client, (Client, )) # Calling threaded_client() on a new thread
ThreadCount += 1
print('Thread Number: ' + str(ThreadCount))
s.close()
_client.py
import socket
import struct # Here to convert Python data types into byte streams (in string) and back
# ---- To Avoid Message Boundary Problem on top of TCP protocol ----
def send_msg(sock: socket, msg): # ---- Use this to send
# Prefix each message with a 4-byte length (network byte order)
msg = struct.pack('>I', len(msg)) + msg
sock.sendall(msg)
def recv_msg(sock: socket): # ---- Use this to receive
# Read message length and unpack it into an integer
raw_msglen = recvall(sock, 4)
if not raw_msglen:
return None
msglen = struct.unpack('>I', raw_msglen)[0]
# Read the message data
return recvall(sock, msglen)
def recvall(sock: socket, n: int):
# Helper function to receive n bytes or return None if EOF is hit
data = bytearray()
while len(data) < n:
packet = sock.recv(n - len(data))
if not packet:
return None
data.extend(packet)
return data
# ---- Client Communication Setup ----
HOST = '127.0.0.1' # The server's hostname or IP address
PORT = 65432 # The port used by the server
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print ("Socket successfully created")
except socket.error as err:
print ("socket creation failed with error %s" %(err))
print('Waiting for connection')
try:
s.connect((HOST, PORT))
except socket.error as e:
print(str(e))
Response = s.recv(1024)
while True:
Input = input('Say Something: ')
# s.send(str.encode(Input))
send_msg(s, str.encode(Input))
# Response = s.recv(1024)
Response = recv_msg(s)
print(Response.decode('utf-8'))
s.close()
You only need to Encrypt the Message itself. I would use RSA to Encrypt the Messages.
If you plan on using multiple Servers, you can open a second Port and if someone connects to it, the Server/Host. Sends the Public Key to the Client.
After that, the Client sends his Public key to the Server.
Now Server and Client switch Ports, and Communicate over there while using the Public key of the other to Encrypt the Messages and decrypt them with their Private key.
You can also Hard-code the Public Keys, if you only have one Server and one Client.
A good Module for RSA Encryption is PyCrytodome.
Here is an Example of to encrypt messages with PyCrytodome.
from Crypto.PublicKey import RSA
from Crypto.Cipher import PKCS1_OAEP
def encrypt(pk_receiver, message):
key = RSA.import_key(pk_receiver)
cipher = PKCS1_OAEP.new(key)
c = cipher.encrypt(message.encode())
return c
def decrypt(sk, c):
key = RSA.import_key(sk)
cipher = PKCS1_OAEP.new(key)
m = cipher.decrypt(c)
return m
def generate_sk(key_length):
key = RSA.generate(key_length)
with open('./secret_key.pem', 'wb') as f:
f.write(key.export_key(format='PEM'))
return key
def generate_pk(sk):
pk = sk.public_key()
with open('./public_key.pem', 'wb') as f:
f.write(pk.export_key(format='PEM'))
return
I recently bought the book Black Hat Python, 2nd Edition, by Justin Seitz, which seems to be a very good book about networking and all that (i am writing my code on Kali Linux)
I have a problem on the TCP Proxy Tool on chapter 2 :
Here is the code :
import sys
import socket
import threading
HEX_FILTER = ''.join(
[(len(repr(chr(i))) == 3) and chr(i) or '.' for i in range(256)])
def hexdump(src, length = 16, show = True):
# basically translates hexadecimal characters to readable ones
if isinstance(src, bytes):
src = src.decode()
results = list()
for i in range(0, len(src), length):
word = str(src[i:i+length])
printable = word.translate(HEX_FILTER)
hexa = ' '.join(['{ord(c):02X}' for c in word])
hexwidth = length*3
results.append('{i:04x} {hexa:<{hexwidth}} {printable}')
if show :
for line in results :
print(line)
else :
return results
def receive_from(connection):
buffer = b""
connection.settimeout(10)
try :
while True :
data = connection.recvfrom(4096)
if not data :
break
buffer += data
except Exception as e:
pass
return buffer
def request_handler(buffer):
# perform packet modifications
return buffer
def response_handler(buffer):
# perform packet modifications
return buffer
def proxy_handler(client_socket, remote_host, remote_port, receive_first):
remote_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote_socket.connect((remote_host, remote_port))
if receive_first :
# Check for any data to receive before
going into the main loop (i guess)
remote_buffer = receive_from(remote_socket)
hexdump(remote_buffer)
remote_buffer = response_handler(remote_buffer)
if len(remote_buffer):
print("[<==] Sending %d bytes to localhost." % len(remote_buffer))
client_socket.send(remote_buffer)
while True : # Start the loop
local_buffer = receive_from(client_socket)
if len(local_buffer):
line = "[==>] Received %d bytes from localhost." % len(local_buffer)
print(line)
hexdump(local_buffer)
local_buffer = request_handler(local_buffer)
remote_socket.send(local_buffer)
print("[==>] Sent to remote.")
remote_buffer = receive_from(remote_socket)
if len(remote_buffer):
print("[==>] Received %d bytes from remote." % len(remote_buffer))
hexdump(remote_buffer)
remote_buffer=response_handler(remote_buffer)
client_socket.send(remote_buffer)
print("[<==] Sent to localhost.")
if not len(local_buffer) or not len(remote_buffer):
# If no data is passed, close the sockets and breaks the loop
client_socket.close()
remote_socket.close()
print("[*] No more data. Closing connections. See you later !")
break
def server_loop(local_host, local_port, remote_host, remote_port, receive_first):
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try :
server.bind((local_host, local_port)) # Bind the local host and the local port
except Exception as e:
print('Problem on bind : %r' %e)
# If an error occurs, prints a
print("[!] Failed to listen on %s:%d" % (local_host, local_port))
print("[!] Check for other listening sockets or correct permissions.")
sys.exit(0)
print("[*] Listening on %s:%d" % (local_host, local_port))
server.listen(5)
while True :
client_socket, addr = server.accept()
# print out the local connection information
line = "> Received incoming connection from %s:%d" % (addr[0], addr[1])
print(line)
# start a thread to talk to the remote host
proxy_thread = threading.Thread(
target = proxy_handler,
args=(client_socket,remote_host,
remote_port, receive_first))
proxy_thread.start()
def main():
if len(sys.argv[1:]) != 5:
print("Usage: ./proxy.py [localhost] [localport]")
print("[remotehost] [remoteport] [receive_first]")
print("Example : ./proxy.py 127.0.0.1 9000 192.168.56.1 9000 True")
sys.exit(0)
loca l_host = sys.argv[1]
local_port = int(sys.argv[2])
remote_host = sys.argv[3]
remote_port = int(sys.argv[4])
receive_first = sys.argv[5]
if "True" in receive_first:
receive_first = True
else :
receive_first = False
server_loop(local_host, local_port,
remote_host, remote_port, receive_first)
if __name__ == '__main__':
main()
(sorry, i had a bit of a trouble formatting it and it's quite long)
Now, normally, i just need to open 2 terminals and run the code with the command line :
sudo python proxy.py 127.0.0.1 21 ftp.dlptest.com 21 True
in one terminal, and :
ftp 127.0.0.1 21
in the other one.
My code seems to be working fine, except that... I receive no data. I tried different ftp servers (notice that i don't use the one quoted in the book), but it still doesn't work. It just says :
[*] Listening on 127.0.0.1
> Received incoming connection from 127.0.0.1:55856
but it doesn't actually displays anything until the connexion times out or that i stop the command with Ctrl + C.
I know this question has already been asked, but they don't resolve my problem.
Please tell me if i forgot a line of code (for example the one that prints the data on the screen lol) or did anything wrong :)
one the hexa variable you need to put and f'{ord(c):02x}' because you just have a string and not using the 'c' variable from the list comprehension. That's a small typo you missed fix that and try the whole process again.
hexa = ' '.join([f'{ord(c):02X}' for c in word])
The f should be here ^
I'm working on a program that receives a string from an Android app sent through WiFi, the program was originally written for Python 2.7, but after adding some additional functionalities I changed it to Python 3.7. However, after making that change, my data had an extra letter at the front and for the life of me I can't figure out why that is.
Here's a snippet of my code, it's a really simple if statement to see which command was sent from the Android app and controls Raspberry Pi (4) cam (v.2) with the command.
This part sets up the connections and wait to see which command I send.
isoCmd = ['auto','100','200','300','400','500','640','800']
HOST = ''
PORT = 21567
BUFSIZE = 1024
ADDR = (HOST,PORT)
brightness = 50
timelapse = 0
tcpSerSock = socket(AF_INET, SOCK_STREAM)
tcpSerSock.bind(ADDR)
tcpSerSock.listen(5)
while True:
print ('Waiting for connection')
tcpCliSock,addr = tcpSerSock.accept()
try:
while True:
data = ''
brightness = ' '
data = tcpCliSock.recv(BUFSIZE)
dataStr = str(data[1:])
print ("Here's data ",dataStr)
if not data:
break
if data in isoCmd:
if data == "auto":
camera.iso = 0
print ('ISO: Auto')
else:
camera.iso = int(data)
print ('ISO: '), data
When I start the program this is what I see:
Waiting for connection
#If I send command '300'
Here's data b'300'
Here's data b''
Waiting for connection
I'm not sure why there's the extra b'' is coming from. I have tested the code by just adding the "b" at the beginning of each items in the array which worked for any commands that I defined, not for any commands to control the Pi camera since well, there's no extra b at the beginning. (Did that make sense?) My point is, I know I'm able to send commands no problem, just not sure how to get rid of the extra letter. If anyone could give me some advice that would be great. Thanks for helping.
Byte strings are represented by the b-prefix.
Although you can see the string in output on printing, inherently they are bytes.
To get a normal string out of it, decode function can help.
dataStr.decode("utf-8")
b'data' simply means the data inside quotes has been received in bytes form, as mentioned in other answers also, you have to decode that with decode('utf-8') to get it in string form.
I have updated your program below, to be compatible for v3.7+
from socket import *
isoCmd = ['auto','100','200','300','400','500','640','800']
HOST = ''
PORT = 21567
BUFSIZE = 1024
ADDR = (HOST,PORT)
brightness = 50
timelapse = 0
tcpSerSock = socket(AF_INET, SOCK_STREAM)
tcpSerSock.bind(ADDR)
tcpSerSock.listen(5)
while True:
print ('Waiting for connection')
tcpCliSock,addr = tcpSerSock.accept()
try:
while True:
data = ''
brightness = ' '
data = tcpCliSock.recv(BUFSIZE).decode('utf-8')
print ("Here's data "+data)
if not data:
break
if data in isoCmd:
if data == "auto":
camera.iso = 0
print ('ISO: Auto')
else:
camera.iso = int(data)
print ('ISO: '+ data)
except Exception as e:
print(e)
I'm using Python 2.7.5 to send a message to a modbus tcp simulator. What I do not understand is why python is sending the last byte twice on sendall:
This is what the python script prints out per my code (below)...
S: 00020000000c1110000400030600790083008d
R: 000200000006111000040003
R: 8d0000000003d8f603
The simulator logs this where (TX is received from script and RX is sent to script)...
TX: 00020000000c1110000400030600790083008d
RX: 000200000006111000040003
TX: 8d
Modbus message in error x03
RX: 8d0000000003d8f603
So question is...why is 8d sent twice (according to the modbus simulator listening)? The simulator is throwing an error because of it...
Code excerpt:
def writeMultipleHoldingRegister(socket, transactionid, unitid, address,valuesToWrite):
TRANSACT_ID_BYTES = convertIntToBytes(int(transactionid))
DATA_ADDRESS_BYTES = convertIntToBytes(int(address))
FUNCTIONCODE_BYTES = '10'
NUMREGISTERS_BYTES = convertIntToBytes(len(valuesToWrite))
NUMBYTES = len(valuesToWrite)*2
NUMBYTES_BYTES = str(('%04X' % NUMBYTES)[2:4])
MESSAGE_LENGTH_BYTES = convertIntToBytes(int(6+(len(valuesToWrite)*2)))
modbustcp_write = (TRANSACT_ID_BYTES + '0000' + MESSAGE_LENGTH_BYTES + unitid + FUNCTIONCODE_BYTES + DATA_ADDRESS_BYTES + NUMREGISTERS_BYTES + NUMBYTES_BYTES).decode('hex')
for value in valuesToWrite:
modbustcp_write = modbustcp_write + convertIntToBytes(int(value)).decode('hex')
print 'S:', binascii.hexlify(modbustcp_write)
socket.sendall(modbustcp_write) #modbus tcp
data = socket.recv(12)
print 'R:', binascii.hexlify(data)
HOST = 'localhost'
PORT = 502
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
writeMultipleHoldingRegister(s, 2, '11', 4, [121, 131, 141])
print 'R:', binascii.hexlify(s.recv(100))
s.close()
Followed advice left in comment. Process Monitor showed me the traffic but not the bytes themselves. So I downloaded and ran RawCAP and viewed the output in Wireshark. To paraphrase...where 'S' is sent by simulator and 'R' is received by simulator...
R: 00020000000c1110000400030600790083008d (modbus request recv)
S: 000200000006111000040003 (modbus ack good request sent)
S: 8d0000000003d8f603 (modbus error notification sent...wtf)
I never see a '8d' received twice...so I don't know what's going on here. Maybe a bug in the simulator... http://www.plcsimulator.org/