Size of PKCS#7 padded string changes after AES encryption - python

I am trying to implement some mechanism in the tor network for educational purposes. The mechanisem that I am trying to implement is that the messages need to be in a fixed size of 512 bytes before they are sent over the network. but when I am padding some string with PKCS#7 to be 512 bytes, after encryption it changes to be 750 bytes for some reason. Do someone know why the encryption of the speaciel characters from the padding changes the length of the message?
This is my main code
def main():
data = "A" * 600
cells = build_message(data) # divided the data to be messages of 512 bytes
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect(("127.0.0.1", 4000))
tls = establish_tls_connection(sock, "client") # establishing a tls session
for cell in cells:
print(len(str(cell))) # printing the len message before encrypting
encrypted = tls.encrypt(str(cell))
print(len(encrypted[0])) # printing the len message after encrypting
This is an example of a padded before encryption
{
'sequence': 2, 'next_hop_ip': '127.0.0.1',
'next_hop_port': 443, 'padded': 1, 'payload':
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
îîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîî
îîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîî
îîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîî
îîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîîî'
}
As you can see I have a lot unprintable characters in the message, Please tell me if you need me to provide any more parts of my code.
Thank you.

Related

Socket Fragmented Received Data

I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1
As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.
One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.
You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()

Why is the time-to-live(ttl) packed using struct?

I stumbled upon some code in the book PyMOTW3 (https://pymotw.com/3/socket/multicast.html) and I could not understand why the time to live(ttl) argument was packed as struct.pack('b',1).
I tried searching the manuals to see if the arguments are supposed to be packed but it states that it can be integers. I tried inputting a normal integer and it seemed to work fine. It gave the same output as the code below. Then is there a specific reason why it has been packed like this ?
I do know that packing it as 1 is unnecessary because the default value is 1, but what if I need to use some other number. Do I need to pack it?
I have included the code from the book below.
import socket
import struct
import sys
message = b'very important data'
multicast_group = ('224.3.29.71', 10000)
# Create the datagram socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Set a timeout so the socket does not block
# indefinitely when trying to receive data.
sock.settimeout(0.2)
# Set the time-to-live for messages to 1 so they do not
# go past the local network segment.
ttl = struct.pack('b', 1)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl)
print(s.getsockopt(socket.IPPROTO_IP,socket.IP_MULTICAST_TTL))
try:
# Send data to the multicast group
print('sending {!r}'.format(message))
sent = sock.sendto(message, multicast_group)
# Look for responses from all recipients
while True:
print('waiting to receive')
try:
data, server = sock.recvfrom(16)
except socket.timeout:
print('timed out, no more responses')
break
else:
print('received {!r} from {}'.format(
data, server))
finally:
print('closing socket')
sock.close()
This is the output I get whether I pack it or use a normal integer.
1
Sending very important message
Timed Out!
Closing Socket
All packets consist of bytes.
For the AF_INET or an IPv4 you can see the distribution of bits among the properties of the packet in this picture. You can see in the IPv4 header, TTL is the 9th octet of 20. TTL is an 8-bit (1-byte) field and can be from 0 to 255.
That is, each property should take a certain number of bits.
For this purpose, Python 3.7 has a module struct. This module performs conversions between Python values and C structs represented as Python bytes objects.
Since TTL is a positive integer of 1 byte in size, we have to select the unsigned char type, which is 'B'.
And so now the code looks like this:
ttl = struct.pack('B', 25) # unsigned char as 1-byte object
sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, ttl)

How to check if socket connection is working and buffering or is the request incorrect?

I have been trying to figure this out and can't seem to wrap my head around it.
import socket
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server = '192.168.1.7'
port = 8000
buffer_size = 4096
request = '1|gkdgdfg49|sdfsdfsd*|sdsfsf*|'+server
request = request.encode()
s.connect((server,port))
s.send(request)
x = s.recv(buffer_size)
The documentation for the server side is:
https://www.sharekhan.com/Upload/General/TradeTigerAPIForClient.pdf
The log of api server shows:
[I] "API Client Socket Connected with error Code=0"2018-09-22 16:07:23.555+05:30
[I] "Hurraay You have Connected."2018-09-22 16:07:23.555+05:30
[I] "Received Bytes Count = 49 Data Length = 1801944113 Thread State = Background, WaitSleepJoin , Queue Count = 0"2018-09-22 16:07:23.555+05:30
[I] "API Client Request Data Length 1801944113 bytes needs to read"
2018-09-22 16:08:21.984+05:30
[I] "Received Bytes Count = 49 Data Length = 1801944113 Thread State = Background, WaitSleepJoin , Queue Count = 0"
2018-09-22 16:08:21.984+05:30
[I] "API Client Request Data Length 1801944113 bytes needs to read"|Reserved = |
Is the request correct? Is the data being transferred?
If I rerun
s.send(request)
x = s.recv(buffer_size)
Nothing happens. The interpreter is stuck at running the above two lines.
I would appreciate any help. Very new to `sockets and the documentation is not very helpful.
As I pointed out in your other question, this is a binary protocol.
You're sending it ASCII text. Note that big number (1801944113), when interpreted as text, is equal to 'kg|1'. This is (the big-endian ordering of) the first four characters you sent it. In other words, it's taken the first four bytes you sent it, interpreted them as a binary value, and is now using it as the message length value -- which is obviously all wrong.
You need to use the python struct module's pack and unpack methods to encode and decode binary values to build up your buffer.
That would look something like this:
import struct
trans_code = 1 # LOGIN
login_id = b'MyName'
mpassword = b'MyPass'
tpassword = b'MyTradingPass'
my_ip = b'192.168.1.31' # No idea what the point of this is
reserved = b''
msg_length = 196
msg_buffer = struct.pack("!IH30s20s20s20s100s", msg_length, trans_code,
login_id, mpassword, tpassword, my_ip, reserved)
assert len(msg_buffer) == msg_length
print("Login request to send '{}'".format(msg_buffer))
You will need to learn to use the struct module. It's well-specified in the python documentation.
Your results (in particular that big number) establish that the server is expecting numeric fields to be encoded in big-endian byte order (AKA "network byte order").

AES encrypted File transfer over TCP socket; padding issue

I'm trying to do a file transfer using encrypted TCP socket with AES 256.
If I transfer a file without encryption, it works fine.
If I send small commands (such as 'ipconfig') to the client or server, the encryption works fine.
No matter the file size is, I keep receiving the following error message:
DecodeAES = lambda c, e: c.decrypt(base64.b64decode(e)).rstrip(PADDING)
File "/usr/lib/python2.6/base64.py", line 76, in b64decode
raise TypeError(msg)
TypeError: Incorrect padding
My function for encoding and decoding is the following (the hat variable is the message):
def AESENC(hat,typ):
BLOCK_SIZE = 32
PADDING = '{'
pad = lambda s: s + (BLOCK_SIZE - len(s) % BLOCK_SIZE) * PADDING
EncodeAES = lambda c, s: base64.b64encode(c.encrypt(pad(s)))
DecodeAES = lambda c, e: c.decrypt(base64.b64decode(e)).rstrip(PADDING)
secret = '01234567890123456789012345678912'
IV = 'wir&/>H54mgd9a";'
cipher = AES.new(secret,AES.MODE_CFB,IV)
if typ == 0:
encoded = EncodeAES(cipher, hat)
return encoded
else:
decoded = DecodeAES(cipher, hat)
return decoded
Client side
if os.path.exists(df):
print ' found the file '
f = open(df, 'rb')
packet = f.read(1024)
while packet != '':
s.send(AESENC(packet,0))
s.send( AESENC('123XXX',0) )
s.send('123XXX')
f.close()
Server side
f = open('/root/Desktop/Transfer.mp3','wb')
while True:
bits = AESENC ( conn.recv(1024) , 1 )
while (bits):
f.write(bits)
bits = AESENC ( conn.recv(1024) , 1 )
if bits.endswith('123XXX'):
print '[+] Transfer completed '
break
f.close()
break
return
Does anybody know how to fix this?
Apologies for the misunderstanding of your sample code, and thank you for adding the server and client calls!
Please note that the error message that you're getting is unrelated to the encryption. You can read the relevant part of the base64.py library: the error message indicates that the base64 data is not valid. In particular, it's not properly padded at the end.
With the additional call site information, I believe the problem is that you're encrypting and then separately encoding each 1024 byte block of the data on the client. Then, on the server, you're reading 1024 bytes from the network and trying to decode it. However, base64-encoding will increase the length of the original data, so what you'll read will only be the first 1024 bytes of the encoded form, which will be a truncated base64 message (hence improperly padded).
As for how to solve it, a base64 message needs to be decoded all as one piece. So you either need to calculate how much data to read from the network to get a complete encoded block (a calculation that you can do reliably except for the possibly short last packet, since the base64 length is always (length + 2) / 3 * 4), encode the entire data stream at once (which is likely a problem due to memory use if you want to handle arbitrary-sized files), or define your network protocol so that the server can tell that it's seen a complete block that can be decoded with base64.

Add length prefix to DNS request using TCP and Scapy

In the RFC 1035 about DNS, it's written :
4.2.2. TCP usage
Messages sent over TCP connections use server port 53 (decimal). The
message is prefixed with a two byte length field which gives the
message length, excluding the two byte length field. This length
field allows the low-level processing to assemble a complete message
before beginning to parse it.
I want to send a DNS request with TCP but I don't know how to add these two bytes before the DNS request. I try with that code :
from scapy.all import *
ip=IP(dst="216.239.32.10")
request = DNS(rd=1, qd=DNSQR(qname = "google.be", qtype="A")) #size = 27(dec) = 1b (hex)
twoBytesRequestSize = "\x1b\x00"
completeRequest = str(request) + twoBytesRequestSize
SYN=ip/TCP(sport=RandNum(1024,65535), dport=53, flags="S", seq=42)
SYNACK=sr1(SYN)
ACK=ip/TCP(sport=SYNACK.dport, dport=53, flags="A", seq=SYNACK.ack, ack=SYNACK.seq + 1)
send(ACK)
DNSRequest = ip/TCP(sport=SYNACK.dport, dport=53, flags="PA", seq=SYNACK.ack, ack=SYNACK.seq + 1) / completeRequest
DNSReply = sr1(DNSRequest, timeout = 1)
But my paquet is interpreted like a simple TCP packet without DNS layer.
Have you an idea to add these two bytes prefix before the DNS request?
Thank you !
The solution uses Big endian notation. \x00\x1b instead of \x1b\x00. But the rest of the code above is correct. Thank you Armin.

Categories