Base64 and non standard - python

I try to create a python client for bacula, but I have some problem with the authentication.
The algorithm is :
import hmac
import base64
import re
...
challenge = re.search("auth cram-md5 ()", data)
#exemple ''
passwd = 'b489c90f3ee5b3ca86365e1bae27186e'
hm = hmac.new(passwd, challenge).digest()
rep = base64.b64encode(hm).strp().rstrip('=')
#result with python : 9zKE3VzYQ1oIDTpBuMMowQ
#result with bacula client : 9z+E3V/YQ1oIDTpBu8MowB'
There's a way more simple than port the bacula's implemenation of base 64?
int
bin_to_base64(char *buf, int buflen, char *bin, int binlen, int compatible)
{
uint32_t reg, save, mask;
int rem, i;
int j = 0;
reg = 0;
rem = 0;
buflen--; /* allow for storing EOS */
for (i=0; i >= (rem - 6);
if (j


To verify your CRAM-MD5 implementation, it is best to use some simple test vectors and check combinations of (challenge, password, username) inputs against the expected output.
Here's one example (from http://blog.susam.in/2009/02/auth-cram-md5.html):
import hmac
username = 'foo#susam.in'
passwd = 'drowssap'
encoded_challenge = 'PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg=='
challenge = encoded_challenge.decode('base64')
digest = hmac.new(passwd, challenge).hexdigest()
response = username + ' ' + digest
encoded_response = response.encode('base64')
print encoded_response
# Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy
That said, I've certainly found examples on the net where the response generated by the above code differs from the expected response stated on the relevant site, so I'm still not entirely clear as to what is happening in those cases.


I HAVE CRACKED THIS.
I ran into exactly the same problem you did, and have just spent about 4 hours identifying the problem, and reimplementing it.
The problem is the Bacula's base64 is BROKEN, AND WRONG!
There are two problems with it:
The first is that the incoming bytes are treated as signed, not unsigned. The effect of this is that, if a byte has the highest bit set (>127), then it is treated as a negative number; when it is combined with the "left over" bits from previous bytes are all set to (binary 1).
The second is that, after b64 has processed all the full 6-bit output blocks, there may be 0, 2 or 4 bits left over (depending on input block modulus 3). The standard Base64 way to handle this is to multiply the remaining bits, so they are the HIGHEST bits in the last 6-bit block, and process them - Bacula leaves them as the LOWEST bits.
Note that some versions of Bacula may accept both the "Bacula broken base64 encoding" and the standard ones, for incoming authentication; they seem to use the broken one for their authentication.
def bacula_broken_base64(binarystring):
b64_chars="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
remaining_bit_count=0
remaining_bits=0
output=""
for inputbyte in binarystring:
inputbyte=ord(inputbyte)
if inputbyte>127:
# REPRODUCING A BUG! set all the "remaining bits" to 1.
remaining_bits=(1 << remaining_bit_count) - 1
remaining_bits=(remaining_bits<<8)+inputbyte
remaining_bit_count+=8
while remaining_bit_count>=6:
# clean up:
remaining_bit_count-=6
new64=(remaining_bits>>remaining_bit_count) & 63 # 6 highest bits
output+=b64_chars[new64]
remaining_bits&=(1 << remaining_bit_count) - 1
if remaining_bit_count>0:
output+=b64_chars[remaining_bits]
return output
I realize it's been 6 years since you asked, but perhaps someone else will find this useful.

Related

How can I find the big endian key in a message?

I am trying to read a binary message from an ESP32 using a broker; i wrote a phyton script where I subscribe the topic. the message that i actually receive is:
b'\x00\x00\x00?'
this is a float binary little endian message but I don't the key to decode it. Is there a way to find the decode key based on this data?
This is my python code:
import paho.mqtt.client as mqtt
def on_connect1(client1, userdata1, flags1, rc1):
client1.subscribe("ESP32DevKit123/mytopic")
def on_message1(client1, userdata1, msg1):
print(msg1.topic+" "+ "TESTENZA: "+str(msg1.payload))
client1 = mqtt.Client()
client1.username_pw_set(username="myuser",password="mypassword")
client1.on_connect = on_connect1
client1.on_message = on_message1
client1.connect("linkclient", portnumber, 60)
def twosComplement_hex(hexval):
bits = 16 # Number of bits in a hexadecimal number format
on_message1 = int(hexval, bits)
if on_message1 & (1 << (bits-1)):
on_message1 -= 1 << bits
return on_message1
client1.loop_forever()
It also gives me an error in the line on_message1 -= 1 << bits; the error says: Expected intended block pylance. Any solutions?

The data you provided is b'\x00\x00\x00?' - I'm going to assume that this is 0000003f (please output hex with msg1.payload.hex()).
I'll also assume that by "float binary little endian" you mean a big endian floating point (IEE754) - note that this does not match up with the algorithm you are using (twos compliment). Plugging this input into an online tool indicates that the expected result ("Float - Big Endian (ABCD)") is 8.82818e-44 (it's worth checking with this tool; sometimes the encoding may not be what you think it is!).
Lets unpack this using python (see the struct docs for more information):
>>> from struct import unpack
>>> unpack('>f', b'\x00\x00\x00\x3f')[0]
8.828180325246348e-44
Notes:
The [0] is there because unpack returns an array (you can unpack more than one item from the input)
>f - the > means big-endian and the f float (standard size = 4 bytes)
The reason your original code gives the error "Expected intended block" is due to the lack of indentation in the line on_message1 -= 1 << bits (as it follows an if it needs to be indented). The algorithm does not appear relevant to your task (but there may be details I'm missing).

How to turn python AES function into C# code

I have a part of python code that I want to use in my C# project, but I can't find a right way to achieve it.
python code:
def getCiphertext(plaintext, key = key_, cfb_iv = iv_, size = 128):
message = plaintext.encode('utf-8')
cfb_cipher_encrypt = AES.new(key, AES.MODE_CFB, cfb_iv, segment_size = size)
mid = cfb_cipher_encrypt.encrypt(message)
return hexlify(mid).decode()
I have tried the C# code below, but the result is different:
using System.Security.Cryptography;
public static string AesEncrypt(string str, string key, string IVString)
{
Encoding encoder = Encoding.UTF8;
byte[] toEncryptArray = Encoding.UTF8.GetBytes(str);
RijndaelManaged rm = new RijndaelManaged
{
Key = encoder.GetBytes(key),
Mode = CipherMode.CFB,
BlockSize = 128,
Padding = PaddingMode.PKCS7,
IV = encoder.GetBytes(IVString),
};
ICryptoTransform cTransform = rm.CreateEncryptor();
byte[] resultArray = cTransform.TransformFinalBlock(toEncryptArray, 0, toEncryptArray.Length);
return ToBCDStringLower(resultArray);//result
}
public static string ToBCDStringLower(byte[] buffer)
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < buffer.Length; i++)
{
sb.Append(buffer[i].ToString("x2"));
}
return sb.ToString();
}
Thanks guys all!

.NET's CFB implementation:
CFB in .NET is problematic. In .NET Framework it is supported, in .NET Core only from .NET 5.0.
In addition, .NET Framework and .NET Core allow different segment sizes, but both support 8-bit and 128-bit, which corresponds to the most common variants, namely CFB8 and CFB128 (or full block CFB). The segment size is an additional parameter in CFB which corresponds to the bits encrypted per encryption step, see CFB.
Another peculiarity is that in .NET the plaintext size must be an integer multiple of the segment size. This is remarkable (actually already a bug), since CFB is a stream cipher mode that does not require padding.
Therefore, for CFB, with the exception of CFB8, padding is generally required. In the case of CFB128 to the full block, allowing the default padding PKCS7 to be applied.
Thus, to obtain the ciphertext that corresponds to the unpadded plaintext, the ciphertext must be truncated to the plaintext size.
Comparison of Python and C# code:
In the posted Python code, the segment size in the argument list defaults to 128 bits (the PyCryptodome default is 8 bits). In the C# code, the segment size (here denoted as FeedbackSize) is not specified, so the default value of 128 bits is used.
Thus, unless a segment size other than 128 bits is explicitly specified in the Python code, both codes apply the same segment size.
Also, in the C# code, padding (PKCS7) is done as required by the C# implementation. Therefore, when the ciphertext of the C# code is truncated to the plaintext size, it matches the ciphertext of the Python code.
The following example uses the code you posted unchanged:
string plaintext = "The quick brown fox jumps over the lazy dog";
string key = "01234567890123456789012345678901";
string cfb_iv = "0123456789012345";
string ciphertext = AesEncrypt(plaintext, key, cfb_iv);
string ciphertextTrunc = ciphertext.Substring(0, plaintext.Length * 2); // *2 since AesEncryptOP returns the ciphertext hex encoded
Console.WriteLine(ciphertext);
Console.WriteLine(ciphertextTrunc);
Output:
09f1e464983a7d25305d5b865386e477d97b34b9a6365372ef83b78e495692489c1848a124345eb808eb66d268c6d1ad
09f1e464983a7d25305d5b865386e477d97b34b9a6365372ef83b78e495692489c1848a124345eb808eb66
As you can verify, the shortened ciphertext corresponds to the output of the Python code.
Note that as explained in the 1st section, padding is required for CFB128. Changing the padding to PaddingMode.None will result in a CryptographicException: The input data is not a complete block. However, with CFB8 this would be possible.
BouncyCastle:
An alternative to the .NET built-in implementation is BouncyCastle, which implements CFB as stream cipher mode so that no padding is needed. The following code:
using Org.BouncyCastle.Crypto.Engines;
using Org.BouncyCastle.Crypto.Modes;
using Org.BouncyCastle.Crypto.Parameters;
using Org.BouncyCastle.Crypto;
...
public static string Encrypt(string str, string keyString, string IVString)
{
byte[] inputBytes = Encoding.UTF8.GetBytes(str);
byte[] IV = Encoding.UTF8.GetBytes(IVString);
byte[] key = Encoding.UTF8.GetBytes(keyString);
AesEngine engine = new AesEngine();
CfbBlockCipher blockCipher = new CfbBlockCipher(engine, 128);
BufferedBlockCipher cipher = new BufferedBlockCipher(blockCipher);
KeyParameter keyParam = new KeyParameter(key);
ParametersWithIV keyParamWithIv = new ParametersWithIV(keyParam, IV);
cipher.Init(true, keyParamWithIv);
byte[] outputBytes = new byte[cipher.GetOutputSize(inputBytes.Length)];
int length = cipher.ProcessBytes(inputBytes, outputBytes, 0);
cipher.DoFinal(outputBytes, length);
string encryptedInput = ToBCDStringLower(outputBytes);
return encryptedInput;
}
directly (i.e. without truncation) returns the result of the Python code.

CRC implementation in python

Here is my code for the implementation of a CRC in python:
import math
divisors = [0b1100000001111, 0b11000000000000101, 0b10001000000100001, 0b1011, 0b10011,0b00000111, 0b11001]
def get_Length(arg):
return math.floor(math.log2(arg)) +1
def CRC(message, type):
print("Message ",bin(message)[2:], hex(message))
# int message_length = get_Length(message);
divisor_length = get_Length(divisors[type])
divisor = divisors[type]
print("Divisor: ",bin(divisor)[2:], hex(divisor))
message = message << (divisor_length-1)
old_message = message
while( (message >> (divisor_length-1)) !=0 ):
ml = get_Length(message)
divisor_copy = divisor << (ml-divisor_length)
message = message ^ divisor_copy
print(bin(message)[2:], hex(message))
print(bin(old_message| message)[2:], hex(old_message|message), end="\n\n")
def main():
CRC(0b1101011011, 4)
CRC(0x34ec, 1)
main()
The first message is from this Wikipedia example and gives the correct result. However the second one (x34ec), which demonstrates a CRC-16 is not giving the correct result (correct result). I'm attaching the output snapshot as well:
.
It would be appreciative if somebody can shed some light on it.
Thanks in advance.

There are many CRC-16's. I count 30 here alone, and I'm sure there are more in use that are not in that catalog.
The one you have implemented is the CRC-16/UMTS in that catalog, also known as CRC-16/BUYPASS and CRC-16/VERIFONE. It is the most straightforward of definitions, with the bits in and bits out not reflected, with an initial value of zero, and with no final exclusive-or.
The result CRC you implemented on the message 34 ec can in fact be found directly in your linked "correct result", on the fourth row of the table which is labeled "CRC-16/BUYPASS".
If you want to implement a different CRC-16, the first thing you need to do is specify which one. That specification is the polynomial, reflection of input and output, initial value, and final exclusive-or value.

Incorrect CRC calculation in protocol. One is implemented using zlib and the other one is calculated in function

I am implementing a protocol in an STM32F412 board. It's almost done, I just need to do a CRC check for the received data.
I tried using the internal CRC module for calculating the CRC but I could not match the result to any online CRC algorithm online, so I decided to do a simple implementation of the Ethernet CRC.
static const uint32_t crc32_tab[] =
{
0x00000000L, 0x77073096L, 0xee0e612cL, 0x990951baL, 0x076dc419L,
0x706af48fL, 0xe963a535L, 0x9e6495a3L, 0x0edb8832L, 0x79dcb8a4L,
0xe0d5e91eL, 0x97d2d988L, 0x09b64c2bL, 0x7eb17cbdL, 0xe7b82d07L,
0x90bf1d91L, 0x1db71064L, 0x6ab020f2L, 0xf3b97148L, 0x84be41deL,
0x1adad47dL, 0x6ddde4ebL, 0xf4d4b551L, 0x83d385c7L, 0x136c9856L,
0x646ba8c0L, 0xfd62f97aL, 0x8a65c9ecL, 0x14015c4fL, 0x63066cd9L,
0xfa0f3d63L, 0x8d080df5L, 0x3b6e20c8L, 0x4c69105eL, 0xd56041e4L,
0xa2677172L, 0x3c03e4d1L, 0x4b04d447L, 0xd20d85fdL, 0xa50ab56bL,
0x35b5a8faL, 0x42b2986cL, 0xdbbbc9d6L, 0xacbcf940L, 0x32d86ce3L,
0x45df5c75L, 0xdcd60dcfL, 0xabd13d59L, 0x26d930acL, 0x51de003aL,
0xc8d75180L, 0xbfd06116L, 0x21b4f4b5L, 0x56b3c423L, 0xcfba9599L,
0xb8bda50fL, 0x2802b89eL, 0x5f058808L, 0xc60cd9b2L, 0xb10be924L,
0x2f6f7c87L, 0x58684c11L, 0xc1611dabL, 0xb6662d3dL, 0x76dc4190L,
0x01db7106L, 0x98d220bcL, 0xefd5102aL, 0x71b18589L, 0x06b6b51fL,
0x9fbfe4a5L, 0xe8b8d433L, 0x7807c9a2L, 0x0f00f934L, 0x9609a88eL,
0xe10e9818L, 0x7f6a0dbbL, 0x086d3d2dL, 0x91646c97L, 0xe6635c01L,
0x6b6b51f4L, 0x1c6c6162L, 0x856530d8L, 0xf262004eL, 0x6c0695edL,
0x1b01a57bL, 0x8208f4c1L, 0xf50fc457L, 0x65b0d9c6L, 0x12b7e950L,
0x8bbeb8eaL, 0xfcb9887cL, 0x62dd1ddfL, 0x15da2d49L, 0x8cd37cf3L,
0xfbd44c65L, 0x4db26158L, 0x3ab551ceL, 0xa3bc0074L, 0xd4bb30e2L,
0x4adfa541L, 0x3dd895d7L, 0xa4d1c46dL, 0xd3d6f4fbL, 0x4369e96aL,
0x346ed9fcL, 0xad678846L, 0xda60b8d0L, 0x44042d73L, 0x33031de5L,
0xaa0a4c5fL, 0xdd0d7cc9L, 0x5005713cL, 0x270241aaL, 0xbe0b1010L,
0xc90c2086L, 0x5768b525L, 0x206f85b3L, 0xb966d409L, 0xce61e49fL,
0x5edef90eL, 0x29d9c998L, 0xb0d09822L, 0xc7d7a8b4L, 0x59b33d17L,
0x2eb40d81L, 0xb7bd5c3bL, 0xc0ba6cadL, 0xedb88320L, 0x9abfb3b6L,
0x03b6e20cL, 0x74b1d29aL, 0xead54739L, 0x9dd277afL, 0x04db2615L,
0x73dc1683L, 0xe3630b12L, 0x94643b84L, 0x0d6d6a3eL, 0x7a6a5aa8L,
0xe40ecf0bL, 0x9309ff9dL, 0x0a00ae27L, 0x7d079eb1L, 0xf00f9344L,
0x8708a3d2L, 0x1e01f268L, 0x6906c2feL, 0xf762575dL, 0x806567cbL,
0x196c3671L, 0x6e6b06e7L, 0xfed41b76L, 0x89d32be0L, 0x10da7a5aL,
0x67dd4accL, 0xf9b9df6fL, 0x8ebeeff9L, 0x17b7be43L, 0x60b08ed5L,
0xd6d6a3e8L, 0xa1d1937eL, 0x38d8c2c4L, 0x4fdff252L, 0xd1bb67f1L,
0xa6bc5767L, 0x3fb506ddL, 0x48b2364bL, 0xd80d2bdaL, 0xaf0a1b4cL,
0x36034af6L, 0x41047a60L, 0xdf60efc3L, 0xa867df55L, 0x316e8eefL,
0x4669be79L, 0xcb61b38cL, 0xbc66831aL, 0x256fd2a0L, 0x5268e236L,
0xcc0c7795L, 0xbb0b4703L, 0x220216b9L, 0x5505262fL, 0xc5ba3bbeL,
0xb2bd0b28L, 0x2bb45a92L, 0x5cb36a04L, 0xc2d7ffa7L, 0xb5d0cf31L,
0x2cd99e8bL, 0x5bdeae1dL, 0x9b64c2b0L, 0xec63f226L, 0x756aa39cL,
0x026d930aL, 0x9c0906a9L, 0xeb0e363fL, 0x72076785L, 0x05005713L,
0x95bf4a82L, 0xe2b87a14L, 0x7bb12baeL, 0x0cb61b38L, 0x92d28e9bL,
0xe5d5be0dL, 0x7cdcefb7L, 0x0bdbdf21L, 0x86d3d2d4L, 0xf1d4e242L,
0x68ddb3f8L, 0x1fda836eL, 0x81be16cdL, 0xf6b9265bL, 0x6fb077e1L,
0x18b74777L, 0x88085ae6L, 0xff0f6a70L, 0x66063bcaL, 0x11010b5cL,
0x8f659effL, 0xf862ae69L, 0x616bffd3L, 0x166ccf45L, 0xa00ae278L,
0xd70dd2eeL, 0x4e048354L, 0x3903b3c2L, 0xa7672661L, 0xd06016f7L,
0x4969474dL, 0x3e6e77dbL, 0xaed16a4aL, 0xd9d65adcL, 0x40df0b66L,
0x37d83bf0L, 0xa9bcae53L, 0xdebb9ec5L, 0x47b2cf7fL, 0x30b5ffe9L,
0xbdbdf21cL, 0xcabac28aL, 0x53b39330L, 0x24b4a3a6L, 0xbad03605L,
0xcdd70693L, 0x54de5729L, 0x23d967bfL, 0xb3667a2eL, 0xc4614ab8L,
0x5d681b02L, 0x2a6f2b94L, 0xb40bbe37L, 0xc30c8ea1L, 0x5a05df1bL,
0x2d02ef8dL
};
uint32_t calc_crc_calculate(uint8_t *pData, uint32_t uLen)
{
uint32_t val = 0xFFFFFFFFU;
int i;
for(i = 0; i < uLen; i++) {
val = crc32_tab[(val ^ pData[i]) & 0xFF] ^ ((val >> 8) & 0x00FFFFFF);
}
return val^0xFFFFFFFF;
}
I calculated the crc of 0x6F and compared the result to the online calculators and it apparently matches.
When I try to test the protocol with my python code I'm just unable to match the CRCs. On python I'm using the following code:
d = 0x6f
crc = zlib.crc32(bytes(d))&0xFFFFFFFF
I'm now unable to tell which is right. Apparently my algorithm is OK because it matches the online calculator. BUT those online calculators do not seem to be reliable sometimes and I doubt that python's zlib implementation is wrong .. I may be using it wrong at worst.

Actually you can compute the Ethernet CRC32 with the builtin module of the STM32. It took me quite a while to make it match up as well.
This code should match up for sizes divisible by 4 (I also used python zlib on the other end):
#include "stm32l4xx_hal.h"
uint32_t CRC32_Compute(const uint32_t *data, size_t sizeIn32BitWords)
{
CRC_HandleTypeDef hcrc = {
.Instance = CRC,
.Init.DefaultPolynomialUse = DEFAULT_POLYNOMIAL_ENABLE,
.Init.DefaultInitValueUse = DEFAULT_INIT_VALUE_ENABLE,
.Init.InputDataInversionMode = CRC_INPUTDATA_INVERSION_WORD,
.Init.OutputDataInversionMode = CRC_OUTPUTDATA_INVERSION_ENABLE,
.InputDataFormat = CRC_INPUTDATA_FORMAT_WORDS,
};
HAL_StatusTypeDef status = HAL_CRC_Init(&hcrc);
assert (status == HAL_OK)
uint32_t checksum = HAL_CRC_Calculate(&hcrc, data, sizeIn32BitWords);
uint32_t checksumInverted = ~checksum;
return checksumInverted;
}
The challenge with sizes not divisible by 4 is to get the "inversion/reversal" (changing the bit order) right. There is an example how the hardware handles this in the "RM0394 Reference manual STM32L43xxx STM32L44xxx STM32L45xxx STM32L46xxx advanced ARM®-based 32-bit MCUs Rev 3" on page 333.
The essence is that reversal reverses the bit order. For CRC32 this reversal must happen on the word level, i.e. over 32 bits.

Ok. It certainly was a bug on my part. But it was happening in my python code.
I suddenly realized that I was practically doing bytes(0x6F) which just creates an array with 111 positions.
What I actually needed to do was
import struct
d = pack('B', 0x6F)
crc = zlib.crc32(bytes(d))&0xFFFFFFFF
This question could have been avoided had I just done a little bit of rubber duck debugging. Hopefuly this will help someone else.

TLS MAC message verification

I'm developing a SSL de-cipher in python but I'm having some problems on HMAC verification:
I've extracted all keyring related material (client IV, MAC, Key and Server IV, MAC, key).
When I receive the first Application_Data message (0x17), I am able to decrypt it, but unable to verify message integrity.
On RFC 2246 (https://www.ietf.org/rfc/rfc2246.txt), tells:
The MAC is generated as:
HMAC_hash(MAC_write_secret, seq_num + TLSCompressed.type +
TLSCompressed.version + TLSCompressed.length +
TLSCompressed.fragment));
where "+" denotes concatenation.
seq_num
The sequence number for this record.
hash
The hashing algorithm specified by
SecurityParameters.mac_algorithm.
Taking this as an example:
Chosen cipher_suite is TLS_RSA_WITH_AES_256_CBC_SHA256
client_mac = "some random stuff"
message_type = 0x17
message_version = 0x0303
encrypted_message_length = 1184 (IV|Message|MAC|Offset)
decrypted_message_length = 1122 (removing IV, MAC and offset)
message = "some message of length 1122"
client_mac is extracted from keyring_material
message_type is 0x17, because as an Application_data message type, the correct value should be 0x17
message version is 0x0303 as it's TLS 1.2
message length is 1122, removing preceding IV, offset and MAC verification, message, gets a final length of 1122
seq_number is 1 as it's the first message
HMAC_SHA256 calculation, in python, is as follows:
import hashlib
import hmac
hmac.new(<client_mac>,label+message,hashlib.sha256).digest()
My question is, how do I calculate label?
As RFC mentions, "+" denotes concatenation, but concatenation of what
HEX values converted to string
"1" + "17" + "0303" + "462"
INT values converted to strings
"1" + "23" + "771" + "1122"
And other thing to mention, TLSCompressed.version means:
0x0303
771
"1.2"
"12"
"TLS 1.2"
In this maillist (http://www.ietf.org/mail-archive/web/tls/current/msg14357.html) I found a supposed clarification of MAC values,
MAC(MAC_write_key, seq_num +
TLSCipherText.type +
TLSCipherText.version +
length of ENC(content + padding + padding_length) +
IV +
ENC(content + padding + padding_length));
where the length is encoded as two bytes in the usual way.
but it makes no sense to me, because it's useless to re-encode decrypted values to check to compute MAC. And from last line "where length is encoded as two bytes in the usual way", does it means that I should use
struct.pack("!H",length)
Then remove "\x" and use this value? or should I encode this value in HEX and then concatenate it?
I'm a bit lost, because RFC are not clear about how values should be used.
I've been trying several combinations (even brute forcing), but none of them worked, I hope you can light my way.

Well, after diggin' a bit I've managed to solve the issue.
RFC 5246, in section 6.2.3.1 (https://www.rfc-editor.org/rfc/rfc5246#section-6.2.3.1)
The MAC is generated as:
MAC(MAC_write_key, seq_num +
TLSCompressed.type +
TLSCompressed.version +
TLSCompressed.length +
TLSCompressed.fragment);
where "+" denotes concatenation.
But it does not points the data size, either representation format (hex, string...).
The way every field must be represented is as follows:
seq_num:
Description: A int counter, starting in 0, which will be incremented every frame received or sended. For a TCP Session, two seq_numbers must be used, one for the server and other for the client, incrementing everytime each of them sends a frame.
Representation: This value must be represented as Unsigned Long Long with 8 bytes
Representation example:
struct.pack("!Q",seq_num)
TLSCompressed.type
Description: This field is extracted from TLS Record layer (the encrypted payload). For example, if it's an Application Data frame, we must use 0x17.
Representation: This value must be represented as Signed Char, with 2 bytes.
Representation example:
struct.pack("!b",TLSCompressed.type)
TLSCompressed.version
Description: This field is also extracted from TLS Record layer (the encrypted payload). For example, if the frame is transferred using TLS 1.2, we must use it's hex representation 0x0303.
Representation: This value must be represented as Unsigned Short, with 2 bytes.
Representation example:
struct.pack("!H",TLSCompressed.version)
TLSCompressed.length
Description: This field represents the actual length of the decrypted payload.
Representation: This value must be represented as Unsigned Short, with 2 bytes.
Representation example:
struct.pack("!H",TLSCompressed.length)
TLSCompressed.fragment
Description: This field **is the actual decrypted payload.
Representation: This value must be represented as a string
As a python example, the HMAC hashing will be as follows for our previous example:
hmac_digest = hmac.new(mac_secret,'',digestmod=hashlib.sha256)
hmac_digest.update(struct.pack('!QbHH',seq_num,TLSCompressed.type,TLSCompressed.version, len(decrypted)))
hmac_digest.update(decrypted)
hmac_digest.digest()

Categories