We Keep Coding

Python email module behaves unexpected when trying to parse "raw" subject lines

I have trouble parsing an email which is encoded in win-1252 and contains the following header (literally like that in the file):
Subject: Счета на оплату по заказу . .
Here is a hexdump of that area:
000008a0 56 4e 4f 53 41 52 45 56 40 41 43 43 45 4e 54 2e |VNOSAREV#ACCENT.|
000008b0 52 55 3e 0d 0a 53 75 62 6a 65 63 74 3a 20 d1 f7 |RU>..Subject: ..|
000008c0 e5 f2 e0 20 ed e0 20 ee ef eb e0 f2 f3 20 ef ee |... .. ...... ..|
000008d0 20 e7 e0 ea e0 e7 f3 20 20 20 2e 20 20 2e 20 20 | ...... . . |
000008e0 20 20 0d 0a 58 2d 4d 61 69 6c 65 72 3a 20 4f 73 | ..X-Mailer: Os|
000008f0 74 72 6f 53 6f 66 74 20 53 4d 54 50 20 43 6f 6e |troSoft SMTP Con|
I realize that this encoding doesn't adhere to the usual RFC 1342 style encoding of =?charset?encoding?encoded-text?= but I assume that many email clients will still correctly display the subject and hence I would like to extract it correctly as well. For context: I am not making these emails up or creating them, they are given and I need to deal with them as is.
My approach so far was to use the email module that comes with Python:
import email
with open('data.eml', 'rb') as fp:
content = fp.read()
mail = email.message_from_bytes(content)
print(mail.get('subject'))
# ����� �� ������ �� ������ . .
print(mail.get('subject').encode())
# '=?unknown-8bit?b?0ffl8uAg7eAg7u/r4PLzIO/uIOfg6uDn8yAgIC4gIC4gICAg?='
My questions are:
can I somehow convince the email module to parse mails with subjects like this correctly?
if not, can I somehow access the "raw" data of this header? i.e. the entries of mail._headers without accessing private properties?
if not, can someone recommend a more versatile Python module for email parsing?
Some random observations:
a) Poking around in the internal data structure of mail, I arrived at [hd[1] for hd in mail._headers if hd[0] == 'Subject'] which is:
['\udcd1\udcf7\udce5\udcf2\udce0 \udced\udce0 \udcee\udcef\udceb\udce0\udcf2\udcf3 \udcef\udcee \udce7\udce0\udcea\udce0\udce7\udcf3 . . ']
b) According to the docs, mail.get_charsets() returns a list of character sets in case of multipart message, and it returns [None, 'windows-1251', None] here. So at least theoretically, the modules does have a chance to guessing the correct charset.
For completeness, the SHA256 has of the email file is 1aee4d068c2ae4996a47a3ae9c8c3fa6295a14b00d9719fb5ac0291a229b4038 (and I uploaded it to MalShare and VirusTotal).

The string you are seeing is just a normal unicode string which contains a lot of characters from the low surrogate range. I am quite sure that in this case, the string came about by using the .decode method with a surrogateescape error handler. Indeed:
In [1]: a = "Счета на оплату по заказу"
In [2]: a.encode("windows-1251").decode("utf8", "surrogateescape")
Out[2]: '\udcd1\udcf7\udce5\udcf2\udce0 \udced\udce0 \udcee\udcef\udceb\udce0\udcf2\udcf3 \udcef\udcee \udce7\udce0\udcea\udce0\udce7\udcf3'
To undo the damage, you should be able to use .encode("utf8", "surrogateescape").decode("windows-1251").
It is unclear to me whether they actually used utf8 with the surrogateescape handler, and you would have to match the charset that they (incorrectly) decode with. However, since the string matches yours perfectly, I think utf8 is what is being used.

mail.get_charsets() returns probably right values (with hard-coded the hexdump provided):
x = '56 4e 4f 53 41 52 45 56 40 41 43 43 45 4e 54 2e' + \
'52 55 3e 0d 0a 53 75 62 6a 65 63 74 3a 20 d1 f7' + \
'e5 f2 e0 20 ed e0 20 ee ef eb e0 f2 f3 20 ef ee' + \
'20 e7 e0 ea e0 e7 f3 20 20 20 2e 20 20 2e 20 20' + \
'20 20 0d 0a 58 2d 4d 61 69 6c 65 72 3a 20 4f 73' + \
'74 72 6f 53 6f 66 74 20 53 4d 54 50 20 43 6f 6e'
print(bytes.fromhex(x).decode('windows-1251'))
VNOSAREV#ACCENT.RU>
Subject: Счета на оплату по заказу . .
X-Mailer: OstroSoft SMTP Con

Your mail.get('subject').encode() does return exactly the bytes you put in. There is no "correctly" beyond this point; you have to know, or guess, the correct encoding.
mail.raw_items() returns what purports to be the "raw" headers from the message, but they are actually encoded. #Jesko's answer shows how to take the encoded value and transform it back to the original bytes, provided you know which encoding to use.
(The surrogate encoding is apparently a hack to allow Python to keep the raw bytes in a form which cannot accidentally leak back into a proper decoded string. You have to know how it was assembled and explicitly request it to be undone.)
Going out on a limb, you can try all the encodings of the body of the message, and check if any of them return a useful decoding.
The following uses the modern EmailMessage API where mail.get('subject').encode() no longer behaves like in your example (I think perhaps this is a bug?)
import email
from email.policy import default
content = b'''\
From: <VNOSAREV#ACCENT.example.RU>
Subject: \xd1\xf7\xe5\xf2\xe0 \xed\xe0 \xee\xef\xeb\xe0\xf2\xf3 \xef\xee \xe7\xe0\xea\xe0\xe7\xf3 . .
Content-type: text/plain; charset="windows-1251"
\xef\xf0\xe8\xe2\xe5\xf2
'''
# notice use of modern EmailMessage API, by specifying a policy
mail = email.message_from_bytes(content, policy=default)
# print(mail.get("subject"))
charsets = set(mail.get_charsets()) - {None}
for header, value in mail.raw_items():
if header == "Subject":
value = value.encode("utf-8", "surrogateescape")
for enc in charsets:
try:
print(value.decode(enc))
break
except (UnicodeEncodeError, UnicodeDecodeError):
pass
This crude heuristic could still misfire in a number of situations. If you know the encoding, probably hardcode it.
To the extent that mail clients are able to display the raw header correctly, I'm guessing it's mainly pure luck. If your system is set up to use code page 1251 by default, that probably helps some clients. Some mail clients also let you manually select an encoding for each message, so you can play around until you get the right one (and perhaps leave it at that setting if you receive many messages with this problem).

What's a less bad way to write this hex formatting function in Python?

With Python, I wanted to format a string of hex characters:
spaces between each byte (easy enough): 2f2f -> 2f 2f
line breaks at a specified max byte width (not hard): 2f 2f 2f 2f 2f 2f 2f 2f\n
address ranges for each line (doable): 0x7f8-0x808: 2f 2f 2f 2f 2f 2f 2f 2f\n
replace large ranges of sequential 00 bytes with: ... trimmed 35 x 00 bytes [0x7 - 0x2a] ... ... it was at this point that I knew I was doing some bad coding. The function got bloated and hard to follow. Too many features piled up in a non-intuitive way.
Example output:
0x0-0x10: 5a b6 f7 6e 7c 65 45 a0 bc 6a e5 f5 77 2b 92 48
0x10-0x20: 47 d7 33 ea 40 15 44 ac 6b a4 50 78 6e f2 10 d4
0x20-0x30: 9c 7c c1 f7 5a bf ec 9f b0 2b b7 29 97 ee 56 31
0x30-0x40: ff 23 d9 1a 0b 4e fd 65 50 92 42 eb b2 77 7a 55
0x40-0x50:
I'm pretty sure the address ranges aren't correct anymore in certain cases (particularly when the 00 replacement occurs), the function just looks disgusting, and I'm embarrassed to even show it.
def pretty_print_hex(hex_str, byte_width=16, line_start=False, addr=0):
out = ''
condense_min = 12
total_bytes = int(len(hex_str) / 2)
line_width = False
if byte_width is not False:
line_width = byte_width * 2
if line_start is not False:
out += line_start
end = addr + byte_width
if (end > addr + total_bytes):
end = addr + total_bytes
out += f"{hex(addr)}-{hex(end)}:\t"
addr += byte_width
i = 0
if len(hex_str) == 1:
print('Cannot pretty print < 1 byte', hex_str)
return
condensing = False
cond_start_addr = 0
cond_end_addr = 0
condense_cache = []
while i < len(hex_str):
byte = hex_str[i] + hex_str[i + 1]
i += 2
if byte == '00':
condensing = True
cond_start_addr = (addr - byte_width) + ((i + 1) % byte_width)
condense_cache.append(byte)
else:
if condensing is True:
condensed_count = len(condense_cache)
if condensed_count >= condense_min:
cond_end_addr = cond_start_addr + condensed_count
out += f"... trimmed {condensed_count} x 00 bytes [{hex(cond_start_addr)} - {hex(cond_end_addr)}] ..."
else:
for byte in condense_cache:
out += f"{byte} "
condense_cache = []
condensing = False
if condensing is False:
out += byte + ' '
if (line_width is not False) and (i) % line_width == 0:
out += '\n'
if line_start is not False:
out += line_start
end = addr + byte_width
if end > addr + total_bytes:
end = addr + total_bytes
if (addr - end) != 0:
out += f"{hex(addr)}-{hex(end)}:\t"
addr += byte_width
if condensing is True:
condensed_count = len(condense_cache)
if condensed_count >= condense_min:
cond_end_addr = cond_start_addr + condensed_count
out += f"... trimmed {condensed_count} x 00 bytes [{hex(cond_start_addr)} - {hex(cond_end_addr)}] ..."
else:
for byte in condense_cache:
out += f"{byte} "
return out.rstrip()
example input / output:
hex_str = 'c8d8fb631cc7d072b62aaf9cd47bc270d4341e35f23b7a94acf24f33397a6cb4145b6eacfd56653d79bea10d2842023155e5b14bec3b5851a0a58cb3a523c476b126486e1392bdd2e3bcb6cbc333b23de387ae8624123009'
byte_width=16
line_start='\t'
addr=0
print(pretty_print_hex(hex_str , byte_width=16, line_start='\t', addr=0))
0x0-0x10: c8 d8 fb 63 1c c7 d0 72 b6 2a af 9c d4 7b c2 70
0x10-0x20: d4 34 1e 35 f2 3b 7a 94 ac f2 4f 33 39 7a 6c b4
0x20-0x30: 14 5b 6e ac fd 56 65 3d 79 be a1 0d 28 42 02 31
0x30-0x40: 55 e5 b1 4b ec 3b 58 51 a0 a5 8c b3 a5 23 c4 76
0x40-0x50: b1 26 48 6e 13 92 bd d2 e3 bc b6 cb c3 33 b2 3d
0x50-0x60: e3 87 ae 86 24 12 30 09
It gets much worse when you involve some 00 replacement, here's an example of that:
hex_str = 'c8000000000000000000000000000aaf9cd47bc270d4341e35f23b7a94acf24f33397a6cb4145b6eacfd56653d79bea10d2842023155e5b14bec3b5851a0a58cb3a523c476b126486e1392bdd2e3bcb6cbc333b23de387ae8624123009'
byte_width=16
line_start='\t'
addr=0
print(pretty_print_hex(hex_str, byte_width=16, line_start='\t', addr=0))
0x0-0x10: c8 ... trimmed 13 x 00 bytes [0xd - 0x1a] ...0a af
0x10-0x20: 9c d4 7b c2 70 d4 34 1e 35 f2 3b 7a 94 ac f2 4f
0x20-0x30: 33 39 7a 6c b4 14 5b 6e ac fd 56 65 3d 79 be a1
0x30-0x40: 0d 28 42 02 31 55 e5 b1 4b ec 3b 58 51 a0 a5 8c
0x40-0x50: b3 a5 23 c4 76 b1 26 48 6e 13 92 bd d2 e3 bc b6
0x50-0x60: cb c3 33 b2 3d e3 87 ae 86 24 12 30 09
It would also make more sense to make the address range (`0x0-0x10) portray the true range, to include the trimmed bytes on that line, but I couldn't even begin to think of how to add that in.
Rather than patch this bad looking function, I thought I might ask for a better approach entirely, if one exists.

I would suggest to not start a "trimmed 00 bytes" series in the middle of an output line, but only apply this compacting when it applies to complete output lines with only zeroes.
This means that you will still see non-compacted zeroes in a line that also contains non-zeroes, but in my opinion this results in a cleaner output format. For instance, if a line would end with just two 00 bytes, it really does not help to replace that last part of the line with the longer "trimmed 2 x 00 bytes" message. By only replacing complete 00-lines with this message, and compress multiple such lines with one message, the output format seems cleaner.
To produce that output format, I would use the power of regular expressions:
to identify a block of bytes to be output on one line: either a line with at least one non-zero, or a range of zero bytes which either runs to the end of the input, or else is a multiple of the "byte width" argument.
to insert spaces in a line of bytes
All this can be done through iterations in one expression:
def pretty_print_hex(hex_str, byte_width=16, line_start='\t', addr=0):
return "\n".join(f"{hex(start)}-{hex(last)}:{line_start}{line}"
for start, last, line in (
(match.start() // 2, match.end() // 2 - 1,
f"...trimmed {(match.end() - match.start()) // 2} x 00 bytes..." if match[1]
else re.sub("(..)(?!$)", r"\1 ", match[0])
)
for match in re.finditer(
f"(0+$|(?:(?:00){{{byte_width}}})+)|(?:..){{1,{byte_width}}}",
hex_str
)
)
)

If you want to use it rather than write it (not sure - tell me to delete if required), you can use the excellent (I am not associated with it) hexdump:
https://pypi.org/project/hexdump
python -m hexdump binary.dat
It is super cool - I guess you could also inspect the source for ideas.
It doesn't, however, look like it is still maintained...

I liked the challenge in this function, and this is what I could come up with this evening. It is somewhat shorter than your original one, but not as short as trincot's answer.
def hexpprint(
hexstring: str,
width: int = 16,
hexsep: str = " ",
addr: bool = False,
addrstart: int = 0,
linestart: str = "",
compress: bool = False,
):
# if address get hex address length size
if addr:
addrlen = len(f"{addrstart+len(hexstring):x}")
# compression buffer just count hex 0 chars
cbuf = 0
for i in range(0, len(hexstring), width):
j = i + width
row = hexstring[i:j]
# if using compression and compressable
if compress and row.count("0") == len(row):
cbuf += len(row)
continue
# if not compressable and has cbuf, flush it
if cbuf:
line = linestart
if addr:
beg = f"0x{addrstart+i-cbuf:0{addrlen}x}"
end = f"0x{addrstart+i:0{addrlen}x}"
line += f"{beg}-{end} "
line += f"compressed {cbuf//2} NULL bytes"
print(line)
cbuf = 0
# print formatted hex row
line = linestart
if addr:
beg = f"0x{addrstart+i:0{addrlen}x}"
end = f"0x{addrstart+i+len(row):0{addrlen}x}"
line += f"{beg}-{end} "
line += hexsep.join(row[i : i + 2] for i in range(0, width, 2))
print(line)
# flush cbuf if necessary
if cbuf:
line = linestart
if addr:
beg = f"0x{addrstart+i-cbuf:0{addrlen}x}"
end = f"0x{addrstart+len(hexstring):0{addrlen}x}"
line += f"{beg}-{end} "
line += f"compressed {cbuf//2} NULL bytes"
print(line)
PS: I don't really like the code repetition to print things, so I might come back and edit later.

How to find out address in binary file with python code only?

I have binary for example https://github.com/andrew-d/static-binaries/blob/master/binaries/linux/x86_64/nmap
1) How to find what is the address of this series of bytes :48 8B 45 A8 48 8D 1C 02 48 8B 45 C8 ? , the result need to be 0x6B0C67
2)How to find out the 12 bytes that in address 0x6B0C67 ? the result need to be 48 8B 45 A8 48 8D 1C 02 48 8B 45 C8 .
3) How to find which address call to specific string? for example i + 1 == features[i].index that locate in 0x6FC272 ? the result need to be 0x4022F6
How can I find all of this without open Ida? only with python/c code?
thanks

For 1) Is your file small enough to be loaded into memory? Then it's as simple as
offset = open(file, 'rb').read().find(
bytes.fromhex("48 8B 45 A8 48 8D 1C 02 48 8B 45 C8")
)
# offset will be -1 if not found
If not, you will need to read it in chunks.
For 2), do
with open(file, 'rb') as stream:
stream.seek(0x6b0c67)
data = stream.read(12)
I'm afraid I don't understand the question in 3)...

Sum Hex Values using Python to get CheckSum

My question maybe is simple but i'm not good with bytes/hex operations. I need to do a checksum from a Serial Port data with this Values:
55 55 3A 0B 47 09 3E 08 FF 0F 93
The last value 93 is the sum value but i don't know how to do this.
55 + 55 + 3A + 0B + 47 + 09 + 3E + 08 + FF + 0F = 93

Convert the raw bytestring into a sequence of numbers, then add all but the last number, mask to byte-length, and compare the result with the last number in the sequence.
>>> data = bytearray('\x55\x55\x3a\x0b\x47\x09\x3e\x08\xff\x0f\x93')
>>> sum(data[:-1]) & 0xff == data[-1]
True

Telegram Api - Creating an Authorization Key 404 error

I am trying to write a simple program in python to use telegram api, (not bot api, main messaging api) Now i have written this code
#!/usr/bin/env python
import socket
import random
import time
import struct
import requests
def swap64(i):
return struct.unpack("<L", struct.pack(">L", i))[0]
MESSAGE = '0000000000000000'+format(swap32(int(time.time()*1000%1000)<<21|random.randint(0,1048575)<<3|4),'x')+format(swap32(int(time.time())),'x')+'140000007897466068edeaecd1372139bbb0394b6fd775d3'
res = requests.post(url='http://149.154.167.40',
data=bytes.fromhex(MESSAGE),
headers={'connection': 'keep-alive'})
print("received data:", res)
For payload of post data i used the source code of telegram web, The 0 auth id, message id is generated using the algo in telegram web, next is length (14000000) just like in the source and main doc and then the method and so on,
When i run this code i get received data: <Response [404]> i have used both tcp and http transport with this and tcp one gives me nothing as answer from server, i don't know where i'm wrong in my code
i would be glad if someone can show the error in my code
btw here is hex dump of my generated req:
0000 34 08 04 17 7a ec 48 5d 60 84 ba ed 08 00 45 00
0010 00 50 c6 07 40 00 40 06 76 28 c0 a8 01 0d 95 9a
0020 a7 28 c9 62 00 50 0d 1a 3b df 41 5a 40 7f 50 18
0030 72 10 ca 39 00 00 00 00 00 00 00 00 00 00 6c 28
0040 22 4a 94 a9 c9 56 14 00 00 00 78 97 46 60 68 ed
0050 ea ec d1 37 21 39 bb b0 39 4b 6f d7 75 d3
i have already read this and this and many other docs but cant find out my problem
thanks in advance
update
i used this code as suggested
TCP_IP = '149.154.167.40'
TCP_PORT = 80
MESSAGE = 'ef0000000000000000'+"{0:0{1}x}".format(int(time.time()*4294.967296*1000),16)+'140000007897466068edeaecd1372139bbb0394b6fd775d3'
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(bytes.fromhex(MESSAGE))
data = s.recv(BUFFER_SIZE)
s.close()
and i still get no response
hex dump of my request:
0000 34 08 04 17 7a ec 48 5d 60 84 ba ed 08 00 45 00
0010 00 51 e1 44 40 00 40 06 5a ea c0 a8 01 0d 95 9a
0020 a7 28 df 8c 00 50 e4 0d 12 46 e2 98 bf a3 50 18
0030 72 10 af 66 00 00 ef 00 00 00 00 00 00 00 00 00
0040 16 37 dc e1 28 39 23 14 00 00 00 78 97 46 60 68
0050 ed ea ec d1 37 21 39 bb b0 39 4b 6f d7 75 d3
Fixed code
Finally got it working with this code
import socket
import random
import time
import struct
import requests
def swap32(i):
return struct.unpack("<L", struct.pack(">L", i))[0]
TCP_IP = '149.154.167.40'
TCP_PORT = 80
z = int(time.time()*4294.967296*1000000)
z = format(z,'x')
q = bytearray.fromhex(z)
e = q[::-1].hex()
MESSAGE = 'ef0a0000000000000000'+e+'140000007897466068edeaecd1372139bbb0394b6fd775d3'
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(bytes.fromhex(MESSAGE))
data = s.recv(BUFFER_SIZE)
s.close()
print(data)

here is sample data from a simple TCP handshake with Telegram Servers:
Connect:Success:0
Connected to 149.154.167.40:443
raw_data: 000000000000000000F011DB3B2AA9561400000078974660A9729A4F5B51F18F7943F9C0D61B1315
auth_key_id: 0000000000000000 0
message_id: 56A92A3BDB11F000 6244568794892726272
data_length: 00000014 20
message_data: 78974660A9729A4F5B51F18F7943F9C0D61B1315
message_type: 60469778
>> EF0A000000000000000000F011DB3B2AA9561400000078974660A9729A4F5B51F18F7943F9C0D61B1315
Send:Success:42
Receive:Success:85
<< 15000000000000000001CC0CC93D2AA9564000000063241605A9729A4F5B51F18F7943F9C0D61B1315B4445B94718B3C6DD4136466FAC62DCD082311272BE9FF8F9700000015C4B51C01000000216BE86C022BB4C3
raw_data: 000000000000000001CC0CC93D2AA9564000000063241605A9729A4F5B51F18F7943F9C0D61B1315B4445B94718B3C6DD4136466FAC62DCD082311272BE9FF8F9700000015C4B51C01000000216BE86C022BB4C3
auth_key_id: 0000000000000000 0
message_id: 56A92A3DC90CCC01 6244568803180334081
data_length: 00000040 64
message_data: 63241605A9729A4F5B51F18F7943F9C0D61B1315B4445B94718B3C6DD4136466FAC62DCD082311272BE9FF8F9700000015C4B51C01000000216BE86C022BB4C3
message_type: 05162463
classid: resPQ#05162463
nonce: A9729A4F5B51F18F7943F9C0D61B1315
server_nonce: B4445B94718B3C6DD4136466FAC62DCD
pq: 2311272BE9FF8F97 2526843935494475671
count: 00000001 1
fingerprints: C3B42B026CE86B21 14101943622620965665
Lets break it down:
We are using the TCP abridged version, so we start off with 0xEF
The format for plain-text Telegram messages is auth_ke_id + msg_id + msg_len + msg
auth_key_id is always 0 for plain-text messages hence we always start with 0000000000000000
msg_id must approximately equal unixtime*2^32(see here) I have also seen that some variant of this works quite well for msg_id in any language on any platform: whole_part_of(current_micro_second_time_stamp * 4294.967296)
The first message you start with for Auth_key generation is reqPQ which is defined as: reqPQ#0x60469778 {:nonce, :int128} so it is simply a TL-header + a 128-bit random integer the total length will always be 4 + 16 = 20 encoded as little-endian that would be msg_len = 14000000
say we have a 128-bit random integer= 55555555555555555555555555555555, then our reqPQ message would be 7897466055555555555555555555555555555555, which is simply TL-type 60469778 or 78974660 in little-endian followed by your randomly chooses 128-bit nonce.
Before you send out the packet, again recall that TCP-abridged mode required you to include the total packet length in front of the other bytes just after the initial 0xEA . This packet length is computed as follows
let len = total_length / 4
a) If len < 127 then len_header = len as byte
b) If len >=127 then len_header = 0x7f + to_3_byte_little_endian(len)
finally we have:
EF0A000000000000000000F011DB3B2AA956140000007897466055555555555555555555555555555555
or
EF0A
0000000000000000
00F011DB3B2AA956
14000000
78974660
55555555555555555555555555555555
compared to yours:
0000000000000000
6C28224A94A9C956
14000000
78974660
68EDEAECD1372139BBB0394B6FD775D3
I would say, try using TCP-abriged mode by include the 0xEF starting bit and re-check your msg_id computation
cheers.