We Keep Coding

Client Hello data printed out in ASCII

I am trying to extract some Client Hello information from a network packet.
I am actually printing some values just for testing.
def parse_client_hello(handshake):
if isinstance(handshake.data, dpkt.ssl.TLSClientHello):
client = dpkt.ssl.TLSClientHello(str(handshake.data))
print(client.random)
print(client.version)
The result is as the above :
Is the printed out data represented in ASCII?
Why is the data printed out in ASCII when in fact the captured packet is in binary?
How can I print it in another form , for ex. in hexadecimal?
Thank you!

New in Python version 3.5: bytes.hex():
...
print(client.random.hex())
print(client.version)
If you're on anything older than that, binascii.hexlify will work:
from binascii import hexlify
...
...
print(hexlify(client.random))
print(client.version)
Or a bilingual solution:
if hasattr(bytes, 'hex'):
b2h = bytes.hex
else:
from binascii import hexlify as b2h
...
...
print(b2h(client.random))
print(client.version)

Python imaplib: Display non-ASCII characters correctly

I am using Python 3.5 and imaplib to fetch an e-mail from GMail and print its body. The body contains non-ASCII characters.
These are 'encoded' in a strange way and I cannot find out how to fix this.
import email
import imaplib
c = imaplib.IMAP4_SSL('imap.gmail.com')
c.login('example#gmail.com', 'password')
c.select('Inbox')
_, data = c.fetch(b'12345', '(RFC822)')
mail = data[0][1]
message = email.message_from_bytes(mail)
payload = message.get_payload()
body = mail[0].as_string()
print(body)
Gives
>> ... Mit freundlichen Gr=C3=BC=C3=9Fen ...
instead of the desired
>> ... Mit freundlichen Grüßen ...
It looks to me like this is not an issue of encoding but one of conversion. But how do I tell Python to convert the characters correctly? Is there a more convenient library?

The text is encoded with quoted-printable encoding, which is a way to encode non-ascii characters in ascii text. You can decode it using python's quopri module.
>>> import quopri
>>> bs = b'Gr=C3=BC=C3=9Fen'
>>> # Decode quoted-printable to raw bytes.
>>> utf8 = quopri.decodestring(bs)
>>> # Decode bytes to text.
>>> s = utf8.decode('utf-8')
>>> print(s)
Grüßen
You may find that quoted-printable is the value of the email's content-transfer-encoding header.

CryptoJS HmacSHA256 Encryption results differ from Python

CryptoJS encrypted string can pass different parameters in python can only pass a string? How to implement the second CryptoJS implementation in python
,how to get clientKey2,This will only give the first result.Thanks!
> saltedPassword=CryptoJS.PBKDF2("key", "salt", {keySize: 8,iterations:500,hasher: CryptoJS.algo.SHA256});
> clientKey1=CryptoJS.HmacSHA256(saltedPassword.toString(), "Client Key")
> clientKey2=CryptoJS.HmacSHA256(saltedPassword, "Client Key")
> clientKey1.toString()
> "857ef8988876a3bb6bcadb85ca257787074e73e830d7dc14c1f838ba46aef1f5"
> clientKey2.toString()
> "9a8574da9b276ee1162dcb92071df587f4513bc03060bda1e9b3897d46233416"
> saltedPassword.toString()
> "6e441ccd26e6b35198b4b17457dc0266d36b751d0062b5850b0e302ceb1d6dcc"
i use this way can get clientKey1,
import hashlib
import hmac
def HmacSHA256(k,v):
message = bytes(k).encode('utf-8')
secret = bytes(v).encode('utf-8')
signature = hmac.new(secret, message, digestmod=hashlib.sha256).hexdigest()
return signature
signature = HmacSHA256("6e441ccd26e6b35198b4b17457dc0266d36b751d0062b5850b0e302ceb1d6dcc","Client Key")
print signature
How to get the second result in Python,Thanks!

To get the desired clientKey2 you need to encode the hex digits of your saltedPassword string to bytes. One way to do that which works on both Python 2 & Python 3 is to use binascii.unhexlify.
Your HmacSHA256 function is a bit odd. It won't work on Python 3, since bytes objects don't have an .encode method. In Python 2, bytes is just a synonym for str.
Anyway, here's some code that works on both Python 2 & Python 3.
from __future__ import print_function
import hashlib
import hmac
import binascii
key = "Client Key".encode()
salted = "6e441ccd26e6b35198b4b17457dc0266d36b751d0062b5850b0e302ceb1d6dcc"
raw = binascii.unhexlify(salted)
signature = hmac.new(key, salted.encode(), digestmod=hashlib.sha256).hexdigest()
print(signature)
signature = hmac.new(key, raw, digestmod=hashlib.sha256).hexdigest()
print(signature)
output
857ef8988876a3bb6bcadb85ca257787074e73e830d7dc14c1f838ba46aef1f5
9a8574da9b276ee1162dcb92071df587f4513bc03060bda1e9b3897d46233416
The output is identical on Python 2 & Python 3.
BTW, it would be simpler to do this task in Python 3, which makes a clean distinction between text and byte strings. Also, the Python 3 hashlib module has a pbkdf2 function.

Python, scrapy : bad utf8 characters writed in file from scraped html page with charset iso-8859-1

I want to scrap a webpage with charset iso-8859-1 with Scrapy, in python 2.7. The text i'm interesting in on the webpage is : tempête
Scrapy returns response as an UTF8 unicode with characters correctly encoded :
>>> response
u'temp\xc3\xaate'
Now, I want to write the word tempête in a file, so I'm doing the following :
>>> import codecs
>>> file = codecs.open('test', 'a', encoding='utf-8')
>>> file.write(response) #response is the above var
When I open the file, the resulting text is tempête. It seems that python does not detect proper encoding and can't read the two bytes encoded char and thinks it's two one-coded char.
How can I handle this simple use case ?

In your example, response is a (decoded) Unicode string with \xc3\xa inside, then something is wrong at scrapy encoding detection level.
\xc3\xa is character ê encoded as UTF-8, so you should only see those character for (encoded) non-Unicode/str strings (in Python 2 that is)
Python 2.7 shell session:
>>> # what your input should look like
>>> tempete = u'tempête'
>>> tempete
u'temp\xeate'
>>> # UTF-8 encoded
>>> tempete.encode('utf-8')
'temp\xc3\xaate'
>>>
>>> # latin1 encoded
>>> tempete.encode('iso-8859-1')
'temp\xeate'
>>>
>>> # back to your sample
>>> s = u'temp\xc3\xaate'
>>> print s
tempête
>>>
>>> # if you use a non-Unicode string with those characters...
>>> s_raw = 'temp\xc3\xaate'
>>> s_raw.decode('utf-8')
u'temp\xeate'
>>>
>>> # ... decoding from UTF-8 works
>>> print s_raw.decode('utf-8')
tempête
>>>
Something is wrong with Scrapy interpreting page as iso-8859-1 encoded.
You can force the encoding by re-building a response from response.body:
>>> import scrapy.http
>>> hr1 = scrapy.http.HtmlResponse(url='http://www.example', body='<html><body>temp\xc3\xaate</body></html>', encoding='latin1')
>>> hr1.body_as_unicode()
u'<html><body>temp\xc3\xaate</body></html>'
>>> hr2 = scrapy.http.HtmlResponse(url='http://www.example', body='<html><body>temp\xc3\xaate</body></html>', encoding='utf-8')
>>> hr2.body_as_unicode()
u'<html><body>temp\xeate</body></html>'
>>>
Build a new reponse
newresponse = response.replace(encoding='utf-8')
and work with newresponse instead

You need to encode your response as iso-8859-1 first and then decode (convert) it to utf-8 before writing to a file opened as utf-8
response = u'temp\xc3\xaate'
r1 = response.encode('iso-8859-1')
r2 = r1.decode('utf-8')
Interesting read: http://farmdev.com/talks/unicode/

URL Decode with Python 3

Is there a way to URL decode a string in Python 3
to take something like this
id%253D184ff84d27c3613d%26quality%3Dmedium
and decode it twice to get
id=184ff84d27c3613d&quality=medium

Just use urllib.parse.unquote():
>>> import urllib.parse
>>> urllib.parse.unquote('id%253D184ff84d27c3613d%26quality%3Dmedium')
'id%3D184ff84d27c3613d&quality=medium'
>>> urllib.parse.unquote('id%3D184ff84d27c3613d&quality=medium')
id=184ff84d27c3613d&quality=medium

Try this:
from urllib.parse import unquote
s = 'id%253D184ff84d27c3613d%26quality%3Dmedium'
unquote(unquote(s))
It will return:
> 'id=184ff84d27c3613d&quality=medium'