We Keep Coding

Upload file to Databricks DBFS with Python API

I'm following the Databricks example for uploading a file to DBFS (in my case .csv):
import json
import requests
import base64
DOMAIN = '<databricks-instance>'
TOKEN = '<your-token>'
BASE_URL = 'https://%s/api/2.0/dbfs/' % (DOMAIN)
def dbfs_rpc(action, body):
""" A helper function to make the DBFS API request, request/response is encoded/decoded as JSON """
response = requests.post(
BASE_URL + action,
headers={'Authorization': 'Bearer %s' % TOKEN },
json=body
)
return response.json()
# Create a handle that will be used to add blocks
handle = dbfs_rpc("create", {"path": "/temp/upload_large_file", "overwrite": "true"})['handle']
with open('/a/local/file') as f:
while True:
# A block can be at most 1MB
block = f.read(1 << 20)
if not block:
break
data = base64.standard_b64encode(block)
dbfs_rpc("add-block", {"handle": handle, "data": data})
# close the handle to finish uploading
dbfs_rpc("close", {"handle": handle})
When using the tutorial as is, I get an error:
Traceback (most recent call last):
File "db_api.py", line 65, in <module>
data = base64.standard_b64encode(block)
File "C:\Miniconda3\envs\dash_p36\lib\base64.py", line 95, in standard_b64encode
return b64encode(s)
File "C:\Miniconda3\envs\dash_p36\lib\base64.py", line 58, in b64encode
encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'
I tried doing with open('./sample.csv', 'rb') as f: before passing the blocks to base64.standard_b64encode but then getting another error:
TypeError: Object of type 'bytes' is not JSON serializable
This happens when the encoded block data is being sent into the API call.
I tried skipping encoding entirely and just passing the blocks into the post call. In this case the file gets created in the DBFS but has 0 bytes size.
At this point I'm trying to make sense of it all. It doesn't want a string but it doesn't want bytes either. What am I doing wrong? Appreciate any help.

In Python we have strings and bytes, which are two different entities note that there is no implicit conversion between them, so you need to know when to use which and how to convert when necessary. This answer provides nice explanation.
With the code snippet I see two issues:
This you already got - open by default reads the file as text. So your block is a string, while standard_b64encode expects bytes and returns bytes. To read bytes from file it needs to be opened in binary mode:
with open('/a/local/file', 'rb') as f:
Only strings can be encoded as JSON. There's no source code available for dbfs_rpc (or I can't find it), but apparently it expects a string, which it internally encodes. Since your data is bytes, you need to convert it to string explicitly and that's done using decode:
dbfs_rpc("add-block", {"handle": handle, "data": data.decode('utf8')})

Decoding Mail Subject Thunderbird in Python 3.x

For a workaround, see below
/Original Question:
Sorry, I am simply too dumb to solve this on my own. I am trying to read the "subjects" from several emails stored in a .mbox folder from Thunderbird. Now, I am trying to decode the header with decode_header(), but I am still getting UnicodeErrors.
I am using the following function (I am sure there is a smarter way to do this, but this is not the point of this post)
import mailbox
from email.header import decode_header
mflder = mailbox.mbox("mailfolder")
for message in mflder:
print(header_to_string(message["subject"]))
def header_to_string(header):
try:
header, encoding = decode_header(header)[0]
except:
return "something went wrong {}".format(header)
if encoding == None:
return header
else:
return header.decode(encoding)
The first 100 outputs or so are perfectly fine, but then this error message appears:
---------------------------------------------------------------------------
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-97-e252df04c215> in <module>
----> 1 for message in mflder:
2 try:
3 print(header_to_string(message["subject"]))
4 except:
5 print("0")
~\anaconda3\lib\mailbox.py in itervalues(self)
107 for key in self.iterkeys():
108 try:
--> 109 value = self[key]
110 except KeyError:
111 continue
~\anaconda3\lib\mailbox.py in __getitem__(self, key)
71 """Return the keyed message; raise KeyError if it doesn't exist."""
72 if not self._factory:
---> 73 return self.get_message(key)
74 else:
75 with contextlib.closing(self.get_file(key)) as file:
~\anaconda3\lib\mailbox.py in get_message(self, key)
779 string = self._file.read(stop - self._file.tell())
780 msg = self._message_factory(string.replace(linesep, b'\n'))
--> 781 msg.set_from(from_line[5:].decode('ascii'))
782 return msg
783
UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 4: ordinal not in range(128)
How can I force mailbox.py to decode a different encoding? Or is the header simply broken? And if I understood this correctly, headers are supposed to be "ASCII", right? I mean, this is the point of this entire MIME thing, no?
Thanks for your help!
/Workaround
I found a workaround by just avoiding to directly iterate over the .mbox mailfolder representation. Instead of using ...
for message in mflder:
# do something
... simply use:
for x in range(len(mflder)):
try:
message = mflder[x]
print(header_to_string(message["subject"]))
except:
print("Failed loading message!")
This skips the broken messages in the .mbox folder. Yet, I stumbled upon several other issues while working with the .mbox folder subjects. For instance, the headers are sometimes split into several tuples when using the decode_header() function. So, in order to receive the full subjects, one needs to add more stuff to the header_to_string() function as well. But this is not related to this question anymore. I am a noob and a hobby prgrammer, but I remember working with the Outlook API and Python, which was MUCH easier...

Solution
It looks like either you have corrupted "mailfolder" mbox file or there is a bug in Python's mailbox module triggered by something in your file. I can't tell what is going on without having the mbox input file or a minimal example input file that reproduces the issue.
You could do some debugging yourself. Each message in the file starts with a "From" line that should look like:
From - Mon Mar 30 18:18:04 2020
From the stack trace you posted, it looks like that line is malformed in one of the messages. Personally, I would use an IDE debugger (PyCharm) track down what the malformed line was, but it can be done with Python's built-in pdb. Wrap your loop like this:
import pdb
try:
for message in mflder:
print(header_to_string(message["subject"]))
except:
pdb.post_mortem()
When you run the code now, it will drop into the debugger when the exception occurs. At that prompt, you can enter l to list the code where the debugger stopped; this should match the last frame printed in your stack trace you originally posted. Once you are there, there are two commands that will tell you what is going on:
p from_line
will show you the malformed "From" line.
p start
will show you at what offset in the file the mailbox code thinks the message was supposed to be.
Previous answer that didn't solve the original problem but still applies
In the real world, there will be messages that don't comply with the standards. You can try to make the code more tolerant if you don't want to reject the bad messages. Decoding with "latin-1" is one way to handle these headers with bytes outside ASCII. This cannot fail because the all possible byte values map to valid Unicode characters (one-to-one mapping of the first 256 codes of Unicode vs. ISO/IEC 8859-1, a.k.a. "latin-1"). This may or may not give you the text the sender intended.
import mailbox
from email.header import decode_header
mflder = mailbox.mbox("mailfolder")
def get_subject(message):
header = message["subject"]
if not header:
return ''
header, encoding = decode_header(header)[0]
if encoding is not None:
try:
header = header.decode(encoding)
except:
header = header.decode('latin-1')
return header
for message in mflder:
print(get_subject(message))

Encode in Laravel, decode in Python

I'm using Laravel's encryptString method to encrypt some data on my website. This uses OpenSSL's 256-bit AES-CBC encryption without any serialization. I'm now trying to decrypt that data in Python but I keep getting errors about key length and can't seem to figure out why.
Example data to decrypt: eyJpdiI6ImdxY0VcLzFodmpISFV4allSWmJDdEpRPT0iLCJ2YWx1ZSI6IkxXd0ZJaUd2bTUweW5pNm0wUjQwOFM2N1wvWEs5SlYrNB4xNlR7Qkh1U3FvPSIsIm1hYyI6Ijc5ZWM0YTYxYjljZGFiNzgwNjY2NDU1ZmQ5Yjc1ZmJlOGU4NzBkMjQzMzA3MmVhYzE3NzY4ZmU1MWIyMjZlOTQifQ==
Example Key to use for decryption (from laravel .env):
base64:/AZejP0lh3McL/+Vy5yZcADdTcR65qnx5Jqinuw7raK=
I changed those values around so actually decrypting with those values won't give any real data, just figured it'd be good for example. I then try to decrypt this data in Python 3.7 with:
import base64
from Crypto.Cipher import AES
def decrypt(enc, key):
IV = 16 * '\x00'
decobj = AES.new(key, AES.MODE_CBC, IV)
data = decobj.decrypt(base64.b64decode(enc))
print(str(data.decode()))
if __name__ == "__main__":
key = b"/AZejP0lh3McL/+Vy5yZcADdTcR65qnx5Jqinuw7raK="
decrypt("eyJpdiI6ImdxY0VcLzFodmpISFV4allSWmJDdEpRPT0iLCJ2YWx1ZSI6IkxXd0ZJaUd2bTUweW5pNm0wUjQwOFM2N1wvWEs5SlYrNB4xNlR7Qkh1U3FvPSIsIm1hYyI6Ijc5ZWM0YTYxYjljZGFiNzgwNjY2NDU1ZmQ5Yjc1ZmJlOGU4NzBkMjQzMzA3MmVhYzE3NzY4ZmU1MWIyMjZlOTQifQ==", key)
And it seems like this should work, but when I run it I get the error: ValueError: Incorrect AES key length (60 bytes) so I'm not sure what I'm doing wrong. I've tried padding/unpadding the data/key but that doesn't seem to change anything. I'm wondering if I'm getting the wrong key to use for decryption from Laravel, but from what I could tell in the linked documentation, it should just be the APP_KEY in my .env file.
If anyone could help me or point me in the right direction, that would be amazing!
This question is unique to other similar questions because I'm trying to figure out primarily if I'm getting the correct AES key from Laravel, I don't actually overly need help decrypting, I just think I'm grabbing the wrong key from Laravel.
EDIT: New code that seems like it works:
import base64
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
def decrypt(enc, key):
IV = 16 * '\x00'.encode()
decobj = AES.new(key, AES.MODE_CBC, IV)
data = decobj.decrypt(pad(base64.b64decode(enc), 16))
print(base64.b64decode(data))
if __name__ == "__main__":
key = base64.b64decode(b"/AZejP0lh3McL/+Vy5yZcADdTcR65qnx5Jqinuw7raK=")
decrypt("eyJpdiI6ImdxY0VcLzFodmpISFV4allSWmJDdEpRPT0iLCJ2YWx1ZSI6IkxXd0ZJaUd2bTUweW5pNm0wUjQwOFM2N1wvWEs5SlYrNB4xNlR7Qkh1U3FvPSIsIm1hYyI6Ijc5ZWM0YTYxYjljZGFiNzgwNjY2NDU1ZmQ5Yjc1ZmJlOGU4NzBkMjQzMzA3MmVhYzE3NzY4ZmU1MWIyMjZlOTQifQ==", key)
The print statement now prints some bytes, but when I run .decode() on it I get the error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfa in position 0: invalid start byte and can't seem to figure out what I need to do to make it be able to be printed as a string.

Question: ...trying to decrypt that data in Python but I keep getting errors about key length
I can use your key, in the code of the linked answer, after doing .b64decode(....
The example code .encode(... and decode(... works as expecte.
Conclusion: There is nothing wrong, with your Key!
key = b"/AZejP0lh3McL/+Vy5yZcADdTcR65qnx5Jqinuw7raK="
key = base64.b64decode(key)
But with your code, I got TypeError, related to the IV parameter:
expect_byte_string(iv)
File "/usr/local/lib/python3.4/dist-packages/Crypto/Util/_raw_api.py", line 172, in expect_byte_string
TypeError: Only byte strings can be passed to C code
Fixed with IV = 16 * '\x00'.encode(), results in ValueError, related to enc:
data = decobj.decrypt(base64.b64decode(enc))
File "/usr/local/lib/python3.4/dist-packages/Crypto/Cipher/_mode_cbc.py", line 209, in decrypt
ValueError: Error 3 while decrypting in CBC mode
This leads to github issues:10
Error 3 means "ERR_NOT_ENOUGH_DATA"
According to the linked GitHub page, you have to reread the documentation, about padding data while you are encoding.
Working example from GitHub:
import base64
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
key = b"/AZejP0lh3McL/+Vy5yZcADdTcR65qnx5Jqinuw7raK="
key = base64.b64decode(key)
BLOCK_SIZE = 32
encryption_suite = AES.new(key, AES.MODE_CBC, b'This is an IV...')
cipher_text = encryption_suite.encrypt(pad(b'A really secret message...', BLOCK_SIZE))
decryption_suite = AES.new(key, AES.MODE_CBC, b'This is an IV...')
print(unpad(decryption_suite.decrypt(cipher_text), BLOCK_SIZE).decode())
>>> A really secret message...
Tested with Python: 3.4.2

Full Working Code with MAC Validation
Adding to Gonzalo's submission, Laravel's encrypted message is a base64 json-encoded array consisting of the following key pairs:
[
'iv' => 'generated initialization vector (iv)',
'value' => 'encrypted, base64ed, signed value',
'mac' => 'message authentication code (mac)'
]
The 'value' is signed using a message authentication code (MAC) to verify that the value has not changed during transit.
The MAC extracted from the payload (encrypted message) should be checked against the
mac extracted from the 'value' (this can be done using the key, iv, and value). Laravel's encryption scheme can be found on GitHub: src/Illuminate/Encryption/Encrypter.php
Referencing a discussion thread on Laravel, I traced a partial solution on Github: orian/crypt.py (which is a fork of fideloper/crypt.py).
I have forked off of Orian's code and fixed an input parameter issue. The code works as expected as long as the encryption key (passed in as an input to decrypt()) is base64 decoded and does not include the 'base64:' that is typically prepended to the APP_KEY environmental variable string assignment found in the .env file.
Solution: pilatuspc12ng/crypt.py
The code snippet of the crypt.py is shown below:
import base64
import json
from Crypto.Cipher import AES
from phpserialize import loads
import hashlib
import hmac
def decrypt(payload, key):
"""
Decrypt strings that have been encrypted using Laravel's encrypter (AES-256 encryption).
Plain text is encrypted in Laravel using the following code:
>>> ciphertext = Crypt::encrypt('hello world');
The ciphertext is a base64's json-encoded array consisting of the following keys:
[
'iv' => 'generated initialization vector (iv)',
'value' => 'encrypted, base64ed, signed value',
'mac' => 'message authentication code (mac)'
]
The 'value' is signed using a message authentication code (MAC) so verify that the value has not changed during
transit.
Parameters:
payload (str): Laravel encrypted text.
key (str): Encryption key (base64 decoded). Make sure 'base64:' has been removed from string.
Returns:
str: plaintext
"""
data = json.loads(base64.b64decode(payload))
if not valid_mac(key, data):
return None
value = base64.b64decode(data['value'])
iv = base64.b64decode(data['iv'])
return unserialize(mcrypt_decrypt(value, iv, key)).decode("utf-8")
def mcrypt_decrypt(value, iv, key):
AES.key_size=128
crypt_object=AES.new(key=key,mode=AES.MODE_CBC,IV=iv)
return crypt_object.decrypt(value)
def unserialize(serialized):
return loads(serialized)
def valid_mac(key, data):
dig = hmac.new(key, digestmod=hashlib.sha256)
dig.update(data['iv'].encode('utf8'))
dig.update(data['value'].encode('utf8'))
dig = dig.hexdigest()
return dig==data['mac']

I notice this haven't been active for a while but I'm trying to do the same and can't seem to make it work.
What I noticed is that the encoded password string stored by Laravel is a base 64 encoded json object, which wasn't taken into consideration in the original question:
pass_obj = base64.b64decode('eyJpdiI6ImdxY0VcLzFodmpISFV4allSWmJDdEpRPT0iLCJ2YWx1ZSI6IkxXd0ZJaUd2bTUweW5pNm0wUjQwOFM2N1wvWEs5SlYrNB4xNlR7Qkh1U3FvPSIsIm1hYyI6Ijc5ZWM0YTYxYjljZGFiNzgwNjY2NDU1ZmQ5Yjc1ZmJlOGU4NzBkMjQzMzA3MmVhYzE3NzY4ZmU1MWIyMjZlOTQifQ==')
print(pass_obj)
>>> b'{"iv":"gqcE\\/1hvjHHUxjYRZbCtJQ==","value":"LWwFIiGvm50yni6m0R408S67\\/XK9JV+4\x1e16T{BHuSqo=","mac":"79ec4a61b9cdab780666455fd9b75fbe8e870d2433072eac17768fe51b226e94"}'
From that you can get the IV and the encrypted value, both seem to be base64 encoded. But I still get a decoding error at the end like;
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfa in position 0: invalid start byte
Here is my complete code;
password = 'eyJpdiI6ImJGNDNNZjN3YWtpcDQ5VEJVXC9IazF3PT0iLCJ2YWx1ZSI6IkNVRW1VQUY1dXArYlFkU3NlY1pnZUE9PSIsIm1hYyI6ImM3ODk0NWQ0NjgxMzM4YjE0M2JhN2MzZWRmOWEwMWJiMjI2Y2FhYmUxYjFhYzAyYjY4YWZkZGE3N2EyMDYwNWYifQ=='
key = 'some secret key that i can't share'.encode()
p_obj = json.loads(base64.b64decode(password).decode())
decobj = AES.new(key, AES.MODE_CBC, base64.b64decode(p_obj['iv']))
data = decobj.decrypt(base64.b64decode(p_obj['value']))
print(data)
>>> b'l\xee:f\x9eZ\x90rP\x99\xca&#\x1d1\x9f'
data.decode()
>>> Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xee in position 1: invalid continuation byte
#Pecans did you ever figure this out?
Thank you.
Full working code EDIT
I fugured it out, I had a problem with my key initially. So here's the full snippet for future reference;
import base64
import json
from phpserialize import unserialize
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
key = b'my secret key'
enc_pass = 'eyJpdiI6ImJGNDNNZjN3YWtpcDQ5VEJVXC9IazF3PT0iLCJ2YWx1ZSI6IkNVRW1VQUY1dXArYlFkU3NlY1pnZUE9PSIsIm1hYyI6ImM3ODk0NWQ0NjgxMzM4YjE0M2JhN2MzZWRmOWEwMWJiMjI2Y2FhYmUxYjFhYzAyYjY4YWZkZGE3N2EyMDYwNWYifQ=='
p_obj = json.loads(base64.b64decode(password).decode())
decobj = AES.new(key, AES.MODE_CBC, base64.b64decode(p_obj['iv']))
data = decobj.decrypt(base64.b64decode(p_obj['value']))
dec_pass = unserialize(unpad(data, 16)).decode()
You will have the decrypted password in dec_pass.
Note that sometimes Laravel generates the key in base64. In that case the string will be something like base64:sdfsdjfhjsdf32, then you have to decode first.
Cheers!

read certificate(.crt) and key(.key) file in python

So i'm using the JIRA-Python module to connect to my company's instance on JIRA and it requires me to pass the certificate and key for this.
However using the OpenSSL module,i'm unable to read my local certificate and key to pass it along the request.
the code for reading is below
import OpenSSL.crypto
c = open('/Users/mpadakan/.certs/mpadakan-blr-mpsot-20160704.crt').read()
cert = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, c)
the error i get is
Traceback (most recent call last):
File "flaskApp.py", line 19, in <module>
cert = OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, c)
TypeError: must be X509, not str
could someone tell me how to read my local .crt and .key file into x509 objects?

#can-ibanoglu was right on:
import OpenSSL.crypto
cert = OpenSSL.crypto.load_certificate(
OpenSSL.crypto.FILETYPE_PEM,
open('/tmp/server.crt').read()
)
>>> cert
<OpenSSL.crypto.X509 object at 0x7f79906a6f50>

Which format in your .crt file.
Are there:
text starting with -----BEGIN CERTIFICATE-----
base64 text started with MI chars
binary data starting with \x30 byte?
In first two case there are PEM format, but in second one you are missing staring line, just add it to get correct PEM file or convert file to binary with base64 and get third case.
In third case you have DER format, so to load it you should use OpenSSL.crypto.FILETYPE_ASN1

IMAPClient - How to get subject and sender?

I'm trying to make a simple program to check and show unread messages, but I have problem while trying to get subject and sender adress.
For sender I've tried this method:
import email
m = server.fetch([a], ['RFC822'])
#a is variable with email id
msg = email.message_from_string(m[a], ['RFC822'])
print msg['from']
from email.utils import parseaddr
print parseaddr(msg['from'])
But it didn't work. I was getting this error:
Traceback (most recent call last):
File "C:/Users/ExampleUser/AppData/Local/Programs/Python/Python35-32/myprogram.py", line 20, in <module>
msg = email.message_from_string(m[a], ['RFC822'])
File "C:\Users\ExampleUser\AppData\Local\Programs\Python\Python35-32\lib\email\__init__.py", line 38, in message_from_string
return Parser(*args, **kws).parsestr(s)
File "C:\Users\ExampleUser\AppData\Local\Programs\Python\Python35-32\lib\email\parser.py", line 68, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
TypeError: initial_value must be str or None, not dict
I also used this:
print(server.fetch([a], ['BODY[HEADER.FIELDS (FROM)]']))
but the result was like:
defaultdict(<class 'dict'>, {410: {b'BODY[HEADER.FIELDS ("FROM")]': b'From: "=?utf-8?q?senderexample?=" <sender#example.com>\r\n\r\n', b'SEQ': 357}, 357: {b'SEQ': 357, b'FLAGS': (b'\\Seen',)}})
Is there a way to repair the first method, or make the result of second look like:
Sender Example <sender#example.com>
?
And I also don't know how to get email subject. But I guess it's the same as sender, but with other arguments. So the only thing I need are these arguments.

You should start by reviewing various IMAP libraries which are available for Python and use one which fits your needs. There are multiple ways of fetching the data you need in IMAP (the protocol), and by extension also in Python (and its libraries).
For example, the most straightforward way of getting the data you need in IMAP the protocol is through fetching the ENVELOPE object. You will still have to perform decoding of RFC2047 encoding of the non-ASCII data (that's that =?utf-8?q?... bit that you're seeing), but at least it would save you from parsing RFC5322 header structure with multiple decades of compatibility syntax rules.