We Keep Coding

Exchanging encrypted messages between python and swift

I need to have a python code and a swift code exchange encrypted message.
Here's what I tried:
Fernet
After a review of the options, I thought that a symetric key algorithm could work well.
In python (as usual), it is straightforward to encrypt and decrypt:
Fernet(key).encrypt(b"mdg") # encrypt
Fernet(key).decrypt(encryptedMsg) # decrypt
In swift, it seemed initially straightforward with something along the lines of:
func encrypt(key: String, msg: String) throws -> String {
let data = Data(base64URL: key)!
let symetricKey = try! SymmetricKey(data: d)
let msgUtf8 = msg.data(using: .utf8)!
let sealBox = try! AES.GCM.seal(msgUtf8, using: symetricKey, nonce: nil)
return sealBox.combined.base64EncodedString();
}
However, I have been unable to find the algorithm in swift matching python's Fernet.
ChaCha
While searching for the problem, I landed on this amazing answer from Bram. Very unfortunately it only solves one side of my problem : encrypting messages in python and decoding them in swift. I also need the reverse process.
How to solve this?

To start, we first need a way to create secure random values to generate the IV and keys. You can also generate the keys using CryptoKit's SymmetricKey and extract the data from them, but for now, I'll use this function.
extension Data {
static func secureRandom(ofSize size: Int) -> Data {
var output = [UInt8](repeating: 0, count: size)
_ = SecRandomCopyBytes(kSecRandomDefault, size, &output)
return Data(output)
}
}
We then require the possibility to compute the AES CBC ciphertext, which can be done using CommonCrypto.
func encrypt(plaintext: Data, key: Data, iv: Data) -> Data {
var encryptor: CCCryptorRef?
defer {
CCCryptorRelease(encryptor)
}
var key = Array(key)
var iv = Array(iv)
var plaintext = Array(plaintext)
CCCryptorCreate(CCOperation(kCCEncrypt), CCAlgorithm(kCCAlgorithmAES), CCOperation(kCCOptionPKCS7Padding), &key, key.count, &iv, &encryptor)
var outputBytes = [UInt8](repeating: 0, count: CCCryptorGetOutputLength(encryptor, plaintext.count, false))
CCCryptorUpdate(encryptor, &plaintext, plaintext.count, &outputBytes, outputBytes.count, nil)
var movedBytes = 0
var finalBytes = [UInt8](repeating: 0, count: CCCryptorGetOutputLength(encryptor, 0, true))
CCCryptorFinal(encryptor, &finalBytes, finalBytes.count, &movedBytes)
return Data(outputBytes + finalBytes[0 ..< movedBytes])
}
and the HMAC with the SHA-256 hash function. I recommend using CryptoKit's HMAC implementation here, but to keep things simple, I went with the CommonCrypto implementation.
func computeHMAC(_ data: Data, using key: Data) -> Data {
var data = Array(data)
var key = Array(key)
var macOut = [UInt8](repeating: 0, count: Int(CC_SHA256_DIGEST_LENGTH))
CCHmac(CCHmacAlgorithm(kCCHmacAlgSHA256), &key, key.count, &data, data.count, &macOut)
return Data(macOut)
}
This brings all of this together into the following
let plaintext = Data("Hello world!".utf8)
let signingKey = Data.secureRandom(ofSize: kCCKeySizeAES128)
let cryptoKey = Data.secureRandom(ofSize: kCCKeySizeAES128)
let fernetKey = (signingKey + cryptoKey).base64EncodedString()
let version: [UInt8] = [0x80]
let timestamp: [UInt8] = {
let timestamp = Int(Date().timeIntervalSince1970).bigEndian
return withUnsafeBytes(of: timestamp, Array.init)
}()
let iv = Data.secureRandom(ofSize: kCCBlockSizeAES128)
let ciphertext = encrypt(plaintext: plaintext, key: cryptoKey, iv: iv)
let hmac = computeHMAC(version + timestamp + iv + ciphertext, using: signingKey)
let fernetToken = (version + timestamp + iv + ciphertext + hmac).base64EncodedString()
print("Fernet key: \(fernetKey)")
print("Fernet token: \(fernetToken)")
An example output can be
Fernet key: 7EwFlYNKTGfj+2fSgL3AUqtrRqRs4D1TWNK7t2XbGJQ=
Fernet token: gAAAAABivCLM0y0poDtGOohT1yK4XTDJppYPJdu4fuDTZ5tb9P9KP5ACgX8aJq4imsSdbzOCcvY3Tueo4FYbwyG+ZugozILL+Q==
We can use this in python using cryptography.io's implementation
from cryptography.fernet import Fernet
key = b'7EwFlYNKTGfj+2fSgL3AUqtrRqRs4D1TWNK7t2XbGJQ='
token = b'gAAAAABivCLM0y0poDtGOohT1yK4XTDJppYPJdu4fuDTZ5tb9P9KP5ACgX8aJq4imsSdbzOCcvY3Tueo4FYbwyG+ZugozILL+Q=='
Fernet(key).decrypt(token)
# b'Hello world!'

nodejs crypto-js vs python sha256

I have a problem with crypto-js vs python sha256. It would like to write a nodejs client for duplicati. So I tried to port some python code to js.
https://github.com/Pectojin/duplicati-client/blob/master/auth.py#L136
Javascript
const CryptoJS = require('crypto-js');
function sha256(to_sign) {
var hash = CryptoJS.SHA256(to_sign.toString());
var hashInBase64 = CryptoJS.enc.Base64.stringify(hash);
return hashInBase64.toString('utf-8');
}
let salt = "ZAwQqEAAwR78oZOxFu0nVH2FLy/BnulVxhuu9IOnBwg="
let salt2 = "YQ=="
let password = "abc"
let saltedpwd = sha256(Buffer.concat([Buffer.from(password),Buffer.from(salt,'base64')]));
let saltedpwd2 = sha256(Buffer.concat([Buffer.from(password),Buffer.from(salt2,'base64')]));
let new_password = saltedpwd.toString('base64');
let new_password2 = saltedpwd2.toString('base64');
console.log(new_password)
console.log(new_password2)
returns:
pw1: 0udYFffMXd2QWW9dVXbFl3qp/6lnRcnspr4M1VEtgJA=
pw2: XD9nt/qE374RGDh8rRR5OSmEWlvHwAgMTYMJ03uqaNA=
Python
import base64
import hashlib
import sys
password = "abc"
salt = "ZAwQqEAAwR78oZOxFu0nVH2FLy/BnulVxhuu9IOnBwg="
salt2 = "YQ=="
salt_password = password.encode() + base64.b64decode(salt)
saltedpwd = hashlib.sha256(salt_password).digest()
print (base64.b64encode(saltedpwd).decode('utf-8'))
salt_password2 = password.encode() + base64.b64decode(salt2)
saltedpwd2 = hashlib.sha256(salt_password2).digest()
print (base64.b64encode(saltedpwd2).decode('utf-8'))
returns:
pw1: v9bAzxPatGzA2W7ORkraUvh+nyXotWXItAKpawGSo+A=
pw2: XD9nt/qE374RGDh8rRR5OSmEWlvHwAgMTYMJ03uqaNA=
as you can see, pw2 with the very simple base64 salt is both the same.
the salt from pw1 comes from the duplicati server, so I can't control it...
I've tried some many combinations of encoding, CryptoJS options, so I'm at the point where I will soon stop my 'project'... :(
Can you please give me any advice, what I'm doing wrong? I would be glad about any information.
Regards,
Benjamin

You're 99% of the way there, I think the fix is a one liner (isn't it so often!).
We just have to change
var hash = CryptoJS.SHA256(to_sign.toString());
to
var hash = CryptoJS.SHA256(CryptoJS.lib.WordArray.create(to_sign));
I believe this is because we want to convert directly from the buffer we created by concatenating the password and salt, rather than converting to a string, which is causing the wrong hash to be computed.
In any case, we get the same output as the Python code, which is what we want, ie
v9bAzxPatGzA2W7ORkraUvh+nyXotWXItAKpawGSo+A=
XD9nt/qE374RGDh8rRR5OSmEWlvHwAgMTYMJ03uqaNA=
The new code would look like this in Node.js:
const CryptoJS = require('crypto-js');
function sha256(to_sign) {
var hash = CryptoJS.SHA256(CryptoJS.lib.WordArray.create(to_sign));
var hashInBase64 = CryptoJS.enc.Base64.stringify(hash);
return hashInBase64.toString('utf-8');
}
let salt = "ZAwQqEAAwR78oZOxFu0nVH2FLy/BnulVxhuu9IOnBwg="
let salt2 = "YQ=="
let password = "abc"
let saltedpwd = sha256(Buffer.concat([Buffer.from(password),Buffer.from(salt,'base64')]));
let saltedpwd2 = sha256(Buffer.concat([Buffer.from(password),Buffer.from(salt2,'base64')]));
let new_password = saltedpwd.toString('base64');
let new_password2 = saltedpwd2.toString('base64');
console.log(new_password)
console.log(new_password2)

Making 700 API calls together and merging the data using AWS Lambda

I have a task wherein I have to make 700 REST API calls. I have used a loop, but the run time of the loop is more than the timeout of AWS Lambda service. Is there any way I can make the calls concurrently and merge the results. I am using node JS, but a solution in python is also welcome
Here is a sample of the code I am running now:
HitBTCData = {}
for item in exchanges:
itemlist = item.split('-')
response = requests.get('https://min-
api.cryptocompare.com/data/histominute?
fsym='+itemlist[0]+'&tsym='+itemlist[1]+'&limit=2000&aggregate=1
&e=HitBTC').json()
if itemlist[0]+itemlist[1] not in HitBTCData:
HitBTCData[itemlist[0]+itemlist[1]] = []
HitBTCData[itemlist[0]+itemlist[1]].append(response['Data'])
while response['Type'] != 1:
time = response['TimeFrom']
response = requests.get('https://min-
api.cryptocompare.com/data/histominute?
fsym='+itemlist[0]+'&tsym='+itemlist[1]+
'&limit=2000&aggregate=1&toTs='+str(time)+'&e=HitBTC').json()
if response['Type'] != 1:
HitBTCData[itemlist[0]+itemlist[1]]
.append(response['Data'])
HitBTCData[itemlist[0]+itemlist[1]] =
HitBTCData[itemlist[0]+itemlist[1]][::-1]

Use a loop in Node.js and the calls will run concurrently, but to make sure all of your calls are processed before exiting the lambda, you need to have some form of counter to make sure all of the queries have ended.
example:
var request = require('request');
var results = [];
var runningQueries = 0;
exchanges.forEach(function (item) {
++runningQueries;
callAPI (item, function (err, data) {
if (err) {
results.push(err)
--runningQueries;
} else {
results.push(data)
--runningQueries;
}
if (runningQueries == 0) {
// all call results have been collated, so do something
}
})
})
You'll need to build your own callAPI function. Also, I'm being a bit lazy with what I'm pushing to the array here, but it's just to give you a framework.

You could very easily do this with Promise.all():
const urls = [...];
const requestFunction = async (url) => {
// all your ifs go here
let data = await request(url);
// transform/sanitize your data
return data;
}
const allResults = await Promise.all(urls.map(requestFunction));
// you now have an array of results
The only downside to Promise.all() is that if any of the promises rejects, everything fails. So I would suggest catching errors and resolving with null or something similar.
You could do a for loop to generate your array of URLs, new Array(700).map(...), or since it looks like you already have some kind of array, just .map() that directly and do more transforms inside the request function.

Hashing algorithm Node js vs Python

I was trying to convert a hash algorithm which is written on Python to node.js
The python code looks something as
import uuid
import hashlib
import struct
CLIENT_ID = uuid.UUID('c5f92e0d-e762-32cd-98cb-8c546c410dbe')
SECRET = uuid.UUID('2cf26ff5-bd06-3245-becf-4d5a3baa704f')
data = CLIENT_ID.bytes_le + SECRET.bytes_le + struct.pack("I", 2017) + struct.pack("I", 9) + struct.pack("I", 2)
token = str(uuid.UUID(bytes_le=hashlib.sha256(data).digest()[0:16]))
The token generated is 32d86f00-eb49-2739-e957-91513d2b9969
Here the date values struct.pack values are generated using datetime but for convenient I have hard coded here.
I tried to convert the same by looking at the python doc for the respective libraries and did so far as
let CLIENT_ID = new Buffer('c5f92e0d-e762-32cd-98cb-8c546c410dbe');
let SECRET = new Buffer('2cf26ff5-bd06-3245-becf-4d5a3baa704f');
let d = new Buffer(2);
let m = new Buffer(9);
let y = new Buffer(2017);
let data = CLIENT_ID+SECRET+y+m+d;
const uuidv4 = require('uuid/v4');
const hash = crypto.createHash('sha256');
let token = uuidv4({random: hash.update(data, 'utf8').digest().slice(0, 16)}, 0);
And the hash it generates is b7b82474-eab4-4295-8318-cc258577ff9b
So, basically I am miserably missing something for the nodejs part.
Could you please guide me on where what went wrong. Thanks for the help

There's a lot of missed parts actually it tuned out.
###node parts:
new Buffer('c5')
does not represent <Buffer c5>, but <Buffer 63 35>.
To write c5 you would need to use Buffer.from([0xc5]) or Buffer.from([197]) (dec).
new Buffer(2)
does not represent <Buffer 02>, it just allocates 2 bytes.
CLIENT_ID+SECRET+y+m+d
concatenation of buffers does not work that way.
Use array of buffers and Buffer.concat([buffers]) to concatenate buffers.
###uuid parts:
it turned out that uuid operates modified version of buffers (bytes_le part in python code)
#the most interesting part:
in the python version of uuid, if no version argument is passed to uuid.UUID(...), uuid would generate an ID without fixing bits
According to the RFC-4122 4.4 uuid should fix that bits.
uuid.py skips RFC-4122 4.4
node-uuid/v4.js fixes required bits
that way even with the same results for sha256 hashing, the results between python and node implementation still would differ
python: 32d86f00-eb49-2739-e957-91513d2b9969
node: 32d86f00-eb49-4739-a957-91513d2b9969
^ ^
So, I see here 2 options
to pass version to python uuid (only for the last uuid call uuid.UUID(bytes_le=..., version=4)), that way python would return 32d86f00-eb49-4739-a957-91513d2b9969
if there's no way to change source code in python project, I guess there's an option to fork uuid and remove two lines of code in node-uuid/v4.js?
##See node version of your code below:
const uuidv4 = require('uuid/v4');
const crypto = require('crypto');
const hash = crypto.createHash('sha256');
const client_id_hex_str = "c5f92e0d-e762-32cd-98cb-8c546c410dbe".replace(/-/g, "");
const secret_hex_str = "2cf26ff5-bd06-3245-becf-4d5a3baa704f".replace(/-/g, "");
let CLIENT_ID = Buffer.from(to_bytes_le(to_bytes(client_id_hex_str, null, 16, 'big')));
let SECRET = Buffer.from(to_bytes_le(to_bytes(secret_hex_str, null, 16, 'big')));
let d = Buffer.from(to_bytes(null, 2, 4));
let m = Buffer.from(to_bytes(null, 9, 4));
let y = Buffer.from(to_bytes(null, 2017, 4));
let data = Buffer.concat([CLIENT_ID, SECRET, y, m, d]);
let hashBytes = hash.update(data, 'utf8').digest().slice(0, 16);
hashBytes = [].slice.call(hashBytes, 0);
hashBytes = Buffer.from(to_bytes_le(hashBytes));
let token = uuidv4({random: hashBytes});
console.log(token);
// https://stackoverflow.com/questions/16022556/has-python-3-to-bytes-been-back-ported-to-python-2-7
function to_bytes(hexString, number, length, endianess) {
if (hexString == null && number == null) {
throw new Error("Missing hex string or number.");
}
if (!length || isNaN(length)) {
throw new Error("Missing or invalid bytes array length number.");
}
if (hexString && typeof hexString != "string") {
throw new Error("Invalid format for hex value.");
}
if (hexString == null) {
if (isNaN(number)) {
throw new Error("Invalid number.");
}
hexString = number.toString(16);
}
let byteArray = [];
if (hexString.length % 2 !== 0) {
hexString = '0' + hexString;
}
const bitsLength = length * 2
hexString = ("0".repeat(bitsLength) + hexString).slice(-1 * bitsLength);
for (let i = 0; i < hexString.length; i += 2) {
const byte = hexString[i] + hexString[i + 1];
byteArray.push(parseInt(byte, 16));
}
if (endianess !== "big") {
byteArray = byteArray.reverse();
}
return byteArray;
}
// https://github.com/python/cpython/blob/master/Lib/uuid.py#L258
function to_bytes_le(bytes) {
const p1 = bytes.slice(0, 4).reverse();
const p2 = bytes.slice(4, 6).reverse();
const p3 = bytes.slice(6, 8).reverse();
const p4 = bytes.slice(8);
const bytes_le = [].concat.apply([], [p1, p2, p3, p4]);
return bytes_le;
}

Do you want the hashing of the data to be the same as the Python code above?
If not, you can take a look at the the Sha256 module below in NodeJS
https://www.npmjs.com/package/sha256

Python convert C header file to dict

I have a C header file which contains a series of classes, and I'm trying to write a function which will take those classes, and convert them to a python dict. A sample of the file is down the bottom.
Format would be something like
class CFGFunctions {
class ABC {
class AA {
file = "abc/aa/functions"
class myFuncName{ recompile = 1; };
};
class BB
{
file = "abc/bb/functions"
class funcName{
recompile=1;
}
}
};
};
I'm hoping to turn it into something like
{CFGFunctions:{ABC:{AA:"myFuncName"}, BB:...}}
# Or
{CFGFunctions:{ABC:{AA:{myFuncName:"string or list or something"}, BB:...}}}
In the end, I'm aiming to get the filepath string (which is actually a path to a folder... but anyway), and the class names in the same class as the file/folder path.
I've had a look on SO, and google and so on, but most things I've found have been about splitting lines into dicts, rather then n-deep 'blocks'
I know I'll have to loop through the file, however, I'm not sure the most efficient way to convert it to the dict.
I'm thinking I'd need to grab the outside class and its relevant brackets, then do the same for the text remaining inside.
If none of that makes sense, it's cause I haven't quite made sense of the process myself haha
If any more info is needed, I'm happy to provide.
The following code is a quick mockup of what I'm sorta thinking...
It is most likely BROKEN and probably does NOT WORK. but its sort of the process that I'm thinking of
def get_data():
fh = open('CFGFunctions.h', 'r')
data = {} # will contain final data model
# would probably refactor some of this into a function to allow better looping
start = "" # starting class name
brackets = 0 # number of brackets
text= "" # temp storage for lines inside block while looping
for line in fh:
# find the class (start
mt = re.match(r'Class ([\w_]+) {', line)
if mt:
if start == "":
start = mt.group(1)
else:
# once we have the first class, find all other open brackets
mt = re.match(r'{', line)
if mt:
# and inc our counter
brackets += 1
mt2 = re.match(r'}', line)
if mt2:
# find the close, and decrement
brackets -= 1
# if we are back to the initial block, break out of the loop
if brackets == 0:
break
text += line
data[start] = {'tempText': text}
====
Sample file
class CfgFunctions {
class ABC {
class Control {
file = "abc\abc_sys_1\Modules\functions";
class assignTracker {
description = "";
recompile = 1;
};
class modulePlaceMarker {
description = "";
recompile = 1;
};
};
class Devices
{
file = "abc\abc_sys_1\devices\functions";
class registerDevice { recompile = 1; };
class getDeviceSettings { recompile = 1; };
class openDevice { recompile = 1; };
};
};
};
EDIT:
If possible, if I have to use a package, I'd like to have it in the programs directory, not the general python libs directory.

As you detected, parsing is necessary to do the conversion. Have a look at the package PyParsing, which is a fairly easy-to-use library to implement parsing in your Python program.
Edit: This is a very symbolic version of what it would take to recognize a very minimalistic grammer - somewhat like the example at the top of the question. It won't work, but it might put you in the right direction:
from pyparsing import ZeroOrMore, OneOrMore, \
Keyword, Literal
test_code = """
class CFGFunctions {
class ABC {
class AA {
file = "abc/aa/functions"
class myFuncName{ recompile = 1; };
};
class BB
{
file = "abc/bb/functions"
class funcName{
recompile=1;
}
}
};
};
"""
class_tkn = Keyword('class')
lbrace_tkn = Literal('{')
rbrace_tkn = Literal('}')
semicolon_tkn = Keyword(';')
assign_tkn = Keyword(';')
class_block = ( class_tkn + identifier + lbrace_tkn + \
OneOrMore(class_block | ZeroOrMore(assignment)) + \
rbrace_tkn + semicolon_tkn \
)
def test_parser(test):
try:
results = class_block.parseString(test)
print test, ' -> ', results
except ParseException, s:
print "Syntax error:", s
def main():
test_parser(test_code)
return 0
if __name__ == '__main__':
main()
Also, this code is only the parser - it does not generate any output. As you can see in the PyParsing docs, you can later add the actions you want. But the first step would be to recognize the what you want to translate.
And a last note: Do not underestimate the complexities of parsing code... Even with a library like PyParsing, which takes care of much of the work, there are many ways to get mired in infinite loops and other amenities of parsing. Implement things step-by-step!
EDIT: A few sources for information on PyParsing are:
http://werc.engr.uaf.edu/~ken/doc/python-pyparsing/HowToUsePyparsing.html
http://pyparsing.wikispaces.com/
(Particularly interesting is http://pyparsing.wikispaces.com/Publications, with a long list of articles - several of them introductory - on PyParsing)
http://pypi.python.org/pypi/pyparsing_helper is a GUI for debugging parsers
There is also a 'tag' Pyparsing here on stackoverflow, Where Paul McGuire (the PyParsing author) seems to be a frequent guest.
* NOTE: *
From PaulMcG in the comments below: Pyparsing is no longer hosted on wikispaces.com. Go to github.com/pyparsing/pyparsing