We Keep Coding

Send bytes encoded data over JSON

I'm working on an Python RESTful API and I need to send bytes-encoded data (specifically data encrypted using a public RSA key from the package rsa) over the network, via JSON forms.
Here is what it looks like :
>>> import rsa
>>> pubkey, privkey = rsa.newkeys(512, True) # Create a pair of public/private rsa keys ; 512 is for the example
>>> encStr = rsa.encrypt(b"Test string", pubkey) # Encrypt a bytes object using the public key
>>> encStr
b'r\x10\x03e\xc6*\xa8\xb1\xee\xbd\x18\x0f\x7f\xecz\xcex\xabP~\xb3]\x8f)R\x9b>i\x03\xab-m\x0c\x19\xd7\xa5f$\x07\xc1;X\x0b\xaa2\x99\xa8&\xfc/\x9f\x05!nk\x93%\xc0\xf5\x1d\xf8C\x1fo'
"encStr" is what I need to send, however, I can't tell what encoding it is, and the package documentation doesn't mention it. If you have any idea, please share it :)
>>> encStr.decode("utf-8")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec cannot decode byte 0x8e in position 0: invalid start byte
>>> encStr.decode("latin1")
'\x8eM\x96Æ\'zÈZ\x89\x85±\x98Z¯Ûzùæ¯;£zñ8\x9b§Ù\x9dÏ\x8eâ0®\x89ó(*?\x92ªg\x12ôsä\x1d\x96\x19\x82\x19-3\x15SBýh"3òÖß\x91Ô' # This could be it
>>> encStr.decode("latin1").encode("latin1")
b'\x8eM\x96\xc6\'z\xc8Z\x89\x85\xb1\x98Z\xaf\xdbz\xf9\xe6\xaf;\xa3z\xf18\x9b\xa7\xd9\x9d\xcf\x8e\xe20\xae\x89\xf3(*?\x92\xaag\x12\xf4s\xe4\x1d\x96\x19\x82\x19-3\x15SB\xfdh"3\xf2\xd6\xdf\x91\xd4' # Nop, garbage
After manipulating for a while, I found a way to get a proper string using base64.
>>> import base64
>>> b64_encStr = base64.b64encode(encStr)
>>> b64_encStr
b'jk2Wxid6yFqJhbGYWq/bevnmrzujevE4m6fZnc+O4jCuifMoKj+SqmcS9HPkHZYZghktMxVTQv1oIjPy1t+R1A=='
>>> b64_encStr.decode("utf-8")
'jk2Wxid6yFqJhbGYWq/bevnmrzujevE4m6fZnc+O4jCuifMoKj+SqmcS9HPkHZYZghktMxVTQv1oIjPy1t+R1A=='
Now I'd just have to send this, however, I would like to know if there is a more efficient way of doing this (shorter string ; less operations, considering the client has to encode and the server decode, etc).
Thanks !
Shawn

Base64 is a relatively efficient method of sending bytes as text (6 bits per character, or 8 bits per byte for normal single byte character encoding). These bytes may have any value, such as found in ciphertext. There are more efficient encodings such as basE91, but they only provide few advantages for the complexity that they bring.
However, I often see ciphertext being "stringified" while there is no need. Files, HTTP, sockets etc. all handle any byte values well. If you want to use it in a GET request then you should use base64url instead of the normal base 64 encoding. Often developers encode strings needlessly so that the values can be easily seen in traces and such, but in that case only the trace printout itself needs to be encoded.
Note that I'd advise you to use OAEP padding rather than PKCS#1 and a key size of at least 3072 bits, especially if you want to encrypt data that is transported rather than encrypted "in place".

Encryption not working properly

i have this piece of code for encryption.
from cryptography.fernet import Fernet
key = Fernet.generate_key()
f = Fernet(key)
token = f.encrypt(b"something cool")
k = f.decrypt(token)
print(k) `
This is the output
b'something cool'
According to the example on the website, that "b" should've gone. I'm very new at this and would like to know or understand how exactly the solution works.
Thanks

That ‘b’ means bytes. So instead of working with strings encryption algorythms are actually using bytes. My experience is that what you give a library (str/bytes/array) it should give you back, which Fernet is doing. I would simply convert the bytes back to a string k.decode(“utf-8”)

The encryption functions are doing what they should: bytes in and bytes out.
Cryptography and encryption work with bytes, not strings or other encoding, decrypt returns bytes. The actual low level decrypt has no idea of encodings, it can't the decryption could be a string, it could be an image, etc.
It is up to the caller to provide encodings in and out that are appropriate to the data being encrypted/decrypted.
As the caller wrap the encryption in a function you write that provides the correct encodings, in this case a string to bytes on encryption and bytes back to a string on decryption.

ECC key encoding in pyecc and seccure

I am working with pyecc which uses the seccure C library.
When a public key is generated I get something back like this:
#Gp}7RRWK5Dyg&-m5yHve1p{?<o0xi.M8-?W^]xb))oA]|qO%[5v?#IxteV?
Are these the 'raw bytes' or is this encoded in some form? When I use os.urandom(16) I get all kinds of messed up characters, but not from this generate public key function. Does that mean that there is some encoding? I've looked at the seccure source code, but I still don't understand why I get all these 'normal' characters.
How do I turn this into a byte array (Python bytearray) of the exact key?
Code:
from pyecc import ECC
ecc = ECC.generate()
print ecc._public

It looks like it might be a proprietary format from quickly looking at their code. They serialize and compress the key information into a printable string. You can download the source code and see an example of how to use it here: http://point-at-infinity.org/seccure/.

PyCrypto: Encode chinese charaters with RSA asymmetric key

I am trying to use PyCrypto to encrypt/decrypt some strings, and I am having troubles with Chinese characters.
When I try to encrypt "ni-hao" (hello)...
pemFile = open("/home/borrajax/keys/myKey.pem", "r")
encryptor = RSA.importKey(pemFile, passphrase="f00")
return encryptor.encrypt("你好", 0)[0]
... I keep getting errors:
Module Crypto.PublicKey.pubkey:64 in encrypt
>> ciphertext=self._encrypt(plaintext, K)
Module Crypto.PublicKey.RSA:92 in _encrypt
>> return (self.key._encrypt(c),)
ValueError: Plaintext too large
I have tried many combinations,
encryptor.encrypt(u"你好"...
encryptor.encrypt(u"你好".encode("utf-8")...
without any luck.
I guess I could always try to use base64 before encoding, but I'd like to leave that as a "last resource"... I was hoping for a more "elegant" way of doing this.
Has anyone encountered the same problems? Any hint will be appreciated. Thank you in advance.

First, you should be encrypting only binary data, not Unicode text. That means str type (in Python 2.x) or bytes (in Python 3.x and most recent Python 2.x). You must encode text before encrypting it, and you must decode it after you decrypt.
Second, the byte string you are encrypting must be smaller than the RSA modulus (e.g. less than 256 bytes for RSA2048). If your data is longer, think of using an intermediate AES session key.
Third, if you use PyCrypto 2.5, there is no good reason for using the .encrypt/.decrypt methods of the RSA key object. It is better and more secure to use one of the PKCS#1 methods: OAEP or v1.5. With them, the plaintext must be even shorter.

I tested with PyCrypto v2.5 installed from pip on Ubuntu Linux 10.04 on python 2.6.5 in the interactive interpreter from the yakuake terminal.
I'm not able to reproduce the error you're seeing, especially the "Plaintext too large" bit. Some errors I have seen:
encryptor.encrypt(u"你好",0)[0]
TypeError: argument 1 must be long, not unicode
Seems like it doesn't like unicode objects - only wants str.
Both of these work on my setup, and both produce the same output, however the first solution is more correct:
encryptor.encrypt(u"你好".encode("utf-8"), 0)[0]
encryptor.encrypt("你好", 0)[0]
Are you trying this from the interactive interpreter, or from a file? If file, is the file UTF-8 encoded? If console, does it have proper UTF-8 support?

I checked related codes of PyCrypto, this error is only thrown when plain-text (after converted to long) is bigger than one of the key parameter. Assuming encoding of the script is correctly set, it may be because your RSA key is invalid or too short. I tried this snippet, it works with no problem:
rsa = RSA.generate(1024)
print(rsa.encrypt("你好", 0))

Radix 64 and encryption

Need to share my problem that is :
A PGP public key server gives me the key in Radix64 format .
And i searching for any method which can encrypt my message using this Radix64 format public key .
any alternate suggestions or documents are welcome .........

exPyCrypto looks good.
This previous SO question addresses Radix64 format specifically for public keys.
To convert the actual base/radix64 encoded characters, see this question:
import base64
decoded_bytes = base64.b64decode(ascii_chars)

You can decode the key by using the base64 module and then encrypt the message.