I want to apply an AES 128b encryption (probably CBC + Padding) on a data stream.
In case it matters, I'm sending chunks of around 1500bits each.
I work in Python, and I did a small test with M2Crypto with AES encrypt in one side and decrypt at the other side. It works perfect, but probably don't really secures anything since I use the same key, same IVS and all that.
So, the question is: What the best approach for AES encryption on large data streams?
I thought about loading a new 'keys' file from time to time. Then, the application will use this file to expend and extract AES keys or something like that, but it still sounds awful to build a new AES object for each chunk, so there must be a better way.
I believe I can also use the IVS here, but not quite sure where and how.
Related
I am trying to create a smart contract where I have to encrypt (and decrypt) data that I receive from a Python script running on a client. The data will be encrypted on the client-side, and I will have to decrypt it on the smart contract.
I got a smart contract code which can generate a key pair here. I am new to cryptography, and I tried looking up how I can encrypt plaintext using the public key and decrypt using the private key, but I am not getting a proper answer anywhere.
It will be great if someone can guide me to a relevant resource. Thank you.
Actually, this document does a pretty good job of going through the steps of using one particular Python elliptic curve library.
Note that you don't actually encrypt or decrypt large amounts of data using the public/private key. Instead, one uses ECC to create a shared secret key, and then that shared secret key is used with AES or whatever is your encryption algorithm of choice.
I've recently tried to create my own Vigenère encryption script for files. It works without issues with both encryption and decryption mechanisms. But it goes REALLY slow when it comes to "big files". For example, a 772 Ko file took 13.71s and it doesn't look linear as I would expect it to be. supposed it has to do with the fact that I try to read the entire document in a buffer.
the critical part is in only one line, for each byte in the buffer (f.read()) I add the value of the key corresponding. I think this is not optimal but I don't think of another method at the moment.
for i, v in enumerate(buf):
output_buffer += bytes([(v+m*key[i%keylength])%256])
"m" is just a parameter that is set to 1 for encryption and -1 for decryption.
The file buffer "buf" and key are bytes, not strings.
If you have any suggestion on how to work with the buffer (not reading and putting into ram the entire document) or with the code for the actual encryption I would be glad to know about it.
Thanks in advance.
I struggle to understand the correct use of an IV (Initialization Vector) when encrypting data in AES.
Precisely, I'm not sure where to store my randomly generated IV: in my script, the data will be encrypted, then saved to a file, then the program will terminate. During the next session, the previously saved data must be decrypted. If my understanding of IVs is correct, I have to use the same IV for decryption as for encryption (but another random-IV for every single encryption process).
Thus, I have to store the IV somewhere - some people recommend prepending it to the encrypted data, but if I get it right that won't work in my case, because I need the IV in order to be able to decrypt it.
Is this correct or did I misunderstand something? I want to avoid saving the encrypted/hashed key and the IV (even if hashed itself) inside some unencrypted plain-text-settings-file or something.
The IV itself is not sensitive data; its just used to scramble the state of the first cipher-text block and the scrambling is not recoverable from the IV itself (the key adds in the "secret" factor).
For "proper" chaining modes the IV is separate from the cipher text (you need the initial IV for both encryption and decryption) and must be stored separately and passed separately to the crypto library API. After encryption you can store the IV however you like - just don't lose it ;).
You can certainly "prepend" / "append" to the cipher text so you only have to store a single blob of data - but you'll just have to split it off prior to decryption as that is what the API will expect.
The "unproper" way to do an IV (e.g. if your crypto library API doesn't have native IV support, but does support chaining) is just to prepend a single block of random data to the plaintext before encrypting it. In this case there isn't any IV to store separately - you simply encrypt the entire IV+message binary pair - and then you simply delete the first block of data after decrypting it. The "random data" you prepend has the same constraints as a real IV (don't reuse the same random data with the same key, etc).
The two approaches are semantically different at the API level, but the effect on the actual encryption is the same (scamble the first block of real payload unpredictably).
In terms of how the IV is used - there are many possible schemes. See the wikipedia article on block chaining here for a convenient picture showing how the IV can be used in various chaining modes when it is really store separately.
https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation
I'm having a download application and I want to encrypt the links for file downloads, so that the user doesn't know the id of the file. Furthermore I'd like to include date/time in the link, and check when serving the file if the link is still valid.
There's a similar question asked here, but I'm running into problems with the character encodings, since I'd like to have urls like /file/encrypted_string/ pointing to the views for downloading, so best would be if the encrypted result only contains letters and numbers. I prefer not using a hash, because I do not want to store a mapping hash <> file somewhere. I do not know if there's a good encryption out there that fulfills my needs...
Sounds like it would be easy, especially if you don't mind using the same encryption key forever. Just delimit a string (/ or : works as well as anything) for the file name, the date/time, and anything else you want to include, then encrypt and b64 it! Remember to use urlsafe_b64encode, not the regular b64encode, which will produce broken urls. It'll be a long string, but so what?
I've done this a few times, using a slight variation: Add a few random characters as the last piece of the key and include that at the beginning or end of the string - more secure than always reusing the same key, without the headaches of a database mapping. As long as your key is complex enough the exposed bits won't be enough to let crackers generate requests at will.
Of course, if the file doesn't exist, don't let them see the decoded result...
By far the easiest way to handle this is to generate a random string for each file, and store a mapping between the key strings and the actual file name or file id. No complex encryption required.
Edit:
You will need to store the date anyway to implement expiring the links. So, you can store the expiration date, a long with the key, and periodically cull out expired links from the table.
If your problem is just one of encryption and decryption of short strings, Python's Crypto module makes it a breeze.
You can encode any character into the url, with django, you may use it's urlencode filter.
However, generating a random string and saving the mapping is more secure.
I need to be able to send encrypted data between a Ruby client and a Python server (and vice versa) and have been having trouble with the ruby-aes gem/library. The library is very easy to use but we've been having trouble passing data between it and the pyCrypto AES library for Python. These libraries seem to be fine when they're the only one being used, but they don't seem to play well across language boundaries. Any ideas?
Edit: We're doing the communication over SOAP and have also tried converting the binary data to base64 to no avail. Also, it's more that the encryption/decryption is almost but not exactly the same between the two (e.g., the lengths differ by one or there is extra garbage characters on the end of the decrypted string)
(e.g., the lengths differ by one or there is extra garbage characters on the end of the decrypted string)
I missed that bit. There's nothing wrong with your encryption/decryption. It sounds like a padding problem. AES always encodes data in blocks of 128 bits. If the length of your data isn't a multiple of 128 bits the data should be padded before encryption and the padding needs to be removed/ignored after encryption.
Turns out what happened was that ruby-aes automatically pads data to fill up 16 chars and sticks a null character on the end of the final string as a delimiter. PyCrypto requires you to do multiples of 16 chars so that was how we figured out what ruby-aes was doing.
It's hard to even guess at what's happening without more information ...
If I were you, I'd check that in your Python and Ruby programs:
The keys are the same (obviously). Dump them as hex and compare each byte.
The initialization vectors are the same. This is the parameter IV in AES.new() in pyCrypto. Dump them as hex too.
The modes are the same. The parameter mode in AES.new() in pyCrypto.
There are defaults for IV and mode in pyCrypto, but don't trust that they are the same as in the Ruby implementation. Use one of the simpler modes, like CBC. I've found that different libraries have different interpretations of how the mode complex modes, such as PTR, work.
Wikipedia has a great article about how block cipher modes.
Kind of depends on how you are transferring the encrypted data. It is possible that you are writing a file in one language and then trying to read it in from the other. Python (especially on Windows) requires that you specify binary mode for binary files. So in Python, assuming you want to decrypt there, you should open the file like this:
f = open('/path/to/file', 'rb')
The "b" indicates binary. And if you are writing the encrypted data to file from Python:
f = open('/path/to/file', 'wb')
f.write(encrypted_data)
Basically what Hugh said above: check the IV's, key sizes and the chaining modes to make sure everything is identical.
Test both sides independantly, encode some information and check that Ruby and Python endoded it identically. You're assuming that the problem has to do with encryption, but it may just be something as simple as sending the encrypted data with puts which throws random newlines into the data. Once you're sure they encrypt the data correctly, check that you receive exactly what you think you sent. Keep going step by step until you find the stage that corrupts the data.
Also, I'd suggest using the openssl library that's included in ruby's standard library instead of using an external gem.