We Keep Coding

S3 InvalidDigest when calling the PutObject operation [duplicate]

I have tried to upload an XML File to S3 using boto3. As recommended by Amazon, I would like to send a Base64 Encoded MD5-128 Bit Digest(Content-MD5) of the data.
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.put
My Code:
with open(file, 'rb') as tempfile:
body = tempfile.read()
tempfile.close()
hash_object = hashlib.md5(body)
base64_md5 = base64.encodebytes(hash_object.digest())
response = s3.Object(self.bucket, self.key + file).put(
Body=body.decode(self.encoding),
ACL='private',
Metadata=metadata,
ContentType=self.content_type,
ContentEncoding=self.encoding,
ContentMD5=str(base64_md5)
)
When i try this the str(base64_md5) create a string like 'b'ZpL06Osuws3qFQJ8ktdBOw==\n''
In this case, I get this Error Message:
An error occurred (InvalidDigest) when calling the PutObject operation: The Content-MD5 you specified was invalid.
For Test purposes I copied only the Value without the 'b' in front: 'ZpL06Osuws3qFQJ8ktdBOw==\n'
Then i get this Error Message:
botocore.exceptions.HTTPClientError: An HTTP Client raised and unhandled exception: Invalid header value b'hvUe19qHj7rMbwOWVPEv6Q==\n'
Can anyone help me how to save Upload a File to S3?
Thanks,
Oliver

Starting with #Isaac Fife's example, stripping it down to identify what's required vs not, and to include imports and such to make it a full reproducible example:
(the only change you need to make is to use your own bucket name)
import base64
import hashlib
import boto3
contents = "hello world!"
md = hashlib.md5(contents.encode('utf-8')).digest()
contents_md5 = base64.b64encode(md).decode('utf-8')
boto3.client('s3').put_object(
Bucket="mybucket",
Key="test",
Body=contents,
ContentMD5=contents_md5
)
Learnings: first, the MD5 you are trying to generate will NOT look like what an 'upload' returns. We actually need a base64 version, it returns a md.hexdigest() version. hex is base16, which is not base64.

(Python 3.7)
Took me hours to figure this out because the only error you get is "The Content-MD5 you specified was invalid." Super useful for debugging... Anyway, here is the code I used to actually get the file to upload correctly before refactoring.
json_results = json_converter.convert_to_json(result)
json_results_utf8 = json_results.encode('utf-8')
content_md5 = md5.get_content_md5(json_results_utf8)
content_md5_string = content_md5.decode('utf-8')
metadata = {
"md5chksum": content_md5_string
}
s3 = boto3.resource('s3', config=Config(signature_version='s3v4'))
obj = s3.Object(bucket, 'filename.json')
obj.put(
Body=json_results_utf8,
ContentMD5=content_md5_string,
ServerSideEncryption='aws:kms',
Metadata=metadata,
SSEKMSKeyId=key_id)
and the hashing
def get_content_md5(data):
digest = hashlib.md5(data).digest()
return base64.b64encode(digest)
The hard part for me was figuring out what encoding you need at each step in the process and not being very familiar with how strings are stored in python at the time.
get_content_md5 takes a utf-8 bytes-like object only, and returns the same. But to pass the md5 hash to aws, it needs to be a string. You have to decode it before you give it to ContentMD5.
Pro-tip - Body on the other hand, needs to be given bytes or a seekable object. Make sure if you pass a seekable object that you seek(0) to the beginning of the file before you pass it to AWS or the MD5 will not match. For that reason, using bytes is less error prone, imo.

base64 HMAC with SHA256 in Python

I am having a hard time creating a signature.
I am needing to make a signature using HMAC with SHA256 using a Checkout Request JSON and a secret key. I need to do it by concatenating signature, pipe character (|) and Checkout Request JSON and then encoding it with BASE64.
This is a formula I found in the documentations:
$signed_checkout_request = base64( hmac_sha256( $checkout_request, $private_key ) + "|" + $checkout_request )
I have made this based on some online code:
import hashlib
import hmac
import base64
checkout_request = '{"charge":{"amount":499,"currency":"EUR"}}'.encode('utf-8');
private_key = b'44444444444';
digest = hmac.new(private_key, msg=checkout_request, digestmod=hashlib.sha256).digest()
signature = base64.b64encode(digest).decode()
However I am not sure how to get the "|" into it. I am also not sure if I am even on the right track if I am honest... I don't have much experience in this section and I have failed at googling.

private_key = 'blahblahblah'
checkout_request = json.dumps({"charge":{"amount":4999,"currency":"EUR"}}, sort_keys=True, separators=(",", ":"))
digest = hmac.new(private_key.encode(), msg=checkout_request.encode(), digestmod=hashlib.sha256,).hexdigest()
signature = base64.b64encode((digest + "|" + checkout_request).encode()).decode()
I was able to get it to work with that :)

Read JSON data from UTF-8 encoded byte string

I have a script that sends a JSON UTF-8 encoded Byte string to a socket. (A github project: https://github.com/alios/raildriver). Now I'm writing the python script that needs to read the incoming data. Right now I can receive the data and print it to the terminal. With the following script: https://www.binarytides.com/code-telnet-client-sockets-python/
Output:
data = '{"Current": 117.42609405517578, "Accelerometer": -5.394751071929932, "SpeedometerKPH": 67.12493133544922, "Ammeter": 117.3575210571289, "Amp": 117.35590362548828, "Acceleration": -0.03285316377878189, "TractiveEffort": -5.394751071929932, "Effort": 48.72163772583008, "RawTargetDistance": 3993.927734375, "TargetDistanceBar": 0.9777777791023254, "TargetDistanceDigits100": -1.0, "TargetDistanceDigits1000": -1.0}'
The problem is that I can't find how to read the JSON array. For example read "Ammeter" and return its value 117.357521057289 to a new variable.
All the data is being received in the variable data
The code I have right now:
decodedjson = data.decode('utf-8')
dumpedjson = json.dumps(decodedjson)
loadedjson = json.loads(dumpedjson)
Can you please help me?

You are encoding to JSON then decoding again. SImply not encode, remove the second line:
decodedjson = data.decode('utf-8')
loadedjson = json.loads(decodedjson)
If you are using Python 3.6 or newer, you don't actually have to decode from UTF-8, as the json.loads() function knows how to deal with UTF-encoded JSON data directly. The same applies to Python 2:
loadedjson = json.loads(data)
Demo using Python 3.7:
>>> data = b'{"Current": 117.42609405517578, "Accelerometer": -5.394751071929932, "SpeedometerKPH": 67.12493133544922, "Ammeter": 117.3575210571289, "Amp": 117.35590362548828, "Acceleration": -0.03285316377878189, "TractiveEffort": -5.394751071929932, "Effort": 48.72163772583008, "RawTargetDistance": 3993.927734375, "TargetDistanceBar": 0.9777777791023254, "TargetDistanceDigits100": -1.0, "TargetDistanceDigits1000": -1.0}'
>>> loadedjson = json.loads(data)
>>> loadedjson['Ammeter']
117.3575210571289

CryptoJS HmacSHA256 Encryption results differ from Python

CryptoJS encrypted string can pass different parameters in python can only pass a string? How to implement the second CryptoJS implementation in python
,how to get clientKey2,This will only give the first result.Thanks!
> saltedPassword=CryptoJS.PBKDF2("key", "salt", {keySize: 8,iterations:500,hasher: CryptoJS.algo.SHA256});
> clientKey1=CryptoJS.HmacSHA256(saltedPassword.toString(), "Client Key")
> clientKey2=CryptoJS.HmacSHA256(saltedPassword, "Client Key")
> clientKey1.toString()
> "857ef8988876a3bb6bcadb85ca257787074e73e830d7dc14c1f838ba46aef1f5"
> clientKey2.toString()
> "9a8574da9b276ee1162dcb92071df587f4513bc03060bda1e9b3897d46233416"
> saltedPassword.toString()
> "6e441ccd26e6b35198b4b17457dc0266d36b751d0062b5850b0e302ceb1d6dcc"
i use this way can get clientKey1,
import hashlib
import hmac
def HmacSHA256(k,v):
message = bytes(k).encode('utf-8')
secret = bytes(v).encode('utf-8')
signature = hmac.new(secret, message, digestmod=hashlib.sha256).hexdigest()
return signature
signature = HmacSHA256("6e441ccd26e6b35198b4b17457dc0266d36b751d0062b5850b0e302ceb1d6dcc","Client Key")
print signature
How to get the second result in Python,Thanks!

To get the desired clientKey2 you need to encode the hex digits of your saltedPassword string to bytes. One way to do that which works on both Python 2 & Python 3 is to use binascii.unhexlify.
Your HmacSHA256 function is a bit odd. It won't work on Python 3, since bytes objects don't have an .encode method. In Python 2, bytes is just a synonym for str.
Anyway, here's some code that works on both Python 2 & Python 3.
from __future__ import print_function
import hashlib
import hmac
import binascii
key = "Client Key".encode()
salted = "6e441ccd26e6b35198b4b17457dc0266d36b751d0062b5850b0e302ceb1d6dcc"
raw = binascii.unhexlify(salted)
signature = hmac.new(key, salted.encode(), digestmod=hashlib.sha256).hexdigest()
print(signature)
signature = hmac.new(key, raw, digestmod=hashlib.sha256).hexdigest()
print(signature)
output
857ef8988876a3bb6bcadb85ca257787074e73e830d7dc14c1f838ba46aef1f5
9a8574da9b276ee1162dcb92071df587f4513bc03060bda1e9b3897d46233416
The output is identical on Python 2 & Python 3.
BTW, it would be simpler to do this task in Python 3, which makes a clean distinction between text and byte strings. Also, the Python 3 hashlib module has a pbkdf2 function.

How to save email image attachment using Jython

I'm attempting to grab an image attached to an email using Jython 2.5.3. I get the email (using they Jython version of the Python imap library). I can get the attachment by looping through the parts, finding the correct part type using get_content_type():
image, img_ext = None, None
for part in self.mail.get_payload():
part_type, part_ext = part.get_content_type().split('/')
part_type = part_type.lower().strip()
part_ext = part_ext.lower().strip()
if part_type == 'image':
image = part.get_payload(decode=True)
img_ext = part_ext
return image, img_ext
'image' is returned as a big block of bytes, which in regular Python I'd write out directly to a file. However when I try the same thing in Jython I get the following error:
TypeError: write(): 1st arg can't be coerced to java.nio.ByteBuffer[], java.nio.ByteBuffer
What's the right way to make Jython recognize my big blob of data as a byte array?
PS: the writing code uses tempfile.mkstmp(), which defaults to writing binary...

For future readers, here's how I got around it. In the code tha does the writing:
from org.python.core.util import StringUtil
from java.nio import ByteBuffer
tmp, filename = tempfile.mkstemp(suffix = "." + extension, text=True)
bytes = StringUtil().toBytes(attachment)
bb = ByteBuffer.wrap(bytes)
tmp.write(bb)
tmp.close()