We Keep Coding

S3 InvalidDigest when calling the PutObject operation [duplicate]

I have tried to upload an XML File to S3 using boto3. As recommended by Amazon, I would like to send a Base64 Encoded MD5-128 Bit Digest(Content-MD5) of the data.
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.put
My Code:
with open(file, 'rb') as tempfile:
body = tempfile.read()
tempfile.close()
hash_object = hashlib.md5(body)
base64_md5 = base64.encodebytes(hash_object.digest())
response = s3.Object(self.bucket, self.key + file).put(
Body=body.decode(self.encoding),
ACL='private',
Metadata=metadata,
ContentType=self.content_type,
ContentEncoding=self.encoding,
ContentMD5=str(base64_md5)
)
When i try this the str(base64_md5) create a string like 'b'ZpL06Osuws3qFQJ8ktdBOw==\n''
In this case, I get this Error Message:
An error occurred (InvalidDigest) when calling the PutObject operation: The Content-MD5 you specified was invalid.
For Test purposes I copied only the Value without the 'b' in front: 'ZpL06Osuws3qFQJ8ktdBOw==\n'
Then i get this Error Message:
botocore.exceptions.HTTPClientError: An HTTP Client raised and unhandled exception: Invalid header value b'hvUe19qHj7rMbwOWVPEv6Q==\n'
Can anyone help me how to save Upload a File to S3?
Thanks,
Oliver

Starting with #Isaac Fife's example, stripping it down to identify what's required vs not, and to include imports and such to make it a full reproducible example:
(the only change you need to make is to use your own bucket name)
import base64
import hashlib
import boto3
contents = "hello world!"
md = hashlib.md5(contents.encode('utf-8')).digest()
contents_md5 = base64.b64encode(md).decode('utf-8')
boto3.client('s3').put_object(
Bucket="mybucket",
Key="test",
Body=contents,
ContentMD5=contents_md5
)
Learnings: first, the MD5 you are trying to generate will NOT look like what an 'upload' returns. We actually need a base64 version, it returns a md.hexdigest() version. hex is base16, which is not base64.

(Python 3.7)
Took me hours to figure this out because the only error you get is "The Content-MD5 you specified was invalid." Super useful for debugging... Anyway, here is the code I used to actually get the file to upload correctly before refactoring.
json_results = json_converter.convert_to_json(result)
json_results_utf8 = json_results.encode('utf-8')
content_md5 = md5.get_content_md5(json_results_utf8)
content_md5_string = content_md5.decode('utf-8')
metadata = {
"md5chksum": content_md5_string
}
s3 = boto3.resource('s3', config=Config(signature_version='s3v4'))
obj = s3.Object(bucket, 'filename.json')
obj.put(
Body=json_results_utf8,
ContentMD5=content_md5_string,
ServerSideEncryption='aws:kms',
Metadata=metadata,
SSEKMSKeyId=key_id)
and the hashing
def get_content_md5(data):
digest = hashlib.md5(data).digest()
return base64.b64encode(digest)
The hard part for me was figuring out what encoding you need at each step in the process and not being very familiar with how strings are stored in python at the time.
get_content_md5 takes a utf-8 bytes-like object only, and returns the same. But to pass the md5 hash to aws, it needs to be a string. You have to decode it before you give it to ContentMD5.
Pro-tip - Body on the other hand, needs to be given bytes or a seekable object. Make sure if you pass a seekable object that you seek(0) to the beginning of the file before you pass it to AWS or the MD5 will not match. For that reason, using bytes is less error prone, imo.

ValueError("Expected: ASCII-armored PGP data") when using pgp_key.from_blob(key_string)

I am getting ValueError("Expected: ASCII-armored PGP data") when using pgp_key.from_blob(key_string) when trying to parse the key.
pgp_key = pgpy.PGPKey()
key = pgp_key.from_blob(key_string);
I tried using parse method as well but getting the same error.

I fixed this error by:
With your key as a file, run base64 /path/to/file_name new_encoded_file_name
Put your encoded key in your desired place (AWS Secrets Manager in my case)
Within your program, add the following line BEFORE getting your pgp key:
key_string = base64.decode(key_string)
Now key = pgp_key.from_blob(key_string) will no longer throw an error as the decoded string will be an ASCII-armored bytearray.

xml signature verification against custom CA

I need to verify an xml signature contained in the answer to a POST request.
The signature is defined by:
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
<SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
<Reference URI="">
<Transforms>
<Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
<Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
<DigestValue>htti3M3ikfm2RooDTNo3Kv7g0K2ongShUfCDUAWpytc=</DigestValue>
</Reference>
</SignedInfo>
and is against a custom CA.
I have no way to change the answer (it is issued by a State Agency) and all my attempts to verify against the certificate supposedly used to sign did result in errors.
One of my attempts has been using the following short test program:
#!/usr/bin/python3
import signxml
cf = 'certificate.cer'
with open(cf, 'r') as fi:
cer = fi.read()
ver = signxml.XMLVerifier()
f = 'response.xml'
with open(f, 'rb') as fi:
xml = fi.read()
try:
vd = ver.verify(xml, x509_cert=cer)
print('OK')
except signxml.exceptions.InvalidSignature as e:
print(e)
This results in:
Signature verification failed: wrong signature length
Other variations have different errors including:
Signature verification failed: invalid padding
and:
unable to get local issuer certificate
I am a seasoned programmer, but NOT a cryptography expert, so it's quite likely I forgot something trivial (to the knowledgeable).
Please point me in the right direction.
Note:: if required I can provide a full example (of failure) as certificate/answer are not "secret".

Unfortunately things are always a bit more complex than expected.
Answer from #stovfl completely missed the relevant point: I need to verify against a non-standard CA.
I already was struggling to use sigxml package, and I had to overcome the following problems:
sigxml will not work (for me) with plain xml.etree; I had to use lxml.etree with a different syntax.
sigxml installed by plain pip3 install sigxml would bomb with a deprecation error; I had to get latest (non-tagged master) from github with pip3 install git+https://github.com/XML-Security/signxml.git.
Examples on sigxml site completely disregard python3 issues with str vs. bytes.
CA Authority used to sign the Certificate I used to sign the outgoing message is different from the CA Authority used to sign response (I have no way to change this!).
This story has an Happy Ending though.
The following test program works (for me):
from signxml import XMLSigner, XMLVerifier, InvalidCertificate
from lxml import etree
outCAroot = 'outCAroot.pem'
inCAroot = 'inCAroot.pem'
cert = open("example.pem").read().encode()
key = open("example.key").read().encode()
file_name = 'test.tosend'
resp_name = 'test.rsp'
xml = etree.parse(file_name) # (data_to_sign)
signer = XMLSigner(c14n_algorithm='http://www.w3.org/TR/2001/REC-xml-c14n-20010315')
signed_xml = signer.sign(xml, key=key, cert=cert)
try:
result = XMLVerifier().verify(signed_xml, ca_pem_file=outCAroot)
except InvalidCertificate as e:
print(e)
else:
print('outgoing signature Ok.')
# here I send signed_xml to remote server and get the response (NO ERRORS!)
answer_xml = etree.parse(resp_name) # (signed answer)
try:
result = XMLVerifier().verify(answer_xml, ca_pem_file=inCAroot)
except InvalidCertificate as e:
print(e)
else:
print('incoming signature Ok.')
print('===================')
print(result.signed_data.decode())
print('===================')
I hope this will help whoever is (or will be) in my situation.

Question: I need to verify an xml signature contained in the answer to a POST request.
Do it, like the example in the Documentation SignXML: XML Signature in Python:
SignXML uses the ElementTree API (also supported by lxml) to work with XML data.
from signxml import XMLSigner, XMLVerifier
from xml.etree import ElementTree
cert = open("example.pem").read()
key = open("example.key").read()
xml = ElementTree.parse(file_name) #(data_to_sign)
signed_xml = XMLSigner().sign(xml, key=key, cert=cert)
result = XMLVerifier().verify(signed_xml)
XMLVerifier().verify(...)
Verify the XML signature supplied in the data and return the XML node signed by the signature, or raise an exception if the signature is not valid.
class signxml.VerifyResult
The results of a verification return the signed data, the signed xml and the signature xml
Note: Necessarily read about See what is signed and Establish trust!
Relevant: Python elementtree find function reads Signature as empty (None)

How to make secure connection with python?

I'm using python3. i need to use certificates file to make secure connection.
In this case, i used Httpsconnection class from http.client...
this class get certs file path and use it. like this:
import http.client
client=http.client.HTTPSConnection\
("epp.nic.ir",key_file="filepath\\nic.pem",cert_file="filepath\\nic.crt")
As you see, this class get path of files and works correctly.
But I need to give contents of these files. Because I want to put contents of crt file and pem file into DB. the reason is that maybe files path changes...
so i tried this:
import http.client
import base64
cert = b'''
content of cert file
'''
pem = b'''
content of pem file
'''
client=http.client.HTTPSConnection("epp.nic.ir" ,pem, cert)
as expected, i got this error:
TypeError: certfile should be a valid filesystem path
is there any way to make this class to get content of file instead of file path ?!
Or Is it possible to make changes in source codes of http for this purpose ?!

It is possible to modify Python source code, but it is not the recommended way as it definitely brings about portability, maintainability and other issues.
Consider you want to update Python version, you have to apply your modification each time you update it.
Consider you want to run your code in another machine, again the same problem.
Instead of modifying the source code, there is a better and preferrable way: extending the API.
You can subclass the existing HTTPSConnection class and override its constructor method by your own implementation.
There are plenty of ways to achieve what you need.
Here is a possible solution with subclassing:
import http.client
import tempfile
class MyHTTPSConnection(http.client.HTTPSConnection):
"""HTTPSConnection with key and cert files passed as contents rather than file names"""
def __init__(self, host, key_content=None, cert_content=None, **kwargs):
# additional parameters are also optional so that
# so that this class can be used with or without cert/key files
# as a replacement of standard HTTPSConnection
self.key_file = None
self.cert_file = None
# here we write the content of cert & pem into a temporary file
# delete=False keeps the file in the file system
# but, this time we need to remove it manually when we are done
if key_content:
self.key_file = tempfile.NamedTemporaryFile(delete=False)
self.key_file.write(key_content)
self.key_file.close()
# NamedTemporaryFile object provides 'name' attribute
# which is a valid file name in the file system
# so we can use those file names to initiate the actual HTTPSConnection
kwargs['key_file'] = self.key_file.name
# same as above but this time for cert content and cert file
if cert_content:
self.cert_file = tempfile.NamedTemporaryFile(delete=False)
self.cert_file.write(cert_content)
self.cert_file.close()
kwargs['cert_file'] = self.cert_file.name
# initialize super class with host and keyword arguments
super().__init__(host, **kwargs)
def clean(self):
# remove temp files from the file system
# you need to decide when to call this method
os.unlink(self.cert_file.name)
os.unlink(self.pem_file.name)
host = "epp.nic.ir"
key_content = b'''content of key file'''
cert_content = b'''content of cert file'''
client = MyHTTPSConnection(host, key_content=key_content, cert_content=cert_content)
# ...

SAML signature verification using Python/M2Crypto

I'm attempting to use M2Crypto to verify a signature contained in an XML response returned from my SSO/SAML provider in my django/python app, but I can't seem to get it to work.
My XML response looks sort of like the second example here.
ETA: And here's a pastebin of my actual XML.
I'm using some code like this to attempt the verification:
def verify_signature(signed_info, cert, signature):
from M2Crypto import EVP, RSA, X509
x509 = X509.load_cert_string(base64.decodestring(cert), X509.FORMAT_DER)
pubkey = x509.get_pubkey().get_rsa()
verify_EVP = EVP.PKey()
verify_EVP.assign_rsa(pubkey)
verify_EVP.reset_context(md='sha1')
verify_EVP.verify_init()
verify_EVP.verify_update(signature.decode('base64'))
result = verify_EVP.verify_final(signed_info)
return result
I can successfully get the NameID from the response, and I know I'm successfully loading the certificate, because I can pull the issuer, etc. out of it.
As for the signature, though, I've tried hashing the passed in XML, encoding/not encoding various pieces, and passing in various bits of XML for the signed_info parameter (the SignedInfo tag, the Response tag, the whole thing), and I've tried using ElementTree/ElementC14N.py to ensure the XML is exclusively canonicalized, as the Transform implies should be done, but I'm not getting a positive result.
What am I missing here? Am I trying to validate against the wrong XML? Something wrong with my verification technique?

You were so close! You should pass to verify_update the signed_info, and then to verify_final pass the signature.
You do need to make sure that your signed_info is correctly canonicalized before verifying the signature.
Here is the correct method:
def verify_signature(signed_info, cert, signature):
from M2Crypto import EVP, RSA, X509
x509 = X509.load_cert_string(base64.decodestring(cert), X509.FORMAT_DER)
pubkey = x509.get_pubkey().get_rsa()
verify_EVP = EVP.PKey()
verify_EVP.assign_rsa(pubkey)
verify_EVP.reset_context(md='sha1')
verify_EVP.verify_init()
verify_EVP.verify_update(signed_info)
result = verify_EVP.verify_final(signature.decode('base64'))
return result

FYI, I was facing the same problem as you, and found no useful software for validating XML signatures in Python, so I wrote a new library: https://github.com/kislyuk/signxml.
from lxml import etree
from signxml import xmldsig
with open("saml2_idp_metadata.xml", "rb") as fh:
cert = etree.parse(fh).find("//ds:X509Certificate").text
root = ElementTree.fromstring(signature_data)
xmldsig(root).verify(x509_cert=cert)