I am trying to see if I can get the python-gnupg module working to sign and verify a file using a python script. I have the following code, which does not interpret any errors when called.
However the code prints "unverified" at the end, when I thought that I had signed the file (example.txt).
I must be missing something in the documentation but after I read it this is what I came up with for signing and verifying. Any help please?
import gnupg
gpg = gnupg.GPG(gnupghome="/home/myname")
stream = open("example.txt", "rb")
signed_data = gpg.sign_file(stream)
verified = gpg.verify_file(stream)
print "Verified" if verified else "Unverified"
There are a few issues with your code,
1.) gpg = gnupg.GPG(gnupghome="/home/myname") needs to be gpg = gnupg.GPG(gnupghome="/home/myname/.gnupg")
2.) You are attempting to verify the stream, using verify_file(stream), however the stream is still a handle to the original, unsigned file. You would first need to either write the signed data to a new file and call verify_file() against a handle to that file, or verify the result sign_file.
Below is a working example of your demo, using the result of sign_file - but before we get to that, the way to troubleshoot what is happening in your script, you can review the output of stderr on the returned object of the gnupg methods. for example, you can review the result of the signed data by printing out signed_data.stderr. Likewise for return of the verify_file method.
On to the code -
import gnupg
gpg = gnupg.GPG(gnupghome="/home/myname/.gnupg")
stream = open("example.txt", "rb")
signed_data = gpg.sign_file(stream)
verified = gpg.verify(signed_data.data)
print "Verified" if verified else "Unverified"
I hope this helps!
Related
I am trying to compare a signature with a certificate for a pdf file in python.
I found this very nice package called endesive.
I followed the example for verifying a pdf signature and I have something like this:
pdf_file_path = "/workspaces/test.pdf"
data = open(pdf_file_path, 'rb').read()
certificates = (
open("/workspaces/certificates/pki.pem", 'rt').read(),
open("/workspaces/certificates/pki-chain.pem", 'rt').read()
)
(hashok, signatureok, certok) = pdf.verify(data, certificates)
print('signature ok?', signatureok)
print('hash ok?', hashok)
print('cert ok?', certok)
This should be pretty straight forward. I read the pdf, I open the certificates and then I 'pdf.verify' to see that everything is in order.
pdf.verify, at one point calls this: signed_data = cms.ContentInfo.load(bcontents)['content'].native which makes ans1crypto raise this error File "/home/vscode/.local/lib/python3.9/site-packages/asn1crypto/core.py", line 4060, in native raise e repeatedly until it gets to
ValueError: Unknown element - context class, constructed method, tag 0
while parsing asn1crypto.core.Sequence
while parsing asn1crypto.cms.SetOfAny
while parsing asn1crypto.cms.CMSAttribute
while parsing asn1crypto.cms.CMSAttributes
while parsing asn1crypto.cms.SignerInfo
What could go wrong here?
Instead of addressing signer data info like this:
signature = signed_data['signer_infos'][0].native['signature']
It should have been addressed like this:
signature = signed_data['signer_infos'][0]['signature'].native
This has been addressed here.
I have tried to upload an XML File to S3 using boto3. As recommended by Amazon, I would like to send a Base64 Encoded MD5-128 Bit Digest(Content-MD5) of the data.
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.put
My Code:
with open(file, 'rb') as tempfile:
body = tempfile.read()
tempfile.close()
hash_object = hashlib.md5(body)
base64_md5 = base64.encodebytes(hash_object.digest())
response = s3.Object(self.bucket, self.key + file).put(
Body=body.decode(self.encoding),
ACL='private',
Metadata=metadata,
ContentType=self.content_type,
ContentEncoding=self.encoding,
ContentMD5=str(base64_md5)
)
When i try this the str(base64_md5) create a string like 'b'ZpL06Osuws3qFQJ8ktdBOw==\n''
In this case, I get this Error Message:
An error occurred (InvalidDigest) when calling the PutObject operation: The Content-MD5 you specified was invalid.
For Test purposes I copied only the Value without the 'b' in front: 'ZpL06Osuws3qFQJ8ktdBOw==\n'
Then i get this Error Message:
botocore.exceptions.HTTPClientError: An HTTP Client raised and unhandled exception: Invalid header value b'hvUe19qHj7rMbwOWVPEv6Q==\n'
Can anyone help me how to save Upload a File to S3?
Thanks,
Oliver
Starting with #Isaac Fife's example, stripping it down to identify what's required vs not, and to include imports and such to make it a full reproducible example:
(the only change you need to make is to use your own bucket name)
import base64
import hashlib
import boto3
contents = "hello world!"
md = hashlib.md5(contents.encode('utf-8')).digest()
contents_md5 = base64.b64encode(md).decode('utf-8')
boto3.client('s3').put_object(
Bucket="mybucket",
Key="test",
Body=contents,
ContentMD5=contents_md5
)
Learnings: first, the MD5 you are trying to generate will NOT look like what an 'upload' returns. We actually need a base64 version, it returns a md.hexdigest() version. hex is base16, which is not base64.
(Python 3.7)
Took me hours to figure this out because the only error you get is "The Content-MD5 you specified was invalid." Super useful for debugging... Anyway, here is the code I used to actually get the file to upload correctly before refactoring.
json_results = json_converter.convert_to_json(result)
json_results_utf8 = json_results.encode('utf-8')
content_md5 = md5.get_content_md5(json_results_utf8)
content_md5_string = content_md5.decode('utf-8')
metadata = {
"md5chksum": content_md5_string
}
s3 = boto3.resource('s3', config=Config(signature_version='s3v4'))
obj = s3.Object(bucket, 'filename.json')
obj.put(
Body=json_results_utf8,
ContentMD5=content_md5_string,
ServerSideEncryption='aws:kms',
Metadata=metadata,
SSEKMSKeyId=key_id)
and the hashing
def get_content_md5(data):
digest = hashlib.md5(data).digest()
return base64.b64encode(digest)
The hard part for me was figuring out what encoding you need at each step in the process and not being very familiar with how strings are stored in python at the time.
get_content_md5 takes a utf-8 bytes-like object only, and returns the same. But to pass the md5 hash to aws, it needs to be a string. You have to decode it before you give it to ContentMD5.
Pro-tip - Body on the other hand, needs to be given bytes or a seekable object. Make sure if you pass a seekable object that you seek(0) to the beginning of the file before you pass it to AWS or the MD5 will not match. For that reason, using bytes is less error prone, imo.
How can I select a stream as the output of a decrypt_file operation in gnupg?
The docs and the code seem to suggest this is not possible. If I am correct (see below), what workarounds are possible?
~~~
The documentation seems to suggest it is not possible:
decrypt_file(filename, always_trust=False, passphrase=None, output=None)¶
with "output (str) – A filename to write the decrypted output to."
~~~
Opening up the code, I see:
def decrypt_file(self, file, always_trust=False, passphrase=None,
output=None, extra_args=None):
args = ["--decrypt"]
if output: # write the output to a file with the specified name
self.set_output_without_confirmation(args, output)
if always_trust: # pragma: no cover
args.append("--always-trust")
if extra_args:
args.extend(extra_args)
result = self.result_map['crypt'](self)
self._handle_io(args, file, result, passphrase, binary=True)
logger.debug('decrypt result: %r', result.data)
return result
which points to set_output_without_confirmation, confirming the idea is that you pass a string filename:
def set_output_without_confirmation(self, args, output):
"If writing to a file which exists, avoid a confirmation message."
if os.path.exists(output):
# We need to avoid an overwrite confirmation message
args.extend(['--yes'])
args.extend(['--output', no_quote(output)])
To output the decrypted data to a variable use decrypt instead of decrypt_file, as shown here in the "Decrypt a string" paragraph.
So the original code:
status = gpg.decrypt_file(input_file, passphrase='my_passphrase', output='my_output_file')
is substituted by:
decrypted_data = gpg.decrypt(input_file.read(), passphrase='my_passphrase')
# decrypted_data.data contains the data
decrypted_stream = io.BytesIO(decrypted_data.data)
# this is py3, in py2 BytesIO is imported from BytesIO
As an example for the specific use case for csv data, building on this SO post, you could then do:
my_df = pandas.read_csv(decrypted_stream)
I was playing around with oauth2 to get a better understanding of it. For this reason, I've installed offlineimap which should act as a third-party app. I've found a nice way to read encrypted credentials here on stackexchange.
Based on the linked post I've modified/copied the following python script:
import subprocess
import os
import json
def passwd(file_name):
acct = os.path.basename(file_name)
path = "/PATHTOFILE/%s" % file_name
args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
try:
return subprocess.check_output(args).strip()
except subprocess.CalledProcessError:
return ""
def oauthpasswd(acct, key):
acct = os.path.basename(acct)
path = "/PATHTOFILE/%s_oauth2.gpg" % acct
args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
try:
return str(json.loads(subprocess.check_output(args).strip())['installed'][key])
except subprocess.CalledProcessError:
return ""
def prime_gpg_agent():
ret = False
i = 1
while not ret:
ret = (passwd("prime.gpg") == "prime")
if i > 2:
from offlineimap.ui import getglobalui
sys.stderr.write("Error reading in passwords. Terminating.\n")
getglobalui().terminate()
i += 1
return ret
prime_gpg_agent()
In the corresponding offlineimaprc file I call the function with the correct arguments:
oauth2_client_id = oauthpasswd('gmail', 'client_id')
oauth2_client_secret = oauthpasswd('gmail', 'client_secret')
oauth2_request_url = https://accounts.google.com/o/oauth2/token
oauth2_refresh_token = passwd('gmail_rf_token.gpg')
Please note in the local file the PATHTOFILE is set correctly. What I've done was downloaded the JSON file from Google including the oauth2 credentials and encrypted it. I've stored the refresh token in a separate file.
However, if I run offlineimap I get an authentication error:
ERROR: While attempting to sync account 'gmail'
('http error', 401, 'Unauthorized', <httplib.HTTPMessage instance at 0x7f488c214320>) (configuration is: {'client_secret': "oauthpasswd('gmail', 'client_secret')", 'grant_type': 'refresh_token', 'refresh_token': "passwd('gmail_rf_token.gpg')", 'client_id': "oauthpasswd('gmail', 'client_id')"})
I've tried then to check the outputs of the two python functions passwd and oauthpasswd in a python interpreter. I get the desired outputs. Even more, I've copied the output from the functions within the python interpreter to the offlineimaprc config file and I was able to sync to Gmail. This implies that there must be a mistake when offlineimap executes the file but I can't see what's wrong.
If I only encrypt my Gmail password everything is working. This means there is something going wrong from the details downloaded from Google (client_id, client_secret and refresh token). As pointed out above, the values itself are correct. I've really copied the output of
oauthpasswd('gmail', 'client_id')
oauthpasswd('gmail', 'client_secret')
passwd('gmail_rf_token.gpg')
from a python console to the offlineimaprc file and it worked.
The problem which happens is the following. According to this answer offlineimap does not allow for encryption of all keys within the offlinemaprc file. That's why the python function never gets evaluated and the wrong strings are handed over.
I am trying to decrypt a PGP file using this module:
http://packages.python.org/python-gnupg/
Here is my code snippet:
#!/usr/bin/python
import gnupg
gpg = gnupg.GPG(gnupghome='C:\\Users\\GSquire\\Desktop\\GnuPG',
gpgbinary='C:\\Users\\GSquire\\Desktop\\GnuPG\\pub\\gpg.exe',
keyring='C:\\Users\\GSquire\\Desktop\\GnuPG\\secring.skr')
with open('.\\tranx08022012.txt.pgp', 'rb') as f:
status = gpg.decrypt_file(f, passphrase='passphrase', output='out.txt')
I am using the latest version of the module, and Python 2.6.6. I thought I could just use the secure ring file to decrypt it because that is obviously needed by the file. It outputs this when I run the script:
ok: False
status:
stderr:
gpg: expected public key but found secret key - must stop
Isn't it true that the secure key is what decrypts the file? Thanks for the help!
The error you're getting is because you're passing the secret keyring's filename in the keyring parameter. That parameter is only for the public keyring. Unfortunately, there doesn't seem to be an alternative parameter to specify a secret keyring file.
By default, GnuPG will look for secret keys in secring.gpg in the gnupghome folder you specify, so you can probably rename your secret key file and get it to work.