I need to use client certificate file in format p12(PKCS12) to talk to a webserver with scrapy, is there a way to do that ?
I can't offer you a tested and complete solution here, but I know a few places where some adjustments might give you what you need.
The starting point is scrapy's ContextFactory object which defines the SSL/TLS configuration. The standard implementation ScrapyClientContextFactory doesn't use client certificates and also doesn't do any server certificate verification, it just accepts any certificate. (More details)
When looking into the source code however you see the alternative BrowserLikeContextFactory is creating an optionsForClientTLS object.
This object can also take a clientCertificate parameter to authenticate to the server. (Details)
So in theory you need to subclass BrowserLikeContextFactory, write there your own creatorForNetloc method and make it create optionsForClientTLS that also have a clientCertificate
In a gist:
#implementer(IPolicyForHTTPS)
class ClientCertContextFactory(BrowserLikeContextFactory):
def creatorForNetloc(self, hostname, port):
with open('yourcert.pem') as keyAndCert:
myClientCert = twisted.internet.ssl.PrivateCertificate.load(keyAndCert.read())
return optionsForClientTLS(hostname.decode("ascii"),
trustRoot=platformTrust(),
clientCertificate=myClientCert,
extraCertificateOptions={
'method': self._ssl_method,
})
Activating the context factory in settings.py:
DOWNLOADER_CLIENTCONTEXTFACTORY = 'your.package.ClientCertContextFactory'
According to the docs twisted.internet.ssl.PrivateCertificate can only load pem or asn.1 format keys, means you will have to convert your key into pem format:
openssl pkcs12 -in client_ssl.pfx -out client_ssl.pem -clcerts
(Borrowed from Converting pfx to pem using openssl)
Update Conversion for PKCS12 files in p12 format:
openssl pkcs12 -in client_cert.p12 -out client_cert.pem -clcerts
Related
A basic Confluent-Kafka producer and consumer have been created to send plaintext messages.
After successfully sending messages from producer to consumer, additional configs were added to use SSL rather than PLAINTEXT.
The following Configs have been implemented, which result in the following error.
"Message Timed Out"
Producer Configs:
bootstrap.servers: localhost9093
security.protocol: SSL
ssl.keystore.location: ../keystore.p12
ssl.keystore.password: [password]
ssl.ca.location: ../CARoot.pem
ssl.key.location: ../key.pem
ssl.certificate.location: ../cert.pem
ssl.key.password: [password]
Server Configs:
ssl.keystore.type= PKCS12
ssl.keystore.location= ../keystore.p12
ssl.keystore.password= [password]
ssl.ca.location= ..\\CARoot.pem
ssl.certificate.location= ..\\cert.pem
ssl.key.password= [password]
ssl.key.location= ../key.pem
security.inter.broker.protocol= SSL
listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
Are there additional configs required to implement SSL?
Additionally, can anyone summarize the CARoot?
From what I have read, it is a chain of certificates.
Therefore, if there is only one certificate, should CARoot.pem and cert.pem be identical?
This file might be the problem.
The certificate and private key were created in PyOpenSSL.
keystore.p12 was converted from a .jks keystore using keytool.
Is there a way to create a CARoot.pem file using this library?
Thank you.
The producer was timing out due to the formatting of the CA Certificate file.
The solution to the following question was used to resolve the time-out error, which uses OpenSSL rather than PyOpenSSL.
Note: OpenSSL is available in Git Bash.
How to export CA certificate chain from PFX in PEM format without bag attributes
Additionally, there were some changes that were made to the configuration of both the server and producer.
Producer Configurations:
bootstrap.servers: localhost9093
security.protocol: SSL
ssl.ca.location: ../CARoot.pem
ssl.key.location: ../key.pem
ssl.certificate.location: ../cert.pem
Server Configurations:
ssl.protocol= SSL
ssl.keystore.type= JKS
ssl.keystore.location= ../keystore.jks
ssl.keystore.password= [password]
ssl.client.auth= required
security.inter.broker.protocol= SSL
listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
Using python-requests, how can I pin a self-signed .pem certificate for a specific server directly, without using CA root bundles?
Is this currently possible? If yes, can you please provide an example?
I read https://2.python-requests.org/en/v2.8.1/user/advanced/#ssl-cert-verification but am not sure if this applies to what I'm trying to do:
You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file’s path: requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key')) Response [200]
Because the certificate file is self-signed, this works just as you would do it normally with requests. Below is a step-by-step procedure:
Obtain the self-signed certificate, ideally in some secure, out-of-band manner. For example, I run a webserver that offers HTTPS access via a self-signed certificate, so I downloaded the certificate using scp:
scp <username>#<server>:/path/to/certfile.crt .
Because I use nginx this is already in PEM format, but if it's not you'll need to convert it. That's outside the scope of this answer.
Use the certificate file from inside requests:
r = requests.get('https://yoursite.com/', verify='certfile.crt')
That's all you need to do.
If you can't obtain the certificate in an out-of-band manner you trust, you can obtain the certificate using your browser. All browsers will let you export the certificate via their UIs. This is less-secure: if someone is going to MITM you then they may well have already started, and can offer you their MITM root CA instead of your self-signed cert.
You can also verify certificates against their fingerprints.
For this you need a custom transport adapter for requests.
An example for a simple one can be found here:
https://github.com/untitaker/vdirsyncer/blob/9d3a9611b2db2e92f933df30dd98c341a50c6211/vdirsyncer/utils/init.py#L198
import requests
from requests.packages.urllib3.poolmanager import PoolManager
class _FingerprintAdapter(requests.adapters.HTTPAdapter):
def __init__(self, fingerprint=None, **kwargs):
self.fingerprint = str(fingerprint)
super(_FingerprintAdapter, self).__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
assert_fingerprint=self.fingerprint)
What is the difference between cert and verify?
From Documentation:
verify – (optional) if True, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.
Does this mean I can do the following:
CA_BUNDLE='path/to/.pem'
requests.get(url=google.com, verify= CA_BUNDLE)
or
Cert='path/to/.pem'
requests.get(url=google.com, cert=Cert)
They both look like they do the same thing. except verify can disable ssl verification.
I am trying to compile my code to an exe using PYinstaller. I am using certifi module that I see already has a cacert.pem file but I guess I still have to bundle it with my code.
In my code do I modify ...verify or cert?...with a path to cacert.pem or just 'cacert.pem'?
I think it is clearly stated in the documentation: SSL Cert Verification
The option cert is to send you own certificate, e.g. authenticate yourself against the server using a client certificate. It needs a certificate file and if the key is not in the same file as the certificate also the key file.
The option verify is used to enable (default) or disable verification of the servers certificate. It can take True or False or a name of a file which contains the trusted CAs. If not given I think (not documented?) it will take the default CA path/file from OpenSSL, which works usually on UNIX (except maybe OS X) and not on windows.
if the *.pem file has this section
-----BEGIN PRIVATE KEY-----
....
-----END PRIVATE KEY-----
then use cert
and if not, then use verify
I'm trying to implement push notifications for iphone based on PyAPNs
When I run it on local but it blocks and prompts me to enter the passphrase manually and doesn't work until I do
I don't know how to set it up so to work without prompt
This is my code:
from apns import APNs, Payload
import optparse
import os
certificate_file = here(".." + app.fichier_PEM.url )
token_hex = '0c99bb3d077eeacdc04667d38dd10ca1a'
pass_phrase = app.mot_de_passe
apns = APNs(use_sandbox=True, cert_file= certificate_file)
payload = Payload(alert = message.decode('utf-8'), sound="default", badge=1)
apns.gateway_server.send_notification(token_hex, payload)
# Get feedback messages
for (token_hex, fail_time) in apns.feedback_server.items():
print "fail: "+fail_time
When you create a .pem file without phrase specify -nodes
To Create .pem file without phrase
openssl pkcs12 -nocerts -out Pro_Key.pem -in App.p12 -nodes
To Create .pem file with phrase
openssl pkcs12 -nocerts -out Pro_Key.pem -in App.p12
If you have a .pem file with password you can get rid of its password for PyAPNs
using the following
openssl rsa -in haspassword.pem -out nopassword.pem
Refer
Raywenderlich
Apple Push Notification Step By Step Guide
for make certificates and other configurations.
Some Python library for interacting with the Apple Push Notification service (APNs)
djacobs->PyAPNs
samuraisam->pyapns
apns-client
Try to use
apns = APNs(use_sandbox=True, cert_file='XYZCert.pem', key_file='XYZKey.pem')
where you specify both the certificate and the private key.
I've created a key pair using the following code in python with pyOpenSSL:
from OpenSSL import crypto
k = crypto.PKey()
k.generate_key(crypto.TYPE_RSA, 2048)
Now how can I create the private and public key .pem files from the key object?
If there is any tutorial available please let me know. I found none. From the manual, it's difficult to know as I'm new to OpenSSL.
What are the chances that the same code will create two same key pairs is there is no specific unique key is being used in RSA?
I know this is an old question - but as I've just found it I thought I'd add an answer.
The easiest way to do this with Python 3.x is to use PyCryptodome.
The in Python (for a 2048-bit key):
from Cryptodome.PublicKey import RSA
key = RSA.generate(2048)
pv_key_string = key.exportKey()
with open ("private.pem", "w") as prv_file:
print("{}".format(pv_key_string.decode()), file=prv_file)
pb_key_string = key.publickey().exportKey()
with open ("public.pem", "w") as pub_file:
print("{}".format(pb_key_string.decode()), file=pub_file)
If you want to check the private key on the (Linux) command-line use:
$ openssl rsa -check -inform pem -noout -in private.pem
RSA key ok
...
I hope this will help people in the future, because I had this same need and couldn't find an answer so I did it myself. Thought I would share it with you.
1. Creating a PEM file
bio_pub = _new_mem_buf() # Memory buffers to write to
bio_priv = _new_mem_buf()
helper = OpenSSL.crypto._PassphraseHelper(OpenSSL.crypto.FILETYPE_PEM, None)
pk = OpenSSL.crypto.PKey()
pk.generate_key(OpenSSL.crypto.TYPE_RSA, n)
# Convert from EVP_PKEY type to RSA type
rsa_pkey = _lib.EVP_PKEY_get1_RSA(pk._pkey)
result_code = _lib.PEM_write_bio_RSAPublicKey(bio_pub, rsa_pkey)
result_code = _lib.PEM_write_bio_RSAPrivateKey(
bio_priv, rsa_pkey, _ffi.NULL, _ffi.NULL, 0,
helper.callback, helper.callback_args)
After this part you will have the public and private keys in your buffers.
To get it as a string you can call the functions:
_bio_to_string(bio_pub), _bio_to_string(bio_priv)
I used these imports for the special "private" functions of OpenSSL.crypto:
import OpenSSL
from OpenSSL._util import lib as _lib, ffi as _ffi
from OpenSSL.crypto import _new_mem_buf, _bio_to_string
You can create a .pem key by follow this tutorial at:
https://help.ubuntu.com/community/OpenSSL
that suppose you want to create a CA(certificate authority) certificate, that
is little complicate because you already have to get a CA from somewhere
because it's not free.
if you only want to create a key juste for your ssl connection test it
better to create
a self-sign certificate.
then make sure first you have install openssl and you have resolve the CN (Common Name) on your serve. without that you will be in trouble to use the created certificate.
for the Self-sign certificate use this command line:
$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048
$ openssl rsa -passin pass:x -in server.pass.key -out server.key
$ rm server.pass.key
$ openssl req -new -key server.key -out server.csr (list of question to answer)
$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
after you got the certificate create you have to activate your
server mod-ssl and add the line where is locate your certificate.
later you have to insert that certificate in your IE certificate
list to get it work with you apache ssl connection daemon.