We Keep Coding

How to authenticate HashiCorp Vault without certificate?

Below is my code:
import hvac
client = hvac.Client(
url='https://vault-abc.net',token='s.d0AGS4FE3o6UxUpVTQ0h0RRd',verify='False'
)
print(client.is_authenticated())
ERROR in output:
in cert_verify
raise IOError("Could not find a suitable TLS CA certificate bundle, " OSError: Could not find a suitable TLS CA certificate
bundle, invalid path: False
I got only token and URL to login on console from client no certificates shared! In other java applications code without using any certificate authentication working but in python code under hvac module or CURL or vault CLI expecting certificates to be passed. Any way I can handle this and fix above error?
Do we have any certificate check skip option?
Agenda is authenticate and do fetch vault secrets using python program, without any certificates need to fetch just with Token & vault URL.

You can disable certificate checks, but for something like Vault that's generally a bad idea (disabling security checks on a security service).
In any case, your problem is simple: You are passing 'False' (a string) where you should be passing False (a boolean) as the verify argument.
Passing a string causes the library to look for a certificate at that path; since there is no certificate at the path 'False', you get the error that you are seeing.

How to use truststore file & password to let KafkaConsumer get topics content in python?

I have been provided truststore file(.jks) & password.
I'm using below code to get messages from kafka topic:
from kafka import KafkaConsumer
consumer = KafkaConsumer('Topicname',security_protocol = 'SSL', bootstrap['x.com:9092'])
from message in consumer:
print(message)
Above code is printing below error:
SSLCertVerificationError: certificate verify failed
I know i have not used the truststore file & password.. but unable to figure how to use these information to make above code working.
Basically, i need the python code snippet to use the file.
I would appreciate if anybody can help me.
Thanks,
N

This error can come up if you've got a certificate but the certificate authority is not included in the default CA list.
So, you could embed the root CA certificate by including it in your code. Try the following steps:
Extract the root certificate from the Kafka trust store by using the below command:
keytool -exportcert -alias CARoot -keystore <TRUST-STORE-PATH> -rfc -file CARoot.pem
Then in your consumer code, you'll need to add the following configuration parameters:
from kafka import KafkaConsumer
consumer = KafkaConsumer(bootstrap_servers='<SERVER-LIST-WITH-PORTS>',
security_protocol='SSL',
ssl_check_hostname=True,
ssl_cafile='<PATH-TO-CARoot.pem>',
...
)
You might also want to verify the ports for SSL communication in listeners as it is generally 9093 for SSL instead of 9092.
Hope this helps.

How to use ssl client certificate (p12) with Scrapy?

I need to use client certificate file in format p12(PKCS12) to talk to a webserver with scrapy, is there a way to do that ?

I can't offer you a tested and complete solution here, but I know a few places where some adjustments might give you what you need.
The starting point is scrapy's ContextFactory object which defines the SSL/TLS configuration. The standard implementation ScrapyClientContextFactory doesn't use client certificates and also doesn't do any server certificate verification, it just accepts any certificate. (More details)
When looking into the source code however you see the alternative BrowserLikeContextFactory is creating an optionsForClientTLS object.
This object can also take a clientCertificate parameter to authenticate to the server. (Details)
So in theory you need to subclass BrowserLikeContextFactory, write there your own creatorForNetloc method and make it create optionsForClientTLS that also have a clientCertificate
In a gist:
#implementer(IPolicyForHTTPS)
class ClientCertContextFactory(BrowserLikeContextFactory):
def creatorForNetloc(self, hostname, port):
with open('yourcert.pem') as keyAndCert:
myClientCert = twisted.internet.ssl.PrivateCertificate.load(keyAndCert.read())
return optionsForClientTLS(hostname.decode("ascii"),
trustRoot=platformTrust(),
clientCertificate=myClientCert,
extraCertificateOptions={
'method': self._ssl_method,
})
Activating the context factory in settings.py:
DOWNLOADER_CLIENTCONTEXTFACTORY = 'your.package.ClientCertContextFactory'
According to the docs twisted.internet.ssl.PrivateCertificate can only load pem or asn.1 format keys, means you will have to convert your key into pem format:
openssl pkcs12 -in client_ssl.pfx -out client_ssl.pem -clcerts
(Borrowed from Converting pfx to pem using openssl)
Update Conversion for PKCS12 files in p12 format:
openssl pkcs12 -in client_cert.p12 -out client_cert.pem -clcerts

Python-Requests: direct .pem pinning with self-signed cert

Using python-requests, how can I pin a self-signed .pem certificate for a specific server directly, without using CA root bundles?
Is this currently possible? If yes, can you please provide an example?
I read https://2.python-requests.org/en/v2.8.1/user/advanced/#ssl-cert-verification but am not sure if this applies to what I'm trying to do:
You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file’s path: requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key')) Response [200]

Because the certificate file is self-signed, this works just as you would do it normally with requests. Below is a step-by-step procedure:
Obtain the self-signed certificate, ideally in some secure, out-of-band manner. For example, I run a webserver that offers HTTPS access via a self-signed certificate, so I downloaded the certificate using scp:
scp <username>#<server>:/path/to/certfile.crt .
Because I use nginx this is already in PEM format, but if it's not you'll need to convert it. That's outside the scope of this answer.
Use the certificate file from inside requests:
r = requests.get('https://yoursite.com/', verify='certfile.crt')
That's all you need to do.
If you can't obtain the certificate in an out-of-band manner you trust, you can obtain the certificate using your browser. All browsers will let you export the certificate via their UIs. This is less-secure: if someone is going to MITM you then they may well have already started, and can offer you their MITM root CA instead of your self-signed cert.

You can also verify certificates against their fingerprints.
For this you need a custom transport adapter for requests.
An example for a simple one can be found here:
https://github.com/untitaker/vdirsyncer/blob/9d3a9611b2db2e92f933df30dd98c341a50c6211/vdirsyncer/utils/init.py#L198
import requests
from requests.packages.urllib3.poolmanager import PoolManager
class _FingerprintAdapter(requests.adapters.HTTPAdapter):
def __init__(self, fingerprint=None, **kwargs):
self.fingerprint = str(fingerprint)
super(_FingerprintAdapter, self).__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
assert_fingerprint=self.fingerprint)

Requests: what is the difference between cert and verify?

What is the difference between cert and verify?
From Documentation:
verify – (optional) if True, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.
Does this mean I can do the following:
CA_BUNDLE='path/to/.pem'
requests.get(url=google.com, verify= CA_BUNDLE)
or
Cert='path/to/.pem'
requests.get(url=google.com, cert=Cert)
They both look like they do the same thing. except verify can disable ssl verification.
I am trying to compile my code to an exe using PYinstaller. I am using certifi module that I see already has a cacert.pem file but I guess I still have to bundle it with my code.
In my code do I modify ...verify or cert?...with a path to cacert.pem or just 'cacert.pem'?

I think it is clearly stated in the documentation: SSL Cert Verification
The option cert is to send you own certificate, e.g. authenticate yourself against the server using a client certificate. It needs a certificate file and if the key is not in the same file as the certificate also the key file.
The option verify is used to enable (default) or disable verification of the servers certificate. It can take True or False or a name of a file which contains the trusted CAs. If not given I think (not documented?) it will take the default CA path/file from OpenSSL, which works usually on UNIX (except maybe OS X) and not on windows.

if the *.pem file has this section
-----BEGIN PRIVATE KEY-----
....
-----END PRIVATE KEY-----
then use cert
and if not, then use verify