We Keep Coding

python ssl (eqivalent of openssl s_client -showcerts ) How to get list of CAs for client certs from server

I have a group of nginx servers, that accept client certificates.
They have the ssl_client_certificate option with a file containing one or more CAs
If I use a web browser, then the web browser seems to receive a list of valid CAs for client certs. The browser shows only client certs signed by one of these CAs.
Following openssl command gives me a list of CA certs:
openssl s_client -showcerts -servername myserver.com -connect myserver.com:443 </dev/null
The lines I am interested in look following way:
---
Acceptable client certificate CA names
/C=XX/O=XX XXXX
/C=YY/O=Y/OU=YY YYYYYL
...
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
How can I get the same information with python?
I do have following code snippet, that allows to obtain a server's certificate, but this code does not return the list of CAs for client certs.
import ssl
def get_server_cert(hostname, port):
conn = ssl.create_connection((hostname, port))
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
sock = context.wrap_socket(conn, server_hostname=hostname)
cert = sock.getpeercert(True)
cert = ssl.DER_cert_to_PEM_cert(cert)
return cerft
I expected to find a functional equivalent of getpeercert(), something like getpeercas() but didn't find anything.
Current workaround:
import os
import subprocess
def get_client_cert_cas(hostname, port):
"""
returns a list of CAs, for which client certs are accepted
"""
cmd = [
"openssl",
"s_client",
"-showcerts",
"-servername", hostname,
"-connect", hostname + ":" + str(port),
]
stdin = open(os.devnull, "r")
stderr = open(os.devnull, "w")
output = subprocess.check_output(cmd, stdin=stdin, stderr=stderr)
ca_signatures = []
state = 0
for line in output.decode().split("\n"):
print(state, line)
if state == 0:
if line == "Acceptable client certificate CA names":
state = 1
elif state == 1:
if line.startswith("Client Certificate Types:"):
break
ca_signatures.append(line)
return ca_signatures
Update:Solution with pyopenssl (Thanks Steffen Ullrich)
#Steffen Ulrich suggested to use pyopenssl, which has a method get_client_ca_list() and this helped me to write a small code snippet.
Below code seems to work. Not sure if it can be improved or whether there are any pit falls.
If nobody is answering within the next days I will post this as a potential answer.
import socket
from OpenSSL import SSL
def get_client_cert_cas(hostname, port):
ctx = SSL.Context(SSL.SSLv23_METHOD)
# If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
# an empty result
ctx.set_options(SSL.OP_NO_TLSv1_3)
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
# next line for SNI
sock.set_tlsext_host_name(hostname.encode("utf-8"))
sock.connect((hostname, port))
# without handshake get_client_ca_list will be empty
sock.do_handshake()
return sock.get_client_ca_list()
Update: 2021-03-31
Above suggested solution using pyopenssl works in most cases.
However sock.get_client_ca_list()) cannot be called immediately after performing a sock.connect((hostname, port))
Some actions seem to be required in between these two commands.
Initially I used sock.send(b"G"), but now I use sock.do_handshake(), which seems a little cleaner.
Even stranger, the solution doesn't work with TLSv1.3 so I had to exclude it.

As a generic example in python
first you need to contact the server to learn which issuer CA subjects it accepts:
from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509Name
from certifi import where
import idna
def get_server_expected_client_subjects(host :str, port :int = 443) -> list[X509Name]:
expected_subjects = []
ctx = SSL.Context(method=SSL.SSLv23_METHOD)
ctx.verify_mode = SSL.VERIFY_NONE
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
conn.do_handshake()
expected_subjects :list[X509Name] = conn.get_client_ca_list()
except SSL.Error as err:
raise SSL.Error from err
finally:
conn.close()
return expected_subjects
This did not have the client certificate, so the TLS connection would fail. There are a lot of bad practices here, but unfortunately they are necessary and the only way to gather the message from the server before we actually want to attempt client authentication using hte correct certificate.
Next you load the cert based on the server:
from pathlib import Path
from OpenSSL.crypto import load_certificate, FILETYPE_PEM
from pathlib import Path
def check_client_cert_issuer(client_pem :str, expected_subjects :list) -> str:
client_cert = None
if len(expected_subjects) > 0:
client_cert_path = Path(client_pem)
cert = load_certificate(FILETYPE_PEM, client_cert_path.read_bytes())
issuer_subject = cert.get_issuer()
for check in expected_subjects:
if issuer_subject.commonName == check.commonName:
client_cert = client_pem
break
if client_cert is None or not isinstance(client_cert, str):
raise Exception('X509_V_ERR_SUBJECT_ISSUER_MISMATCH') # OpenSSL error code 29
return client_cert
In a real app (not an example snippet) you would have a database of some sort to take the server subject and lookup the location of the cert to load - this example does it in reverse for demonstration only.
Make the TLS connection, and capture any OpenSSL errors:
from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509, FILETYPE_PEM
from certifi import where
import idna
def openssl_verifier(conn :SSL.Connection, server_cert :X509, errno :int, depth :int, preverify_ok :int):
ok = 1
verifier_errors = conn.get_app_data()
if not isinstance(verifier_errors, list):
verifier_errors = []
if errno in OPENSSL_CODES.keys():
ok = 0
verifier_errors.append((server_cert, OPENSSL_CODES[errno]))
conn.set_app_data(verifier_errors)
return ok
client_pem = '/path/to/client.pem'
client_issuer_ca = '/path/to/ca.pem'
host = 'example.com'
port = 443
ctx = SSL.Context(method=SSL.SSLv23_METHOD) # will negotiate TLS1.3 or lower protocol, what every is highest possible during negotiation
ctx.load_verify_locations(cafile=where())
if client_pem is not None:
ctx.use_certificate_file(certfile=client_pem, filetype=FILETYPE_PEM)
if client_issuer_ca is not None:
ctx.load_client_ca(cafile=client_issuer_ca)
ctx.set_verify(SSL.VERIFY_NONE, openssl_verifier)
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
conn.do_handshake()
verifier_errors = conn.get_app_data()
except SSL.Error as err:
raise SSL.Error from err
finally:
conn.close()
# handle your errors in your main app
print(verifier_errors)
Just make sure you handle those OPENSSL_CODES errors if any are encountered, the lookup dictionary is here.
Many validations occur pre verification inside OpenSSL itself and all PyOpenSSL will do is a basic validation. so we need to access these codes from OpenSSL if we want to do Client Authentication, i.e. on the client and throw away the response from an untrusted server if it fails any authentication checks on the client side, per Client Authorisation or rather mutual-TLS dictates

#Stof's solution is more complete than this one.
So I selected his answer as 'official' answer.
This answer predates his, but might still be of some interest.
With #Steffen Ullrich's help I found following solution,
which works for all the (nginx with a ssl_client_certificate setting) servers that I tested with.
It requires to install an external package
pip install pyopenssl
Then following work will work:
import socket
from OpenSSL import SSL
def get_client_cert_cas(hostname, port):
ctx = SSL.Context(SSL.SSLv23_METHOD)
# If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
# an empty result
ctx.set_options(SSL.OP_NO_TLSv1_3)
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
# next line for SNI
sock.set_tlsext_host_name(hostname.encode("utf-8"))
sock.connect((hostname, port))
# without handshake get_client_ca_list will be empty
sock.do_handshake()
return sock.get_client_ca_list()
The line sock.do_handshake() is required to trigger enough of the SSL protocol. Otherwise client_ca_list information doesn't seem to be populated.
At least for the servers, that I tested I had to make sure TLSv1.3 is not used. I don't know whether this is a bug, a feature or whether with TLSv1.3 another function has to be called prior to calling get_client_ca_list()
I am no pyopenssl expert, but could imagine, that there is a more elegant / more explicit way to get the same behavior.
but so far this works for me for all the servers, that I encountered.

SSH/SCP through Paramiko with key in string

I got a great Paramiko Python script to transfer file over SCP protocol. But My need is a single file (script.py) without any other file dependencies so I don't want to have an external file for my SSH private key.
What I'm trying to do is to embed the private key into a string variable and use this variable as the file needed by the script to connect. I tried with the StringIO library but it doesn't seems to work :
hostname = '1.1.1.1' # remote hostname where SSH server is running
port = 22
username = 'user'
password = 'pswd'
dir_local='/home/paramikouser/local_data'
dir_remote = "remote_machine_folder/subfolder"
glob_pattern='*.*'
import os
import glob
import paramiko
import md5
import StringIO
private_key = StringIO.StringIO('-----BEGIN RSA PRIVATE KEY-----\n-----END RSA PRIVATE KEY-----')
def agent_auth(transport, username):
"""
Attempt to authenticate to the given transport using any of the private
keys available from an SSH agent or from a local private RSA key file (assumes no pass phrase).
"""
try:
ki = paramiko.RSAKey.from_private_key_file(private_key)
except Exception, e:
print 'Failed loading' % (private_key, e)
agent = paramiko.Agent()
agent_keys = agent.get_keys() + (ki,)
if len(agent_keys) == 0:
return
for key in agent_keys:
print 'Trying ssh-agent key %s' % key.get_fingerprint().encode('hex'),
try:
transport.auth_publickey(username, key)
print '... success!'
return
except paramiko.SSHException, e:
print '... failed!', e
# get host key, if we know one
hostkeytype = None
hostkey = None
files_copied = 0
try:
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
except IOError:
try:
# try ~/ssh/ too, e.g. on windows
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/ssh/known_hosts'))
except IOError:
print '*** Unable to open host keys file'
host_keys = {}
if host_keys.has_key(hostname):
hostkeytype = host_keys[hostname].keys()[0]
hostkey = host_keys[hostname][hostkeytype]
print 'Using host key of type %s' % hostkeytype
# now, connect and use paramiko Transport to negotiate SSH2 across the connection
try:
print 'Establishing SSH connection to:', hostname, port, '...'
t = paramiko.Transport((hostname, port))
t.start_client()
agent_auth(t, username)
if not t.is_authenticated():
print 'RSA key auth failed! Trying password login...'
t.connect(username=username, password=password, hostkey=hostkey)
else:
sftp = t.open_session()
sftp = paramiko.SFTPClient.from_transport(t)
# dirlist on remote host
# dirlist = sftp.listdir('.')
# print "Dirlist:", dirlist
How can I use this string as a key?

Use RSAKey.from_private_key:
ki = paramiko.RSAKey.from_private_key(private_key)
See How do use paramiko.RSAKey.from_private_key()?
The answer on the above question shows code for Python 3.
In Python 2.7, this works:
import os
import glob
import paramiko
import StringIO
private_key_file = StringIO.StringIO()
private_key_file.write('-----BEGIN RSA PRIVATE KEY-----\nMIIEoQIBAAKCAQEAvG9YlF2da0jJ5PvvlmVnVnYYFc7kkJuC0wvsACVuvep/sds5\nIEX0e+/rq9UBj/V3rzsvbHzb6IVulSjEqcM32NA4SyqR1m5jAj/WVDXQcxzruBDO\nZbdNhDS1T4+HckTWzttAE4o83bRju+3BhR9CtrDtt+7CSei4MccSMEH7yxo1BGuL\nONfkhB6qAWh55T6tamTyjLg9R9xqBkG6x3ZmoOB9j/11P5awuUoE1DfbqQ3KMLSR\nr64unavECfBaBYvuxlWbxRJFAN8C95oE+Kbc1g1bEL9du6FIeHZ/eaBbkcl84Fm7\nswdHBnd7+tqfo4TGzvbEW4H2ZQLiuiGK23ao0wIBJQKCAQEAiYGvV4KVd81VDuFb\nzp0GOCypyrmSCKjVFowowdYgYRLnj5/5QRB0IxbcaKJbFgYm56e60qBNclOIC/sn\nuiasNm5uRLBclY7SoMbM1aq6tN3AxJakc70c4+8chip3mJMZStdYRZw6QOtrX5+o\n5JpFcI7yqNDS96nShTBnN/jMf3K6yjQjvTv/DJi9SHJ6dTtrY1AuUNEoiO3cwgeH\nFks169756L+fpweL4VjQl4UyClL0bwHWpe579XzjBV0AlGu1tHaE5zslTPtGw1lg\nnZhj/7skZKAIGQxIzfmGv4QEcvePKYzM8EUhOr/0O3BHjLC0lp5hMwmsPJfaHlMb\nBak0JQKBgQD0cRu65WNkCcRlpuUvp5/kiMvu7PmcFUsY6dMeV0bL4oQ+PCqfwXFj\nhkyS7V2DJnllYPwi6E68soie+IL1blmY7hWcoznJ48PWJ0bJmqBgzhpC623RtTKS\ny/O0IbrGKPpaRGfD/PAvqJOpwx7Im2k6/UVQ0OYSurC8CB3BDRTCXQKBgQDFWEq7\ny2SntPFA9zu+31bW57lb26l8nNmUXmRLnXyvqomAkCGSadiW/i5nBEBDV/zJ/rXp\n0QWrmrpfvjnMF6g26m4sj6Pfs5zoSV1+FEidqYDcytUPJnpR55Ulpshf57TGuFbx\n1SCnda1dmm3TzAzzKTc6MbSPV0krMyAgCP7E7wKBgFXijTPUDioRRQEe9pQz+eiD\nFzhFbHUcPPrqXu78EfSbsexaVCpK4qZtdNmtWDT/rhyzX4Hi6zthUpi4LgM0nAVM\nu3w5WX5JG0s+O3dEKoLgoXF1UBk/qfw50iqIZDfJNV38W889McuOQbgvzIusObrH\nsJIENSks1k/nLQx6N7npAoGAUAEzDdzVx3LeWJuUwwCY06oM4Azxrw8nxochvco5\nd6YAZI11ZN7NbaVRFQG5MA7p8QZlbKDYyQdgUFQJl+3qP8bSuB6Oix9Ncu1Panbt\nAaWVGz14+E3ej+hDYkqIlZVJSaStoE978NyuETDEvaXAD40/5yjoVclwsKYGGtM2\n2jcCgYA6v1tvd2QdDeijiSRnXAeJ1hDLB8Jj2WJqnDZ7dQ5+XTIKfY4POIpHCPx8\n6Uk4NCnyJGmBHog1M7Bjb/o0c1UTid6CNBI4ciVaRyXXcy6Czup2EhkiNGom2883\n8+9pdxShKf0pJCqdZxJdVmg1NHZnr20PwN7PASbVcRg3t+wt2g==\n-----END RSA PRIVATE KEY-----')
private_key_file.seek(0)
ki = paramiko.RSAKey.from_private_key(private_key_file)

Error in SSL wrapper while cloning with Mercurial

I've recently started working with Bitbucket and mercurial. I can work with git repositories just fine, but when I try to clone a mercurial one, it crashes in /Users/foobar/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mercurial-3.4.2-py2.7-macosx-10.6-x86_64.egg/mercurial/sslutil.py on line 36, with the error:
ValueError: can't clear options before OpenSSL 0.9.8m
I am running OpenSSL 1.0.2c.
As is apparent from the file path, I downloaded python (and mercurial) through Enthought Canopy
I searched around and I found someone with a similar problem: https://bitbucket.org/durin42/hgsubversion/issues/439/unknown-exception-in-dispatchpy-value. His partial solution (commenting out the offending line) was enough to make my download work, but I confess I am out of my depth in trying to determine whether this will cause any security issues with me using Mercurial. Are there any potential security issues with this? Is there any additional information you'd need to begin to answer this?
The file I edited is below (search for "==========" to find the line I removed).
# sslutil.py - SSL handling for mercurial
#
# Copyright 2005, 2006, 2007, 2008 Matt Mackall <mpm#selenic.com>
# Copyright 2006, 2007 Alexis S. L. Carvalho <alexis#cecm.usp.br>
# Copyright 2006 Vadim Gelfer <vadim.gelfer#gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
import os, sys
from mercurial import util
from mercurial.i18n import _
_canloaddefaultcerts = False
try:
# avoid using deprecated/broken FakeSocket in python 2.6
import ssl
CERT_REQUIRED = ssl.CERT_REQUIRED
try:
ssl_context = ssl.SSLContext
_canloaddefaultcerts = util.safehasattr(ssl_context,
'load_default_certs')
def ssl_wrap_socket(sock, keyfile, certfile, cert_reqs=ssl.CERT_NONE,
ca_certs=None, serverhostname=None):
# Allow any version of SSL starting with TLSv1 and
# up. Note that specifying TLSv1 here prohibits use of
# newer standards (like TLSv1_2), so this is the right way
# to do this. Note that in the future it'd be better to
# support using ssl.create_default_context(), which sets
# up a bunch of things in smart ways (strong ciphers,
# protocol versions, etc) and is upgraded by Python
# maintainers for us, but that breaks too many things to
# do it in a hurry.
sslcontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
# ================================================
# LINE I REMOVED
sslcontext.options &= ssl.OP_NO_SSLv2 & ssl.OP_NO_SSLv3
# ================================================
if certfile is not None:
sslcontext.load_cert_chain(certfile, keyfile)
sslcontext.verify_mode = cert_reqs
if ca_certs is not None:
sslcontext.load_verify_locations(cafile=ca_certs)
elif _canloaddefaultcerts:
sslcontext.load_default_certs()
sslsocket = sslcontext.wrap_socket(sock,
server_hostname=serverhostname)
# check if wrap_socket failed silently because socket had been
# closed
# - see http://bugs.python.org/issue13721
if not sslsocket.cipher():
raise util.Abort(_('ssl connection failed'))
return sslsocket
except AttributeError:
def ssl_wrap_socket(sock, keyfile, certfile, cert_reqs=ssl.CERT_NONE,
ca_certs=None, serverhostname=None):
sslsocket = ssl.wrap_socket(sock, keyfile, certfile,
cert_reqs=cert_reqs, ca_certs=ca_certs,
ssl_version=ssl.PROTOCOL_TLSv1)
# check if wrap_socket failed silently because socket had been
# closed
# - see http://bugs.python.org/issue13721
if not sslsocket.cipher():
raise util.Abort(_('ssl connection failed'))
return sslsocket
except ImportError:
CERT_REQUIRED = 2
import socket, httplib
def ssl_wrap_socket(sock, keyfile, certfile, cert_reqs=CERT_REQUIRED,
ca_certs=None, serverhostname=None):
if not util.safehasattr(socket, 'ssl'):
raise util.Abort(_('Python SSL support not found'))
if ca_certs:
raise util.Abort(_(
'certificate checking requires Python 2.6'))
ssl = socket.ssl(sock, keyfile, certfile)
return httplib.FakeSocket(sock, ssl)
def _verifycert(cert, hostname):
'''Verify that cert (in socket.getpeercert() format) matches hostname.
CRLs is not handled.
Returns error message if any problems are found and None on success.
'''
if not cert:
return _('no certificate received')
dnsname = hostname.lower()
def matchdnsname(certname):
return (certname == dnsname or
'.' in dnsname and certname == '*.' + dnsname.split('.', 1)[1])
san = cert.get('subjectAltName', [])
if san:
certnames = [value.lower() for key, value in san if key == 'DNS']
for name in certnames:
if matchdnsname(name):
return None
if certnames:
return _('certificate is for %s') % ', '.join(certnames)
# subject is only checked when subjectAltName is empty
for s in cert.get('subject', []):
key, value = s[0]
if key == 'commonName':
try:
# 'subject' entries are unicode
certname = value.lower().encode('ascii')
except UnicodeEncodeError:
return _('IDN in certificate not supported')
if matchdnsname(certname):
return None
return _('certificate is for %s') % certname
return _('no commonName or subjectAltName found in certificate')
# CERT_REQUIRED means fetch the cert from the server all the time AND
# validate it against the CA store provided in web.cacerts.
#
# We COMPLETELY ignore CERT_REQUIRED on Python <= 2.5, as it's totally
# busted on those versions.
def _plainapplepython():
"""return true if this seems to be a pure Apple Python that
* is unfrozen and presumably has the whole mercurial module in the file
system
* presumably is an Apple Python that uses Apple OpenSSL which has patches
for using system certificate store CAs in addition to the provided
cacerts file
"""
if sys.platform != 'darwin' or util.mainfrozen() or not sys.executable:
return False
exe = os.path.realpath(sys.executable).lower()
return (exe.startswith('/usr/bin/python') or
exe.startswith('/system/library/frameworks/python.framework/'))
def _defaultcacerts():
"""return path to CA certificates; None for system's store; ! to disable"""
if _plainapplepython():
dummycert = os.path.join(os.path.dirname(__file__), 'dummycert.pem')
if os.path.exists(dummycert):
return dummycert
if _canloaddefaultcerts:
return None
return '!'
def sslkwargs(ui, host):
kws = {}
hostfingerprint = ui.config('hostfingerprints', host)
if hostfingerprint:
return kws
cacerts = ui.config('web', 'cacerts')
if cacerts == '!':
pass
elif cacerts:
cacerts = util.expandpath(cacerts)
if not os.path.exists(cacerts):
raise util.Abort(_('could not find web.cacerts: %s') % cacerts)
else:
cacerts = _defaultcacerts()
if cacerts and cacerts != '!':
ui.debug('using %s to enable OS X system CA\n' % cacerts)
ui.setconfig('web', 'cacerts', cacerts, 'defaultcacerts')
if cacerts != '!':
kws.update({'ca_certs': cacerts,
'cert_reqs': CERT_REQUIRED,
})
return kws
class validator(object):
def __init__(self, ui, host):
self.ui = ui
self.host = host
def __call__(self, sock, strict=False):
host = self.host
cacerts = self.ui.config('web', 'cacerts')
hostfingerprint = self.ui.config('hostfingerprints', host)
if not getattr(sock, 'getpeercert', False): # python 2.5 ?
if hostfingerprint:
raise util.Abort(_("host fingerprint for %s can't be "
"verified (Python too old)") % host)
if strict:
raise util.Abort(_("certificate for %s can't be verified "
"(Python too old)") % host)
if self.ui.configbool('ui', 'reportoldssl', True):
self.ui.warn(_("warning: certificate for %s can't be verified "
"(Python too old)\n") % host)
return
if not sock.cipher(): # work around http://bugs.python.org/issue13721
raise util.Abort(_('%s ssl connection error') % host)
try:
peercert = sock.getpeercert(True)
peercert2 = sock.getpeercert()
except AttributeError:
raise util.Abort(_('%s ssl connection error') % host)
if not peercert:
raise util.Abort(_('%s certificate error: '
'no certificate received') % host)
peerfingerprint = util.sha1(peercert).hexdigest()
nicefingerprint = ":".join([peerfingerprint[x:x + 2]
for x in xrange(0, len(peerfingerprint), 2)])
if hostfingerprint:
if peerfingerprint.lower() != \
hostfingerprint.replace(':', '').lower():
raise util.Abort(_('certificate for %s has unexpected '
'fingerprint %s') % (host, nicefingerprint),
hint=_('check hostfingerprint configuration'))
self.ui.debug('%s certificate matched fingerprint %s\n' %
(host, nicefingerprint))
elif cacerts != '!':
msg = _verifycert(peercert2, host)
if msg:
raise util.Abort(_('%s certificate error: %s') % (host, msg),
hint=_('configure hostfingerprint %s or use '
'--insecure to connect insecurely') %
nicefingerprint)
self.ui.debug('%s certificate successfully verified\n' % host)
elif strict:
raise util.Abort(_('%s certificate with fingerprint %s not '
'verified') % (host, nicefingerprint),
hint=_('check hostfingerprints or web.cacerts '
'config setting'))
else:
self.ui.warn(_('warning: %s certificate with fingerprint %s not '
'verified (check hostfingerprints or web.cacerts '
'config setting)\n') %
(host, nicefingerprint))

ValueError: can't clear options before OpenSSL 0.9.8m
I am running OpenSSL 1.0.2c.
Just a guess, but it sounds like the script is using Apple's version of OpenSSL, which is 0.9.8 (and not OpenSSL 1.0.2 that you built things against).
To ensure you use your OpenSSL 1.0.2, use DYLD_LIBRARY_PATH. Its similar to LD_LIBRARY_PATH on Linux.
The other option is to omit shared from the OpenSSL Configure, so that your gear can only perform static linking. Apple's linker always uses the *.dylib if available (even on iOS, where its not allowed), so you have to be careful with it.

Assuming OS X.
Try checking what python you're using:
which python
/usr/local/bin/python
Then see where that's pointing
ls -al /usr/local/bin/python
/usr/local/bin/python -> /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python
Make sure you're using /System/Library/Frameworks python, if not, delete the /usr/local/bin/python symlink and recreate it:
ln -s /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python /usr/local/bin/python

The line commented out was one that disabled use of some older and weaker encryption protocols; commenting it out allows you to use those older protocols. Possibly the server doesn't support the more recent protocols. Unless you are worried about the NSA getting their hands on your code, this is probably not that big a deal.

Django ssh failed during sftp connection

This is the error am getting
ri#ri-desktop:~/workspace/ssh$ python ssh.py
Establishing SSH connection to: upload.partner.com 19321 ...
Failed loading ~/workspace/ssh/rsa_private_key
No handlers could be found for logger "paramiko.transport"
Trying ssh-agent key b747f6899b3a450e63bc6faab1625686 ... failed! key cannot be used for signing
*** Caught exception: <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'get_fingerprint'
============================================================
Total files copied: 0
All operations complete!
============================================================
ri#ri-desktop:~/workspace/ssh$
This is my setup code. i took this setup code from activstate . I given correct path only and my doubt is firstly the error showing Failed loading ~/workspace/ssh/rsa_private_key. but its showing some ssh-agent key failed what is that?
hostname = 'upload.partner.com' # remote hostname where SSH server is running
port = 19321
username = 'music--test'
password = 'grg'
rsa_private_key = r"~/workspace/ssh/rsa_private_key"
dir_local='~/workspace/ssh/New/'
dir_remote = "remote_machine_folder/subfolder"
glob_pattern='*.*'
import os
import glob
import paramiko
import md5
rsa = None
def agent_auth(transport, username):
ki = None
#ppk = None
Attempt to authenticate to the given transport using any of the private
keys available from an SSH agent or from a local private RSA key file (assumes no pass phrase).
try:
ki = paramiko.RSAKey.from_private_key_file(rsa_private_key)
except Exception, e:
print 'Failed loading {0}'.format (rsa_private_key, e)
agent = paramiko.Agent()
agent_keys = agent.get_keys() + (ki,)
if len(agent_keys) == 0:
return
for key in agent_keys:
print 'Trying ssh-agent key {0}'.format(key.get_fingerprint().encode('hex')),
try:
transport.auth_publickey(username, key)
print '... success!'
return
except paramiko.SSHException, e:
print '... failed!', e
hostkeytype = None
hostkey = None
files_copied = 0
try:
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
except IOError:
try:
# try ~/ssh/ too, e.g. on windows
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/ssh/known_hosts'))
except IOError:
print ' Unable to open host keys file'
host_keys = {hostname:'upload.partner.com'}
if hostname in host_keys:
hostkeytype = host_keys[hostname].keys()[0]
hostkey = host_keys[hostname][hostkeytype]
print 'Using host key of type {0}'.format(hostkeytype)
try:
print 'Establishing SSH connection to:', hostname, port, '...'
t = paramiko.Transport((hostname, port))
t.connect()
agent_auth(t, username)
if not t.is_authenticated():
print 'RSA key auth failed! Trying password login...'
t.connect(username=username, password=password, hostkey=hostkey)
else:
sftp = t.open_session()
sftp = paramiko.SFTPClient.from_transport(t)
try:
sftp.mkdir(dir_remote)
except IOError, e:
print '(assuming ', dir_remote, 'exists)', e
for fname in glob.glob(dir_local + os.sep + glob_pattern):
is_up_to_date = False
if fname.lower().endswith('xml'):
local_file = os.path.join(dir_local, fname)
remote_file = dir_remote + '/' + os.path.basename(fname)
try:
if sftp.stat(remote_file):
local_file_data = open(local_file, "rb").read()
remote_file_data = sftp.open(remote_file).read()
md1 = md5.new(local_file_data).digest()
md2 = md5.new(remote_file_data).digest()
if md1 == md2:
is_up_to_date = True
print "UNCHANGED:", os.path.basename(fname)
else:
print "MODIFIED:", os.path.basename(fname),
except:
print "NEW: ", os.path.basename(fname),
if not is_up_to_date:
print 'Copying', local_file, 'to ', remote_file
sftp.put(local_file, remote_file)
files_copied += 1
t.close()
except Exception, e:
print '*** Caught exception: %s: %s' % (e.__class__, e)
try:
t.close()
except:
pass
print '=' * 60
print 'Total files copied:',files_copied
print 'All operations complete!'
print '=' * 60
can you help me to solve this error

EDIT After much to and fro, the solution was simply to add user's private key to she ssh agent via ssh-add.
Actually, you might be trying to use your public key, not your private key.
The file name "id-rsa.pub" looks suspiciously like a public key. The corresponding private key is usually named "id-rsa". Public keys look something like this:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCUmHZlySQqmZCGgE0NKWYyYlPHb3g1Bwdll2ztngUWNkrDSWGmLq6IqJP9EwwxungwJkdkR/U86gFv5MQfQ92+0ote7pUXOACwHfqvIoUXXFI3ZLo/C2cuqDIO7fcO50KKGFAuWbjTd3rugbpoMnNqT99wz/1lrCkTsJLd0YxtRo/QsJ8jiZXRuaEzbdXKwZJaP8G3eBHSMiEa1781yWklk50xxLk2qtpWVzen+Om6InbQ2cR6bBvfiA4B3LES53ccmMfzrCygjl8B6yaV3NI60Re5oNdyNrPZgH4VXf5p4VwrKpY4dCcJZyNmHlFhJlTgZu25uKAbp8Wk4u1ky0vJ mhawke#localhost.localdomain
Whereas unencypted private keys look something like this:
-----BEGIN RSA PRIVATE KEY-----
MIICXQIBAAKBgQCfqBLoK4Vec7r0df4a2CYNzmhJn74qIDqbJnkGasHcN5/GYuDv
.
.
xLCNrVMXYPd1I7L5NGzZalaTrS+DkgLwNvGhkVGKUGao
-----END RSA PRIVATE KEY-----
and encrypted private keys like this:
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,1ACD09B7F078AEB84B9A109979B77CBB
LDEuf08/xMGsHyesZxFGkRNHf8C78vpdDJyBBManOt/mRf/5fkjOel8RgoH4oYEz
.
.
tKjd+PlR4WLluMPFeHujdqhvyC4fQeFzWqak+rlUG5o3lm+TAcKqjypAEU4RVUuW
-----END RSA PRIVATE KEY-----
Check to see if you are using the correct key file. You can test it by:
openssl rsa -text -noout -in private_key_file
You shouldn't see any errors.

The first problem is occurring in this block of code
rsa_private_key = r"~/workspace/ssh/rsa_private_key"
.
.
.
try:
ki = paramiko.RSAKey.from_private_key_file(rsa_private_key)
except Exception, e:
print 'Failed loading {0}'.format (rsa_private_key, e)
I don't think that from_private_key_file() handles file expansion, so the ~ is interpreted literally as a tilde. Try changing rsa_private_key to the absolute file name.
Also the print statement in your except clause doesn't print out the exception. I suggest changing it to print 'Failed loading {0} : {1}'.format (rsa_private_key, e). You will probably then see the cause of the problem in the following error message:
Failed loading ~/workspace/ssh/rsa_private_key : [Errno 2] No such file or directory: '~/workspace/ssh/rsa_private_key'
Also, you may need to supply a password to from_private_key_file() if your key is encrypted.

How can I connect to a mail server using SMTP over SSL using Python?

So I have been having a hard time sending email from my school's email address. It is SSL and I could only find this code online by Matt Butcher that works with SSL:
import smtplib, socket
__version__ = "1.00"
__all__ = ['SMTPSSLException', 'SMTP_SSL']
SSMTP_PORT = 465
class SMTPSSLException(smtplib.SMTPException):
"""Base class for exceptions resulting from SSL negotiation."""
class SMTP_SSL (smtplib.SMTP):
"""This class provides SSL access to an SMTP server.
SMTP over SSL typical listens on port 465. Unlike StartTLS, SMTP over SSL
makes an SSL connection before doing a helo/ehlo. All transactions, then,
are done over an encrypted channel.
This class is a simple subclass of the smtplib.SMTP class that comes with
Python. It overrides the connect() method to use an SSL socket, and it
overrides the starttles() function to throw an error (you can't do
starttls within an SSL session).
"""
certfile = None
keyfile = None
def __init__(self, host='', port=0, local_hostname=None, keyfile=None, certfile=None):
"""Initialize a new SSL SMTP object.
If specified, `host' is the name of the remote host to which this object
will connect. If specified, `port' specifies the port (on `host') to
which this object will connect. `local_hostname' is the name of the
localhost. By default, the value of socket.getfqdn() is used.
An SMTPConnectError is raised if the SMTP host does not respond
correctly.
An SMTPSSLError is raised if SSL negotiation fails.
Warning: This object uses socket.ssl(), which does not do client-side
verification of the server's cert.
"""
self.certfile = certfile
self.keyfile = keyfile
smtplib.SMTP.__init__(self, host, port, local_hostname)
def connect(self, host='localhost', port=0):
"""Connect to an SMTP server using SSL.
`host' is localhost by default. Port will be set to 465 (the default
SSL SMTP port) if no port is specified.
If the host name ends with a colon (`:') followed by a number,
that suffix will be stripped off and the
number interpreted as the port number to use. This will override the
`port' parameter.
Note: This method is automatically invoked by __init__, if a host is
specified during instantiation.
"""
# MB: Most of this (Except for the socket connection code) is from
# the SMTP.connect() method. I changed only the bare minimum for the
# sake of compatibility.
if not port and (host.find(':') == host.rfind(':')):
i = host.rfind(':')
if i >= 0:
host, port = host[:i], host[i+1:]
try: port = int(port)
except ValueError:
raise socket.error, "nonnumeric port"
if not port: port = SSMTP_PORT
if self.debuglevel > 0: print>>stderr, 'connect:', (host, port)
msg = "getaddrinfo returns an empty list"
self.sock = None
for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
try:
self.sock = socket.socket(af, socktype, proto)
if self.debuglevel > 0: print>>stderr, 'connect:', (host, port)
self.sock.connect(sa)
# MB: Make the SSL connection.
sslobj = socket.ssl(self.sock, self.keyfile, self.certfile)
except socket.error, msg:
if self.debuglevel > 0:
print>>stderr, 'connect fail:', (host, port)
if self.sock:
self.sock.close()
self.sock = None
continue
break
if not self.sock:
raise socket.error, msg
# MB: Now set up fake socket and fake file classes.
# Thanks to the design of smtplib, this is all we need to do
# to get SSL working with all other methods.
self.sock = smtplib.SSLFakeSocket(self.sock, sslobj)
self.file = smtplib.SSLFakeFile(sslobj);
(code, msg) = self.getreply()
if self.debuglevel > 0: print>>stderr, "connect:", msg
return (code, msg)
def setkeyfile(self, keyfile):
"""Set the absolute path to a file containing a private key.
This method will only be effective if it is called before connect().
This key will be used to make the SSL connection."""
self.keyfile = keyfile
def setcertfile(self, certfile):
"""Set the absolute path to a file containing a x.509 certificate.
This method will only be effective if it is called before connect().
This certificate will be used to make the SSL connection."""
self.certfile = certfile
def starttls():
"""Raises an exception.
You cannot do StartTLS inside of an ssl session. Calling starttls() will
return an SMTPSSLException"""
raise SMTPSSLException, "Cannot perform StartTLS within SSL session."
And then my code:
import ssmtplib
conn = ssmtplib.SMTP_SSL('HOST')
conn.login('USERNAME','PW')
conn.ehlo()
conn.sendmail('FROM_EMAIL', 'TO_EMAIL', "MESSAGE")
conn.close()
And got this error:
/Users/Jake/Desktop/Beth's Program/ssmtplib.py:116: DeprecationWarning: socket.ssl() is deprecated. Use ssl.wrap_socket() instead.
sslobj = socket.ssl(self.sock, self.keyfile, self.certfile)
Traceback (most recent call last):
File "emailer.py", line 5, in
conn = ssmtplib.SMTP_SSL('HOST')
File "/Users/Jake/Desktop/Beth's Program/ssmtplib.py", line 79, in init
smtplib.SMTP.init(self, host, port, local_hostname)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 239, in init
(code, msg) = self.connect(host, port)
File "/Users/Jake/Desktop/Beth's Program/ssmtplib.py", line 131, in connect
self.sock = smtplib.SSLFakeSocket(self.sock, sslobj)
AttributeError: 'module' object has no attribute 'SSLFakeSocket'
Thank you!

That code you've found seems to be for an older version, considering the deprecation warning. Maybe you can get by with the stdlib: There is a SMTP_SSL class as of Python 2.6, and as of at least 2.4 there is a starttls method on the plaintext SMTP class.