We Keep Coding

OpenSearch Python Client Connection SSL Error

I tried connecting to opensearch using python client but getting SSL Error.
OpenSearch Version: 2.0.1
Python Version:3.6
public IP of the opensearch host: 10.83.5.67
Here is the python code for connecting the client to opensearch:
from opensearchpy import *
host = '10.83.5.67'
port = 9200
auth = ('admin', 'admin') # For testing only. Don't store credentials in code.
ca_certs_path = '/etc/opensearch/cert.pem' # Provide a CA bundle if you use intermediate CAs with your root CA.
# Create the client with SSL/TLS enabled, but hostname verification disabled.
client = OpenSearch(
hosts = [{'host': host, 'port': port}],
http_compress = True, # enables gzip compression for request bodies
http_auth = auth,
use_ssl = True,
verify_certs = True,
ssl_assert_hostname = False,
ssl_show_warn = False,
ca_certs = ca_certs_path
)
The certificates used in opensearch.yml file are:
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/cert.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/privkey.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/fullchain.pem
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/cert.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/privkey.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/fullchain.pem
This is the error I am getting:
[root#ip-10-92-3-240 pyscripts]# python3 ospy4.py
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
ssl_context=context)
File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 367, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
_context=self, _session=session)
File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
self.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/opensearchpy/connection/http_urllib3.py", line 250, in perform_request
method, url, body, retries=Retry(False), headers=request_headers, **kw
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 344, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
ssl_context=context)
File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 367, in ssl_wrap_socket
return context.wrap_socket(sock)
File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
_context=self, _session=session)
File "/usr/lib64/python3.6/ssl.py", line 776, in __init__
self.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "ospy4.py", line 31, in <module>
response = client.indices.create(index_name, body=index_body)
File "/usr/local/lib/python3.6/site-packages/opensearchpy/client/utils.py", line 177, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "/usr/local/lib/python3.6/site-packages/opensearchpy/client/indices.py", line 125, in create
"PUT", _make_path(index), params=params, headers=headers, body=body
File "/usr/local/lib/python3.6/site-packages/opensearchpy/transport.py", line 405, in perform_request
raise e
File "/usr/local/lib/python3.6/site-packages/opensearchpy/transport.py", line 375, in perform_request
timeout=timeout,
File "/usr/local/lib/python3.6/site-packages/opensearchpy/connection/http_urllib3.py", line 261, in perform_request
raise SSLError("N/A", str(e), e)
opensearchpy.exceptions.SSLError: ConnectionError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)) caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897))
[root#ip-10-92-3-240 pyscripts]#
Not sure what's wrong I am using the same certificate in opensearch-dashboard.yml file, but can't connect using python client.
Any suggestion would be really helpful.
Thanks

HTTPS request with Python standard library

UPDATE: I managed to do a request with urllib2, but I'm still wondering what is happening here.
I would like to do a HTTPS request with Python.
This works fine with the requests module, but I don't want to use external dependencies, so I'd like to use the standard library.
httplib
When I follow this example I don't get a response. I get a timeout instead. I'm out of ideas as to what would cause this.
Code:
import requests
print requests.get('https://python.org')
from httplib import HTTPSConnection
conn = HTTPSConnection('www.python.org')
conn.request('GET', '/index.html')
print conn.getresponse()
Output:
<Response [200]>
Traceback (most recent call last):
File "test.py", line 6, in <module>
conn.request('GET', '/index.html')
File "C:\Python27\lib\httplib.py", line 1069, in request
self._send_request(method, url, body, headers)
File "C:\Python27\lib\httplib.py", line 1109, in _send_request
self.endheaders(body)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1282, in connect
HTTPConnection.connect(self)
File "C:\Python27\lib\httplib.py", line 831, in connect
self.timeout, self.source_address)
File "C:\Python27\lib\socket.py", line 575, in create_connection
raise err
socket.error: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
urllib
This fails for a different (but possibly related) reason. Code:
import urllib
print urllib.urlopen("https://python.org")
Output:
Traceback (most recent call last):
File "test.py", line 10, in <module>
print urllib.urlopen("https://python.org")
File "C:\Python27\lib\urllib.py", line 87, in urlopen
return opener.open(url)
File "C:\Python27\lib\urllib.py", line 215, in open
return getattr(self, name)(url)
File "C:\Python27\lib\urllib.py", line 445, in open_https
h.endheaders(data)
File "C:\Python27\lib\httplib.py", line 1065, in endheaders
self._send_output(message_body)
File "C:\Python27\lib\httplib.py", line 892, in _send_output
self.send(msg)
File "C:\Python27\lib\httplib.py", line 854, in send
self.connect()
File "C:\Python27\lib\httplib.py", line 1290, in connect
server_hostname=server_hostname)
File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
_context=self)
File "C:\Python27\lib\ssl.py", line 599, in __init__
self.do_handshake()
File "C:\Python27\lib\ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:727)
What is requests doing that makes it succeed where both of these libraries fail?

requests.get without timeout parameter mean no timeout at all.
httplib.HTTPSConnection accept parameter timeout in Python 2.6 and newer according to httplib docs. If your problem was caused by timeout, setting high enough timeout should help. Please try replacing:
conn = HTTPSConnection('www.python.org')
with:
conn = HTTPSConnection('www.python.org', timeout=300)
which will give 300 seconds (5 minutes) for processing.

python2.7: [SSL: UNKNOWN_PROTOCOL] unknown protocol

I'm trying to install ROS from source.
When I execute the command of installation, I get such an error:
Traceback (most recent call last):
File "/home/zyh/ros_catkin_ws/install_isolated/share/ros/core/rosbuild/bin/download_checkmd5.py", line 126, in <module>
sys.exit(main())
File "/home/zyh/ros_catkin_ws/install_isolated/share/ros/core/rosbuild/bin/download_checkmd5.py", line 73, in main
urllib.urlretrieve('https://github.com/assimp/assimp/archive/v3.1.1.zip', dest)
File "/usr/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/usr/lib/python2.7/urllib.py", line 245, in retrieve
fp = self.open(url, data)
File "/usr/lib/python2.7/urllib.py", line 213, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 443, in open_https
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 882, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 844, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1263, in connect
server_hostname=server_hostname)
File "/usr/lib/python2.7/ssl.py", line 363, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 611, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 840, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:661)
/home/zyh/ros_catkin_ws/install_isolated/share/mk/download_unpack_build.mk:37: recipe for target 'build/assimp-3.1.1/unpacked' failed
make[3]: *** [build/assimp-3.1.1/unpacked] Error 1
I don't know how to solve this issue. Maybe it's because I worked behind a proxy? If so, how to make urllib.urlretrieve work behind the proxy?

Add proxy settings to your global environment to see if it fixes the problem.
sudo gedit /etc/environment
Then add these two lines
http_proxy=http://your_proxy.com:443
https_proxy=https://your_proxy.com:443

Using ftplib to connect to secure ftp server ( EOF occurred in violation of protocol )

What i'm making wrong in this dead simple connect/list example? ( i'm using python 3.3 )
import ftplib
ftp = ftplib.FTP_TLS()
ftp.connect('localhost', 221, 20)
ftp.login()
ftp.prot_p()
ftp.retrlines('LIST')
Source : https://docs.python.org/2/library/ftplib.html#ftplib.FTP_TLS
Error output:
Traceback (most recent call last):
File "C:/projects/nipples/ftps_client.py", line 10, in <module>
ftp.login()
File "C:\Python33\lib\ftplib.py", line 703, in login
self.auth()
File "C:\Python33\lib\ftplib.py", line 719, in auth
ssl_version=self.ssl_version)
File "C:\Python33\lib\ssl.py", line 630, in wrap_socket
ciphers=ciphers)
File "C:\Python33\lib\ssl.py", line 350, in __init__
raise x
File "C:\Python33\lib\ssl.py", line 346, in __init__
self.do_handshake()
File "C:\Python33\lib\ssl.py", line 553, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:548)
Thanks in advance.

How to use tweepy STREAMING API with a proxy?

I am aware there is a patch for using the REST API with a proxy and it works for me. But the Streaming API uses a HTTPConnection which cannot be emulated by urllib and urllib2 (as far as I know). Is there any fix for this?
I tried using proxy with port, but it did not work.
In the streaming.py file, line 153.
if self.scheme == "http":
conn = httplib.HTTPConnection(self.api.proxy_host,self.api.proxy_port, timeout=self.timeout)
else:
conn = httplib.HTTPSConnection(self.api.proxy_host,self.api.proxy_port,timeout=self.timeout)
self.auth.apply_auth(url, 'POST', self.headers, self.parameters)
print conn.host
conn.connect()
conn.request('POST', self.scheme+'://'+self.host+self.url, self.body, headers=self.headers)
resp = conn.getresponse()
And, "self.scheme+'://'+self.host+ self.url" corresponds to - https://stream.twitter.com/1.1/statuses/filter.json?delimited=length
I get this error in return -
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2538, in run_code
exec code_obj in self.user_global_ns, self.user_ns
File "<ipython-input-3-4798d330f7cd>", line 1, in <module>
execfile('main.py')
File "main.py", line 130, in <module>
streamer.filter(track = ['AAP'])
File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 305, in filter
self._start(async)
File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 242, in _start
self._run()
File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 159, in _run
conn.connect()
File "/usr/lib/python2.7/httplib.py", line 1161, in connect
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Tweepy currently does not support using a proxy. However, support for proxies should be coming in the next few days. I'll update this answer with more information.

Finally found a github patch that enabled streaming api via proxy. It works just fine!
https://github.com/shogo82148/tweepy/blob/64c6266018920e0e36c6d8d1600adb6caa0840de/tweepy/streaming.py