We Keep Coding

AWS IOT Data: CERTIFICATE_VERIFY_FAILED

I run test scripts for AWS IOT in a bitbucket pipeline using python + boto3
It worked fine until recently, now i get the following error:
Traceback (most recent call last):
File "/localDebugRepo/tests/aws/test_iot_api.py", line 119, in test_set_get_owner
self.iot_util.set_owner(owner, self.test_thing)
File "/localDebugRepo/aws/iot_api.py", line 176, in set_owner
self.iot_data.update_thing_shadow(thingName=thing, payload=payload)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 663, in _make_api_call
operation_model, request_dict, request_context)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 256, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/usr/local/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/usr/local/lib/python3.6/site-packages/botocore/httpsession.py", line 281, in send
raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://data.iot.eu-central-1.amazonaws.com/things/thing-unittest/shadow [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
While I cannot reproduce this on my local system, reproducing the error with the default python:3.6.4 docker image is successful indicating that there might be an invalid certificate.
Intrestingly, running the following command in pipeline is succesfull:
openssl s_client -connect data.iot.eu-central-1.amazonaws.com:443
root#f30a34330be5:/localDebugRepo# openssl s_client -connect data.iot.eu-central-1.amazonaws.com:443
CONNECTED(00000003)
depth=2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 C = US, O = Symantec Corporation, OU = Symantec Trust Network, CN = Symantec Class 3 Secure Server CA - G4
verify return:1
depth=0 C = US, ST = Washington, L = Seattle, O = "Amazon.com, Inc.", CN = *.iot.eu-central-1.amazonaws.com
verify return:1
140686038922896:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.iot.eu-central-1.amazonaws.com
i:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
1 s:/C=US/O=Symantec Corporation/OU=Symantec Trust Network/CN=Symantec Class 3 Secure Server CA - G4
i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
---
any advice on how can i debug this further would be greatly apreciated

It would appear that AWS has bad certs for the last several hours.
I do not subscribe to a support tier, so I don't know how to tell them.
I am getting the same problem; boto3 reports that bad cert (which you can verify in a browser).
All of my IoT functions are affected, though if I run it locally (not as a lambda), it seems to work.
Perhaps someone has a way to tell Amazon their little problem?
Edit:
See:
https://forums.aws.amazon.com/thread.jspa?messageID=967311&#967311
and
https://github.com/boto/boto3/issues/2686
for the fix. You shouldn't use the defaults for creating your dataplane client, because certifi (python) has been fixed to ignore the Symantec CA for the URL, and Amazon isn't going to fix it.

The solution pointed out by Eric Lyons did not worked for me directly. The problem was with the endpoint provided by:
iot_client = boto3.client("iot", region_name=os.getenv("IOT_REGION"))
iot_client.describe_endpoint(endpointType="iot:Data-ATS").get("endpointAddress")
It fails during authentication:
I fixed it by getting the endpoint directly from the IOT-Core settings page:
client('iot-data',
aws_access_key_id = '<MY ACCESS KEY>',
aws_secret_access_key = '<MY ACCESS SECRET KEY>',
endpoint_url = '<MY ENDPOINT>');

Duplicity Backup: Errno 104 Connection reset by peer when backing up to Amazon S3

I use Duplicity to run backups from my local server to Amazon S3. This has been working fine for over a year. Three days ago, I started getting the following errors:
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/duplicity/backends/_boto_multi.py", line 204, in _upload
num_cb=max(2, 8 * bytes / (1024 * 1024))
File "/usr/lib/python2.7/site-packages/boto/s3/multipart.py", line 260, in upload_part_from_file
query_args=query_args, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1291, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 748, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 949, in _send_file_internal
query_args=query_args
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
retry_handler=retry_handler
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1068, in make_request
retry_handler=retry_handler)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 939, in _mexe
request.body, request.headers)
File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 842, in sender
http_conn.send(chunk)
File "/usr/lib64/python2.7/httplib.py", line 805, in send
self.sock.sendall(data)
File "/usr/lib64/python2.7/ssl.py", line 229, in sendall
v = self.send(data[count:])
File "/usr/lib64/python2.7/ssl.py", line 198, in send
v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer
These are still occurring even after I tried the following:
--added "s3-use-multiprocessing" to my script file
--added the following two lines to /etc/sysctl.conf:
net.ipv4.tcp_wmem = 4096 16384 512000
net.ipv4.tcp_rmem = 4096 87380 512000
-- ran sysctl -p to start using the above.
Three days ago, I started running Duplicity on a couple of other servers, backing up to a different bucket on the same account. That was when THIS server started reporting connection reset errors. The other servers are working fine, and all of them are using the same versions of Duplicity and Python. They are in different locations on different subnets, but that shouldn't make a difference.
The original chunk size on the problem server was 25MB. It's 250MB on the others. What else can I look for? I'm guessing Amazon is resetting the connection, but why single out this server?

IOError: unsupported XML-RPC protocol while running yum command

When I try to turn any yum command I get the follwing message. I disabled and enabled SSL before this error occurred. As the system said RHNS-CA-CERT has expired, I removed the certificate and downloaded it using wget command. Then I tried to update the certificate using the yum command and that's where the problem started.
Here's the error message:
Loaded plugins: rhnplugin
Exception RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in <module>
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 285, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 105, in main
base.getOptionsConfig(args)
File "/usr/share/yum-cli/cli.py", line 228, in getOptionsConfig
self.conf
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 891, in <lambda>
conf = property(fget=lambda self: self._getConfig(),
File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 362, in _getConfig
self.plugins.run('init')
File "/usr/lib/python2.6/site-packages/yum/plugins.py", line 184, in run
func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/share/yum-plugins/rhnplugin.py", line 118, in init_hook
login_info = up2dateAuth.getLoginInfo(timeout=timeout)
File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 219, in getLoginInfo
login(timeout=timeout)
File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 170, in login
server = rhnserver.RhnServer(timeout=timeout)
File "/usr/share/rhn/up2date_client/rhnserver.py", line 154, in __init__
timeout=timeout)
File "/usr/share/rhn/up2date_client/rpcServer.py", line 160, in getServer
timeout=timeout)
File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 169, in __init__
self._reset_host_handler_and_type()
File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 315, in _reset_host_handler_and_type
raise IOError, "unsupported XML-RPC protocol"
IOError: unsupported XML-RPC protocol

OK, my guess is you are running against RHN Classic (rhn.redhat.com). There were erratum fixing this expired certificate and here comes relevant knowleadge base article:
System connection to RHN fails with "The certificate is expired, or certificate verify failed" errors
https://access.redhat.com/solutions/353033
Traceback with IOError: unsupported XML-RPC protocol leads me to guess that you have incorrect serverURL in /etc/sysconfig/rhn/up2date. It should look like this:
serverURL=https://xmlrpc.rhn.redhat.com/XMLRPC

Force SSLv3 or TLSV1 in splinter

I am trying to visit gateway.playneverwinter.com with splinter
from splinter import Browser
browser = Browser()
browser.visit('https://gateway.playneverwinter.com')
if browser.is_text_present('Neverwinter'):
print("Yes, we made it to the entrance of the Prime Material Plane!")
else:
print("Fumble")
browser.quit()
It fails with
File "gateway_bot.py", line 10, in <module>
browser.visit('https://gateway.playneverwinter.com')
File "/usr/local/lib/python3.4/dist-packages/splinter/driver/webdriver/__init__.py", line 53, in visit
self.connect(url)
File "/usr/local/lib/python3.4/dist-packages/splinter/request_handler/request_handler.py", line 23, in connect
self._create_connection()
File "/usr/local/lib/python3.4/dist-packages/splinter/request_handler/request_handler.py", line 53, in _create_connection
self.conn.endheaders()
File "/usr/lib/python3.4/http/client.py", line 1061, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 906, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 841, in send
self.connect()
File "/usr/lib/python3.4/http/client.py", line 1205, in connect
server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 364, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 578, in __init__
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 805, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)
Firefox is able to connect and browse this site without issue, tough. After some diagnostic
$ openssl s_client -connect gateway.playneverwinter.com:443
CONNECTED(00000003)
139745006343840:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
I found that it looked like a fixed issue in OpenSSL and that forcing either SSLv3 or TLSv1 allowed me to connect (and that I could then download the target with cURL) e.g. either of
openssl s_client -ssl3 -connect gateway.playneverwinter.com:443
openssl s_client -tls1 -connect gateway.playneverwinter.com:443
According to the comments in the OpenSSL ticket, I expect that the issue is on the server side, but as I do not have access to it, it is quite unhelpful. So, for a quick fix, is there a way to force splinter to use SSLv3 or TLSv1?

After looking into it, the only way I can think of doing that would to be to go into that client.py file and change the initialization of their ssl stuff.

Following #Natecat suggestion, I wrote a monkey patch to force SSLv3 when this error occurs
# Monkey patch splinter to force SSLv3 on `ssl.SSLEOFError`
from splinter import request_handler
import ssl
from http import client as http_client
_old_req = request_handler.request_handler.RequestHandler._create_connection
def _splinter_sslv3_patch(self):
try:
_old_req(self)
except ssl.SSLEOFError:
self.conn = http_client.HTTPSConnection(self.host, self.port,
context=ssl.SSLContext(ssl.PROTOCOL_SSLv3))
self.conn.putrequest('GET', self.path)
self.conn.putheader('User-agent', 'python/splinter')
if self.auth:
self.conn.putheader("Authorization", "Basic %s" % self.auth)
self.conn.endheaders()
request_handler.request_handler.RequestHandler._create_connection = _splinter_sslv3_patch

Tornado https ssl error

When I kick off my tornado https server, I am asked for a PEM password (which I did not set, so I just hit enter)
Enter PEM pass phrase: 2013-10-17 14:24:46,730 ioloop.py:660 Exception
in I/O handler for fd 3 Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/tornado/ioloop.py", line 653, in
start
self._handlers[fd](fd, events) File "/usr/lib/python2.7/site-packages/tornado/stack_context.py", line 241,
in wrapped
callback(*args, **kwargs) File "/usr/lib/python2.7/site-packages/tornado/netutil.py", line 141, in
accept_handler
callback(connection, address) File "/usr/lib/python2.7/site-packages/tornado/tcpserver.py", line 212, in
_handle_connection
do_handshake_on_connect=False) File "/usr/lib/python2.7/site-packages/tornado/netutil.py", line 322, in
ssl_wrap_socket
return ssl.wrap_socket(socket, **dict(context, **kwargs)) File "/usr/lib64/python2.7/ssl.py", line 387, in wrap_socket
ciphers=ciphers) File "/usr/lib64/python2.7/ssl.py", line 141, in __init__
ciphers) SSLError: [Errno 336265225] _ssl.c:351: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib Enter
PEM pass phrase:
I generated the keys with these instructions: http://www.thegeekstuff.com/2009/07/linux-apache-mod-ssl-generate-key-csr-crt-file/
Then modified the tornado spin-up as the following
SSL_OPTIONS = {
"certfile": "path/to/crt",
"keyfile": "path/to/private/key", }
https_server = tornado.httpserver.HTTPServer(application, ssl_options=SSL_OPTIONS)
I can't find any solution to this problem. I am using the latest tornado version and python 2.7
Thanks!

If you followed the instructions on that page, your key still has a password, it's just empty. I'm not sure if it's possible to use a key with a password non-interactively in Python 2 (the SSLContext.load_cert_chain method for this is new in Python 3.2). You can create a key with no password at all (which will disable the prompt) by changing -des3 to -nodes in the first step: openssl genrsa -nodes -out www.thegeekstuff.com.key 1024 (and then repeating the remaining steps for the new key), or using openssl rsa to strip the password from the key you've already got (see http://www.mnxsolutions.com/apache/removing-a-passphrase-from-an-ssl-key.html)