We Keep Coding

SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY Error

I am sending multiple requests with aiohttp using tor's http proxy (with aiohttp_socks)
After some requests are done I am getting the following error:
Traceback (most recent call last):
File "main.py", line 171, in <module>
loop.run_until_complete(future)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "main.py", line 95, in get_market_pages
async with session.get(active_link, headers=headers) as response:
File "/home/mrlalatg/.local/lib/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
self._resp = await self._coro
File "/home/mrlalatg/.local/lib/python3.8/site-packages/aiohttp/client.py", line 504, in _request
await resp.start(conn)
File "/home/mrlalatg/.local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 847, in start
message, payload = await self._protocol.read() # type: ignore # noqa
File "/home/mrlalatg/.local/lib/python3.8/site-packages/aiohttp/streams.py", line 591, in read
await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 1] [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2745)
I can't find any information about this error, except the git discussion about a similar one - github
There I found a workaround (link) that I can modify to ignore this error, but it didn't work, the error is still there.
The modified version of a workaround:
SSL_PROTOCOLS = (asyncio.sslproto.SSLProtocol,)
try:
import uvloop.loop
except ImportError:
pass
else:
SSL_PROTOCOLS = (*SSL_PROTOCOLS, uvloop.loop.SSLProtocol)
def ignore_aiohttp_ssl_eror(loop):
"""Ignore aiohttp #3535 / cpython #13548 issue with SSL data after close
There is an issue in Python 3.7 up to 3.7.3 that over-reports a
ssl.SSLError fatal error (ssl.SSLError: [SSL: KRB5_S_INIT] application data
after close notify (_ssl.c:2609)) after we are already done with the
connection. See GitHub issues aio-libs/aiohttp#3535 and
python/cpython#13548.
Given a loop, this sets up an exception handler that ignores this specific
exception, but passes everything else on to the previous exception handler
this one replaces.
Checks for fixed Python versions, disabling itself when running on 3.7.4+
or 3.8.
"""
orig_handler = loop.get_exception_handler()
def ignore_ssl_error(loop, context):
if context.get("message") in {
"SSL error in data received",
"Fatal error on transport",
}:
# validate we have the right exception, transport and protocol
exception = context.get('exception')
protocol = context.get('protocol')
if (
isinstance(exception, ssl.SSLError)
and exception.reason == 'APPLICATION_DATA_AFTER_CLOSE_NOTIFY'
and isinstance(protocol, SSL_PROTOCOLS)
):
if loop.get_debug():
asyncio.log.logger.debug('Ignoring asyncio SSL KRB5_S_INIT error')
return
if orig_handler is not None:
orig_handler(loop, context)
else:
loop.default_exception_handler(context)
loop.set_exception_handler(ignore_ssl_error)

Catching Firebase 504 gateway timeout

I'm building a simple IOT device (with a Raspberry Pi Zero) which pulls data from Firebase Realtime Database every 1 second and checks for updates.
However, after a certain time (not sure exactly how much but somewhere between 1 hour and 3 hours) the program exits with a 504 Server Error: Gateway Time-out message.
I couldn't understand exactly why this is happening, I tried to recreate this error by disconnecting the Pi from the internet and I did not get this message. Instead, the program simply paused in a ref.get() line and automatically resumed running once the connection was back.
This device is meant to be always on, so ideally if I get some kind of error, I would like to restart the program / reinitiate the connection / reboot the Pi. Is there a way to achieve something like this?
It seems like the message is actually generated by the firebase_admin package.
Here is the error message:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/firebase_admin/db.py", line 944, in request
return super(_Client, self).request(method, url, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/firebase_admin/_http_client.py", line 105, in request
resp.raise_for_status()
File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://someFirebaseProject.firebaseio.com/someRef/subSomeRef/payload.json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/Desktop/project/main.py", line 94, in <module>
lastUpdate = ref.get()['lastUpdate']
File "/home/pi/.local/lib/python3.7/site-packages/firebase_admin/db.py", line 223, in get
return self._client.body('get', self._add_suffix(), params=params)
File "/home/pi/.local/lib/python3.7/site-packages/firebase_admin/_http_client.py", line 117, in body
resp = self.request(method, url, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/firebase_admin/db.py", line 946, in request
raise _Client.handle_rtdb_error(error)
firebase_admin.exceptions.UnknownError: Internal server error.
>>>

To reboot the whole Raspberry Pi, you can just run a shell command:
import os
os.system("sudo reboot")
I've had this problem too and usually feel safer with that, but there's obvious downsides. I'd try resetting the wifi connection or network interface in a similar way

How to fix invalid URL error while using http.client?

I am trying to hit an api using http.client of pyhton3 to simulate how I would do the same in a web browser.
However, http.client feels that url is inappropriate.
This is what I am trying to do.
import http.client
connection = http.client.HTTPSConnection("https://analyticsapi.zoho.com/api/EmailAddress/WorkspaceName/TableName?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=************&ZOHO_IMPORT_TYPE=APPEND&ZOHO_AUTO_IDENTIFY=TRUE&ZOHO_ON_IMPORT_ERROR=ABORT&ZOHO_CREATE_TABLE=TRUE&ZOHO_FILE=/home/dev1/Desktop/Zoho/temporary.csv")
connection.request("GET", "/")
response = connection.getresponse()
print("Status: {} and reason: {}".format(response.status, response.reason))
connection.close()
And this is the error I am getting.
$ python3 pyToTestPushingCSV.py
Traceback (most recent call last):
File "/usr/lib/python3.5/http/client.py", line 798, in _get_hostport
port = int(host[i+1:])
ValueError: invalid literal for int() with base 10: '//analyticsapi.zoho.com/api/usename/ATable/InsideTable?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=****
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "pyToTestPushingCSV.py", line 3, in <module>
connection = http.client.HTTPSConnection("https://analyticsapi.zoho.com/api/usename/ATable/InsideTable?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=****************&ZOHO_IMPORT_TYPE=APPEND&ZOHO_AUTO_IDENTIFY=TRUE&ZOHO_ON_IMPORT_ERROR=ABORT&ZOHO_CREATE_TABLE=TRUE&ZOHO_FILE=/home/dev1/Desktop/Zoho/temporary.csv")
File "/usr/lib/python3.5/http/client.py", line 1233, in __init__
source_address)
File "/usr/lib/python3.5/http/client.py", line 762, in __init__
(self.host, self.port) = self._get_hostport(host, port)
File "/usr/lib/python3.5/http/client.py", line 803, in _get_hostport
raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
http.client.InvalidURL: nonnumeric port: '//analyticsapi.zoho.com/api/usename/ATable/InsideTable?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=XML&ZOHO_API_VERSION=1.0&authtoken=*************&ZOHO_IMPORT_TYPE=APPEND&ZOHO_AUTO_IDENTIFY=TRUE&ZOHO_ON_IMPORT_ERROR=ABORT&ZOHO_CREATE_TABLE=TRUE&ZOHO_FILE=/home/dev1/Desktop/Zoho/temporary.csv'
When I hit the URL with my Browser it gives a good response back in XML which in short, succeeds.
This is where I referred for documentation
Can you pin point where I went wrong?

As per documentation HTTPS support is only available if Python was compiled with SSL support (through the ssl module).
Also the default port for https connection is 443. The module seems to be hitting that port by default.

How do I dumb down SSL DNS verification?

I have a very low security environment (sensor data on a LAN) that I am trying to add just a little bit of security on.
I am pretty sure this error is asking for proper DNS which is not available. Certs are self-signed with basically junk added to the FQDN.
I get the following error:
$ python3 twistedClientsocketSSL.001.py
Error during info_callback
Traceback (most recent call last):
File "/home/.local/lib/python3.5/site-packages/twisted/protocols/tls.py", line 315, in dataReceived
self._checkHandshakeStatus()
File "/home/.local/lib/python3.5/site-packages/twisted/protocols/tls.py", line 235, in _checkHandshakeStatus
self._tlsConnection.do_handshake()
File "/home/.local/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1906, in do_handshake
result = _lib.SSL_do_handshake(self._ssl)
File "/home/.local/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1288, in wrapper
callback(Connection._reverse_mapping[ssl], where, return_code)
--- <exception caught here> ---
File "/home/.local/lib/python3.5/site-packages/twisted/internet/_sslverify.py", line 1103, in infoCallback
return wrapped(connection, where, ret)
File "/home/.local/lib/python3.5/site-packages/twisted/internet/_sslverify.py", line 1216, in _identityVerifyingInfoCallback
verifyHostname(connection, self._hostnameASCII)
File "/home/.local/lib/python3.5/site-packages/service_identity/pyopenssl.py", line 48, in verify_hostname
obligatory_ids=[DNS_ID(hostname)],
File "/home/.local/lib/python3.5/site-packages/service_identity/_common.py", line 245, in __init__
raise ValueError("Invalid DNS-ID.")
builtins.ValueError: Invalid DNS-ID.
main function encountered error
Traceback (most recent call last):
--- <exception caught here> ---
File "twistedClientsocketSSL.001.py", line 18, in custom_trust
response = yield treqish.get('https://192.168.1.7:1079')
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure builtins.ValueError: Invalid DNS-ID.>]
client code:
import treq
from twisted.internet import defer, ssl, task
from twisted.web import client
#task.react
#defer.inlineCallbacks
def custom_trust(_reactor):
# get root cert from pem file
with open('keys/server.crt') as cert_file:
trust_root = yield ssl.Certificate.loadPEM(cert_file.read())
# ready made browser-like policy
policy = client.BrowserLikePolicyForHTTPS(trustRoot=trust_root)
agent = client.Agent(_reactor, policy)
treqish = treq.client.HTTPClient(agent)
response = yield treqish.get('https://192.168.1.7:1079')
content = yield response.content()
print(content)
corresponding server code:
$ cat twistedServersocketSSL.002.py
import sys
from twisted.internet import endpoints, reactor, ssl
from twisted.web import server, resource
from twisted.python import log
from twisted.python.modules import getModule
class Example(resource.Resource):
isLeaf = True
def render_GET(self, request):
return u'Hello World'.encode('ascii')
# create SSL server from string
https_server = endpoints.serverFromString(
reactor,
'ssl:1079:interface=192.168.1.7:certKey=keys/server.crt:privateKey=keys/server_no_pass.key')
# start server
site = server.Site(Example())
https_server.listen(site)
log.startLogging(sys.stdout)
reactor.run()
How do I dumb down SSL DNS verification so I can get around this error or a decent low security solution thats light weight that gets plain text off a network socket connection?

Handling Non-SSL Traffic in Python/Tornado

I have a webservice running in python 2.7.10 / Tornado that uses SSL. This service throws an error when a non-SSL call comes through (http://...).
I don't want my service to be accessible when SSL is not used, but I'd like to handle it in a cleaner fashion.
Here is my main code that works great over SSL:
if __name__ == "__main__":
tornado.options.parse_command_line()
#does not work on 2.7.6
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("...crt.pem","...key.pem")
ssl_ctx.load_verify_locations("...CA.crt.pem")
http_server = tornado.httpserver.HTTPServer(application, ssl_options=ssl_ctx, decompress_request=True)
http_server.listen(options.port)
mainloop = tornado.ioloop.IOLoop.instance()
print("Main Server started on port XXXX")
mainloop.start()
and here is the error when I hit that server with http://... instead of https://...:
[E 151027 20:45:57 http1connection:700] Uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 691, in _server_request_loop
ret = yield conn.read_response(request_delegate)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 807, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 209, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 810, in run
yielded = self.gen.throw(*sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 166, in _read_message
quiet_exceptions=iostream.StreamClosedError)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 807, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 209, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:590)
Any ideas how I should handle that exception?
And what the standard-conform return value would be when I catch a non-SSL call to an SSL-only API?
UPDATE
This API runs on a specific port e.g. https://example.com:1234/. I want to inform a user who is trying to connect without SSL, e.g. http://example.com:1234/ that what they are doing is incorrect by returning an error message or status code. As it is the uncaught exception returns a 500, which they could interpret as a programming error on my part. Any ideas?

There's an excelent discussion in this Tornado issue about that, where Tornado maintainer says:
If you have both HTTP and HTTPS in the same tornado process, you must be running two separate HTTPServers (of course such a feature should not be tied to whether SSL is handled at the tornado level, since you could be terminating SSL in a proxy, but since your question stipulated that SSL was enabled in tornado let's focus on this case first). You could simply give the HTTP server a different Application, one that just does this redirect.
So, the best solution it's to HTTPServer that listens on port 80 and doesn't has the ssl_options parameter setted.
UPDATE
A request to https://example.com/some/path will go to port 443, where you must have an HTTPServer configured to handle https traffic; while a request to http://example.com/some/path will go to port 80, where you must have another instance of HTTPServer without ssl options, and this is where you must return the custom response code you want. That shouldn't raise any error.