I have a webservice running in python 2.7.10 / Tornado that uses SSL. This service throws an error when a non-SSL call comes through (http://...).
I don't want my service to be accessible when SSL is not used, but I'd like to handle it in a cleaner fashion.
Here is my main code that works great over SSL:
if __name__ == "__main__":
tornado.options.parse_command_line()
#does not work on 2.7.6
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("...crt.pem","...key.pem")
ssl_ctx.load_verify_locations("...CA.crt.pem")
http_server = tornado.httpserver.HTTPServer(application, ssl_options=ssl_ctx, decompress_request=True)
http_server.listen(options.port)
mainloop = tornado.ioloop.IOLoop.instance()
print("Main Server started on port XXXX")
mainloop.start()
and here is the error when I hit that server with http://... instead of https://...:
[E 151027 20:45:57 http1connection:700] Uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 691, in _server_request_loop
ret = yield conn.read_response(request_delegate)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 807, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 209, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 810, in run
yielded = self.gen.throw(*sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/tornado/http1connection.py", line 166, in _read_message
quiet_exceptions=iostream.StreamClosedError)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 807, in run
value = future.result()
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 209, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:590)
Any ideas how I should handle that exception?
And what the standard-conform return value would be when I catch a non-SSL call to an SSL-only API?
UPDATE
This API runs on a specific port e.g. https://example.com:1234/. I want to inform a user who is trying to connect without SSL, e.g. http://example.com:1234/ that what they are doing is incorrect by returning an error message or status code. As it is the uncaught exception returns a 500, which they could interpret as a programming error on my part. Any ideas?
There's an excelent discussion in this Tornado issue about that, where Tornado maintainer says:
If you have both HTTP and HTTPS in the same tornado process, you must be running two separate HTTPServers (of course such a feature should not be tied to whether SSL is handled at the tornado level, since you could be terminating SSL in a proxy, but since your question stipulated that SSL was enabled in tornado let's focus on this case first). You could simply give the HTTP server a different Application, one that just does this redirect.
So, the best solution it's to HTTPServer that listens on port 80 and doesn't has the ssl_options parameter setted.
UPDATE
A request to https://example.com/some/path will go to port 443, where you must have an HTTPServer configured to handle https traffic; while a request to http://example.com/some/path will go to port 80, where you must have another instance of HTTPServer without ssl options, and this is where you must return the custom response code you want. That shouldn't raise any error.
Related
I have been using the Python Jenkins APIs to manager my Jenkins jobs. It has worked for a long time, but it stopped suddenly working. This is the code excerpt:
import jenkins
server = jenkins.Jenkins('https://jenkins.company.com', username='xxxx', password='password')
server._session.verify = False
print(server.jobs_count())
The traceback:
File "", line 1, in
server.jobs_count()
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line
1160, in jobs_count
return len(self.get_all_jobs())
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line
1020, in get_all_jobs
jobs = [(0, [], self.get_info(query=jobs_query)['jobs'])]
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 769,
in get_info
requests.Request('GET', self._build_url(url))
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 557,
in jenkins_open
return self.jenkins_request(req, add_crumb, resolve_auth).text
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 573,
in jenkins_request
self.maybe_add_crumb(req)
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 371,
in maybe_add_crumb
'GET', self._build_url(CRUMB_URL)), add_crumb=False)
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 557,
in jenkins_open
return self.jenkins_request(req, add_crumb, resolve_auth).text
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 576,
in jenkins_request
self._request(req))
File "E:\anaconda3\Lib\site-packages\jenkins_init_.py", line 550,
in _request
return self._session.send(r, **_settings)
File "E:\anaconda3\Lib\site-packages\requests\sessions.py", line
622, in send
r = adapter.send(request, **kwargs)
File "E:\anaconda3\Lib\site-packages\requests\adapters.py", line
507, in send
raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='jenkins.company.com', port=443): Max
retries exceeded with url:
/job/scp/job/sm/job/9218/job/4198/job/SIT/crumbIssuer/api/json (Caused
by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection
failed: 403 Forbidden')))
Note that there isn't any proxy on the Jenkins server, and I can use the user/password logon to the Jenkins server without any issues.
I have the crum id and API token, but I haven't found anything that is indicating how to add the crum into the Python-Jenkins API.
The final part of the traceback says:
ProxyError: HTTPSConnectionPool(host='ebs.usps.gov', port=443)
Which most likely indicates that you have proxy settings that your Python code inherits from somewhere when it runs. It could be environment variables ((HTTP|HTTPS)_PROXY) on POSIX sort of platforms or something similar... If you need to to use a proxy to reach the Jenkins instance, then the issue is in the proxy itself. It blocks your access for some reason. If you do not need to use a proxy, then you should remove the settings affecting your Python code when you run it.
Also, see what J_H said...
tl;dr: You lack connectivity.
The jenkins library depends on import requests,
which is reporting the connectivity error.
Regrettably, it uses ProxyError in the diagnostic.
The rationale goes like this:
We're making a GET request for the application.
Optionally the "GET from server S" will be turned into "GET from proxy P" if proxying is in use.
Eventually we try to contact some host, S or P. Might as well tell a proxy user that state of S is unknown, but state of P is "down".
Here ends the "why mention proxying?" diagnostic rant.
When you say "I'm not using proxying", I believe you.
The diagnostic can be a bit of a red herring for
folks who are not yet familiar with it.
When I probe ebs.usps.gov (56.207.107.97) on ports 443, 80, or with ICMP, I see zero response packets.
You're in a different part of the net, with different
filters between you and server, so your mileage might vary.
I wouldn't describe that host as a "public server",
since it offers me no responses.
It appears you sent SYN to tcp port 443,
and either some network device discarded that packet,
or the server replied with SYN-ACK and that
reply packet was discarded.
Most likely the server is down or your request was discarded.
My web deployment setup is on openBSD and consists of httpd on front with guicorn + uvicorn as the back engine, connected via unix socket.
The setup works, in the sense that requests from httpd are being forwarded to gunicorn over the unix sockets. However, the gunicorn/uvicorn is not able to understand the incoming http request.
The error stack
[2021-11-22 22:52:17 +0530] [1631] [WARNING] Invalid HTTP request received.
Traceback (most recent call last):
File "/home/shared/Builds/Python-3.10.0/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
event = self.conn.next_event()
File "/home/shared/Builds/Python-3.10.0/lib/python3.10/site-packages/h11/_connection.py", line 443, in next_event
exc._reraise_as_remote_protocol_error()
File "/home/shared/Builds/Python-3.10.0/lib/python3.10/site-packages/h11/_util.py", line 76, in _reraise_as_remote_protocol_error
raise self
File "/home/shared/Builds/Python-3.10.0/lib/python3.10/site-packages/h11/_connection.py", line 425, in next_event
event = self._extract_next_receive_event()
File "/home/shared/Builds/Python-3.10.0/lib/python3.10/site-packages/h11/_connection.py", line 367, in _extract_next_receive_event
event = self._reader(self._receive_buffer)
File "/home/shared/Builds/Python-3.10.0/lib/python3.10/site-packages/h11/_readers.py", line 68, in maybe_read_from_IDLE_client
raise LocalProtocolError("illegal request line")
h11._util.RemoteProtocolError: illegal request line
I am not sure what are potential causes for illegal request line.
httpd is not support http proxying.
It is support serving static files as well as FastCGI. And error message, indicate that your httpd try to communicate with gunicorn using FastCGI.
So, if you stick to httpd, find a way to run your app using FastCGI server instead of WSGI (gunicorn). Many years ago flup was a popular choice.
Or, just use Nginx instead of httpd.
I am trying to configure AsyncHTTPClient with auth proxy to access https websites. Is it possible to do with authenticated proxy?
from tornado import httpclient, ioloop
config = {
'proxy_host': proxy_host,
'proxy_port': proxy_post,
"proxy_username": proxy_username,
"proxy_password": proxy_password
}
httpclient.AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
def handle_request(response):
if response.error:
print("Error:", response.error)
else:
print(response.body)
ioloop.IOLoop.instance().stop()
http_client = httpclient.AsyncHTTPClient()
http_client.fetch("https://twitter.com/",
handle_request, **config)
ioloop.IOLoop.instance().start()
I get these errors after running the code above
Traceback (most recent call last):
File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\tornado\curl_httpclient.py", line 130, in _handle_socket
self.io_loop.add_handler(fd, self._handle_events, ioloop_event)
File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\tornado\platform\asyncio.py", line 103, in add_handler
self.asyncio_loop.add_writer(fd, self._handle_events, fd, IOLoop.WRITE)
File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\asyncio\events.py", line 507, in add_writer
raise NotImplementedError
NotImplementedError
Traceback (most recent call last):
File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\tornado\curl_httpclient.py", line 130, in _handle_socket
self.io_loop.add_handler(fd, self._handle_events, ioloop_event)
File "C:\Users\Adam\Anaconda3\envs\sizeer\lib\site-packages\tornado\platform\asyncio.py", line 97, in add_handler
raise ValueError("fd %s added twice" % fd)
ValueError: fd 700 added twice
ERROR:asyncio:Future exception was never retrieved
future: <Future finished exception=HTTP 599: SSL certificate problem: unable to get local issuer certificate>
tornado.curl_httpclient.CurlError: HTTP 599: SSL certificate problem: unable to get local issuer certificate
Process finished with exit code -1
I'm not sure if this is the only problem here, but the NotImplementedError is because Python 3.8 on Windows uses a different event loop implementation that is incompatible with Tornado. You need to add asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) to the beginning of your main file/function.
I suspect you may also need to use the ca_certs argument to tell libcurl where to find the trusted root certificates for your proxy.
I run a simple flask app like this:
from flask import Flask
app = Flask(__name__)
#app.route('/')
def welcome():
return "OK"
app.config.update(
DEBUG = True
)
if __name__ == '__main__':
app.run(use_reloader = False)
when I run it and visit it, sometimes(not always) it could't response the request and throw an except:
Exception happened during processing of request from ('127.0.0.1', 54481)
Traceback (most recent call last):
File "c:\python27\Lib\SocketServer.py", line 295, in _handle_request_noblock
self.process_request(request, client_address)
File "c:\python27\Lib\SocketServer.py", line 321, in process_request
self.finish_request(request, client_address)
File "c:\python27\Lib\SocketServer.py", line 334, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "c:\python27\Lib\SocketServer.py", line 651, in __init__
self.finish()
File "c:\python27\Lib\SocketServer.py", line 710, in finish
self.wfile.close()
File "c:\python27\Lib\socket.py", line 279, in close
self.flush()
File "c:\python27\Lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053]
I can't understand what cause this fault? and how can I solve it?
and how can I use try except to catch it?
I recently ran into this error message while trying to use Flask to serve audio files. I get this error message whenever the client closes the stream before the end of the stream.
This error message doesn't originate from your Flask application, but rather from the underlying SocketServer used to dispatch request data. What is happening is the connection to the client is ending for some reason, but Flask continues to try to write data to the closed socket. You can't catch this exception from your Flask application, because Flask catches it for you. Flask prints it out as a service to you, notifying you that the stream was closed prematurely, i.e. before Flask finished writing data to the stream.
To sum it up, this error message is internal to Flask, Flask is printing it to tell you that it couldn't get all the data to the client before the connection closed. You can't catch it, and you shouldn't have any reason to catch it.
I've found this solution to be a good at least temporary fix.
if __name__ == '__main__':
while True:
try:
app.run(use_reloader = False)
except:
pass
You can add your own exit logic, or leave the program with CTRL + \ which sends SIGQUIT. (important if you're running threaded flask)
You however can't:
except KeyboardInterupt:
Because Flask already catches KeyboardInterupt exceptions and handles them.
error 10052 means you're using windows, so as far as I know, close the command window to exit the program
It is probably due to the port number being used which is 54481 by looking at your error message. It might be clashing with something else. I also suggest not to use the use_reloader parameter since your DEBUG is already set to False. So flask will not reload any code changes. Can you instead do this :
if __name__ == '__main__':
app.run(port=5000)
I ran into an error that was painful to track down, so I thought I'd add the cause + "solution" here.
The setup:
Devbox - Running Google App Engine listening on all ports ("--address=0.0.0.0") serving a URL that launches a task.
Client - Client (Python requests library) which queries the callback URL
App Engine code:
class StartTaskCallback(webapp.RequestHandler):
def post(self):
param = self.request.get('param')
logging.info('STARTTASK: %s' % param)
# launch a task
taskqueue.add(url='/tasks/mytask',
queue_name='myqueue',
params={'param': param})
class MyTask(webapp.RequestHandler):
def post(self):
param = self.request.get('param')
logging.info('MYTASK: param = %s' % param)
When I queried the callback with my browser, everything worked, but the same query from the remote client gave me the following error:
ERROR 2012-03-23 21:18:27,351 taskqueue_stub.py:1858] An error occured while sending the task "task1" (Url: "/tasks/mytask") in queue "myqueue". Treating as a task error.
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py", line 1846, in ExecuteTask
connection.endheaders()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 868, in endheaders
self._send_output()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 740, in _send_output
self.send(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 699, in send
self.connect()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/httplib.py", line 683, in connect
self.timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/socket.py", line 498, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno 8] nodename nor servname provided, or not known
This error would just spin in a loop as the task retried. Though oddly, I could go to Admin -> Task Queues and click 'Run' to get the task to complete successfully.
At first I thought this was an error with the binding. I would not get an error if I queried the StartTaskCallback via the browser or if I ran the client locally.
Finally I noticed that App Engine is using the 'host' field of the request in order to build an absolute URL for the task. In /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py (1829):
connection_host, = header_dict.get('host', [self._default_host])
if connection_host is None:
logging.error('Could not determine where to send the task "%s" '
'(Url: "%s") in queue "%s". Treating as an error.',
task.task_name(), task.url(), queue.queue_name)
return False
connection = httplib.HTTPConnection(connection_host)
In my case, I was using a special name + hosts file on the remote client to access the server.
192.168.1.208 devbox
So the 'host' for the remote client looked like 'devbox:8085' which the local server could not resolve.
To fix the issue, I simply added devbox to my AppEngine server's hosts file, but it sure would have been nice if the gaierror exception had printed the name it failed to resolve, or if App Engine didn't use the 'host' of the incoming request to build a URL for task creation.