uwsgi --reload refuses incoming connections - python

I am trying to setup a uwsgi-hosted app such that I get graceful reloads with uwsgi --reload but I am obviously failing. Here is my test uwsgi setup:
[admin2-prod]
http = 127.0.0.1:9090
pyargv = $* --db=prod --base-path=/admin/
max-requests = 3
listen=1000
http-keepalive = 1
pidfile2 =admin.pid
add-header=Connection: keep-alive
workers = 1
master = true
chdir = .
plugins = python,http,router_static,router_uwsgi,router_http
buffer-size = 8192
pythonpath = admin2
file = admin2/app.py
static-map=/admin/static/=admin2/static/
static-map=/admin/v3/build/=admin2/client/build/
disable-logging = false
http-timeout = 100
(please, note that I ran sysctl net.core.somaxconn=1000 before)
And here is my test python script:
import httplib
connection = httplib.HTTPConnection('127.0.0.1', 9090)
connection.connect()
for i in range(0, 1000):
print 'sending... ', i
try:
connection.request('GET', '/x', '', {'Connection' : ' keep-alive'})
response = connection.getresponse()
d = response.read()
print ' ', response.status
except:
connection = httplib.HTTPConnection('127.0.0.1', 9090)
connection.connect()
The above client fails during --reload:
sending... 920
Traceback (most recent call last):
File "./test.py", line 15, in <module>
connection.connect()
File "/usr/lib64/python2.7/httplib.py", line 836, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
raise err
socket.error: [Errno 111] Connection refused
From a tcpdump, it looks like uwsgi is indeed accepting the second incoming TCP request which happens upon the --reload, the client is sending the GET, the server is TCP ACKing it but the connection is finally RSTed by the server before sending back the HTTP response. So, what am I missing that is needed to make the server queue this incoming connection until it is ready to process it and get a real graceful reload ?

you are managing both the app and the proxy in the same uWSGI instance, so when you reload the stack you are killing the frontend web server too (the one you start with the 'http' option).
You have to split the http router in another uWSGI instance, or use nginx/haproxy or similar. Once you have two different stacks you can reload the application without closing the socket

Your exceptions happens when uwsgi process cant accept connections obviously... So, your process have to wait until server restarted - you can use loop with timeout in except block to properly handle this situation. Try this:
import httplib
import socket
import time
connection = httplib.HTTPConnection('127.0.0.1', 8000)
# connection moved below... connection.connect()
for i in range(0, 1000):
print 'sending... ', i
try:
connection.request('GET', '/x', '', {'Connection' : ' keep-alive'})
response = connection.getresponse()
d = response.read()
print ' ', response.status
except KeyboardInterrupt:
break
except socket.error:
while True:
try:
connection = httplib.HTTPConnection('127.0.0.1', 8000)
connection.connect()
except socket.error:
print 'cant connect, will try again in a second...'
time.sleep(1)
else:
break
before restart:
sending... 220
404
sending... 221
404
sending... 222
404
restarting server:
cant connect, will try again in a second...
cant connect, will try again in a second...
cant connect, will try again in a second...
cant connect, will try again in a second...
server up again:
sending... 223
404
sending... 224
404
sending... 225
404
update for your comment:
Obviously, in the real world, you can't rewrite the code of all the
http clients that connect to your server. My question is: what can I
do to get a graceful reload (no failures) for arbitrary clients.
One of universal solutions that i think can handle such problems with clients - simple proxy between client and server. With proxy you can restart server independently of clients (implies that proxy is always on).
And in fact this is commonly used - 502 (bad gateway) errors from web applications frontend proxies - exactly same situation - client receives error from proxy while application server is down! Try nginx, varnish or something similar.
Btw, uwsgi have builtin "proxy/load-balancer/router" plugin:
The uWSGI FastRouter
For advanced setups uWSGI includes the “fastrouter” plugin, a proxy/load-balancer/router speaking the uwsgi
protocol. It is built in by default. You can put it between your
webserver and real uWSGI instances to have more control over the
routing of HTTP requests to your application servers.
docs here: http://uwsgi-docs.readthedocs.io/en/latest/Fastrouter.html

Related

Python - [Errno 111] Connection refused on client side of the connection

I'm trying to create a chat between client and server written in Python, using SSL protocols with mutual authentication (i.e: server authenticates client and client authenticates server using certificates). My host machine is being used as the server, and my laptop is the client.
When attempting to connect to my host ip, I keep getting this error on my laptop:
Traceback (most recent call last):
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 88, in <module>
main()
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 75, in main
connection(ip, port, SSLSock)
File "/home/icarus/Codes/RealtimeChat/Chat.py", line 35, in connection
sock.connect((ip, port))
File "/usr/lib/python3.10/ssl.py", line 1375, in connect
self._real_connect(addr, False)
File "/usr/lib/python3.10/ssl.py", line 1362, in _real_connect
super().connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused
And in the server - which was supposed to print a message saying that a connection was refused - nothing happens, it keeps listening for connections as if nothing happened
Connection function on client side:
def connection(ip, port, sock):
try:
sock.connect((ip, port))
print(f"Connected with {ip}")
except Exception as e:
print("Connection failed: ", e)
sock.close()
Server side:
def acceptConnection(self):
while True:
con, senderIP = self.sock.accept()
# Attempting to wrap connection with SSL socket
try:
SSLSock = self.getSSLSocket(con)
# If exception occurs, close socket and continue listening
except Exception as e:
print("Connection refused: ", e)
con.close()
continue
print(f"{senderIP} connected to the server")
# Adding connection to clients list
self.clients.append(SSLSock)
# Initializing thread to receive and communicate messages
# to all clients
threading.Thread(target=self.clientCommunication, args=(SSLSock, ), daemon=True).start()
This is the main function on my server:
def main():
serverIP = "127.0.0.1"
port = int(input("Port to listen for connections: "))
server = Server()
server.bindSocket(serverIP, port)
server.socketListen(2)
server.acceptConnection()
Everything works fine when I connect from my localhost (e.g I open a server on my host machine on one terminal and use another one on the same machine to connect to it). Both machines have the required certificates to authenticate each other, so I don't think that's the problem. Also, without the SSL implementation, the connection between this two different computers was refused by the server
I've tried using sock.bind('', port) on server side, disabling my firewall, used telnet 127.0.0.1 54321 (on my host machine) to check if the connection was working on the specified port (and it is), and also on the client machine (which showed that the connection was refused). I also tried running both scripts with admin privileges (sudo), but it also didn't work. Any suggestions?
I found what was wrong: I was trying to connect to my public IP address (which I found by searching for "What is my ip" on Google), but instead what should be done is to connect to the private IP address (I guess that's the correct name), and you can see yours using ifconfig on Linux and Mac and ipconfig on Windows using a terminal. By doing this, I could connect two computers that are on my network to my desktop server, I still haven't tested for computers in different networks, but the problem has been solved.

why does the request.Session context manager doesn't actually close the connection?

I have 2 containers, client and server. In server, I started a python SimpleHTTPServer:
python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
on client, I started a request to it:
import requests
with requests.Session() as s:
r = s.get("http://172.17.0.3:8000")
The server successfully received the request:
172.17.0.2 - - [26/Oct/2022 00:17:29] "GET / HTTP/1.1" 200 -
Now, I am inspecting the state of the TCP connections in server using ss and netstat and I see the TCP connections are still open for a little while before actually getting discarded, even when my python script already finished.
What's the point of opening a request.Session() in a context manager, if the connection is going to be left open anyways? According to the requests documentation:
Sessions can also be used as context managers:
with requests.Session() as s:
s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')
This will make sure the session is closed as soon as the with block is exited, even if
unhandled exceptions occurred.
This says the Session is closed but doesn't really specify that the TCP connection is closed. What's the point of closing the Session, if the TCP connection is going to be left open anyways? Is there a way to close the TCP connection right after the context manager finishes? Is it even recommended?

SSL raises version error only on re-connect attempt?

We have a server and client both running python3.
The client connects to the server and authenticates upon initialisation of the client. This completes without issue.
However, if the connection drops, the client catches the error (the socket.recv returning 0) and attempts to re-run the code that connects to the server).
The server recieves the initial request whilst listening on its given recieving socket and then once making a connection the next recv call raises the following error:
File "/usr/lib64/python3.7/ssl.py", line 1056, in recv
return self.read(buflen)
File "/usr/lib64/python3.7/ssl.py", line 931, in read
return self._sslobj.read(len)
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:2570)
Why would this succeed upon the first connection but then raise this error thereafter? If the client is closed and restarted the error is avoided upon the next connection. However, if the server is closed and the client tries to connect once the server restarts this error is encountered.
At the client end the following error is raised:
File "\Our_Code", line 100, in make_connection
data = self.ssl_sock.recv(1024)
File "C:\Users\Home\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1226, in recv226, in recv
return self.read(buflen) 101, in read
File "C:\Users\Home\AppData\Local\Programs\Python\Python39\lib\ssl.py", line 1101, in read
return self._sslobj.read(len)
ssl.SSLError: [SSL] internal error (_ssl.c:2633)
This exact formultation suggests to me that the issue is actually with the client and some data that it is saving between reconnects. But it overwrites the sockets for a new connection so I thought all data from the previous connection would be discarded?
The connection function is as:
import socket
import ssl
def make_connection(self, email, password):
try:
self.ssl_sock = []
HOST = "9:9:99:999" # The server's hostname or IP address (not our actual IP)
PORT = 5432 # The port used by the server (not our actual port)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
self.ssl_sock = self.context.wrap_socket(sock)
password = password.encode('utf-8').strip()
password = base64.b64encode(password).decode("utf-8")
tcp_string = (f"^{email}*{password}$")
tcp_string = tcp_string.encode('utf-8')
self.ssl_sock.sendall(tcp_string)
data = self.ssl_sock.recv(1024)
data = data.decode('utf-8')
if data == "^good_connect$":
return True
else:
return False
except Exception:
print(f"make_connection - {traceback.format_exc()}")
I'm still fairly new to python and particularly networking, so I suspect I have made a rookie error. But so far all my searches have returned the obvious, that the TLS version is wrong, but the fact that it authenticates fine on the initial connection suggests to me that that isn't the case.
I'm happy to answer any questions I can about the situation.

simple socket programming exercise for TCP connections in Python; Error: "No connection could be made because the target machine actively refused it"

*Before you mark as duplicate please note that I am referencing this similar question found here:
Python Socket Programming - ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
unfortunately but have found anything in that post that provides a solution to my problem.
I am working on a very basic exercise designed to familiarize students with programming related to networks. This particular assignment is a common one as is described as follows:
In this assignment, you will learn the basics of socket programming for TCP connections in Python: how to create a socket, bind it to a specific address and port, as well as send and receive an HTTP packet. You will also learn some basics of HTTP header format. You can only use Python3.
You will develop a web server that handles one HTTP request at a time. Your web server should accept and parse the HTTP request, get the requested file from the server’s file system, create an HTTP response message consisting of the requested file preceded by header lines, and then send the response directly to the client. If the requested file is not present in the server, the server should send an HTTP “404 Not Found” message back to the client.
Part one specification:
Put the attached HTML file (named HelloWorld.html) in the same directory in which the server webserver.py runs. Run the server program. Determine the IP address of the host that is running the server (e.g., 128.238.251.26 or localhost). From another host, open a browser and provide the corresponding URL. For example: http://128.238.251.26:6789/HelloWorld.html. You can open a browser in the same host where the server runs and use the following http://localhost:6789/HelloWorld.html.
‘HelloWorld.html’ is the name of the file you placed in the server directory. Note also the use of the port number after the colon. You need to replace this port number with the port number that was assigned to you. In the above example, we have used port number 6789. The browser should then display the contents of HelloWorld.html. If you omit “:6789”, the browser will assume port 80 (why?), and you will get the web page from the server only if your server is listening at port 80.
Then try to get a file that is not present on the server (e.g., test.html). You should get a “404 File Not Found” message.
Part Two specification:
Write your own HTTP client to test your server. Your client will connect to the server using a TCP connection, send an HTTP request to the server, and display the server response as an output. You can assume that the HTTP request sent is a GET method. The client should take command line arguments specifying the server IP address or hostname, the port at which the server is listening, and the HTTP file name (e.g., test.html or HelloWorld.html). The following is an input command format to run the client. webclient.py <server_host> <server_port>
My code is for the Webserver is as follows:
#import socket module
from socket import *
import sys # In order to terminate the program
serverSocket = socket(AF_INET, SOCK_STREAM)
# Prepare a sever socket
# Fill in start
serverHost = '192.168.1.4'
serverPort = 56014
serverSocket.bind((serverHost, serverPort))
serverSocket.listen(5)
# Fill in end
while True:
#establish connection
print('The server is ready to receive')
connectionSocket, addr = serverSocket.accept() # Fill in start #Fill in end
try:
message = connectionSocket.recv(4096) # Fill in start #Fill in end
filename = message.split()[1]
f = open(filename[1:])
outputdata = f.readlines() # Fill in start #Fill in end
# send one http header line in to the socket
# Fill in start
connectionSocket.send("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n")
connectionSocket.send("\r\n")
# Fill in end
# Send the content of the requested file to the connection socket
for i in range(0, len(outputdata)):
connectionSocket.send(outputdata[i].encode())
connectionSocket.send("\r\n".encode())
connectionSocket.close()
except IOError:
# Send HTTP response code and message for file not found
# Fill in start
connectionSocket.send("HTTP/1.1 404 Not Found\r\n")
connectionSocket.send("Content-Type: text/html\r\n")
connectionSocket.send("\r\n")
connectionSocket.send("<html><head></head><body><h1>404 Not Found</h1></body></html><\r\n>")
# Fill in end
# Close the client connection socket
# Fill in start
serverSocket.close()
# Fill in end
serverSocket.close()
sys.exit() # Terminate the program after sending the corresponding data
My code for the Webclient is as follows:
from socket import *
import sys
serverName = sys.argv[1]
serverPort = int(sys.argv[2])
fileName = sys.argv[3]
request = "GET "+str(fileName)+" HTTP/1.1"
clientSocket = socket(AF_INET, SOCK_STREAM)
clientSocket.connect((serverName, serverPort))
clientSocket.send(request.encode())
returnFromSever = clientSocket.recv(4096)
while(len(returnFromSever)>0):
print(returnFromSever.decode())
returnFromSever = clientSocket.recv(4096)
clientSocket.close()
The error I am receiving is:
"No connection could be made because the target machine actively refused it"
Admittedly, I know almost nothing about network related programming and on top of that I am not familiar with the Python syntax (my entire degree program was exclusively in Java) so I am very lost here and somewhat desperate.
If anyone could please point me in the right direction as far as how to correct this error, I would be very deeply grateful.
Thanks
The error you are getting (No connection could be made because the target machine actively refused it) means that the port you are trying to connect to is not not being listened on the server.
For example, if you try to connect to 192.168.1.1:80 (IP = 192.168.1.1, port=80) and the server on 192.168.1.1 doesn't listen on port 80, you would receive this error.
A few things I would check in your case:
Is your server IP actually 192.168.1.4 ? If not, set it to the correct IP of the interface you want to listen on. If you want to listen on all the interfaces of the server, use this: serverHost = '0.0.0.0'
Does your client code attempt to connect to the server port? The server port is 56014. You need to pass it as the second parameter of your client program (because of this line serverPort = int(sys.argv[2])).

Dual HTTP and HTTPS Python server based on BaseHTTPServer not working as expected

I'm trying to implement a Python server supporting both HTTP and HTTPS based in BaseHTTPServer. This is my code:
server_class = BaseHTTPServer.HTTPServer
# Configure servers
httpd = server_class(("0.0.0.0", 1044), MyHandler)
httpsd = server_class(("0.0.0.0", 11044), MyHandler)
httpsd.socket = ssl.wrap_socket(httpsd.socket, keyfile="/tmp/localhost.key", certfile="/tmp/localhost.crt", server_side=True)
# Run the servers
try:
httpd.serve_forever()
httpsd.serve_forever()
except KeyboardInterrupt:
print("Closing the server...")
httpd.server_close()
httpsd.server_close()
So, HTTP runs in port 1044 and HTTPS runs in 11044. The MyHandler class is omitted for the sake of briefness.
Using that code, when I send requests to HTTP port (e.g. curl http://localhost:1044/path) it works. However, when I send requests to the HTTPS port (e.g. curl -k https://localhost:11104/path) the server never responses, i.e. the curl terminal gets hanged.
I have observed that if I comment the line starting the HTTP server (i.e. httpd.server_forever()) then the HTTPS server works,.i.e. curl -k https://localhost:11104/path works. Thus, I guess that I'm doing something wrong which is precluding not being able to set both servers at the same time.
Any help is appreciated!
Following feedback comments, I have refactored the code in a multithread way and now it works as expected.
def init_server(http):
server_class = BaseHTTPServer.HTTPServer
if http:
httpd = server_class(("0.0.0.0", 1044), MyHandler)
else: # https
httpd = server_class(("0.0.0.0", 11044), MyHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, keyfile="/tmp/localhost.key", certfile="/tmp/localhost.crt", server_side=True)
httpd.serve_forever()
httpd.server_close()
VERBOSE = "True"
thread.start_new_thread(init_server, (True, ))
thread.start_new_thread(init_server, (False, ))
while 1:
time.sleep(10)

Categories