This question already has answers here:
Display the contents of a log file as it is updated
(3 answers)
Closed 6 years ago.
I am trying to create a web application with Flask.
The problem is that it has been 2 weeks since I am stuck on a problem.
I would like to run a python command that launches a server, retrieve the standard output, and display in real time on the web-site.
I do not know at all how to do because if I use "render_template" I do not see how to update the web site-the values sent in the console.
I use python 2.7, thank you very much
It's gonna take a lot of work to get this done and probably more then you think but I'll still try to help you out.
To get any real time updates to the browser you're going to need something like a socket connection, something which allows you to send multiple messages at any time. Not just when the browser requests it.
So imagine this with a regular http connection you can only receive a message once and once you receive that message you cannot receive a message again. You can only call return once and not again.
Once you call return, you cannot call return again to send another message.
So basically with a regular http request you can only receive the log messages once and once any changes have been made to the log you cannot send those changes to the client again since the connection is end.
The connection is end the moment you call return.
There is a way to fix this by using a socket connection. A socket connection would allow you to open a connection with the user and server and they both can send messages at any time as long as the connection is open. The connection is only not open when you manually close it.
Check this answer for ways you could have real time updates with flask. If you want to do it with sockets (which is what I suggest you to use) then use the websocket interface instead.
There's options like socketio for python which allow you to write websocket applications in python.
Overall this is gonna be split into 5 parts:
Start a websocket server when the Flask application start
Create a javsacript file (one that the browser loads) that connects with the websocket server
Find the function that gets triggered whenever Flask logging occurs
Send a socket message with the log inside of it
Make the browser display the log whenever it receives a websocket message
Here's a sample application written in Flask and socketio which should give you a idea on how to use socketio.
There's a lot to it and there's part you might be new to like websockets but don't let that stop you from doing what you want to do.
I hope this help, if any part confuses you then feel free to respond.
The simple part : server side, you could redirect the stdout and stderr of the server to a file,
import sys
print("output will be redirected")
# stdout is saved
save_stdout = sys.stdout
fh = open("output.txt","w")
sys.stdout = fh
the server itself would then read that file within a subprocess.
f = subprocess.Popen(['tail','-F',"output.txt", '-n1'],\
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p = select.poll()
p.register(f.stdout)
and the following threaded :
while True :
if p.poll(1):
output+=f.stdout.readline()
You can also use the tailhead or tailer libraries instead of the system tail
Now, the problem is that the standard output is a kind of active pipe and output is going to grow forever, so you'll need to keep only a frame of that output buffer.
If you have only one user that can connect to that window, the problem would be different, as you could flush the output as soon as is it send to that only client. See the difference between a terminal window and multiplexed, remote terminal window ?
I don't know flask, but client side, you only need some javascript to poll the server every second with an ajax request asking for the complete log (or --case of unique client-- the buffer to be appended to the DOM). You could also use websockets, but it's not an absolute necessity.
A compromise between the two is possible (infinite log with real time append / multiplexed at different rate) and it requires to keep a separate output buffer for each client.
Related
I have an application to stream a video on web using flask. Something like an example. But sometimes when the user closes its connection flask does not recognize the disconnection and keeps the socket open. Each socket in Linux is a file-descriptor and the maximum number of open file-descriptors in Linux is 1024 by default. After a while (e.g. 24 hours) new users cannot see the video stream because flask cannot create new socket (which is a file-descriptor).
The same happens when I use flask sockets: from flask_sockets import Sockets. I dont know what happens to these sockets. Most of the time when user refreshes the browser or closes it normally, the socket get closed on server.
I made a test and removed the network cable from my laptop (as a client) and realized that in this case sockets continue to be open and flask does not recognize this kind of disconnection.
Any help will be appreciated.
Update 1:
I put this code on top of main script and it can recognize some of disconnections. But the main problem still is there.
import socket
socket.setdefaulttimeout(1)
Update 2:
Seems duplicated with this post but no solution. I checked the sockets status(by sudo lsof -p your_process_id | egrep 'CLOSE_WAIT') and all of them are "CLOSE_WAIT".
I currently have a program that does work on a large set of data, at one point in the process it sends the data to a server for more work to be done, then my program looks for the completed data periodically, sleeping if it is not ready and repeating until it fetches the data, then continuing to do work locally.
Instead of polling repeatedly until the external server has finished, it has the ability to send a simple http post to an address I designate once the work has finished.
So I assume I need flask running at an address that can receive the notification, but I'm unsure of the best way to incorporate flask into the original program. I am thinking just to split my program into 2 parts.
part1.py
does work --> send to external server
part1 ends
flask server.py
receives data --> spawns part2.py with received data
The original program uses multiprocessing pools to offset waiting for the server responses, but with using flask, can I just repeatedly spawn new instances of part2 to do work on the data as it is received?
Am I doing this all completely wrong, I've just put this together with some googling and feel out of my depth
U can use broker with a message queue implemented ex. Celery + Redis or RabbitMQ. Then, when the other server finishes doing whatever it has to do with the data it can produce an event, and the first server will receive a notification.
I'm looking to start a web project using Flask and its SocketIO plugin, which depends on gevent (something something greenlets), but I don't understand how gevent relates to the webserver. Does using gevent restrict my server choice at all? How does it relate to the different levels of web servers that we have in python (e.g. Nginx/Apache, Gunicorn)?
Thanks for the insight.
First, lets clarify what we are talking about:
gevent is a library to allow the programming of event loops easily. It is a way to immediately return responses without "blocking" the requester.
socket.io is a javascript library create clients that can maintain permanent connections to servers, which send events. Then, the library can react to these events.
greenlet think of this a thread. A way to launch multiple workers that do some tasks.
A highly simplified overview of the entire process follows:
Imagine you are creating a chat client.
You need a way to notify the user's screens when anyone types a message. For this to happen, you need someway to tell all the users when a new message is there to be displayed. That's what socket.io does. You can think of it like a radio that is tuned to a particular frequency. Whenever someone transmits on this frequency, the code does something. In the case of the chat program, it adds the message to the chat box window.
Of course, if you have a radio tuned to a frequency (your client), then you need a radio station/dj to transmit on this frequency. Here is where your flask code comes in. It will create "rooms" and then transmit messages. The clients listen for these messages.
You can also write the server-side ("radio station") code in socket.io using node, but that is out of scope here.
The problem here is that traditionally - a web server works like this:
A user types an address into a browser, and hits enter (or go).
The browser reads the web address, and then using the DNS system, finds the IP address of the server.
It creates a connection to the server, and then sends a request.
The webserver accepts the request.
It does some work, or launches some process (depending on the type of request).
It prepares (or receives) a response from the process.
It sends the response to the client.
It closes the connection.
Between 3 and 8, the client (the browser) is waiting for a response - it is blocked from doing anything else. So if there is a problem somewhere, like say, some server side script is taking too long to process the request, the browser stays stuck on the white page with the loading icon spinning. It can't do anything until the entire process completes. This is just how the web was designed to work.
This kind of 'blocking' architecture works well for 1-to-1 communication. However, for multiple people to keep updated, this blocking doesn't work.
The event libraries (gevent) help with this because they accept and will not block the client; they immediately send a response and when the process is complete.
Your application, however, still needs to notify the client. However, as the connection is closed - you don't have a way to contact the client back.
In order to notify the client and to make sure the client doesn't need to "refresh", a permanent connection should be open - that's what socket.io does. It opens a permanent connection, and is always listening for messages.
So work request comes in from one end - is accepted.
The work is executed and a response is generated by something else (it could be a the same program or another program).
Then, a notification is sent "hey, I'm done with your request - here is the response".
The person from step 1, listens for this message and then does something.
Underneath is all is WebSocket a new full-duplex protocol that enables all this radio/dj functionality.
Things common between WebSockets and HTTP:
Work on the same port (80)
WebSocket requests start off as HTTP requests for the handshake (an upgrade header), but then shift over to the WebSocket protocol - at which point the connection is handed off to a websocket-compatible server.
All your traditional web server has to do is listen for this handshake request, acknowledge it, and then pass the request on to a websocket-compatible server - just like any other normal proxy request.
For Apache, you can use mod_proxy_wstunnel
For nginx versions 1.3+ have websocket support built-in
This question already has an answer here:
Closed 10 years ago.
Possible Duplicate:
How to write a twisted server that is also a client?
How can I create a tcp client server app with twisted, where also the server can send requests, not just answer them? Sort of like a p2p app but where clients always initiate the connection. Since I don't know when the requests from the server will occur, I don't see how I can do this once the reactor is started.
The question you have to ask yourself is: why is the server sending a request?
Presumably something has happened in the world that would prompt the server to send a request; it wouldn't just do it at random. Even if it did it at random, the thing that has happened in the world would be "some random amount of time has passed". In other words, callLater(random(...), doSomething).
When you are writing a program with Twisted, you start off by setting up ways to react to events. Then you run the reactor - i.e. the "thing that reacts to events" - forever. At any time you can set up new ways to react to incoming network events (reactor.connectTCP, reactor.listenTCP, reactor.callLater) or tear down existing waiting things (protocol.loseConnection, port.stopListening, delayedCall.cancel). You don't need to re-start the reactor; in fact, really, the only thing you should do before the reactor runs is do reactor.callWhenRunning(someFunctionThatListensOrConnects), and write someFunctionThatListensOrConnects to do all your initial set-up. That set-up then happens once the reactor is already running, which demonstrates that you don't need to do anything in advance; the reactor is perfectly capable of changing its configuration as it runs.
If the event that causes the server to send an event to client B the fact that client A sent it a message, then your question is answered by the FAQ, "how do I make input on one connection result in output on another?"
I am using a server to send some piece of information to another server every second. The problem is that the other server response is few kilobytes and this consumes the bandwidth on the first server ( about 2 GB in an hour ). I would like to send the request and ignore the return ( not even receive it to save bandwidth ) ..
I use a small python script for this task using (urllib). I don't mind using any other tool or even any other language if this is going to make the request only.
A 5K reply is small stuff and is probably below the standard TCP window size of your OS. This means that even if you close your network connection just after sending the request and checking just the very first bytes of the reply (to be sure that request has been really received) probably the server already sent you the whole answer and the packets are already on the wire or on your computer.
If you cannot control (i.e. trim down) what is the server reply for your notification the only alternative I can think to is to add another server on the remote machine waiting for a simple command and doing the real request locally and just sending back to you the result code. This can be done very easily may be even just with bash/perl/python using for example netcat/wget locally.
By the way there is something strange in your math as Glenn Maynard correctly wrote in a comment.
For HTTP, you can send a HEAD request instead of GET or POST:
import urllib2
request = urllib2.Request('https://stackoverflow.com/q/5049244/')
request.get_method = lambda: 'HEAD' # override get_method
response = urllib2.urlopen(request) # make request
print response.code, response.url
Output
200 https://stackoverflow.com/questions/5049244/how-can-i-ignore-server-response-t
o-save-bandwidth
See How do you send a HEAD HTTP request in Python?
Sorry but this does not make much sense and is likely a violation of the HTTP protocol. I consider such an idea as weird and broken-by-design. Either make the remote server shut up or configure your application or whatever is running on the remote server on a different protocol level using a smarter protocol with less bandwidth usage. Everything else is hard being considered as nonsense.