I created a python server on port 8000 using python -m SimpleHTTPServer.
When I visit this url from my web browser it shows the below content
Now, I want to get the above content using python. So, for that what I did is
>>> import socket
>>> s = socket.socket(
... socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(("localhost", 8000))
>>> s.recv(1024)
But after s.recv(1024) nothing happens it just wait there and prints nothing.
So, my question is how to get above directory content output using python. Also can someone suggest me a tutorial on socket programming with python. I didn't liked the official tutorial that much.
I also observed a strange thing when I try to receive contents using python and nothing happens at that time I cannot access localhost:8000 from my web browser but as soon as I kill my python program I can access it.
Arguably the simplest way to get content over http in python is to use the urllib2 module. For example:
from urllib2 import urlopen
f = urlopen('http://localhost:8000')
for line in f:
print line
This will print out the file hosted by SimpleHTTPServer.
But after s.recv(1024) nothing happens it just wait there and prints nothing.
You simply open a socket and waiting for the data, but it's not how HTTP protocol works. You have to send a request first if you want to receive a response (basically, you have to tell the server which directory you want to list or which file to download). If you really want to, you can send the request using raw sockets to train your skills, but the proper library is highly recommended (see Matthew Adams' response and urllib2 example).
I also observed a strange thing when I try to receive contents using python and nothing happens at that time I cannot access localhost:8000 from my web browser but as soon as I kill my python program I can access it.
This is because SimpleHTTServer is single-threaded and doesn't support multiple connections simultaneously. If you would like to fix it, take a look at the answers here: BasicHTTPServer, SimpleHTTPServer and concurrency.
Related
This question already has answers here:
Display the contents of a log file as it is updated
(3 answers)
Closed 6 years ago.
I am trying to create a web application with Flask.
The problem is that it has been 2 weeks since I am stuck on a problem.
I would like to run a python command that launches a server, retrieve the standard output, and display in real time on the web-site.
I do not know at all how to do because if I use "render_template" I do not see how to update the web site-the values sent in the console.
I use python 2.7, thank you very much
It's gonna take a lot of work to get this done and probably more then you think but I'll still try to help you out.
To get any real time updates to the browser you're going to need something like a socket connection, something which allows you to send multiple messages at any time. Not just when the browser requests it.
So imagine this with a regular http connection you can only receive a message once and once you receive that message you cannot receive a message again. You can only call return once and not again.
Once you call return, you cannot call return again to send another message.
So basically with a regular http request you can only receive the log messages once and once any changes have been made to the log you cannot send those changes to the client again since the connection is end.
The connection is end the moment you call return.
There is a way to fix this by using a socket connection. A socket connection would allow you to open a connection with the user and server and they both can send messages at any time as long as the connection is open. The connection is only not open when you manually close it.
Check this answer for ways you could have real time updates with flask. If you want to do it with sockets (which is what I suggest you to use) then use the websocket interface instead.
There's options like socketio for python which allow you to write websocket applications in python.
Overall this is gonna be split into 5 parts:
Start a websocket server when the Flask application start
Create a javsacript file (one that the browser loads) that connects with the websocket server
Find the function that gets triggered whenever Flask logging occurs
Send a socket message with the log inside of it
Make the browser display the log whenever it receives a websocket message
Here's a sample application written in Flask and socketio which should give you a idea on how to use socketio.
There's a lot to it and there's part you might be new to like websockets but don't let that stop you from doing what you want to do.
I hope this help, if any part confuses you then feel free to respond.
The simple part : server side, you could redirect the stdout and stderr of the server to a file,
import sys
print("output will be redirected")
# stdout is saved
save_stdout = sys.stdout
fh = open("output.txt","w")
sys.stdout = fh
the server itself would then read that file within a subprocess.
f = subprocess.Popen(['tail','-F',"output.txt", '-n1'],\
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p = select.poll()
p.register(f.stdout)
and the following threaded :
while True :
if p.poll(1):
output+=f.stdout.readline()
You can also use the tailhead or tailer libraries instead of the system tail
Now, the problem is that the standard output is a kind of active pipe and output is going to grow forever, so you'll need to keep only a frame of that output buffer.
If you have only one user that can connect to that window, the problem would be different, as you could flush the output as soon as is it send to that only client. See the difference between a terminal window and multiplexed, remote terminal window ?
I don't know flask, but client side, you only need some javascript to poll the server every second with an ajax request asking for the complete log (or --case of unique client-- the buffer to be appended to the DOM). You could also use websockets, but it's not an absolute necessity.
A compromise between the two is possible (infinite log with real time append / multiplexed at different rate) and it requires to keep a separate output buffer for each client.
I'm trying to make a simple HTTP request in Python in an SSH terminal:
from requests import get
r = get("https://www.google.com")
However, this command just stalls to infinity. This does not happen when not in SSH.
Is there any way to send the request such that it goes through?
Thanks ahead of time.
EDIT: Running the logging in Joran's link yields only the following line:
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.google.com
First, check that you have ability to reach URL using some system-wide tool, like curl: curl -I "https://www.google.com". In case, you will have not timeout error, and got success response, my answer is not for you :)
You code can run forever, just because there is not timeout defined for socket connections. And if for some reason your system is not able to read from socket (at low level), you have to wait for long time.
http://docs.python-requests.org/en/latest/user/quickstart/#timeouts
Try this (assuming you are using python3):
from urllib.request import urlopen
r = urlopen('https://www.google.com').read()
I'm trying to write an application where I send an initial HTTP post message to server and leave the connection open. The application then sits around until the server sends data back. Once the server sends data back I want to read it and write it to a file (easy enough).
The part I'm having trouble with is actua
Basically I do this:
h=http.HTTPConnection(sever, port, timeout)
h.putrequest('POST', selector)
h.putheaders(...)
h.endheaders()
h.send(body)
buffering = False
while 1:
r = h.getresponse(buffering)
f=open(unique_filename, 'w')
f.write(r.read())
f.close()
What I expect is that the app should block in the loop and when data arrives it gets written to the file. I suspect I'm using read the wrong way, but looking at the httplib source didn't help.
Also, the python documentation site mentions a httplib.fileno() that returns the socket httplib uses. I'm using 2.7.0 and website doc is for 2.7.2, I can't find the fileno() method. I suspect taking the socket over httplib and calling recv myself is the best way to go, is that a good idea?
Any help is appreciated with one exception: please don't tell me to use some other library.
I am trying to use urllib.urlopen to open a web page running on the same host and port as the page I am loading it from and it is just hanging.
For example I have a page at: "http://mydevserver.com:8001/readpage.html" and I have the following code in it:
data = urllib.urlopen("http://mydevserver.com:8001/testpage.html")
When I try and load the page it just hangs. However if I move the testpage.html script to a different port on the same host it works fine. e.g.
data = urllib.urlopen("http://mydevserver.com:8002/testpage.html")
Does anyone know why this might be and how I can solve the problem?
A firewall perhaps? Try opening the page from the command line with wget/curl (assuming you're on Linux) or on the browser, with both ports on settings. Furthermore, you could try a packet sniffer to find out what's going on and where the connection gets stuck. Also, if testpage.html is dynamically generated, see if it is hit, check webserver logs if the request shows up there.
Maybe something is already running on port 8001. Does the page open properly with a browser?
You seem to be implying that you are accessing a web page that is scripted in Python. That implies that the Python script is handling the incoming connections, which could mean that since it's already handling the urllib call, it is not available to handle the connection that results from it as well.
Show the code (or tell us what software) you're using to serve these Python scripts.
url = "http://example.com/file.xml"
data = urllib2.urlopen(url)
data.read()
The question is, when exactly will the file be downloaded from the internet? When i do urlopen or .read()? On my network interface I see high traffic both times.
Witout looking at the code, I'd expect that the following happens:
urlopen() opens the connection, and sends the query. Then the server starts feeding the reply. At this point, the data accumulates in buffers until they are full and the operating system tells the server to hold on for a while.
Then data.read() empties the buffer, so the operating system tells the server to go on, and the rest of the reply gets downloaded.
Naturally, if the reply is short enough, or if the .read() happens quickly enough, then the buffers do not have time to fill up and the download happens in one go.
I agree with ddaa. However, if you want to understand this sort of thing, you can set up a dummy server using something like nc (in *nix) and then open the URL in the interactive Python interpreter.
In one terminal, run nc -l 1234 which will open a socket and listen for connections on port 1234 of the local machine. nc will accept an incoming connection and display whatever it reads from the socket. Anything you type into nc will be sent over the socket to the remote connection, in this case Python's urlopen().
Run Python in another terminal and enter your code, i.e.
data = urllib2.urlopen('http://127.0.0.1:1234')
data.read()
The call to urlopen() will establish the connection to the server, send the request and then block waiting for a response. You will see that nc prints the HTTP request into it's terminal.
Now type something into the terminal that is running nc. The call to urlopen() will still block until you press ENTER in nc, that is, until it receives a new line character. So urlopen() will not return until it has read at least one new line character. (For those concerned about possible buffering by nc, this is not an issue. urlopen() will block until it sees the first new line character.)
So it should be noted that urlopen() will block until the first new line character is received, after which data can be read from the connection. In practice, HTTP responses are short multiline responses, so urlopen() should return quite quickly.