I want to avoid closing connection after receiving response of one request so that i can reuse the connection and save my bytes while using proxy in python request library.
I saw FIN ACK ( in wire shark ) is going every time my request is served.
i have tried python requests session but still not working
Related
I read that HTTP Pipelining is not activated by default in modern browsers
How can I implement a Persistent HTTP Connection with Pipelining in Python (like code socket from scratch) without using requests library to download all pdf file in folder slides from http://web.stanford.edu/class/cs224w/slides/
I tried to send request from scratch many times just using import Socket and Threading (because I don't able to use requests lib or anything else like requests to automatically send request) but don't gain any result.
I made a TCP socket connection like this
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((host, port))
After that I make many sending-thread with the request with the format like this:
request = f"GET {path}{file_name} HTTP/1.1\r\nHost:{host}\r\nConnection: Keep-Alive\r\n\r\n"
Then I make many receiving-thread to receive data but Host return the response by turn.
I'm working with the urllib.request module in Python 3.
My question is:
Do we need to close the connection if I open it using urllib.request.ulropen?
From looking at the documentation for urllib.request.urlopen
It states that
urllib.request module uses HTTP/1.1 and includes Connection:close
header in its HTTP requests.
and when we check this against Request for Comments for the HTTP protocol:
From RFC 2616, Section 14.10
HTTP/1.1 defines the "close" connection option for the sender to
signal that the connection will be closed after completion of the
response. For example,
Connection: close
in either the request or the response header
fields indicates that the connection SHOULD NOT be considered
`persistent' (section 8.1) after the current request/response is
complete.
HTTP/1.1 applications that do not support persistent connections MUST
include the "close" connection option in every message.
So you don't need to close the connection. The connection is closed as soon as you receive your response
I have implemented a simple HTTP server and a client. The latter issues a PUT request using the requests library, sending some arbitrary JSON and then exits.
When I start the server, and then run the client, both the server and the client block. The server however appears to not have gone through the entire handler function yet.
This is what I get on server side:
$ python3 server.py
PUT / HTTP/1.1
That is, after printing the request line, the content JSON string is not printed. At this point both client and server block for some reason.
Interestingly, when I trigger a KeyboardInterrupt to the client, the server proceeds:
$ python3 server.py
PUT / HTTP/1.1
b'{"content": "Hello World"}'
127.0.0.1 - - [25/Feb/2016 11:52:54] "PUT / HTTP/1.1" 200 -
My questions:
Why is it necessary to kill the client to let the server proceed?
Am I using any of these components the wrong way?
How can I make client and server to operate (nearly) instantaneously?
This is the code of the HTTP server. It only handles PUT requests. It prints the request line and the content data and responds using the success code to the client:
import http.server
class PrintPUTRequestHandler(http.server.BaseHTTPRequestHandler):
def do_PUT(self):
print(self.requestline)
print(self.rfile.read())
self.send_response(200)
self.end_headers()
server_address = ('', 8000)
httpd = http.server.HTTPServer(server_address, PrintHTTPRequestHandler)
httpd.serve_forever()
This is the HTTP client. It is intended to connect to the server, write the request and return as soon as possible (but it doesn't):
import requests
server_address = "http://127.1:8000"
data = '{"content": "Hello World"}'
requests.put(server_address, data, headers={"Content-type": "application/json"})
This is how I run it after the server has started (no output observable):
python client.py
The server blocks both itself and the client on this line:
print(self.rfile.read())
That happens because you didn't specify the amount of data to be read so the server reads the input stream until it is closed. And in your case the input stream is closed once you kill the client.
Remember that the server doesn't know a priori when the streaming of data ends because you may want to send data chunk by chunk (for example when you send big files).
The size of request should be passed in Content-Length header so this is what you should do:
length = int(self.headers['Content-Length'])
print(self.rfile.read(length))
That's assuming that the length is small enough to fit in your memory (and in your case it is).
I am trying to connect to a website through a program, whose authentification protocol appears to be the following :
Connect to the websocket.
Receive crucial data for logging in.
Make a HTTP request with username/password/some of the datas THROUGH THIS CONNECTION.
This means that if i open another connection, the data linked to this connection, according to the server, will be different.
For exemple, receiving the data through the websocket, and doing a HTTP request with a browser will not work, as the browser is another connection.
By connection, i mean what is created at the start of the program, for exemple :
ws2= websocket.create_connection("ws://sim.smogon.com:8000/showdown/websocket") for the websocket module
h1 = httplib.HTTPConnection('www.cwi.nl') for the http module.
My problem is that i have to create 2 different connections to connect to websocket and make a http request. This means that the http request will not work with the websocket received data.
How to make these 2 connections the same ? Or, simplier said : how to solve the problem ?
I have these two Python scripts I'm using to attempt to work out how to send and receive POST requests in Python:
The Client:
import httplib
conn = httplib.HTTPConnection("localhost:8000")
conn.request("POST", "/testurl")
conn.send("clientdata")
response = conn.getresponse()
conn.close()
print(response.read())
The Server:
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
ADDR = "localhost"
PORT = 8000
class RequestHandler(BaseHTTPRequestHandler):
def do_POST(self):
print(self.path)
print(self.rfile.read())
self.send_response(200, "OK")
self.end_headers()
self.wfile.write("serverdata")
httpd = HTTPServer((ADDR, PORT), RequestHandler)
httpd.serve_forever()
The problem is that the server hangs on self.rfile.read() until conn.close() has been called on the client but if conn.close() is called on the client the client cannot receive a response from the server. This creates a situation where one can either get a response from the server or read the POST data but never both. I assume there is something I'm missing here that will fix this problem.
Additional information:
conn.getresponse() causes the client to hang until the response is received from the server. The response doesn't appear to be received until the function on the server has finished execution.
There are a couple of issues with your original example. The first is that if you use the request method, you should include the message body you want to send in that call, rather than calling send separately. The documentation notes send() can be used as an alternative to request:
As an alternative to using the request() method described above, you
can also send your request step by step, by using the four functions
below.
You just want conn.request("POST", "/testurl", "clientdata").
The second issue is the way you're trying to read what's sent to the server. self.rfile.read() attempts to read the entire input stream coming from the client, which means it will block until the stream is closed. The stream won't be closed until connection is closed. What you want to do is read exactly how many bytes were sent from the client, and that's it. How do you know how many bytes that is? The headers, of course:
length = int(self.headers['Content-length'])
print(self.rfile.read(length))
I do highly recommend the python-requests library if you're going to do more than very basic tests. I also recommend using a better HTTP framework/server than BaseHTTPServer for more than very basic tests (flask, bottle, tornado, etc.).
Long time answered but came up during a search so I bring another piece of answer. To prevent the server to keep the stream open (resulting in the response never being sent), you should use self.rfile.read1() instead of self.rfile.read()