I'm working with the urllib.request module in Python 3.
My question is:
Do we need to close the connection if I open it using urllib.request.ulropen?
From looking at the documentation for urllib.request.urlopen
It states that
urllib.request module uses HTTP/1.1 and includes Connection:close
header in its HTTP requests.
and when we check this against Request for Comments for the HTTP protocol:
From RFC 2616, Section 14.10
HTTP/1.1 defines the "close" connection option for the sender to
signal that the connection will be closed after completion of the
response. For example,
Connection: close
in either the request or the response header
fields indicates that the connection SHOULD NOT be considered
`persistent' (section 8.1) after the current request/response is
complete.
HTTP/1.1 applications that do not support persistent connections MUST
include the "close" connection option in every message.
So you don't need to close the connection. The connection is closed as soon as you receive your response
Related
I want to avoid closing connection after receiving response of one request so that i can reuse the connection and save my bytes while using proxy in python request library.
I saw FIN ACK ( in wire shark ) is going every time my request is served.
i have tried python requests session but still not working
I'm really new to coding using sockets.
I like the socket library, I get to understand a big part of what's happening in my program, so i you don't mind i would like to stick with it.
So as the title says, I have a socket based client and server and I would like to exchange content through an HTTP proxy(I'm using a Squid proxy). This little piece of code is supposed to bypass the proxy in my campus to simulate a chat over the campus network. This is totally legal since I asked the IT guys that work there.
Here's the deal, I am able to send a POST request through the proxy to my server which receives it and sends it back to client 1, but when I try to send more requests to the proxy none of them gets to the server so I think to my self the connection died but here's the thing, when I send messages from client 2 which is connected directly to the server, the server AND client 1 receive them.
import socket
from _thread import *
def sender(server,h):
b=input("<<--Send--")
b=h
server.send(b.encode())
PROXY_IP="127.0.0.1"
PROXY_PORT=3128
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.settimeout(0.05)
server.connect((PROXY_IP,PROXY_PORT))
header="""POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n"""
server.send(header.encode())
while 1:
try:
start_new_thread(sender,(server,header))
a=server.recv(1024)
print("-->>{}".format(a.decode()))
except KeyboardInterrupt:
break
except:
pass
server.close()
I already tried the CONNECT method which works perfectly, but it's not allowed in my campus network proxy.
What am I doing wrong ?
Is there something I should know about how to re-send content through a proxy ?
Thank you for your time and please bear with me..
Here's what I get on the client that sends a request to the proxy:
~#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
#Received : HTTP/1.1 200 OK
Server: squid/3.5.19
Mime-Version: 1.0
Date: Mon, 10 Oct 2016 00:46:39 GMT
X-Transformed-From: HTTP/0.9
X-Cache: MISS from kali
X-Cache-Lookup: MISS from kali:3128
Transfer-Encoding: chunked
Via: 1.1 kali (squid/3.5.19)
Connection: keep-alive
#Received : B2
POST / HTTP/1.1
Content-Length: 5
Host: 127.0.0.1:3001
Via: 1.1 kali (squid/3.5.19)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=259200
Connection: keep-alive
hello
#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
Nothing is received after this..
POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n
The body of your HTTP response consists of 7 bytes not 5 as you've stated in your Content-length. The \r\n after the 5 byte still belong to the response body. Giving the wrong size might mixup request handling, i.e. the proxy is expecting a new request but is actually getting \r\n, i.e. the 2 bytes after your 5 bytes Content-length.
Apart from that both path and Host header must include the name of the target from the perspective of the proxy. Using 127.0.0.1. like in your example would mean that you try to access a server at the same host of the proxy, i.e. localhost from the view of the proxy. This is probably not what you've intended.
...
X-Transformed-From: HTTP/0.9
This header in the response of the proxy indicates that your server does not properly speak HTTP/1.x. Instead of sending HTTP header and body it just sends the payload back without any HTTP header, like done in the HTTP 0.9 protocol which was obsoleted 20 years ago. With HTTP 0.9 the response will always end only at the end of the TCP connection. This means that you cannot have multiple requests within the same TCP connection.
I'm really new to coding using sockets.
The problem is not caused by the wrong use of sockets but due the wrong implementation of the application protocol, i.e. the data send over the socket. If you really need to implement HTTP please study the standards, i.e. RFC 7230 and following. If you don't want to do this use existing and tested HTTP libraries instead of writing your own.
I am trying to connect to a website through a program, whose authentification protocol appears to be the following :
Connect to the websocket.
Receive crucial data for logging in.
Make a HTTP request with username/password/some of the datas THROUGH THIS CONNECTION.
This means that if i open another connection, the data linked to this connection, according to the server, will be different.
For exemple, receiving the data through the websocket, and doing a HTTP request with a browser will not work, as the browser is another connection.
By connection, i mean what is created at the start of the program, for exemple :
ws2= websocket.create_connection("ws://sim.smogon.com:8000/showdown/websocket") for the websocket module
h1 = httplib.HTTPConnection('www.cwi.nl') for the http module.
My problem is that i have to create 2 different connections to connect to websocket and make a http request. This means that the http request will not work with the websocket received data.
How to make these 2 connections the same ? Or, simplier said : how to solve the problem ?
I have a Tornado HTTPServer initialized like so:
ssl_options = {
"certfile": "mycert.crt",
"keyfile": "mykey.key"
}
server = tornado.httpserver.HTTPServer(application, xheaders=True,
ssl_options=ssl_options)
It has a WebSocketHandler with an open() method.
When I attempt to open a secure websocket connection via Javascript from the Chrome 30 console, like
var sock = new WebSocket("wss://localhost:9001/mywebsocket");
the connection does not open successfully, there is no log output, and the WebSocketHandler's open() is not called.
How can I open and maintain a secure (SSL) WebSocket connection to a Javascript client in Tornado?
Attempting an insecure connection, like
var sock = new WebSocket("ws://localhost:9001/mywebsocket");
results in the following error output in the Tornado log:
2013-10-08 13:59:55,305 tornado.general 820 : SSL Error on 8 ('192.168.149.27', 62851): [Errno 1] _ssl.c:490: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
Also in this case, the connection is not opened successfully and open() isn't called.
Additional info: The number after the IP in the error message (62851 in the above example) increases with every request, secure or otherwise. I don't know what that number is but it does indicate that the request is at least getting to the server.
Also, removing ssl_options from the constructor and making insecure (ws://) requests to the server fixes the issue.
The certificate being used is self-signed. To communicate with an HTTP endpoint of the server via curl, I had to use the --insecure flag.
If you're starting the connection from the javascript console, the browser doesn't have a chance to show you the self-signed certificate warning and give you a chance to accept it. If you go to https://localhost:9001 first and accept the certificate there, does it work?
I am creating a proxy server in python, which is based on BaseHTTPServer.
What it does is create a connection to a squid proxy, identifies the browser request(GET, CONNECT, POST etc) and adds a proxy-authorization header to it, and then forwards this request to the squid proxy.
Problem is, as I understand, when I send a connect request, I should relay all the corresponding traffic to the squid proxy. But, as I can see in wireshark, the squid proxy doesn't reply to the 'Client Hello' part of the handshake, which I think is due to squid proxy not understanding binary data of SSL that I am just forwarding to it.
How do I process HTTPS requests in this case?
The code is more or less similar to TinyHTTPProxy : http://www.oki-osk.jp/esc/python/proxy/
RFC 2817 defines the CONNECT method. It is different from other HTTP methods in that the receiving proxy (your Python proxy) is directed to establish a raw TCP tunnel directly to the destination host (called the authority in the RFC).
A proxy can make no assumptions about the data that will be sent over that tunnel; it will not necessarily be HTTP – the client can use the tunnel to speak any protocol it likes. Indeed, SSL ≠ HTTP.
You have two options:
Open a TCP connection directly to the requested destination host.
Make a CONNECT request to your upstream proxy (Squid). This is within spec:
It may be the case that the proxy itself can only reach the
requested origin server through another proxy. In this case, the
first proxy SHOULD make a CONNECT request of that next proxy,
requesting a tunnel to the authority. A proxy MUST NOT respond
with any 2xx status code unless it has either a direct or tunnel
connection established to the authority.
Make sure that your request includes the required Host header.
CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Proxy-Authorization: ...