Because the new version of openssl removes support for 3DES, I have to use this lib to send raw socket request which not use libopenssl.
I read the docs and test demo from python-gnutls's website page that hosted on github.
Here is my code:
win10 python 3.9 python-gnutls 3.1.3 gnutls 3.5.19
import socket
from gnutls.connection import *
# this is the host only 3DES under TLS1.0, even though there is a # 302 redirection.
# hostname = 'www.mycardtamoil.it'
hostname = 'stackoverflow.com'
ctx = TLSContext(X509Credentials())
sock = socket.create_connection((hostname, 443))
s = ClientSession(sock, ctx)
s.handshake()
print(s.peer_certificate.subject)
print(s.protocol.decode(), s.cipher.decode())
#s.send(f'GET / HTTP/1.1\r\nHost: {hostname}\r\n\r\n') Wrong usage
s.send(memoryview(f'GET / HTTP/1.1\r\nHost: {hostname}\r\n\r\n\r\n'.encode()))
print(s.recv(1024).decode())
s.bye()
s.close()
The result with Wrong usage which not use memoryview type :
CN=*.stackexchange.com
TLS1.2 AES-128-GCM
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 11
content-type: text/plain; charset=utf-8
Now the problem is how to send a GET request to normal website.
I have solved this problem at 2021-04-29.
s.send(memoryview(f'GET / HTTP/1.1\r\nHost: {hostname}\r\n\r\n\r\n'.encode())) Write
s.send(f'GET / HTTP/1.1\r\nHost: {hostname}\r\n\r\n') Wrong type used
Related
When I am using this Python script to send a request to my server:
import requests as r
url = "http://localhost:8070/"
response = r.get(url=url)
It sends the following request:
GET / HTTP/1.1
Host: localhost:8070
User-Agent: python-requests/2.27.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
If I understand it correctly, Connection: keep-alive means that I should not close the client-socket, because the client could use it again.
But if I do not close the client-socket, the Python script just gets stuck until I close the socket. Is there another way to indicate that the request is complete so pythons requests understands it?
If I try to send the request to any other server, the script almost immediately finishes. My guess would be to timeout the client after a few milliseconds e.g. by using select like this:
struct timeval timeout;
timeout.tv_sec = 0;
timeout.tv_usec = 1000;
select_ret = select(this->_maxfds + 1, &this->_readfds, &this->_writefds, NULL, &timeout);
And now I would just close the client-socket after select returns 0:
if (select_ret == 0) {
close(client_socket);
}
Is that a valid approach, or am I missing something?
I am sending the response like this:
char *response = "HTTP/1.1 200 Ok\r\n\r\n";
send(this->_client_socket, response, strlen(response), 0)
BUT this does not terminate the python script. The python script still hangs after I execute this line of code. It only finishes when I close the socket on my side.
So how would I determine if I should close it or not? As I already said my approach was to use a timeout in case no data is getting written in to the socket from the client side.
Your server's response is incomplete.
Your understanding of Connection: keep-alive in the request is correct. However, there is no Content-Length or Transfer-Encoding: chunked header present in your response, so the only way the client has to know when the response is finished is to wait for the socket connection to be closed on the server side. Read the rules outlined in RFC 2616 Section 4.4 and RFC 7230 Section 3.3.3 of the HTTP 1.1 protocol spec.
Try something more like this instead:
const char *response = "HTTP/1.1 200 Ok\r\nContent-Length: 0\r\n\r\n";
send(this->_client_socket, response, strlen(response), 0)
how to get html file into python code using socket. I was able to implement using the requests library. However, it needs to be rewritten to sockets. I don’t understand how. The implementation code through requests will be below. I will also leave pathetic attempts to implement via a socket using Google. However, the decision is not at all correct. ! (Help implement using sockets.
import requests
reg_get = requests.get("https://stackoverflow.blog/")
text = reg_get.text
print(text)
import socket
request = b"GET / HTTP/1.1\nHost: https://stackoverflow.blog/\n\n"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("https://stackoverflow.blog/", 80))
s.send(request)
result = s.recv(10000)
while (len(result) > 0):
print(result)
result = s.recv(10000)
After seeing the comments and listening to you. I have rewritten the following code. However, I never got the html. And I received information about the site. How do I get html structure in python
import socket
import ssl
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
request = "GET /r/AccidentalRenaissance/comments/8ciibe/mr_fluffies_betrayal/ HTTP/1.1\r\nHost: www.reddit.com\r\n\r\n"
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
s = context.wrap_socket(sock, server_hostname = "www.reddit.com")
s.connect(("www.reddit.com", 443))
s.sendall(request.encode())
contest = s.recv(1024).decode()
s.close()
print(contest)
result
HTTP/1.1 200 OK
Connection: keep-alive
Cache-control: private, s-maxage=0, max-age=0, must-revalidate, no-store
Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN
Accept-Ranges: bytes
Date: Sun, 03 Oct 2021 03:34:25 GMT
Via: 1.1 varnish
Vary: Accept-Encoding, Accept-Encoding
A URL is composed of a protocol, a hostname, an optional port, and an optional path. In the URL http://stackoverflow.blog/ , https is the protocol, stackoverflow.blog is the hostname, and no port or path is provided. For http, the port defaults to 80 and the path defaults to /. When using sockets, first establish a connection to the host at the port using connect then send an HTTP command to retrieve the page on the path. The HTTP command to retrieve the page is "GET /" and receive the response from the server.
Note that I used http instead of https because https adds security set up and negotiation to the above that occurs once the connect is done but before the "GET /" is done. It is quite complicated and a good reason to use Requests instead of trying to implement it yourself. If you don't want to use Requests but don't want to go down to the level of sockets, take a look at urllib3
I'm really new to coding using sockets.
I like the socket library, I get to understand a big part of what's happening in my program, so i you don't mind i would like to stick with it.
So as the title says, I have a socket based client and server and I would like to exchange content through an HTTP proxy(I'm using a Squid proxy). This little piece of code is supposed to bypass the proxy in my campus to simulate a chat over the campus network. This is totally legal since I asked the IT guys that work there.
Here's the deal, I am able to send a POST request through the proxy to my server which receives it and sends it back to client 1, but when I try to send more requests to the proxy none of them gets to the server so I think to my self the connection died but here's the thing, when I send messages from client 2 which is connected directly to the server, the server AND client 1 receive them.
import socket
from _thread import *
def sender(server,h):
b=input("<<--Send--")
b=h
server.send(b.encode())
PROXY_IP="127.0.0.1"
PROXY_PORT=3128
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.settimeout(0.05)
server.connect((PROXY_IP,PROXY_PORT))
header="""POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n"""
server.send(header.encode())
while 1:
try:
start_new_thread(sender,(server,header))
a=server.recv(1024)
print("-->>{}".format(a.decode()))
except KeyboardInterrupt:
break
except:
pass
server.close()
I already tried the CONNECT method which works perfectly, but it's not allowed in my campus network proxy.
What am I doing wrong ?
Is there something I should know about how to re-send content through a proxy ?
Thank you for your time and please bear with me..
Here's what I get on the client that sends a request to the proxy:
~#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
#Received : HTTP/1.1 200 OK
Server: squid/3.5.19
Mime-Version: 1.0
Date: Mon, 10 Oct 2016 00:46:39 GMT
X-Transformed-From: HTTP/0.9
X-Cache: MISS from kali
X-Cache-Lookup: MISS from kali:3128
Transfer-Encoding: chunked
Via: 1.1 kali (squid/3.5.19)
Connection: keep-alive
#Received : B2
POST / HTTP/1.1
Content-Length: 5
Host: 127.0.0.1:3001
Via: 1.1 kali (squid/3.5.19)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=259200
Connection: keep-alive
hello
#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
Nothing is received after this..
POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n
The body of your HTTP response consists of 7 bytes not 5 as you've stated in your Content-length. The \r\n after the 5 byte still belong to the response body. Giving the wrong size might mixup request handling, i.e. the proxy is expecting a new request but is actually getting \r\n, i.e. the 2 bytes after your 5 bytes Content-length.
Apart from that both path and Host header must include the name of the target from the perspective of the proxy. Using 127.0.0.1. like in your example would mean that you try to access a server at the same host of the proxy, i.e. localhost from the view of the proxy. This is probably not what you've intended.
...
X-Transformed-From: HTTP/0.9
This header in the response of the proxy indicates that your server does not properly speak HTTP/1.x. Instead of sending HTTP header and body it just sends the payload back without any HTTP header, like done in the HTTP 0.9 protocol which was obsoleted 20 years ago. With HTTP 0.9 the response will always end only at the end of the TCP connection. This means that you cannot have multiple requests within the same TCP connection.
I'm really new to coding using sockets.
The problem is not caused by the wrong use of sockets but due the wrong implementation of the application protocol, i.e. the data send over the socket. If you really need to implement HTTP please study the standards, i.e. RFC 7230 and following. If you don't want to do this use existing and tested HTTP libraries instead of writing your own.
I am trying to simulate Network Address Translation for some test code. I am mapping virtual users to high port numbers, then, when I want to test a request from user 1, I send it from port 6000 (user 2, from 6001, etc).
However, I can't see the port number in the response.
connection = httplib.HTTPConnection("the.company.lan", port=80, strict=False,
timeout=10, source_address=("10.129.38.51", 6000))
connection.request("GET", "/portal/index.html")
httpResponse = connection.getresponse()
connection.close()
httpResponse.status is 200, but I don't see the port number anywhere in the response headers.
Maybe I should be using some lower level socket functionality? If so, which is simplest and supports both HTTP and FTP? Btw, I only want to use built-in modules, nothing which I have to install.
[Update] I should have made it clearer; I really do need to get the actual port number received in the response, not just remember it.
To complete #TimSpence answer, you can use a socket object as an interface for your connection and then treat with some API your data as an HTTP object.
host = 'xxx.xxx.xxx.xxx'
port = 80
address = (host, port)
## socket object interface for a TCP connection
listener_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM,
socket.IPPROTO_TCP)
listener_socket.bind(address)
listener_socker.listen(MAX_CONNECTIONS)
## new_connection is the connection object you use to handle the data echange
## source_address (source_host, source_port) is the address object
## source_port is what you're looking for
new_connection, source_address = listener_socket.accept()
data = new_connection.recv(65536)
## handle data as an HTTP object(for example as an HTTP request)
new_connection.close()
HTTP messages do not contain anything about ports so the httpResponse will not have that information.
However, you will need a different connection object (which will map to a different underlying socket) for each request anyway so you can get that information from the HTTPconnection object.
_, port = connection.source_address
Does that help?
Considring your comments, I had to provide a new answer.
I though you can also put a non standard header host in your HTTPRespose, 'Host: domain/IP:port', so that your client can read it when it receives a response.
Server Response:
HTTP/1.1 200 OK
Date: Day, DD Month YYYY HH:MM:SS GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: LENGTH
Last-Modified: Day, DD Month YYYY HH:MM:SS GMT
Server: Name/Version (Platform)
Accept-Ranges: bytes
Connection: close
Host: domain/IP:port #exapmple: the.company.lan:80
<html>
<head>
<title>Example Response</title>
</head>
<body>
Hello World!
</body>
</html>
Client:
connection = httplib.HTTPConnection("the.company.lan", port=80,
strict=False, timeout=10,
source_address=("10.129.38.51", 6000))
connection.request("GET", "/portal/index.html")
httpResponse = connection.getresponse()
## store a dict with the response headers
## extract your custom header 'host'
res_headers = dict(httpResponse.getheaders());
server_address = tuple(headers['host'].split(':'))
## read the response content
HTMLData = httpResponse.read(CONTENT_LENGTH_HEADER)
connection.close()
This way you got server_address as a tuple (domain, port).
Fir let me clear I don't want to to use higher level APIs, I only want to use socket programming
I have wrote following program to connect to server using POST request.
import socket
import binascii
host = "localhost"
port = 9000
message = "POST /auth HTTP/1.1\r\n"
parameters = "userName=Ganesh&password=pass\r\n"
contentLength = "Content-Length: " + str(len(parameters))
contentType = "Content-Type: application/x-www-form-urlencoded\r\n"
finalMessage = message + contentLength + contentType + "\r\n"
finalMessage = finalMessage + parameters
finalMessage = binascii.a2b_qp(finalMessage)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
s.sendall(finalMessage)
print(s.recv(1024))
I checked online how POST request is created.
Somehow Paramters are not getting passed to the server. Do I have to add or remove "\r\n" in between the request?
Thanks in advance,
Regards,
Ganesh.
This line finalMessage = binascii.a2b_qp(finalMessage) is certainly wrong, so you should remove the line completely, another problem is that there is no new-line missing after Content-Length. In this case the request sent to the socket is (I am showing the CR and LF characters here as \r\n, but also splitting lines for clarity):
POST /auth HTTP/1.1\r\n
Content-Length: 31Content-Type: application/x-www-form-urlencoded\r\n
\r\n
userName=Ganesh&password=pass\r\n
So obviously this does not make much sense to the web server.
But even after adding a newline and removing a2b_qp, there is still the problem is that you are not talking HTTP/1.1 there; the request must have a Host header for HTTP/1.1 (RFC 2616 14.23):
A client MUST include a Host header field in all HTTP/1.1 request
messages . If the requested URI does not include an Internet host name
for the service being requested, then the Host header field MUST be
given with an empty value. An HTTP/1.1 proxy MUST ensure that any
request message it forwards does contain an appropriate Host header
field that identifies the service being requested by the proxy. All
Internet-based HTTP/1.1 servers MUST respond with a 400 (Bad Request)
status code to any HTTP/1.1 request message which lacks a Host header
field.
Also you do not support chunked requests and persistent connections, keepalives or anything, so you must do Connection: close (RFC 2616 14.10):
HTTP/1.1 applications that do not support persistent connections MUST
include the "close" connection option in every message.
Thus, any HTTP/1.1 server that would still respond normally to your messages without Host: header is also broken.
This the data that you should send to the socket with that request:
POST /auth HTTP/1.1\r\n
Content-Type: application/x-www-form-urlencoded\r\n
Content-Length: 29\r\n
Host: localhost:9000\r\n
Connection: close\r\n
\r\n
userName=Ganesh&password=pass
Note that you'd not add the \r\n in the body anymore (thus the length of body 29). Also you should read the response to find out whatever the error is that you're getting.
On Python 3 the working code would say:
host = "localhost"
port = 9000
headers = """\
POST /auth HTTP/1.1\r
Content-Type: {content_type}\r
Content-Length: {content_length}\r
Host: {host}\r
Connection: close\r
\r\n"""
body = 'userName=Ganesh&password=pass'
body_bytes = body.encode('ascii')
header_bytes = headers.format(
content_type="application/x-www-form-urlencoded",
content_length=len(body_bytes),
host=str(host) + ":" + str(port)
).encode('iso-8859-1')
payload = header_bytes + body_bytes
# ...
socket.sendall(payload)