I am trying to go to http://www.py4inf.com/code/romeo.txt, read the contents of romeo.txt and print them back out, am using python 3.6.1.
import socket
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send('GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\n\n'.encode("utf8"))
while True:
data = mysock.recv(512)
if ( len(data) < 1 ) :
break
print (data.decode("utf8"))
mysock.close()
instead of the contents of the page it prints out
TTP/1.1 404 Not Found
Server: nginx
Date: Wed, 21 Jun 2017 03:00:15 GMT
Content-Type: text/html
Content-Length: 162
Connection: close
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html
Why is this? Thanks in advance
In theory, the Host header is only mandatory from HTTP 1.1 onwards, but it appears that particular server requires the Host header to be present, even for HTTP 1.0. I'm not sure if that's the default behaviour of Nginx, or whether the server admin's explicitly configured it that way.
In any case, try changing your request to the following:
mysock.send('GET http://www.py4inf.com/code/romeo.txt HTTP/1.0\nHost: www.py4inf.com\n\n'.encode("utf8"))
I can understand your confusion - IMHO, it should be returning 400 not 404 if it is insisting on the Host header being provided (since it's a client request issue, not a matter of the resource not existing).
Related
how to get html file into python code using socket. I was able to implement using the requests library. However, it needs to be rewritten to sockets. I don’t understand how. The implementation code through requests will be below. I will also leave pathetic attempts to implement via a socket using Google. However, the decision is not at all correct. ! (Help implement using sockets.
import requests
reg_get = requests.get("https://stackoverflow.blog/")
text = reg_get.text
print(text)
import socket
request = b"GET / HTTP/1.1\nHost: https://stackoverflow.blog/\n\n"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("https://stackoverflow.blog/", 80))
s.send(request)
result = s.recv(10000)
while (len(result) > 0):
print(result)
result = s.recv(10000)
After seeing the comments and listening to you. I have rewritten the following code. However, I never got the html. And I received information about the site. How do I get html structure in python
import socket
import ssl
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
request = "GET /r/AccidentalRenaissance/comments/8ciibe/mr_fluffies_betrayal/ HTTP/1.1\r\nHost: www.reddit.com\r\n\r\n"
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
s = context.wrap_socket(sock, server_hostname = "www.reddit.com")
s.connect(("www.reddit.com", 443))
s.sendall(request.encode())
contest = s.recv(1024).decode()
s.close()
print(contest)
result
HTTP/1.1 200 OK
Connection: keep-alive
Cache-control: private, s-maxage=0, max-age=0, must-revalidate, no-store
Content-Type: text/html; charset=utf-8
X-Frame-Options: SAMEORIGIN
Accept-Ranges: bytes
Date: Sun, 03 Oct 2021 03:34:25 GMT
Via: 1.1 varnish
Vary: Accept-Encoding, Accept-Encoding
A URL is composed of a protocol, a hostname, an optional port, and an optional path. In the URL http://stackoverflow.blog/ , https is the protocol, stackoverflow.blog is the hostname, and no port or path is provided. For http, the port defaults to 80 and the path defaults to /. When using sockets, first establish a connection to the host at the port using connect then send an HTTP command to retrieve the page on the path. The HTTP command to retrieve the page is "GET /" and receive the response from the server.
Note that I used http instead of https because https adds security set up and negotiation to the above that occurs once the connect is done but before the "GET /" is done. It is quite complicated and a good reason to use Requests instead of trying to implement it yourself. If you don't want to use Requests but don't want to go down to the level of sockets, take a look at urllib3
I write a simple program to get some information from a website using python.
but when I run the code below, it always returns the following 301 info. At the same time, my browser can visit the website easily.
Please tell me why this happens and how to improve my code to avoid the problem.
HTTP/1.1 301 Moved Permanently
Date: Tue, 28 Aug 2018 14:26:20 GMT
Server: Apache
Referrer-Policy: origin-when-cross-origin
Location: https://www.ncbi.nlm.nih.gov/
Content-Length: 237
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved here.</p>
</body></html>
import socket
searcher = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
searcher.connect(("www.ncbi.nlm.nih.gov", 80))
cmd = "GET https://www.ncbi.nlm.nih.gov/ HTTP/1.0\r\n\r\n".encode()
searcher.send(cmd)
while True:
data = searcher.recv(512)
if len(data)<1: break
print(data.decode())
searcher.close()
You recieve a 301 because site is redirecting to https site.
I don't know if using sockets is mandatory, but if not you can use requests, it's a easy-to-use lib for doing http requests:
import requests
req = requests.get("http://www.ncbi.nlm.nih.gov")
html = req.text
With this, the 301 is performed anyway but it's transparent.
If you want to do it with sockets, you should add the "ssl layer" manually:
import socket
import ssl
searcher = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
searcher.connect(("www.ncbi.nlm.nih.gov", 443))
searcher = ssl.wrap_socket(searcher, keyfile=None, certfile=None, server_side=False, cert_reqs=ssl.CERT_NONE, ssl_version=ssl.PROTOCOL_SSLv23)
cmd = "GET https://www.ncbi.nlm.nih.gov/ HTTP/1.0\r\n\r\n".encode()
searcher.send(cmd)
while True:
data = searcher.recv(512)
if len(data) < 1: break
print(data.decode())
searcher.close()
I'm really new to coding using sockets.
I like the socket library, I get to understand a big part of what's happening in my program, so i you don't mind i would like to stick with it.
So as the title says, I have a socket based client and server and I would like to exchange content through an HTTP proxy(I'm using a Squid proxy). This little piece of code is supposed to bypass the proxy in my campus to simulate a chat over the campus network. This is totally legal since I asked the IT guys that work there.
Here's the deal, I am able to send a POST request through the proxy to my server which receives it and sends it back to client 1, but when I try to send more requests to the proxy none of them gets to the server so I think to my self the connection died but here's the thing, when I send messages from client 2 which is connected directly to the server, the server AND client 1 receive them.
import socket
from _thread import *
def sender(server,h):
b=input("<<--Send--")
b=h
server.send(b.encode())
PROXY_IP="127.0.0.1"
PROXY_PORT=3128
server=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
server.settimeout(0.05)
server.connect((PROXY_IP,PROXY_PORT))
header="""POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n"""
server.send(header.encode())
while 1:
try:
start_new_thread(sender,(server,header))
a=server.recv(1024)
print("-->>{}".format(a.decode()))
except KeyboardInterrupt:
break
except:
pass
server.close()
I already tried the CONNECT method which works perfectly, but it's not allowed in my campus network proxy.
What am I doing wrong ?
Is there something I should know about how to re-send content through a proxy ?
Thank you for your time and please bear with me..
Here's what I get on the client that sends a request to the proxy:
~#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
#Received : HTTP/1.1 200 OK
Server: squid/3.5.19
Mime-Version: 1.0
Date: Mon, 10 Oct 2016 00:46:39 GMT
X-Transformed-From: HTTP/0.9
X-Cache: MISS from kali
X-Cache-Lookup: MISS from kali:3128
Transfer-Encoding: chunked
Via: 1.1 kali (squid/3.5.19)
Connection: keep-alive
#Received : B2
POST / HTTP/1.1
Content-Length: 5
Host: 127.0.0.1:3001
Via: 1.1 kali (squid/3.5.19)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=259200
Connection: keep-alive
hello
#Sent : POST http://127.0.0.1:3001 HTTP/1.1
Host: 127.0.0.1:3001
Proxy-Connection: keep-alive
Content-Length: 5
hello
Nothing is received after this..
POST http://127.0.0.1:3001 HTTP/1.1\r\n
Host: 127.0.0.1:3001\r\n
Proxy-Connection: keep-alive\r\n
Content-Length: 5 \r\n\r\n
hello\r\n
The body of your HTTP response consists of 7 bytes not 5 as you've stated in your Content-length. The \r\n after the 5 byte still belong to the response body. Giving the wrong size might mixup request handling, i.e. the proxy is expecting a new request but is actually getting \r\n, i.e. the 2 bytes after your 5 bytes Content-length.
Apart from that both path and Host header must include the name of the target from the perspective of the proxy. Using 127.0.0.1. like in your example would mean that you try to access a server at the same host of the proxy, i.e. localhost from the view of the proxy. This is probably not what you've intended.
...
X-Transformed-From: HTTP/0.9
This header in the response of the proxy indicates that your server does not properly speak HTTP/1.x. Instead of sending HTTP header and body it just sends the payload back without any HTTP header, like done in the HTTP 0.9 protocol which was obsoleted 20 years ago. With HTTP 0.9 the response will always end only at the end of the TCP connection. This means that you cannot have multiple requests within the same TCP connection.
I'm really new to coding using sockets.
The problem is not caused by the wrong use of sockets but due the wrong implementation of the application protocol, i.e. the data send over the socket. If you really need to implement HTTP please study the standards, i.e. RFC 7230 and following. If you don't want to do this use existing and tested HTTP libraries instead of writing your own.
I am trying to simulate Network Address Translation for some test code. I am mapping virtual users to high port numbers, then, when I want to test a request from user 1, I send it from port 6000 (user 2, from 6001, etc).
However, I can't see the port number in the response.
connection = httplib.HTTPConnection("the.company.lan", port=80, strict=False,
timeout=10, source_address=("10.129.38.51", 6000))
connection.request("GET", "/portal/index.html")
httpResponse = connection.getresponse()
connection.close()
httpResponse.status is 200, but I don't see the port number anywhere in the response headers.
Maybe I should be using some lower level socket functionality? If so, which is simplest and supports both HTTP and FTP? Btw, I only want to use built-in modules, nothing which I have to install.
[Update] I should have made it clearer; I really do need to get the actual port number received in the response, not just remember it.
To complete #TimSpence answer, you can use a socket object as an interface for your connection and then treat with some API your data as an HTTP object.
host = 'xxx.xxx.xxx.xxx'
port = 80
address = (host, port)
## socket object interface for a TCP connection
listener_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM,
socket.IPPROTO_TCP)
listener_socket.bind(address)
listener_socker.listen(MAX_CONNECTIONS)
## new_connection is the connection object you use to handle the data echange
## source_address (source_host, source_port) is the address object
## source_port is what you're looking for
new_connection, source_address = listener_socket.accept()
data = new_connection.recv(65536)
## handle data as an HTTP object(for example as an HTTP request)
new_connection.close()
HTTP messages do not contain anything about ports so the httpResponse will not have that information.
However, you will need a different connection object (which will map to a different underlying socket) for each request anyway so you can get that information from the HTTPconnection object.
_, port = connection.source_address
Does that help?
Considring your comments, I had to provide a new answer.
I though you can also put a non standard header host in your HTTPRespose, 'Host: domain/IP:port', so that your client can read it when it receives a response.
Server Response:
HTTP/1.1 200 OK
Date: Day, DD Month YYYY HH:MM:SS GMT
Content-Type: text/html; charset=UTF-8
Content-Encoding: UTF-8
Content-Length: LENGTH
Last-Modified: Day, DD Month YYYY HH:MM:SS GMT
Server: Name/Version (Platform)
Accept-Ranges: bytes
Connection: close
Host: domain/IP:port #exapmple: the.company.lan:80
<html>
<head>
<title>Example Response</title>
</head>
<body>
Hello World!
</body>
</html>
Client:
connection = httplib.HTTPConnection("the.company.lan", port=80,
strict=False, timeout=10,
source_address=("10.129.38.51", 6000))
connection.request("GET", "/portal/index.html")
httpResponse = connection.getresponse()
## store a dict with the response headers
## extract your custom header 'host'
res_headers = dict(httpResponse.getheaders());
server_address = tuple(headers['host'].split(':'))
## read the response content
HTMLData = httpResponse.read(CONTENT_LENGTH_HEADER)
connection.close()
This way you got server_address as a tuple (domain, port).
Fir let me clear I don't want to to use higher level APIs, I only want to use socket programming
I have wrote following program to connect to server using POST request.
import socket
import binascii
host = "localhost"
port = 9000
message = "POST /auth HTTP/1.1\r\n"
parameters = "userName=Ganesh&password=pass\r\n"
contentLength = "Content-Length: " + str(len(parameters))
contentType = "Content-Type: application/x-www-form-urlencoded\r\n"
finalMessage = message + contentLength + contentType + "\r\n"
finalMessage = finalMessage + parameters
finalMessage = binascii.a2b_qp(finalMessage)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
s.sendall(finalMessage)
print(s.recv(1024))
I checked online how POST request is created.
Somehow Paramters are not getting passed to the server. Do I have to add or remove "\r\n" in between the request?
Thanks in advance,
Regards,
Ganesh.
This line finalMessage = binascii.a2b_qp(finalMessage) is certainly wrong, so you should remove the line completely, another problem is that there is no new-line missing after Content-Length. In this case the request sent to the socket is (I am showing the CR and LF characters here as \r\n, but also splitting lines for clarity):
POST /auth HTTP/1.1\r\n
Content-Length: 31Content-Type: application/x-www-form-urlencoded\r\n
\r\n
userName=Ganesh&password=pass\r\n
So obviously this does not make much sense to the web server.
But even after adding a newline and removing a2b_qp, there is still the problem is that you are not talking HTTP/1.1 there; the request must have a Host header for HTTP/1.1 (RFC 2616 14.23):
A client MUST include a Host header field in all HTTP/1.1 request
messages . If the requested URI does not include an Internet host name
for the service being requested, then the Host header field MUST be
given with an empty value. An HTTP/1.1 proxy MUST ensure that any
request message it forwards does contain an appropriate Host header
field that identifies the service being requested by the proxy. All
Internet-based HTTP/1.1 servers MUST respond with a 400 (Bad Request)
status code to any HTTP/1.1 request message which lacks a Host header
field.
Also you do not support chunked requests and persistent connections, keepalives or anything, so you must do Connection: close (RFC 2616 14.10):
HTTP/1.1 applications that do not support persistent connections MUST
include the "close" connection option in every message.
Thus, any HTTP/1.1 server that would still respond normally to your messages without Host: header is also broken.
This the data that you should send to the socket with that request:
POST /auth HTTP/1.1\r\n
Content-Type: application/x-www-form-urlencoded\r\n
Content-Length: 29\r\n
Host: localhost:9000\r\n
Connection: close\r\n
\r\n
userName=Ganesh&password=pass
Note that you'd not add the \r\n in the body anymore (thus the length of body 29). Also you should read the response to find out whatever the error is that you're getting.
On Python 3 the working code would say:
host = "localhost"
port = 9000
headers = """\
POST /auth HTTP/1.1\r
Content-Type: {content_type}\r
Content-Length: {content_length}\r
Host: {host}\r
Connection: close\r
\r\n"""
body = 'userName=Ganesh&password=pass'
body_bytes = body.encode('ascii')
header_bytes = headers.format(
content_type="application/x-www-form-urlencoded",
content_length=len(body_bytes),
host=str(host) + ":" + str(port)
).encode('iso-8859-1')
payload = header_bytes + body_bytes
# ...
socket.sendall(payload)