I'm using socket to build a simple "web browser" but I'm getting stuck at the start, whit a bad request result, here is my code:
import socket
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
URI = 'data.pr4e.org'
mysocket.connect((URI, 80))
cmd = "GET http://{0}/romeo.txt HTTP/1.0\n\n".format(URI).encode()
mysocket.send(cmd) # send a request
while True:
data = mysocket.recv(512) # recieve 512 bites at time
# if there is no more information to recive, then, close the loop
if (len(data) < 1):
break
print(data.decode())
pass
mysocket.close() # close connection
here is the output
HTTP/1.1 400 Bad Request
Date: Mon, 15 Feb 2021 14:36:06 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 308
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</
h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at do1.dr-chuck.com Port 80</address>
what I'm doing wrong? also, I tryed replacing data.pr4e.org by facebook.com and youtube.com and I get this output:
HTTP/1.1 301 Moved Permanently
Vary: Accept-Encoding
Location: https://facebook.com/
Content-Type: text/html; charset="utf-8"
X-FB-Debug: LPmWQm0VVptVpi8QX8/SxymrJg9ZoL/mL+W+G4pZA4HGj5WI5YIG1s8sgqwp6TIleGvUg3U1eDNEhGoCsaJG5g==
Date: Mon, 15 Feb 2021 14:52:43 GMT
Alt-Svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
Connection: close
Content-Length: 0
thank you
Here the problem is just that you used \n when the server expected \r\n for end of line.
Anyway, as you directly connect to the HTTP host, you should not put the full URI in the request line. This would be better on a HTTP 1.0 conformance point:
cmd = "GET /romeo.txt HTTP/1.0\r\n\r\n".encode()
But if the server could accept more that one virtual server, you should pass the name in a Host header:
cmd = "GET /romeo.txt HTTP/1.0\r\nHost: {}\r\n\r\n".format(URI).encode()
Related
I'm new with rest APIs and I'm trying to set up an OAuth handshake and I need help with requesting the request token. I'm using the requests_oauthlib module in Python. Here is the sample code and it is returning Response [400].
consumer_key, consumer_secret, and request_url are all loaded in properly. I got my code to work using a different Auth module. Can someone explain what http headers are and how they are used in a GET request?
from requests_oauthlib import OAuth1
from variables import *
oauth = OAuth1(consumer_key, client_secret = consumer_secret)
request_token = requests.get(request_url, auth=oauth, params={'oauth_callback':'oob', 'format':'json'})
print request_token
request : your computer sends a http message to another computer usually on port 443 or 80
response : the server listens for any connection requests, and responds if it understands the message.
For example telnet stackoverflow.com 80 you can type in
GET /questions/52350391/can-someone-explain-get-requests-specifically-the-http-header HTTP/2
Host: stackoverflow.com
User-Agent: curl/7.54.0
Accept: */*
And then press enter twice to conclude the request header, at which point the server responds:
➜ mysite telnet stackoverflow.com 80
Trying 151.101.1.69...
Connected to stackoverflow.com.
Escape character is '^]'.
GET /questions/52350391/can-someone-explain-get-requests-specifically-the-http- header HTTP/2
Host: stackoverflow.com
User-Agent: curl/7.54.0
Accept: */*
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=utf-8
Location: https://stackoverflow.com/questions/52350391/can-someone-explain-get-requests-specifically-the-http-
X-Request-Guid: xxx
Content-Security-Policy: upgrade-insecure-requests
Accept-Ranges: bytes
Age: 0
Content-Length: 217
Accept-Ranges: bytes
Date: Sun, 16 Sep 2018 03:29:16 GMT
Via: 1.1 varnish
Age: 0
Connection: close
X-Served-By: cache-ord1744-ORD
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1537068557.736123,VS0,VE25
Vary: Fastly-SSL
X-DNS-Prefetch-Control: off
Set-Cookie: prov=xxx; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
<html><head><title>Object moved</title></head><body>
<h2>Object moved to here.</h2>
</body></html>
Connection closed by foreign host.
The telnet session then prints out the response from the server and closes the connection. The response will include several pieces, the response headers, and the response body.
Your example might look something like:
GET /some/oauth/api?oauth_callback=oob&format=json
Host: someplace.com
Authorization: Bearer asdfasdfasdfasdf
HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store
Pragma: no-cache
{
"access_token":"sdfasasdfasdf",
"token_type":"bearer",
"expires_in":3600,
"refresh_token":"asdfasdfasdfasdf",
"scope":"create"
}
also check out :
curl -Lv https://stackoverflow.com/questions/52350391/can-someone-explain-get-requests-specifically-the-http-header | head -n 100
related:
https://www.oauth.com/oauth2-servers/access-tokens/access-token-response/
HTTP Request\Response Header Grammer
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
https://www.oauth.com/oauth2-servers/authorization/the-authorization-request/
I write a simple program to get some information from a website using python.
but when I run the code below, it always returns the following 301 info. At the same time, my browser can visit the website easily.
Please tell me why this happens and how to improve my code to avoid the problem.
HTTP/1.1 301 Moved Permanently
Date: Tue, 28 Aug 2018 14:26:20 GMT
Server: Apache
Referrer-Policy: origin-when-cross-origin
Location: https://www.ncbi.nlm.nih.gov/
Content-Length: 237
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved here.</p>
</body></html>
import socket
searcher = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
searcher.connect(("www.ncbi.nlm.nih.gov", 80))
cmd = "GET https://www.ncbi.nlm.nih.gov/ HTTP/1.0\r\n\r\n".encode()
searcher.send(cmd)
while True:
data = searcher.recv(512)
if len(data)<1: break
print(data.decode())
searcher.close()
You recieve a 301 because site is redirecting to https site.
I don't know if using sockets is mandatory, but if not you can use requests, it's a easy-to-use lib for doing http requests:
import requests
req = requests.get("http://www.ncbi.nlm.nih.gov")
html = req.text
With this, the 301 is performed anyway but it's transparent.
If you want to do it with sockets, you should add the "ssl layer" manually:
import socket
import ssl
searcher = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
searcher.connect(("www.ncbi.nlm.nih.gov", 443))
searcher = ssl.wrap_socket(searcher, keyfile=None, certfile=None, server_side=False, cert_reqs=ssl.CERT_NONE, ssl_version=ssl.PROTOCOL_SSLv23)
cmd = "GET https://www.ncbi.nlm.nih.gov/ HTTP/1.0\r\n\r\n".encode()
searcher.send(cmd)
while True:
data = searcher.recv(512)
if len(data) < 1: break
print(data.decode())
searcher.close()
Well, I just want to make the following simple program that tries to create an https tunel with www.google.com at port 443. I first tried the following code:
import socket
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("www.google.com", 80))
request = "CONNECT www.google.com:443 HTTP/1.1\n\n"
s.send(request.encode())
print(s.recv(4096).decode())
main()
The result of that was the following:
HTTP/1.1 405 Method Not Allowed
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1592
Date: Wed, 16 Aug 2017 07:56:14 GMT
Connection: close
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 405 (Method Not Allowed)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>405.</b> <ins>That’s an error.</ins>
<p>The request method <code>CONNECT</code> is inappropriate for the URL <code>/</code>. <ins>That’s all we know.</ins>
That means that the server does not allow this request to be executed. So I thought that the problem was the port number. So I changed it to 443(which is the port for https connection). The code is that:
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("www.google.com", 443))
request = "CONNECT www.google.com:443 HTTP/1.1\n\n"
s.send(request.encode())
print(s.recv(4096).decode())
main()
But it does not print out a valid respnse as it should have done. It gives me an empty response.
The question to that is: "Why is that happening? How can I make it work properly?"
Note: I don't want to use built-in urllib or urllib2 libraries. I want to do that with sockets.
HTTP
In your original connection to port 80 you are just using wrong Host:
import socket
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('google.com', 80))
request = b'CONNECT google.com HTTP/1.1\n\n'
s.send(request)
print(s.recv(4096).decode())
main()
Response:
HTTP/1.0 200 Connection established
Or use GET method right away:
request = b'GET http://google.com HTTP/1.1\n\n'
Response is the same as to HTTPS request, google.com host doesn't work for some reason.
HTTPS
You should wrap your socket in ssl tunnel (not sure if correct term) in order to connect using HTTPS, and GET method is ready to use right after connection:
import socket
import ssl
def main():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s = ssl.wrap_socket(s)
s.connect(('google.com', 443))
request = b'GET google.com HTTP/1.1\n\n'
s.send(request)
print(s.recv(4096).decode())
main()
Response:
HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Location: https://www.google.ru/?gfe_rd=cr&ei=WwCUWc66L6qB3APs7ZPABA
Content-Length: 259
Date: Wed, 16 Aug 2017 08:20:43 GMT
Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
here.
</BODY></HTML>
Hi I Want Python Socket Connect Poloniex API.
I ran the code. But I can not get the results I want.
I Made Code:
===================================================================
import requests
import socket
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("poloniex.com",443))
message="GET /public?command=returnTicker HTTP/1.1\r\nHost: poloniex.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.18.1\r\n\r\n"
s.send(message)
print s.recv(5000)
===================================================================
Response Text:
HTTP/1.1 400 Bad Request
Server: cloudflare-nginx
Date: Tue, 20 Jun 2017 02:52:22 GMT
Content-Type: text/html
Content-Length: 275
Connection: close CF-RAY: - 400 The plain HTTP request was sent to HTTPS port
===================================================================
The error message is right - you're sending an HTTP request to port 443 which is the HTTPS port. If you want to send an HTTP request, use port 80. I have just tried to send a request to port 80, and the response says I should be using HTTPS from now on (see Location: https:// part):
HTTP/1.1 301 Moved Permanently
Date: Tue, 20 Jun 2017 13:40:52 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d28a8f446379618a093014a5f13bbcb141497966052; expires=Wed, 20-Jun-18 13:40:52 GMT; path=/; domain=.poloniex.com; HttpOnly
Location: https://poloniex.com/public?command=returnTicker
Server: cloudflare-nginx
CF-RAY: 371f2473b09f5a7a-BOS
In this case you should use whether ssl module instead of socket, or just use requests since it is a simpler option.
I am playing with HTTP transfers, just trying to make something work. I have a GAE server and I'm pretty sure it's working properly because it renders when I go to it with my browser, but here is the python code anyway:
import sys
print 'Content-Type: text/html'
print ''
print '<pre>'
number = -1
data = sys.stdin.read()
try:
number = int(data[data.find('=')+1:])
except:
number = -1
print 'If your number was', number, ', then you are awesome!!!'
print '</pre>'
I am just learning the whole HTTP POST vs GET vs Response process, but this is what I have been doing from the terminal:
$ telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET http://localhost:8080/?number=28 HTTP/1.0
HTTP/1.0 200 Good to go
Server: Development/1.0
Date: Thu, 07 Jul 2011 21:29:28 GMT
Content-Type: text/html
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Length: 61
<pre>
If your number was -1 , then you are awesome!!!
</pre>
Connection closed by foreign host.
I am using a GET here because I stumbled around for about 40 minutes trying to make a telnet POST work - with no success :(
I would appreciate any help on how to get this GET and/or the POST to work. Thanks in advance!!!!
when using GET, no data will be present in the request body, so sys.stdin.read() is bound to fail. instead, you might want to look at the environment, specifically os.environ['QUERY_STRING']
Another thing you're doing a little strangely is you are not using the correct request format. The second part of the request should not include the url scheme, host or port, it should look like:
GET /?number=28 HTTP/1.0
specify the host in a seperate Host: header; the server will determine the scheme on it's own.
When using POST, most servers won't read past the amount of data in the Content-Length header, which if you don't supply one, may be assumed to be zero bytes. The server may try to read any bytes after the point specified by the content-length to be the next request in a persistent connection, and when it doesn't begin with a valid request, it closes the connection. So basically:
POST / HTTP/1.0
Host: localhost: 8080
Content-Length: 2
Content-Type: text/plain
28
But why are you testing this in telnet? How about curl?
$ curl -vs -d'28' -H'Content-Type: text/plain' http://localhost:8004/
* About to connect() to localhost port 8004 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8004 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.20.1 (x86_64-redhat-linux-gnu) libcurl/7.20.1 NSS/3.12.6.2 zlib/1.2.3 libidn/1.16 libssh2/1.2.4
> Host: localhost:8004
> Accept: */*
> Content-Type: text/plain
> Content-Length: 2
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Thu, 07 Jul 2011 22:09:17 GMT
< Server: WSGIServer/0.1 Python/2.6.4
< Content-Type: text/html; charset=UTF-8
< Content-Length: 45
<
* Closing connection #0
{'body': '28', 'method': 'POST', 'query': []}
or better yet, in python:
>>> import httplib
>>> headers = {"Content-type": "text/plain",
... "Accept": "text/plain"}
>>>
>>> conn = httplib.HTTPConnection("localhost:8004")
>>> conn.request("POST", "/", "28", headers)
>>> response = conn.getresponse()
>>> print response.read()
{'body': '28', 'method': 'POST', 'query': []}
>>>