I want to send some HTTP requests to Twitter in Python in order to create a sign in for Twitter users for my app. I am using urllib, and following this link: https://dev.twitter.com/web/sign-in/implementing.
But I am unable to do this. I guess I need to authenticate before requesting a token but I don't know how to do that.
Code:
import urllib.request
req = urllib.request.Request("https://api.twitter.com/oauth/authenticate",
headers={'User-Agent': 'Mozilla/5.0'})
html = urllib.request.urlopen(req).read() //after this statement im
getting the error
Error:
Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
html = urllib.request.urlopen(req).read()
File "C:\Python34\lib\urllib\request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 469, in open
response = meth(req, response)
File "C:\Python34\lib\urllib\request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python34\lib\urllib\request.py", line 507, in error
return self._call_chain(*args)
File "C:\Python34\lib\urllib\request.py", line 441, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
If you go to the url with a browser it shows you that you need a key:
Whoa there!
There is no request token for this page. That's the special key we need from applications asking to use your Twitter account. Please go back to the site or application that sent you here and try again; it was probably just a mistake.
If you go to this link it will let you choose one of your apps and
it will bring you to a signature-generator that will show you the request settings.
To get a request_token you can use requests_oauthlib:
import requests
from requests_oauthlib import OAuth1
REQUEST_TOKEN_URL = "https://api.twitter.com/oauth/request_token"
CONSUMER_KEY = "xxxxxxxx
CONSUMER_SECRET = "xxxxxxxxxxxxxxxxx"
oauth = OAuth1(CONSUMER_KEY, client_secret=CONSUMER_SECRET)
r = requests.post(url=REQUEST_TOKEN_URL, auth=oauth)
print(r.content)
oauth_token=xxxxxxxxxxxxxx&oauth_token_secret=xxxxxxxxxxx&oauth_callback_confirmed=true
You then need to extract the oauth_token oauth_token_secret:
from urlparse import parse_qs
import webbrowser
data = parse_qs(r.content)
oauth_token = data['oauth_token'][0]
oauth_token_secret = data['oauth_token_secret'][0]
AUTH = "https://api.twitter.com/oauth/authorize?oauth_token={}"
auth = AUTH.format(oauth_token)
webbrowser.open(auth)
A webpage will open asking to Authorize your_app to use your account?
For python 3 use:
from urllib.parse import parse_qs
data = parse_qs(r.text)
oauth_token = data['oauth_token'][0]
oauth_token_secret = data['oauth_token_secret'][0]
Related
So recently I came across the Google Testing Tools API - Mobile Friendly Test (https://developers.google.com/webmaster-tools/search-console-api/reference/rest/v1/urlTestingTools.mobileFriendlyTest) but I couldn't work it even when I am trying on the site. I tried to use python for this app and followed the guide (https://developers.google.com/webmaster-tools/search-console-api/v1/samples) and I made some few changes to actually make it work (since urllib was merged into one library). So end of the day my code looked like this:
from __future__ import print_function
import urllib
import urllib.request as urllib2
api_key = 'API_KEY'
request_url = 'https://www.google.com/'
service_url = 'https://searchconsole.googleapis.com/v1/urlTestingTools/mobileFriendlyTest:run'
params = {
'url' : request_url,
'key' : api_key,
}
data = urllib.parse.urlencode(params)
content = urllib2.urlopen(url=service_url, data=str.encode(data)).read()
print(content)
And I got the error:
File ".\script2.py", line 14, in <module>
content = urllib2.urlopen(url=service_url, data=str.encode(data)).read()
File "C:\Python\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Python\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Python\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Python\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Python\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
I also tried using curl command and the online tool (not https://search.google.com/test/mobile-friendly but Try This API section) but neither of them worked.
Well I actually solved my own problem, it is mainly caused by urllib I think. So here is what I did;
from __future__ import print_function
import urllib.parse as parser
import urllib.request as urllib2
import json
import base64
request_url = url
params = {
'url': request_url,
'key': api_key
}
data = bytes(parser.urlencode(params), encoding='utf-8')
content = urllib2.urlopen(url=service_url, data=data).read()
sContent = str(content, encoding='utf-8') #Shorthand for stringContent
I have used requests library and I know how to work with it, but I need to work with standard library only, so I would appreciate if you don't encourage me to use requests instead.
I made a flask server that handles POST requests and then from a different script I call urllib to make POST calls to the flask server. I need to send a raw json in body just like we do in Postman.
Flask Server
from flask import Flask, request
app = Flask(__name__)
#app.route('/random', methods=['POST'])
def random():
if request.method == 'POST':
if request.headers.get('Authorization') and request.headers.get('Content-Type') == 'application/json':
print(request.get_json())
return "Success"
else:
print(request.get_json())
return "Bad request"
app.run(host='0.0.0.0', port=5000, debug=True)
Urllib Client (saved as test.py) -
import urllib.request
import urllib.parse
d = {"spam": 1, "eggs": 2, "bacon": 0}
data = urllib.parse.urlencode(d)
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
print(f.read().decode('utf-8'))
With only Authorization header I get Bad Request as output and the json is None on the flask server side as expected.
With ONLY Content-Type header OR both the headers I get this error -
Traceback (most recent call last):
File "test.py", line 9, in <module>
with urllib.request.urlopen(req) as f:
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: BAD REQUEST
The whole thing is simple enough, but I am not able to understand why is this happening and the error message doesn't help much either.
The server is failing in request.get_json(). It's only happening when the client sends both headers because that's when it reaches this line.
To fix it, change the client to send the data as JSON:
import json # <-- Import json
import urllib.request
import urllib.parse
d = {"spam": 1, "eggs": 2, "bacon": 0}
data = json.dumps(d) # <-- Dump the dictionary as JSON
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
print(f.read().decode('utf-8'))
I hope this helps
I'm trying to download the HTML of a page (http://www.guangxindai.com in this case) but I'm getting back an error 403. Here is my code:
import urllib.request
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
f = opener.open("http://www.guangxindai.com")
f.read()
but I get error response.
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
f = opener.open("http://www.guangxindai.com")
File "C:\Python33\lib\urllib\request.py", line 475, in open
response = meth(req, response)
File "C:\Python33\lib\urllib\request.py", line 587, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python33\lib\urllib\request.py", line 513, in error
return self._call_chain(*args)
File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 595, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
I have tried different request headers, but still can not get correct response. I can view the web through browser. It seems strange for me. I guess the web use some method to block web spider. Does anyone know what is happening? How can I get the HTML of page correctly?
I was having the same problem that you and I found the answer in this link.
The answer provided by Stefano Sanfilippo is quite simple and worked for me:
from urllib.request import Request, urlopen
url_request = Request("http://www.guangxindai.com",
headers={"User-Agent": "Mozilla/5.0"})
webpage = urlopen(url_request).read()
If your aim is to read the html of the page you can use the following code. It worked for me on Python 2.7
import urllib
f = urllib.urlopen("http://www.guangxindai.com")
f.read()
I need help with to Login to webpage using id and password and goto a link within the website and download the complete page code or response that we can see in page source as response for the link. I tried using perl or python but no luck.
I need help with to login to www.server.com:7180 and save cookies and then redirect to ww.server.com:7180/healthissues page or directly download whatever we get response in browser in text file.
import urllib
import urllib2
import webbrowser
import cookielib
data1={
'j_username':'id', 'j_password':'pass'
}
data = urllib.urlencode(data1)
url = 'http://server.intranet.com:7180/cmf/allHealthIssues'
full_url = url + '?' + data
response = urllib2.urlopen(full_url)
with open("results.html", "w") as f:
f.write(response.read())
webbrowser.open("results.html")
The above code downloads the webpage but i always end up with authentication page in the download. I found lot of packages but unfortunately i donot have access to install packages or libraries. Any help is appreciated.
I tried with the code suggested by PM 2Ring but I'm getting the error below. I have python 2.6.6 and I'm not sure if that method will work. Please let me know any workaround or way to resolve the error.
Traceback (most recent call last):
File "a.py", line 15, in <module>
handle = urllib2.urlopen(req)
File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.6/urllib2.py", line 435, in error
return self._call_chain(*args)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Invalid request
Although you are importing cookielib you aren't actually using it, so you can't get past the authentication page of the website. The Python docs for cookielib have some simple examples of how to use it. Eg,
import cookielib, urllib2
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
Obviously, your code will need to be a little more complicated, as you need to send the password.
So you'll need to do something like this (untested):
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
data1={'j_username':'id', 'j_password':'pass'}
data = urllib.urlencode(data1)
headers = {'User-agent' : 'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0'}
req = urllib2.Request(url, data, headers)
handle = urllib2.urlopen(req)
It's a shame you can't install Requests, as it makes things so much simpler than using the native Python modules.
I am trying to connect to a REST resource and retrieve the data using Python script (Python 3.2.3). When I run the script I am getting error as HTTP Error 401: Unauthorized. Please note that I am able to access the given REST resource using REST client using Basic Authentication. In the REST Client I have specified the hostname, user and password details (realm is not required).
Below is the code and complete error. Your help is very much appreciated.
Code:
import urllib.request
# set up authentication info
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(realm=None,
uri=r'http://hostname/',
user='administrator',
passwd='administrator')
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
res = opener.open(r'http://hostname:9004/apollo-api/nodes')
nodes = res.read()
Error
Traceback (most recent call last):
File "C:\Python32\scripts\get-nodes.py", line 12, in <module>
res = opener.open(r'http://tolowa.wysdm.lab.emc.com:9004/apollo-api/nodes')
File "C:\Python32\lib\urllib\request.py", line 375, in open
response = meth(req, response)
File "C:\Python32\lib\urllib\request.py", line 487, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python32\lib\urllib\request.py", line 413, in error
return self._call_chain(*args)
File "C:\Python32\lib\urllib\request.py", line 347, in _call_chain
result = func(*args)
File "C:\Python32\lib\urllib\request.py", line 495, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
Try to give the correct realm name. You can find this out for example when opening the page in a browser - the password prompt should display the name.
You can also read the realm by catching the exception that was raised:
import urllib.error
import urllib.request
# set up authentication info
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(realm=None,
uri=r'http://hostname/',
user='administrator',
passwd='administrator')
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
try:
res = opener.open(r'http://hostname:9004/apollo-api/nodes')
nodes = res.read()
except urllib.error.HTTPError as e:
print(e.headers['www-authenticate'])
You should get the following output:
Basic realm="The realm you are after"
Read the realm from above and set it in your add_password method and it should be good to go.