We Keep Coding

Python urllib.request.urlopen() returning error 403

I'm trying to download the HTML of a page (http://www.guangxindai.com in this case) but I'm getting back an error 403. Here is my code:
import urllib.request
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
f = opener.open("http://www.guangxindai.com")
f.read()
but I get error response.
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
f = opener.open("http://www.guangxindai.com")
File "C:\Python33\lib\urllib\request.py", line 475, in open
response = meth(req, response)
File "C:\Python33\lib\urllib\request.py", line 587, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python33\lib\urllib\request.py", line 513, in error
return self._call_chain(*args)
File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 595, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
I have tried different request headers, but still can not get correct response. I can view the web through browser. It seems strange for me. I guess the web use some method to block web spider. Does anyone know what is happening? How can I get the HTML of page correctly?

I was having the same problem that you and I found the answer in this link.
The answer provided by Stefano Sanfilippo is quite simple and worked for me:
from urllib.request import Request, urlopen
url_request = Request("http://www.guangxindai.com",
headers={"User-Agent": "Mozilla/5.0"})
webpage = urlopen(url_request).read()

If your aim is to read the html of the page you can use the following code. It worked for me on Python 2.7
import urllib
f = urllib.urlopen("http://www.guangxindai.com")
f.read()

Twitter HTTP Requests: 403 error

I want to send some HTTP requests to Twitter in Python in order to create a sign in for Twitter users for my app. I am using urllib, and following this link: https://dev.twitter.com/web/sign-in/implementing.
But I am unable to do this. I guess I need to authenticate before requesting a token but I don't know how to do that.
Code:
import urllib.request
req = urllib.request.Request("https://api.twitter.com/oauth/authenticate",
headers={'User-Agent': 'Mozilla/5.0'})
html = urllib.request.urlopen(req).read() //after this statement im
getting the error
Error:
Traceback (most recent call last):
File "<pyshell#5>", line 1, in <module>
html = urllib.request.urlopen(req).read()
File "C:\Python34\lib\urllib\request.py", line 161, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 469, in open
response = meth(req, response)
File "C:\Python34\lib\urllib\request.py", line 579, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python34\lib\urllib\request.py", line 507, in error
return self._call_chain(*args)
File "C:\Python34\lib\urllib\request.py", line 441, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 587, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

If you go to the url with a browser it shows you that you need a key:
Whoa there!
There is no request token for this page. That's the special key we need from applications asking to use your Twitter account. Please go back to the site or application that sent you here and try again; it was probably just a mistake.
If you go to this link it will let you choose one of your apps and
it will bring you to a signature-generator that will show you the request settings.
To get a request_token you can use requests_oauthlib:
import requests
from requests_oauthlib import OAuth1
REQUEST_TOKEN_URL = "https://api.twitter.com/oauth/request_token"
CONSUMER_KEY = "xxxxxxxx
CONSUMER_SECRET = "xxxxxxxxxxxxxxxxx"
oauth = OAuth1(CONSUMER_KEY, client_secret=CONSUMER_SECRET)
r = requests.post(url=REQUEST_TOKEN_URL, auth=oauth)
print(r.content)
oauth_token=xxxxxxxxxxxxxx&oauth_token_secret=xxxxxxxxxxx&oauth_callback_confirmed=true
You then need to extract the oauth_token oauth_token_secret:
from urlparse import parse_qs
import webbrowser
data = parse_qs(r.content)
oauth_token = data['oauth_token'][0]
oauth_token_secret = data['oauth_token_secret'][0]
AUTH = "https://api.twitter.com/oauth/authorize?oauth_token={}"
auth = AUTH.format(oauth_token)
webbrowser.open(auth)
A webpage will open asking to Authorize your_app to use your account?
For python 3 use:
from urllib.parse import parse_qs
data = parse_qs(r.text)
oauth_token = data['oauth_token'][0]
oauth_token_secret = data['oauth_token_secret'][0]

Tor .exit with polipo from urllib2

I am trying to use Tor with urllib2 and polipo. What I need is a way to switch to specific exit nodes while program is running.
I have set 'AllowDotExit 1' in /etc/tor/torrc and was trying the following approach:
import urllib2
proxy = '127.0.0.1'
port = '8118'
url='http://ifconfig.me.651d7ace80e0b53e6c05eb4db2491264f049df66.exit'
proxyurl = '%s:%s' % (proxy, port)
proxyhandler = urllib2.ProxyHandler({'http': proxyurl})
opener = urllib2.build_opener(proxyhandler)
page = opener.open(url)
print 'Page opened.'
print page.read()
But what I am getting is:
:!/usr/bin/env python tortest.py
Traceback (most recent call last):
File "tortest.py", line 18, in <module>
page = opener.open(url, timeout=20)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 504: Connect to ifconfig.me.651d7ace80e0b53e6c05eb4db2491264f049df66.exit:80 failed: General SOCKS server failure
Could anyone help me with that?

General SOCKS server failure could be anything. For example, your TOR node might not know how to reach the specified exit node. This happens a lot. The exit may be listed on one of the status pages, but still be unreachable from your tor node. Try a different exit, or retry it later. Often when you request a particular exit node, it takes some minutes to establish a link through the network.

Parse.com user login - 404 error

I am fairly inexperienced with user authentication especially through restful apis. I am trying to use python to log in with a user that is set up in parse.com. The following is the code I have:
API_LOGIN_ROOT = 'https://api.parse.com/1/login'
params = {'username':username,'password':password}
encodedParams = urllib.urlencode(params)
url = API_LOGIN_ROOT + "?" + encodedParams
request = urllib2.Request(url)
request.add_header('Content-type', 'application/x-www-form-urlencoded')
# we could use urllib2's authentication system, but it seems like overkill for this
auth_header = "Basic %s" % base64.b64encode('%s:%s' % (APPLICATION_ID, MASTER_KEY))
request.add_header('Authorization', auth_header)
request.add_header('X-Parse-Application-Id', APPLICATION_ID)
request.add_header('X-Parse-REST-API-Key', MASTER_KEY)
request.get_method = lambda: http_verb
# TODO: add error handling for server response
response = urllib2.urlopen(request)
#response_body = response.read()
#response_dict = json.loads(response_body)
This is a modification of an open source library used to access the parse rest interface.
I get the following error:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 464, in post
url = user.login()
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 313, in login
url = self._executeCall(self.username, self.password, 'GET', data)
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 292, in _executeCall
response = urllib2.urlopen(request)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
Can someone point me to where I am screwing up? I'm not quite sure why I'm getting a 404 instead of an access denied or some other issue.

Make sure the "User" class was created on Parse.com as a special user class. When you are adding the class, make sure to change the Class Type to "User" instead of "Custom". A little user head icon will show up next to the class name on the left hand side.
This stumped me for a long time until Matt from the Parse team showed me the problem.

Please change: API_LOGIN_ROOT = 'https://api.parse.com/1/login' to the following: API_LOGIN_ROOT = 'https://api.parse.com/1/login**/**'
I had the same problem using PHP, adding the / at the end fixed the 404 error.

Problem with Facebook OAuth on Google Appengine

I am using the "Server Side" flow to get a user's permissions to access some information using Python on Google Appengine.
I am able to get the server generated code from Facebook after the user clicks on the "Allow" button.
However when I get the access token, I run into the following error:
Traceback (most recent call last):
File
"/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/init.py",
line 515, in call
handler.get(*groups) File "/base/data/home/apps/finisherph/1.348502373491720746/controllers.py",
line 21, in get
data = urllib2.urlopen(access_token_url)
File
"/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 124, in urlopen
return _opener.open(url, data) File
"/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 387, in open
response = meth(req, response) File
"/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 498, in http_response
'http', request, response, code, msg, hdrs) File
"/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 425, in error
return self._call_chain(*args) File
"/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 360, in _call_chain
result = func(*args) File "/base/python_runtime/python_dist/lib/python2.5/urllib2.py",
line 506, in http_error_default
raise HTTPError(req.get_full_url(), code,
msg, hdrs, fp) HTTPError: HTTP Error
400: Bad Request
Here's the code in my controller where the response from facebook goes after user clicks on the "Allow" button. It's still a hack so the code is a little bit dirty. Still trying to make it work.
class Register(webapp.RequestHandler):
def get(self):
code=self.request.get('code')
logging.debug("code: "+code)
accesst_url=["https://graph.facebook.com/oauth/access_token?"]
accesst_url.append("client_id=CLIENT_ID&")
import urllib
accesst_url.append(urllib.urlencode
({'redirect_uri':'http://my.website.com/register/facebook/'}))
accesst_url.append('&')
accesst_url.append("client_secret=CLIENT_SECRET&")
accesst_url.append("".join(["code=",str(code)]))
logging.debug(accesst_url)
access_token_url="".join(accesst_url)
logging.debug(access_token_url)
import urllib2
data = urllib2.urlopen(access_token_url)
...
...
The error occurs here:
data = urllib2.urlopen(access_token_url)
when I copy and paste the access_token_url from my logs, I get the following error:
{ "error": {
"type": "OAuthException",
"message": "Error validating verification code." } }
What am I missing here?

It looks like you are trying to access the access_token as url, which is not quite right.
Here is an example which illustrates how OAuth authentication via FB is done over GAE.
You go to the https://graph.facebook.com/oauth/authorize? with your client_id and redirect_uri
Upon authorization, it gives a code and you use code and client_secret to get an access_token from https://graph.facebook.com/oauth/access_token
And then you use that access_token to operate as the Facebook user.