We Keep Coding

Google login on Pythonanywhere

I'm using this code to connect remotely to my google account using requests. On local it works perfectly but when I'm trying to use on Pythonanywhere (same version of python 3.6 + I do have a paid account, a hacker plan) it doesn't work, it doesn't connect to my google account at all, without any error message in the console, do you have any idea what could be the problem ?
from bs4 import BeautifulSoup
import requests
class SessionGoogle:
def __init__(self, url_login, url_auth, login, pwd):
self.ses = requests.session()
login_html = self.ses.get(url_login)
soup_login = BeautifulSoup(login_html.content).find('form').find_all('input')
my_dict = {}
for u in soup_login:
if u.has_attr('value'):
my_dict[u['name']] = u['value']
# override the inputs without login and pwd:
my_dict['Email'] = login
my_dict['Passwd'] = pwd
self.ses.post(url_auth, data=my_dict)
def get(self, URL):
return self.ses.get(URL).text
url_login = "https://accounts.google.com/ServiceLogin"
url_auth = "https://accounts.google.com/ServiceLoginAuth"
session = SessionGoogle(url_login, url_auth, "myGoogleLogin", "myPassword")
print session.get("http://plus.google.com")

You're never actually looking at the response of the login post request, so it's entirely likely that Google is rejecting your login for some reason and you have no way of knowing.

HTTP Error 401: Unauthorized using urllib.request.urlopen

I'm using urllib.request in python to try and download some build information from Teamcity. This request used to work without username and password, however a recent security change means I must use a username and password. So I have changed tried each of the two solutions below:
Attempt 1)
url = 'http://<domain>/httpAuth/app/rest/buildTypes/<buildlabel>/builds/running:false?count=1&start=0'
# create a password manager
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
# Add the username and password.
top_level_url = "http://<domain>/httpAuth/app/rest/buildTypes/id:<buildlabel>/builds/running:false?count=1&start=0"
password_mgr.add_password(None, top_level_url, username, password)
handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener(handler)
# use the opener to fetch a URL
opener.open(url)
Attempt 2
url = 'http://<username>:<password>#<domain>/httpAuth/app/rest/buildTypes/id:buildlabel/builds/running:false?count=1&start=0'
rest_api = urllib.request.urlopen(url)
Both of these return "HTTP Error 401: Unauthorized". However if I was to print 'url' and copy this output into a browser the link works perfectly. But when used through python I get the above error.
I use something very similar in another Perl script and this works perfectly also.
* SOLVED BELOW *

Solved this using.
credentials(url, username, password)
rest_api = urllib2.urlopen(url)
latest_build_info = rest_api.read()
latest_build_info = latest_build_info.decode("UTF-8")
# Then parse this xml for the information I want.
def credentials(self, url, username, password):
p = urllib2.HTTPPasswordMgrWithDefaultRealm()
p.add_password(None, url, username, password)
handler = urllib2.HTTPBasicAuthHandler(p)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
As a side note, I then want to download a file..
credentials(url, username, password)
urllib2.urlretrieve(url, downloaded_file)
Where Url is:
http://<teamcityServer>/repository/download/<build Label>/<BuildID>:id/Filename.zip

Trying To Download Subversion Code with Python's urllib2 module = Fail

I am trying to download code from a Subversion repository without using the svn Subversion program. I am using Python's urllib2 module instead. Below is my script. The problem with this script is that it returns a web page of sorts and not the actual source-code. I can see the links to the source-code but not the actual source-code.
Does anyone have any suggestions on how I can download the actual source-code from Subversion with urllib2?
#! /usr/bin/env python
import urllib2
def sub():
theurl = 'https://Intranet-Server/svn/FancySoftware/trunk/'
username = 'username'
password = 'password'
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, theurl, username, password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
print "OPENER :", opener
urllib2.install_opener(opener)
pagehandle = urllib2.urlopen(theurl)
print "PAGEHANDLE :", pagehandle
return pagehandle
if __name__ == "__main__":
ret = sub()
for line in ret:
print line

Logging into quora using python

I tried logging into quora using python. But it gives me the following error.
urllib2.HTTPError: HTTP Error 500: Internal Server Error
This is my code till now. I also work behind a proxy.
import urllib2
import urllib
import re
import cookielib
class Quora:
def __init__(self):
'''Initialising and authentication'''
auth = 'http://name:password#proxy:port'
cj = cookielib.CookieJar()
logindata = urllib.urlencode({'email' : 'email' , 'password' : 'password'})
handler = urllib2.ProxyHandler({'http' : auth})
opener = urllib2.build_opener(handler , urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
a = urllib2.urlopen('http://www.quora.com/' , logindata)
def main():
Quora()
Can someone please point out what is wrong?
if __name__ == '__main__':
main()

Try something like this:
# Load proxies
proxies = []
proxies_fp = open('proxies.txt', 'r') # A list of proxies
for proxy in proxies_fp:
proxies.append(proxy)
cookiejar = cookielib.CookieJar()
def perform_request(url, opener, credientials):
# Instantiate our request object
request = urllib2.Request(url)
# Perform the request, returning a pointer to the result set.
result = opener.urlopen(request, credentials)
return result
credentials ={
'username' : 'username',
'password' : 'password'
}
encoded_credentials = urllib.urlencode(credentials)
def main():
# Get random proxy
proxy = random.choice(proxies)
# Install our proxy
opener = urllib2.build_opener(
urllib2.ProxyHandler({'http': proxy}),
urllib2.HTTPRedirectHandler(),
urllib2.HTTPHandler(debuglevel=0),
urllib2.HTTPSHandler(debuglevel=0),
urllib2.HTTPCookieProcessor(cookiejar),
)
urllib2.install_opener(opener)
a = perform_request(url, opener, encoded_credentials)
--untested--
I've had to do something similar to this, and it worked for me this way. (Please note, that this is NOT an exact copy of code I used. I had to manipulate it a bit, and did NOT test)

How to specify an authenticated proxy for a python http connection?

What's the best way to specify a proxy with username and password for an http connection in python?

This works for me:
import urllib2
proxy = urllib2.ProxyHandler({'http': 'http://
username:password#proxyurl:proxyport'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)
conn = urllib2.urlopen('http://python.org')
return_str = conn.read()

Use this:
import requests
proxies = {"http":"http://username:password#proxy_ip:proxy_port"}
r = requests.get("http://www.example.com/", proxies=proxies)
print(r.content)
I think it's much simpler than using urllib. I don't understand why people love using urllib so much.

Setting an environment var named http_proxy like this: http://username:password#proxy_url:port

The best way of going through a proxy that requires authentication is using urllib2 to build a custom url opener, then using that to make all the requests you want to go through the proxy. Note in particular, you probably don't want to embed the proxy password in the url or the python source code (unless it's just a quick hack).
import urllib2
def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"):
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, proxyurl, proxyuser, proxypass)
proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl})
proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr)
return urllib2.build_opener(proxy_handler, proxy_auth_handler)
if __name__ == "__main__":
import sys
if len(sys.argv) > 4:
url_opener = get_proxy_opener(*sys.argv[1:4])
for url in sys.argv[4:]:
print url_opener.open(url).headers
else:
print "Usage:", sys.argv[0], "proxy user pass fetchurls..."
In a more complex program, you can seperate these components out as appropriate (for instance, only using one password manager for the lifetime of the application). The python documentation has more examples on how to do complex things with urllib2 that you might also find useful.

Or if you want to install it, so that it is always used with urllib2.urlopen (so you don't need to keep a reference to the opener around):
import urllib2
url = 'www.proxyurl.com'
username = 'user'
password = 'pass'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
# None, with the "WithDefaultRealm" password manager means
# that the user/pass will be used for any realm (where
# there isn't a more specific match).
password_mgr.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
print urllib2.urlopen("http://www.example.com/folder/page.html").read()

Here is the method use urllib
import urllib.request
# set up authentication info
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})
# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
urllib.request.CacheFTPHandler)
# install it
urllib.request.install_opener(opener)
f = urllib.request.urlopen('http://www.python.org/')
"""