I am using the python shell to test requests together with proxy servers.
After reading documentation (http://docs.python-requests.org/en/master/user/advanced/) and a few stackoverflow threads I am doing the following:
import requests
s = requests.session()
proxies = {'http': 'http://90.178.216.202:3128'}
s.proxies.update(proxies)
req = s.get('http://jsonip.com')
After this, if I print req.text, I get this:
u'{"ip":"my current IP (not the proxy server IP I have inserted before)","about":"/about", ......}'
Can you please explain why I'm getting my computer's IP address and not the proxy server's IP address?
Did I go wrong somewhere or am I expecting the wrong thing to happen here?
I am new to requests + proxy servers so I would like to make sure I am understanding this.
UPDATE
I also have this in my code:
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0'}
s.headers.update(headers)
Thanks
Vittorio
The site ( http://jsonip.com ) broadcasts an 'Upgrade-Insecure-Requests' header. This means that your request gets redirected to https://jsonip.com, so requests doesn't use a proxy because you don't have an https proxy in your proxies dict.
So, all you have to do is add an https proxy in proxies , eg:
proxies = {'http':'http://90.178.216.202:3128', 'https':'https://90.178.216.202:3128'}
Instead of doing this pass user-agent
requests.post(url='abc.com',header={'user-agent':'Mozila 5.0'})
u need to change ur get request to have the proxies used.
something like this:req = s.get('http://jsonip.com', proxies=proxies)
Related
I'm recently working on occasional data intensive projects and I'm in need of gathering data from e-commerce platforms like Amazon so I created a web scraping program in Python. I'm using requests library along with a list of user agents and proxies however I think they are not working and it is causing failure of the program. Note that Amazon Api is limiting in terms of content and access rates and is not suitable for my needs.
Here's how I send requests:
import requests
import random
session = requests.session()
proxies = [{'https:': 'https://' + item.rstrip(), 'http':
'http://' + item.rstrip()} for item in open('proxies.txt').readlines()]
user_agent = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) '
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36'}
print(session.get('https://icanhazip.com', proxies=random.choice(proxies), headers=user_agent).text)
However I keep getting the same ip address printed and this means the proxies are not working this way. and the proxies.txt contains proxies in this format:
ex:
178.168.19.139:30736
342.552.34.456:8080
...
What is the best way to workaround captchas and robot checks presented by Amazon using these tools (or extra tools if you have any suggestions) and why are the proxies failing to work?
I'm not sure if this will work for you, but I found that removing the protocol at the start of the ip within the dictionary solved the problem.
proxies = [{'https': item.rstrip(), 'http': item.rstrip()} for item in open('proxies.txt').readlines()]
I am trying to scrape a website using requests in python.
url = "https://stackoverflow.com/questions/23013220/max-retries-exceeded-with-url"
# set the headers like we are a browser,
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}
# download the homepage
s = requests.Session()
s.trust_env = False
response = s.get(url, headers=headers )
This is working fine when I use my personal wifi. However, when I connect to my company's VPN, I get the following error.
ConnectionError: HTTPSConnectionPool(host='stackoverflow.com', port=443): Max retries exceeded with url: /questions/23013220/max-retries-exceeded-with-url (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it',))
Now, I need this to work over my company's VPN because I need to access a website which works only in that. How to resolve this?
In my case, the problem was related to IPv6.
Our VPN used split tunneling, and it seems the VPN configuration does not support IPv6.
So for example this would hang forever:
requests.get('https://pokeapi.co/api/v2/pokemon')
But if you add a timeout, the request succeeds:
requests.get('https://pokeapi.co/api/v2/pokemon', timeout=1)
But not all machines were having this problem. So I compared the output of this among two different machines:
import socket
for line in socket.getaddrinfo('pokeapi.co', 443):
print(line)
The working one only returned IPv4 addresses. The non-working machine returned both IPv4 and IPv6 addresses.
So with the timeout specified, my theory is that python fails quickly with IPv6 and then moves to IPv4, where the request succeeds.
Ultimately we resolved this by disabling IPv6 on the machine:
networksetup -setv6off "Wi-Fi"
But I assume that this could instead be resolved through VPN configuration.
How about trying like this:
url = "https://stackoverflow.com/questions/23013220/max-retries-exceeded-with-url"
ua = UserAgent()
headers = headers = {"User-Agent": ua.random}
# download the homepage
s = requests.Session()
s.trust_env = False
response = s.get(url, headers=headers)
It seems to be caused by UserAgent() settings difference.
Try to set trust_env = None
trust_env = None #
Trust environment settings for proxy configuration, default authentication and similar.
Or you can disable proxies for a particular domain. The question
import os
os.environ['NO_PROXY'] = 'stackoverflow.com'
In my organization, I have to run my program under VPN for different geo locations. so we have multiple proxy configurations.
I found it simpler to use a package called PyPAC to get my proxy details automatically
from pypac import PACSession
from requests.auth import HTTPProxyAuth
session = PACSession()
# when the username and password is required
# session = PACSession(proxy_auth=HTTPProxyAuth(name, password))
r = session.get('http://example.org')
How does this work:
The package locates the PAC file which is configured by the organization. This file consist of proxy configuration detail (more info).
I have a Python script to get a page using Requests. I need to use proxies to access the page. When I access an http page, it goes through the proxy but when I access an https page, it does not go through the proxy (I used logs to check this, as explained below). I have checked with the proxy service provider (proxymesh) and they said that their proxies can be used for https pages as well. Is there anything I need to change in the script when accessing https sites vs http sites?
My code is presented below. At the end of this question, I have included the log files generated for the http and https sites, which show that the proxy is used for http but not for https.
Any ideas will be really helpful.
import logging
import requests
#set up logging
logging.getLogger('').handlers = []
logging.basicConfig(
filename = "mylog_with_proxy.log", #in my code, the full path is specified
filemode="w",
level = logging.DEBUG)
#specify proxies and headers
proxies = {'http': 'http://fr.proxymesh.com:31280', 'https': 'http://fr.proxymesh.com:31280'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393',}
#the two URLs that I accessed. One is for an http site and the other one is for an https site. These sites are just examples of sites I need to access.
http_url = "http://docs.python-requests.org/en/master/user/quickstart/"
https_url = "https://www.haskell.org/happy/"
#get the page. I executed the script twice - once for http_url and the second time for https_url. Here, it shows http_url
r = requests.get(http_url, headers=headers, proxies=proxies, timeout=5)
r.raise_for_status()
The log files are as shown below:
When accessing the http site (that is, when running the script with http_url):
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): fr.proxymesh.com
DEBUG:requests.packages.urllib3.connectionpool:"GET http://docs.python-requests.org/en/master/user/quickstart/ HTTP/1.1" 200 None
When accessing the https site (that is, when running the script with https_url)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.haskell.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /happy/ HTTP/1.1" 200 None
I'm trying to determine high anonymity proxies. Also called private/elite proxies. From a forum I've read this:
High anonymity Servers don't send HTTP_X_FORWARDED_FOR, HTTP_VIA and
HTTP_PROXY_CONNECTION variables. Host doesn't even know you are using
proxy server and of course it doesn't know your IP address.
A highly anonymous proxy will display the following information:
REMOTE_ADDR = Proxy's IP address
HTTP_VIA = blank
HTTP_X_FORWARDED_FOR = blank
So, how I can check for this headers in Python, to discard them as a HA Proxy ? I have tried to retrieve the headers for 20-30 proxies using the requests package, also with urllib, with the build-in http.client, with urllib2. But I didn't see these headers, never. So I should be doing something wrong...
This is the code I've used to test with requests:
proxies = {'http': 'http://176.100.108.214:3128'}
header = {'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.360',}
s = requests.session()
s.proxies = proxies
r = s.get('http://www.python.org', headers=header)
print(r.status_code)
print(r.request.headers)
print(r.headers)
It sounds like the forum post you're referring to is talking about the headers seen by the server on your proxied request, not the headers seen by the client on the proxied response.
Since you're testing with www.python.org as the server, the only way to see the headers it receives would be to have access to their logs. Which you don't.
But there's a simple solution: run your own HTTP server, make requests against that, and then you can see what it receives. (If you're behind a firewall or NAT that the proxy you're testing won't be able to connect to, you may have to get a free hosted server somewhere; if not, you can just run it on your machine.)
If you have no idea how to set up and configure a web server, Python comes with one of its own. Just run this script with Python 3.2+ (on your own machine, or an Amazon EC2 free instance, or whatever):
from http.server import HTTPServer, SimpleHTTPRequestHandler
class HeaderDumper(SimpleHTTPRequestHandler):
def do_GET(self):
try:
return super().do_GET()
finally:
print(self.headers)
server = HTTPServer(("", 8123), HeaderDumper)
server.serve_forever()
Then run that script with python3 in the shell.
Then just run your client script, with http://my.host.ip instead of http://www.python.org, and look at what the script dumps to the server's shell.
I have Fiddler2 listening on 0.0.0.0:8888.
try:
data = ''
proxy = urllib2.ProxyHandler({'http': '127.0.0.1:8888'}) //also tried {'http': 'http://127.0.0.1:8888/'}
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
req = urllib2.Request('http://www.google.com')
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
except Exception, detail:
print "Err ", detail
I don't see the GET or any request to google in Fiddler (but I can see other requests)
is there a way to debug it? is seems like python bypasses Fiddler or ignores the proxy.
I also configured WinHTTP to work with Fiddler -
C:\Windows\system32>netsh winhttp set proxy 127.0.0.1:8888
Current WinHTTP proxy settings:
Proxy Server(s) : 127.0.0.1:8888
Bypass List : (none)
does is matter if the request it to a SSL address? (Fiddler supports https)
Thanks!
Maybe you can work with the opener directly instead of installing it. turn on your fiddler proxy listener on 8008 (i'm using WebScarab, but they're probably the same) then try this code exactly (also has cookies which you don't need, but lets try as-is and narrow it down later):
cj = cookielib.MozillaCookieJar(cookie_filename)
if os.access(cookie_filename, os.F_OK):
cj.load()
proxy_handler = urllib2.ProxyHandler({'https': 'localhost:8008'})
opener = urllib2.build_opener(
proxy_handler,
urllib2.HTTPCookieProcessor(cj)
)
opener.addheaders = [
('User-agent', ('Mozilla/4.0 (compatible; MSIE 6.0; '
'Windows NT 5.2; .NET CLR 1.1.4322)'))
]
auth = urllib.urlencode({'email':email,'pass':passw})
data = opener.open('https://login.facebook.com/login.php',data=auth)
so - things i'm doing differently: direct usage of the opener, change the port to 8008, add cookies and use WebScarab. let me know which one of these did the trick for you...
proxy_bypass_registry in urllib.py does not handle the ProxyOverride registry value properly: it treats an empty override as *, i.e. bypass the proxy for all hosts. This behavior does not match other programs (e.g. Chrome).
There are a number of possible workarounds:
Set urllib.proxy_bypass = lambda h: 0 to disable bypass checking.
Specify the proxy settings in the http_proxy environment variable (proxy_bypass_registry is not called in this case).
In Fiddler2, go to the page Tools->Fiddler Options ...->Connections, remove the trailing semicolon from the value in the "IE should bypass Fiddler for ..." field and restart Fiddler2.
In Fiddler2, go to the page Tools -> Fiddler Options ... -> Connections, remove the trailing semicolon from the value in the IE should bypass Fiddler for ... field and restart Fiddler2.
This solution is definitely works for me when I using urllib2 proxy, however I still don't understand why removing the trailing semicolon can solve it.
btw, u need use http://www.google.com/ instead of http://www.google.com so that fiddler could figure you are requesting 'get /'
otherwise fiddler cannot figure out the uri. (u may get a 504 receive failure).