I am trying to use Chrome along with python webdriver + selenium, but it seems not working when I set the proxy settings? Here is my code:
from selenium import webdriver
PROXY = 'http://42.115.88.220:53281'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
wbe = webdriver.Chrome(options=chromeOptions)
wbe.get("http://icanhazip.com")
When I run the above codes, the browser gives me: "This site can’t be reached" error:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
Some Efforts: I tried requests with my proxy server, and it works. So it shouldn't be the problem of my proxy server.
import requests
proxies = {"http": "http://42.115.88.220:53281"}
r = requests.get("http://icanhazip.com", proxies = proxies)
print (r.status_code)
This gives me a response code of 200 and good response.
Goal: My final goal is to build a web-crawler with headless chrome with PROXY, so now I am testing a non-headless one first. But it seems there have been something wrong with this PROXY issue.
I would be really appreciated if anyone could help me out with this problem!!!
Try this.
For me it seems that you have used wrong type of headless mode. For chrome selenium browsers it's important to set --headless argument correct.
from selenium import webdriver
PROXY = 'http://ip:port'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
# Headless mode for chrome browser
chromeOptions.add_argument('--headless=chrome')
wbe = webdriver.Chrome('your_driver_path_or_service', options=chromeOptions)
wbe.get("http://icanhazip.com")
print(wbe.title)
print(wbe.current_url)
print(wbe.page_source)
# Output:
# http://icanhazip.com/
# <html><head><meta name="color-scheme" content="light dark"></head><body><pre
# style="word-wrap: break-word; white-space: pre-wrap;">your ip
# </pre></body></html>
Related
I would like to make a program that modifies my ip when I go to consult a site with selenium, I use webdriver firefox, unfortunately the site that I use for my tests returns my ip and not the ip that I indicated in the options, could you tell me the error please.
The program launches and the firefox page opens (I don't use headless for the test), but it's my ip that is returned and not the one from the specified proxy.
here is my program
from selenium import webdriver
options = webdriver.FirefoxOptions()
proxy = f'{"137.74.65.101"}:{"80"}'
options.add_argument(f'--proxy-server={proxy}')
driver = webdriver.Firefox(options=options)
driver.get('https://httpbin.org/ip')
I'm trying to perform a fairly simple action using Selenium, namely opening google images in Firefox browser.
I also use a proxy server running on the localhost.
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType
options = FirefoxOptions()
service = Service()
options.add_argument("--headless")
options.accept_insecure_certs = True
proxy = Proxy({
'httpProxy': proxy_addr,
'sslProxy': proxy_addr,
'proxyType': ProxyType.MANUAL
})
options.proxy = proxy
b = Firefox(service=service, options=options)
b.execute("get", {'url': 'http://images.google.com'})
But unfortunately, I'm getting an error like this:
selenium.common.exceptions.WebDriverException: Message: Reached error
page:
about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
I would be very grateful for any thoughts and advice what exactly might be the problem and at least approximately what should be paid attention to.
I'm using:
debian
firefox-esr
selenium == 4.2.0
geckodriver-v0.31.0
This error message...
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
...implies that there are some configuration settings mismatch while GeckoDriver initiates/spawns a new Browsing Context i.e. firefox session and is often observed as:
Solution
As per the mozilla support docs you need to try out the following steps:
Try to reset the network.http.accept-encoding prefs on the about:config page in case they show as user set (bold). You can open the about:config page via the location/address bar. You can accept the warning and click "I'll be careful" to continue.
If you are having Avast Antivirus or Malwarebytes installed, you may need to disable those in the test machine before executing the tests.
I am having a strange issue. I have used the below code for a while to launch a anonymous wedriver instance, however somehow my settings of Chrome have refreshed and now the traffic is not going through the proxy. I tried making changes to the code and the chrome options but I can't figure out what I am doing wrong.
Also, it has somehow stopped adding the uBlock extension.
def anon_drive():
os.startfile(r"C:\Users\xxx\Downloads\tor-win32-0.2.9.9\Tor\tor.exe")
time.sleep(10)
chrome_path = r'C:\Users\xxx\Desktop\chromedriver.exe'
chop = webdriver.ChromeOptions()
chop.add_extension(r'C:\Users\Sid\Desktop\Ublock\uBlock-Origin_v1.11.0.crx')
driver = webdriver.Chrome(chrome_path, chrome_options=chop)
driver = webdriver.Chrome(chrome_path)
return driver
I have the Chrome options set to "Use a proxy server for your LAN is checked" I have tried setting the Advanced options to 127.0.0.1:9050 and 9051.
Still getting
Socks version 67 not recognised.(Tor is not an http proxy.)
I am not sure what I am doing wrong. There are some answers which I found but I am unable to find one which would perfectly.
#JamesKPolk gave me the answer.
Steps:
Chrome>Settings>Advanced Settings>Open Proxy Settings> LAN Settings>Use a Proxy Server for LAN> Advanced> Socks: 127.0.0.1 Port 9050
Press OK and everything should work.
Note: Leave everything except Socks empty.
With python-selenium I am running the following short python snippet:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.http", "localhost")
profile.set_preference("network.proxy.http_port", "9998")
browser = webdriver.Firefox(firefox_profile = profile)
browser.get("http://www.google.co.uk")
in which I define a proxy running at the localhost at port 9998. The test finishes fine, i.e. the google webpage is being shown.
However, there is no proxy running at port 9998. I was expecting an error.
Question: Why do I not get an error?
The code example is incorrect. With the following code example it works just fine, and when specifying a wrong port you will get an error immediately:
from selenium import webdriver
PROXY = "0.0.0.0:9999"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
"httpProxy":PROXY,
"ftpProxy":PROXY,
"sslProxy":PROXY,
"proxyType":"MANUAL"
}
driver = webdriver.Firefox()
driver.get("http://www.google.co.in")
I am using selenium with python.
I need to configure a proxy.
It is working for HTTP but not for HTTPS.
The code I am using is:
# configure firefox
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", '11.111.11.11')
profile.set_preference("network.proxy.http_port", int('80'))
profile.update_preferences()
# launch
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('https://www.iplocation.net/find-ip-address')
Also. Is there a way for me to completely block any outgoing traffic from my IP and restrict it ONLY to the proxy IP so that I don't accidently mess up the test/stats by accidently switching from proxy to direct connection?
Any tips would help!
Thanks :)
Check out browsermob proxy for setting up a proxies for use with selenium
from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob
server.stop()
driver.quit()
You can use a remote proxy server with the RemoteServer class.
Is there a way for me to completely block any outgoing traffic from my IP and restrict it ONLY to the proxy IP
Yes, just look up how to setup proxies for whatever operating system you're using. Just use caution because some operating systems will ignore proxy rules based on certain conditions, for example, if using a VPN connection.