We Keep Coding

How to address WebDriverException: Message: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED

So the problems have been asked a couple of times here, I've tried multiple solutions none of which worked. So, I am trying to automate the process of signing in to google. But google always throws in "Couldn’t sign you in" after you enter your email, also it presented me with "Chrome is being controlled by automated test software". However, I tried using proxies and such methods to solve the problem, and I've tried different solutions but none is working as I am faced with that error ( in the title), I am guessing that I have setup my proxy incorrectly.Here's my code:
This code is just supposed to connect to a proxy and avoid google automation detection only, and start at https://google.com
try:
import sys
import os
from fp.fp import FreeProxy
from fake_useragent import UserAgent
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time
except Exception as e:
print("Error ->>>: {} ".format(e))
class Spoofer(object):
def __init__(self, country_id=['US'], rand=True, anonym=True):
self.country_id = country_id
self.rand = rand
self.anonym = anonym
self.userAgent, self.ip = self.get()
def get(self):
ua = UserAgent()
proxy = FreeProxy(country_id=self.country_id, rand=self.rand, anonym=self.anonym).get()
ip = proxy.split("://")[1]
print(ip)
return ua.random, ip
class DriverOptions(object):
def __init__(self):
self.options = Options()
self.options.add_argument('--no-sandbox')
self.options.add_argument('--start-maximized')
# self.options.add_argument('--start-fullscreen')
self.options.add_argument('--single-process')
self.options.add_argument('--disable-dev-shm-usage')
self.options.add_argument("--incognito")
self.options.add_argument('--disable-blink-features=AutomationControlled')
self.options.add_argument('--disable-blink-features=AutomationControlled')
self.options.add_experimental_option('useAutomationExtension', False)
self.options.add_experimental_option("excludeSwitches", ["enable-automation"])
self.options.add_argument("disable-infobars")
self.helperSpoofer = Spoofer()
self.options.add_argument('user-agent={}'.format(self.helperSpoofer.userAgent))
self.options.add_argument('--proxy-server=%s' % self.helperSpoofer.ip)
class WebDriver(DriverOptions):
def __init__(self, path=''):
DriverOptions.__init__(self)
self.driver_instance = self.get_driver()
def get_driver(self):
print("""
IP:{}
UserAgent: {}
""".format(self.helperSpoofer.ip, self.helperSpoofer.userAgent))
PROXY = self.helperSpoofer.ip
webdriver.DesiredCapabilities.CHROME['proxy'] = {
"httpProxy":PROXY,
"ftpProxy":PROXY,
"sslProxy":PROXY,
"noProxy":None,
"proxyType":"MANUAL",
"autodetect":False
}
webdriver.DesiredCapabilities.CHROME['acceptSslCerts'] = True
# path = os.path.join(os.getcwd(), )
driver = webdriver.Chrome(executable_path=r'./chromedriver.exe', options=self.options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source":
"const newProto = navigator.__proto__;"
"delete newProto.webdriver;"
"navigator.__proto__ = newProto;"
})
return driver
def main():
driver= WebDriver(path='./chromedriver.exe')
driverinstance = driver.driver_instance
driverinstance.get("https://google.com")
time.sleep(5)
print("done")
if __name__ == "__main__":
main()
I tried different methods, listed in other stackoverflow answers with the same error, but none are working.

This error message...
...implies proxy exception occured and Chrome can't go online due to incorrect proxy settings.
It typically occurs in google-chrome when the web browser refuses to establish an Internet connection through the proxy due to incorrect proxy settings.
Details
When connecting to the Internet over a virtual private network (VPN) or proxy server the header information from user requests is removed/altered to access the websites which helps to conceal your identity online. In majority of the cases, this will not result in any issues and you will be able to maintain a connection to websites without any complications. However there are instances in which there might be a disparity between the data that was expected and the data that was supplied which leads to the ERR_TUNNEL_CONNECTION_FAILED using the Chrome browser.
Solution
A few areas to check can be:
Verify that the proxy settings are proper.
Update Chrome/ChromeDriver to the latest version.
Clear browsing data and cache.
Delete the Cookies and other site data and Cached images and files.
Remove Conflicting Browser Extensions.
Reset Google Chrome Browser.
Reset Internet settings from cmd (Windows):
ipconfig /flushdns
ipconfig /registerdns
ipconfig /release
ipconfig /renew
NETSH winsock reset catalog
NETSH int ipv4 reset reset.log
NETSH int ipv6 reset reset.log
Alternative
As an alternative you can opt to use firefox or opera browser.

contentEncodingError when opening website in Firefox using selenium and python3

I'm trying to perform a fairly simple action using Selenium, namely opening google images in Firefox browser.
I also use a proxy server running on the localhost.
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType
options = FirefoxOptions()
service = Service()
options.add_argument("--headless")
options.accept_insecure_certs = True
proxy = Proxy({
'httpProxy': proxy_addr,
'sslProxy': proxy_addr,
'proxyType': ProxyType.MANUAL
})
options.proxy = proxy
b = Firefox(service=service, options=options)
b.execute("get", {'url': 'http://images.google.com'})
But unfortunately, I'm getting an error like this:
selenium.common.exceptions.WebDriverException: Message: Reached error
page:
about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
I would be very grateful for any thoughts and advice what exactly might be the problem and at least approximately what should be paid attention to.
I'm using:
debian
firefox-esr
selenium == 4.2.0
geckodriver-v0.31.0

This error message...
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
...implies that there are some configuration settings mismatch while GeckoDriver initiates/spawns a new Browsing Context i.e. firefox session and is often observed as:
Solution
As per the mozilla support docs you need to try out the following steps:
Try to reset the network.http.accept-encoding prefs on the about:config page in case they show as user set (bold). You can open the about:config page via the location/address bar. You can accept the warning and click "I'll be careful" to continue.
If you are having Avast Antivirus or Malwarebytes installed, you may need to disable those in the test machine before executing the tests.

WebDriverError#chrome instead of WebDriverError#firefox while using Firefox browser with selenium

I'm trying to write several tests using selenium, but I'm seeing the following strange behavior.
When I run the tests like this:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
options = FirefoxOptions()
service = Service()
brow = Firefox(service=service, options=options)
brow.execute("get", {'url': 'https://python.org'})
I get the result I expected, the python.org website is opened in Firefox browser.
But if I make a mistake in URL, I'm getting the following error:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
options = FirefoxOptions()
service = Service()
brow = Firefox(service=service, options=options)
brow.execute("get", {'url': 'qwerty'})
selenium.common.exceptions.InvalidArgumentException: Message: Malformed URL: URL constructor: qwerty is not a valid URL.
Stacktrace:
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:186:5
InvalidArgumentError#chrome://remote/content/shared/webdriver/Errors.jsm:315:5
GeckoDriver.prototype.navigateTo#chrome://remote/content/marionette/driver.js:804:11
I just want to understand why I see here WebDriverError#chrome, and not WebDriverError#firefox or something like that.
Is this a bug, or am I doing something wrong?

These error messages...
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:186:5
InvalidArgumentError#chrome://remote/content/shared/webdriver/Errors.jsm:315:5
GeckoDriver.prototype.navigateTo#chrome://remote/content/marionette/driver.js:804:11
containing the phrase #chrome may leave an impression of a strange behavior while using GeckoDriver and firefox combo.
However, as per #AutomatedTester's comment in the GitHub discussion Selenium 3.4.0-GeckoDriver 0.17.0 : GeckoDriver producing logs through Chromium/Chrome modules #787:
These errors are nothing to worry about. Mozilla uses different open source projects to build Firefox for different reasons. It showing Chrome errors means nothing in the big picture.
So you can ignore them safely.

Adding proxy support to Selenium multi session opening script in Python

I recently made a script that allow me to create multiple Browser sessions targeting one URL. I would like to add proxy support to it in order to not get banned when running it. I tried to use the Proxy lib from selenium but it just get ignored.
My Question : How can I add proxy support into this script while using Selenium in python ? (each session will get a random proxy)
Here is my code

You could use the stem library which allows you to use Tor in python. Read the docs here to see how to use it.
The two basic parts missing from your code are the following:
from selenium.webdriver.common.proxy import Proxy, ProxyType
chrome_options.add_argument('--proxy-server=#yourproxyhere#'
Tor!
Here you can see how I set up my stem + selenium project:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.proxy import Proxy, ProxyType
from time import sleep
from stem import Signal
from stem.control import Controller
#this gives you a new identity
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
#set the proxy in selenium to 127.0.0.1:9150 and have your Tor Browser open!
link = 'https://some-link.com' #target url
prox= 'socks5://127.0.0.1:9150' #Here you connect to your localhost which connects to a Tor network
#some chrome_options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % prox)
chrome_options.add_argument("--window-size=400,600")
#the following also deactivates location tracking!
prefs = {"profile.default_content_setting_values.geolocation" :2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome("path_to_chromedriver", chrome_options=chrome_options)
driver.get(link)

Here is an updated version of my code including the new proxy support elements, I want it to use proxies form a txt file via a proxies = read_from_txt("proxies.txt") and randomly rotate between them using random lib :
Thank you for quick replies, really appreciate it.

Headless Chrome (with selenium) CANNOT request with PROXY server, but requests can?

I am trying to use Chrome along with python webdriver + selenium, but it seems not working when I set the proxy settings? Here is my code:
from selenium import webdriver
PROXY = 'http://42.115.88.220:53281'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
wbe = webdriver.Chrome(options=chromeOptions)
wbe.get("http://icanhazip.com")
When I run the above codes, the browser gives me: "This site can’t be reached" error:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
Some Efforts: I tried requests with my proxy server, and it works. So it shouldn't be the problem of my proxy server.
import requests
proxies = {"http": "http://42.115.88.220:53281"}
r = requests.get("http://icanhazip.com", proxies = proxies)
print (r.status_code)
This gives me a response code of 200 and good response.
Goal: My final goal is to build a web-crawler with headless chrome with PROXY, so now I am testing a non-headless one first. But it seems there have been something wrong with this PROXY issue.
I would be really appreciated if anyone could help me out with this problem!!!

Try this.
For me it seems that you have used wrong type of headless mode. For chrome selenium browsers it's important to set --headless argument correct.
from selenium import webdriver
PROXY = 'http://ip:port'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
# Headless mode for chrome browser
chromeOptions.add_argument('--headless=chrome')
wbe = webdriver.Chrome('your_driver_path_or_service', options=chromeOptions)
wbe.get("http://icanhazip.com")
print(wbe.title)
print(wbe.current_url)
print(wbe.page_source)
# Output:
# http://icanhazip.com/
# <html><head><meta name="color-scheme" content="light dark"></head><body><pre
# style="word-wrap: break-word; white-space: pre-wrap;">your ip
# </pre></body></html>