I'm trying to perform a fairly simple action using Selenium, namely opening google images in Firefox browser.
I also use a proxy server running on the localhost.
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType
options = FirefoxOptions()
service = Service()
options.add_argument("--headless")
options.accept_insecure_certs = True
proxy = Proxy({
'httpProxy': proxy_addr,
'sslProxy': proxy_addr,
'proxyType': ProxyType.MANUAL
})
options.proxy = proxy
b = Firefox(service=service, options=options)
b.execute("get", {'url': 'http://images.google.com'})
But unfortunately, I'm getting an error like this:
selenium.common.exceptions.WebDriverException: Message: Reached error
page:
about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
I would be very grateful for any thoughts and advice what exactly might be the problem and at least approximately what should be paid attention to.
I'm using:
debian
firefox-esr
selenium == 4.2.0
geckodriver-v0.31.0
This error message...
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
...implies that there are some configuration settings mismatch while GeckoDriver initiates/spawns a new Browsing Context i.e. firefox session and is often observed as:
Solution
As per the mozilla support docs you need to try out the following steps:
Try to reset the network.http.accept-encoding prefs on the about:config page in case they show as user set (bold). You can open the about:config page via the location/address bar. You can accept the warning and click "I'll be careful" to continue.
If you are having Avast Antivirus or Malwarebytes installed, you may need to disable those in the test machine before executing the tests.
Related
I'm trying to write several tests using selenium, but I'm seeing the following strange behavior.
When I run the tests like this:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
options = FirefoxOptions()
service = Service()
brow = Firefox(service=service, options=options)
brow.execute("get", {'url': 'https://python.org'})
I get the result I expected, the python.org website is opened in Firefox browser.
But if I make a mistake in URL, I'm getting the following error:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
options = FirefoxOptions()
service = Service()
brow = Firefox(service=service, options=options)
brow.execute("get", {'url': 'qwerty'})
selenium.common.exceptions.InvalidArgumentException: Message: Malformed URL: URL constructor: qwerty is not a valid URL.
Stacktrace:
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:186:5
InvalidArgumentError#chrome://remote/content/shared/webdriver/Errors.jsm:315:5
GeckoDriver.prototype.navigateTo#chrome://remote/content/marionette/driver.js:804:11
I just want to understand why I see here WebDriverError#chrome, and not WebDriverError#firefox or something like that.
Is this a bug, or am I doing something wrong?
These error messages...
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:186:5
InvalidArgumentError#chrome://remote/content/shared/webdriver/Errors.jsm:315:5
GeckoDriver.prototype.navigateTo#chrome://remote/content/marionette/driver.js:804:11
containing the phrase #chrome may leave an impression of a strange behavior while using GeckoDriver and firefox combo.
However, as per #AutomatedTester's comment in the GitHub discussion Selenium 3.4.0-GeckoDriver 0.17.0 : GeckoDriver producing logs through Chromium/Chrome modules #787:
These errors are nothing to worry about. Mozilla uses different open source projects to build Firefox for different reasons. It showing Chrome errors means nothing in the big picture.
So you can ignore them safely.
I'm learning Python and faced the following problem: While studying the Selenium library, an error occurs:
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_TUNNEL_CONNECTION_FAILED
(Session info: chrome=97.0.4692.99)
Here is the code itself:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time
from selenium.webdriver.common.proxy import Proxy, ProxyType
proxy_ip_port = '91.132.151.232:80'
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = proxy_ip_port
proxy.ssl_proxy = proxy_ip_port
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
s = Service("C:\\Users\\Anton\\PycharmProjects\\pythonProject\\access\\chromedriver.exe")
driver = webdriver.Chrome(service=s, desired_capabilities=capabilities)
driver.get('https://2ip.ru/')
time.sleep(10)
driver.quit()
This error is most likely not generated by Selenium, but by the browser you are using. There can be a number of issues that can throw the aforementioned error. By a quick search on the web I found out the possible causes behind your issue, namely
Incorrect website domain configuration.
Conflicting browser data.
DNS Connection Issues.
Proxy settings incorrectly entered.
These issues are common to chrome, but might be similar for other browsers as well. Go through this list to find your issue.
I also encountered this issue and for me it was a non-working proxy.
I am trying to use Chrome along with python webdriver + selenium, but it seems not working when I set the proxy settings? Here is my code:
from selenium import webdriver
PROXY = 'http://42.115.88.220:53281'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
wbe = webdriver.Chrome(options=chromeOptions)
wbe.get("http://icanhazip.com")
When I run the above codes, the browser gives me: "This site can’t be reached" error:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
Some Efforts: I tried requests with my proxy server, and it works. So it shouldn't be the problem of my proxy server.
import requests
proxies = {"http": "http://42.115.88.220:53281"}
r = requests.get("http://icanhazip.com", proxies = proxies)
print (r.status_code)
This gives me a response code of 200 and good response.
Goal: My final goal is to build a web-crawler with headless chrome with PROXY, so now I am testing a non-headless one first. But it seems there have been something wrong with this PROXY issue.
I would be really appreciated if anyone could help me out with this problem!!!
Try this.
For me it seems that you have used wrong type of headless mode. For chrome selenium browsers it's important to set --headless argument correct.
from selenium import webdriver
PROXY = 'http://ip:port'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
# Headless mode for chrome browser
chromeOptions.add_argument('--headless=chrome')
wbe = webdriver.Chrome('your_driver_path_or_service', options=chromeOptions)
wbe.get("http://icanhazip.com")
print(wbe.title)
print(wbe.current_url)
print(wbe.page_source)
# Output:
# http://icanhazip.com/
# <html><head><meta name="color-scheme" content="light dark"></head><body><pre
# style="word-wrap: break-word; white-space: pre-wrap;">your ip
# </pre></body></html>
I'm trying to capture all network logs using seleniumwire. When chromedriver is in normal mode, it is able to capture all requests. But when it is in headless mode, it is not capturing all requests.
I tried adding sleep(10), assert driver.last_request.response.status_code == 200
but neither helped.
Since seleniumwire is not that popular, I'm adding a sample guide below in the hope of getting people with knowledge of selenium to try a hand to help me fix the problem.
Working with seleniumwire
Installing seleniumwire
pip install seleniumwire
Sample script:
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver
driver = webdriver.Chrome()
# Go to the YouTube homepage.
driver.get('https://www.youtube.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.path,
request.response.status_code,
request.response.headers['Content-Type']
)
try to capture all requests
options = {
'ignore_http_methods': [] # Capture all requests, including OPTIONS requests
}
driver = webdriver.Chrome("C:\chromedriver.exe",seleniumwire_options=options)
In default it ignores OPTIONS method
When chrome browser is opened by selenium, it uses it's own profile rather than the default one present. Try using custom profile, for chrome you can use ChromeOptions class use a custom profile and try.
I used to set up proxy on chrome like in a code below, but when i updated to selenium 3.8.1 proxy stops working, i dont get any errors it just doesn't use proxy server and i dont know why. My chromedriver is also up to date.
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=192.99.55.120:3128')
driver = webdriver.Chrome(executable_path='C:\chromedriver_win32\chromedriver.exe', chrome_options=options)
driver.get("http://google.com/")
Would like to receive any advice, maybe alternative way to set up proxy for chromedriver.
If someone still interested, this is how i have finally solved the problem
from selenium.webdriver import Proxy
settings = {
"httpProxy": "192.99.55.120:3128",
"sslProxy": "192.99.55.120:3128"
}
proxy = Proxy(settings)
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
cap = DesiredCapabilities.CHROME.copy()
cap['platform'] = "WINDOWS"
cap['version'] = "10"
proxy.add_to_capabilities(cap)
from selenium.webdriver.chrome.webdriver import WebDriver as ChromeDriver
driver = ChromeDriver(desired_capabilities=cap, executable_path='C:\chromedriver_win32\chromedriver.exe')
try
options.add_argument('--proxy-server="http=192.99.55.120:3128;https=192.99.55.120:3128"')
also try running your chrome binary directly with these params to see whether it works or not
chrome.exe --proxy-server="http=192.99.55.120:3128"
If the navigator asks for the credentials username and password for the proxy and you need to handle this : (only if the alert come up)
driver.get("http://username:password#google.com/")