We Keep Coding

using proxy with selenium webdriver firefox in python not working

I would like to make a program that modifies my ip when I go to consult a site with selenium, I use webdriver firefox, unfortunately the site that I use for my tests returns my ip and not the ip that I indicated in the options, could you tell me the error please.
The program launches and the firefox page opens (I don't use headless for the test), but it's my ip that is returned and not the one from the specified proxy.
here is my program
from selenium import webdriver
options = webdriver.FirefoxOptions()
proxy = f'{"137.74.65.101"}:{"80"}'
options.add_argument(f'--proxy-server={proxy}')
driver = webdriver.Firefox(options=options)
driver.get('https://httpbin.org/ip')

contentEncodingError when opening website in Firefox using selenium and python3

I'm trying to perform a fairly simple action using Selenium, namely opening google images in Firefox browser.
I also use a proxy server running on the localhost.
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType
options = FirefoxOptions()
service = Service()
options.add_argument("--headless")
options.accept_insecure_certs = True
proxy = Proxy({
'httpProxy': proxy_addr,
'sslProxy': proxy_addr,
'proxyType': ProxyType.MANUAL
})
options.proxy = proxy
b = Firefox(service=service, options=options)
b.execute("get", {'url': 'http://images.google.com'})
But unfortunately, I'm getting an error like this:
selenium.common.exceptions.WebDriverException: Message: Reached error
page:
about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
I would be very grateful for any thoughts and advice what exactly might be the problem and at least approximately what should be paid attention to.
I'm using:
debian
firefox-esr
selenium == 4.2.0
geckodriver-v0.31.0

This error message...
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=contentEncodingError&u=https%3A//images.google.com/%3Fgws_rd%3Dssl&c=UTF-8&d=The%20page%20you%20are%20trying%20to%20view%20cannot%20be%20shown%20because%20it%20uses%20an%20invalid%20or%20unsupported%20form%20of%20compression.
...implies that there are some configuration settings mismatch while GeckoDriver initiates/spawns a new Browsing Context i.e. firefox session and is often observed as:
Solution
As per the mozilla support docs you need to try out the following steps:
Try to reset the network.http.accept-encoding prefs on the about:config page in case they show as user set (bold). You can open the about:config page via the location/address bar. You can accept the warning and click "I'll be careful" to continue.
If you are having Avast Antivirus or Malwarebytes installed, you may need to disable those in the test machine before executing the tests.

Automate WhatsApp with Python & store WhatsApp login

I am trying to automate WhatsApp. I use selenium but the issue is that I have to scan QRCode every time I launch a chrome instance. I tried by saving the cookies of logged browser and then load the cookies but it didn't work and still show the QRcode to sign in.
I wanted to store WhatsApp login so that I don't have to log in every time I run the script.
Is this something possible with Selenium or if there is another better way?

Add chrome profile , that will allows reusing session data:
chrom_options.add_argument("user-data-dir=C:\\Users\\AppData\\Local\\Google\\Chrome\\User Data")
To get user data drectory:
open : chrome://version in address bar
copy till user data don't need profile part

To store WhatsApp login u have to open chrome in debug mode. following procedure i used in windows and it works.
first add chrome.exe in environment variable. then open chrome in debug mode from cmd. I use following vbscript to open chrome in debug mode. you can just save the code snip as chrome_launch.vbs and use this script to lauch chrome, only first time u would be asked for scan qr code.
Set objStdOut = WScript.StdOut
set osh = createObject("Wscript.shell")
user_profile = """C:\Users\" & osh.ExpandEnvironmentStrings("%USERNAME%") & "\AppData\Local\Google\Chrome\User Data\selenium_chrome_profile"""
osh.run "cmd /c " & "chrome.exe https://web.whatsapp.com/ --remote-debugging-port=9222 --user-data-dir=" & user_profile, 0, True
set osh = Nothing
after login whatsapp web, used following python snip to attach with existing session
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
driver = webdriver.Chrome('chromedriver.exe', options=chrome_options)

InvalidArgumentException: Message: invalid argument: user data directory is already in use error while initiating Chrome with ChromeDriver Selenium

I am trying to open a web page using the selenium python library with my default user, it is critical that the script uses the default user but if my chrome browser is already open the script crashes and gives me this error:
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
I have tried all the solutions given here :
Selenium chromedriver won't launch URL if another chrome instance is open
Selenium won't open a new URL in a new tab (Python & Chrome)
and read that there was a bug in older chromedriver versions but it was fixed in chrome 74 (which im using ) :
https://github.com/SeleniumHQ/docker-selenium/issues/741
from selenium import webdriver
import time
from getpass import getuser
def run():
# Chrome driver path
chromedriver = r'C:\Users\user1\Downloads\chromedriver_win32\chromedriver_new.exe'
# Get chrome webdriver options and set open the browser as headless
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument("--headless")
# Fix for selenium Issue 2907
#chrome_options.add_argument('--log-level=3')
#chrome_options.add_experimental_option('excludeSwitches', ['enable-logging'])
# Load current user default profile
current_user = getuser()
chrome_options.add_argument(
r"--user-data-dir=C:\Users\{}\AppData\Local\Google\Chrome\User Data".format(current_user))
# didable "Chrome is being controled by an automated test software"
chrome_options.add_argument('disable-infobars')
# get Chrome to stay open
chrome_options.add_experimental_option("detach", True)
# open browser with options and driver
driver = webdriver.Chrome(options=chrome_options, executable_path=chromedriver)
driver.get(r'https://www.youtube.com/watch?v=dQw4w9WgXcQ')
if __name__ == '__main__':
run()
if i run it without a chrome browser open its fine if not it crashes

This error message...
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session using the specified user data directory as it was already in use.
I was able to reproduce the error in my local windows-10box as follows:
Code Block:
from selenium import webdriver
import getpass
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument(r"--user-data-dir=C:\Users\{}\AppData\Local\Google\Chrome\User Data".format(getpass.getuser()))
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.google.com/")
Complete relevant traceback:
[18516:23156:0204/032227.883:ERROR:cache_util_win.cc(21)] Unable to move the cache: Access is denied. (0x5)
[18516:23156:0204/032227.898:ERROR:cache_util.cc(141)] Unable to move cache folder C:\Users\Soma Bhattacharjee\AppData\Local\Google\Chrome\User Data\ShaderCache\GPUCache to C:\Users\Soma Bhattacharjee\AppData\Local\Google\Chrome\User Data\ShaderCache\old_GPUCache_000
[18516:23156:0204/032227.898:ERROR:disk_cache.cc(178)] Unable to create cache
[18516:23156:0204/032227.898:ERROR:shader_disk_cache.cc(605)] Shader Cache Creation failed: -2
Opening in existing browser session.
Traceback (most recent call last):
.
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: user data directory is already in use, please specify a unique value for --user-data-dir argument, or don't use --user-data-dir
Analysis
The error stack trace clearly complains of Access is denied as the program was unable to move the cache folder ..\ShaderCache\GPUCache to ..\ShaderCache\old_GPUCache_000. hence the creation of cache failed and subsequently creation of Shader Cache Creation failed. Though these issues raises the InvalidArgumentException but forcefully able to open a new window within the existing Chrome Browser Session.
Snapshot of existing Chrome Browser Session:
Snapshot of new window within the existing Chrome Browser Session:
Conclusion
Though the error is thrown still the new Chrome window gets initiated but remains attached with the already opened Chrome session but the new window can't be controlled by the WebDriver instance.
Solution
You need to take care of a couple of things:
If you are using the Default Chrome Profile to access webpages for your other work on the same Test Machine, you shouldn't set user-data-dir as the User Data as it remains locked by the other Chrome process you have initiated manually.
In the above scenario you need to create and use another Chrome Profile and you can find a detailed discussion in How to open a Chrome Profile through Python
If you are executing your tests in a isolated test system, you can set user-data-dir as ..\User Data\Default to access the Default Chrome Profile.
In the above scenario you need to create and use another Chrome Profile and you can find a detailed discussion in How to use Chrome Profile in Selenium Webdriver Python 3
However as per best practices you must always create a new Chrome Profile to execute your tests as the Default Chrome Profile may contain Extensions, Bookmarks, Browsing History, etc, and may not load properly.
You can find a detailed discussion in How to open a Chrome Profile through --user-data-dir argument of Selenium

I also wanted to run Selenium using my default Chrome profile but I came across the same issue. I solved it by copying my UserData folder to another location then I used the new location. Here is my complete code:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\myusername\\Desktop\\User Data")
options.add_argument("--profile-directory=Profile 1");
browser = webdriver.Chrome(options=options)
browser.get('https://www.google.com')
Remove the following line from the code if you would like to use your default Chrome profile not a specific profile you created for Selenium.
options.add_argument("--profile-directory=Profile 1");

Setting default setting to 'no proxy' in Selenium Firefox

Whenever I open Firefox using Selenium in python using the command
browser = webdriver.Firefox()
The default proxy configuration is set to "use system proxy setting". I have not configured any proxy in the system. Still whenever the browser opens, it says "The proxy server is refusing connection".
How do I open the browser so that the default proxy setting is set to "no proxy" ?
Please help. Thanks in advance.

I will post from my memory
import os
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference('network.proxy.Kind','Direct')
webdriver.Firefox(profile)
To use the default profile you have to specify it
profile = webdriver.FirefoxProfile(path_to_profile_in_your_pc)
webdriver.Firefox(profile)