We Keep Coding

Prevent selenium detection on firefox (Python) [duplicate]

I followed this post on Stackoverflow to disable Firefox WebDriver detection.
Launch Geckodriver:
System.setProperty("webdriver.gecko.driver", geckdriverExecutableFilePath);
File firefoxProfileFile = new File(fullPathOfFirefoxInstallationFolder);
FirefoxProfile firefoxProfile = null;
try {
firefoxProfile = new FirefoxProfile(firefoxProfileFile);
} catch (Exception e) {
e.printStackTrace();
}
I disabled WebDriver:
WebDriver Disabled
FirefoxOptions firefoxOptions = new FirefoxOptions();
firefoxOptions.setProfile(firefoxProfile);
// Disables WebRTC
firefoxProfile.setPreference("media.peerconnection.enabled", false);
I disabled Automation Extensions:
Automation Extension Disabled
// Disables Automation Extension
firefoxProfile.setPreference("useAutomationExtension", false);
I added Proxy:
DesiredCapabilities dc = DesiredCapabilities.firefox();
Proxy proxy = new Proxy();
proxy.setHttpProxy(ipAddress + ":" + port);
proxy.setFtpProxy(ipAddress + ":" + port);
proxy.setSslProxy(ipAddress + ":" + port);
dc.setCapability(CapabilityType.PROXY, proxy);
firefoxOptions.merge(dc);
driver = new FirefoxDriver(firefoxOptions);
Yet BotD still detects my browser as being controlled by automation tool.
BotD Detection
How can I solve this?

When using Selenium driven GeckoDriver initiated firefox Browsing Context
The webdriver-active flag is set to true when the user agent is under remote control. It is initially false.
where, webdriver returns true if webdriver-active flag is set, false otherwise.
As:
navigator.webdriver Defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, for
example so that alternate code paths can be triggered during
automation.
Further #whimboo in his comments confirmed:
This implementation have to be conformant to this requirement. As such
we will not provide a way to circumvent that.
Conclusion
So, the bottom line is:
Selenium identifies itself
and there is no way to conceal the fact that the browser is WebDriver driven.
Recommendations
However some pundits have suggested some different approaches which can conceal the fact that the Mozilla Firefox browser is WebDriver controled through the usage of Firefox Profiles and Proxies as follows:
selenium4 compatible python code
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
profile_path = r'C:\Users\Admin\AppData\Roaming\Mozilla\Firefox\Profiles\s8543x41.default-release'
options=Options()
options.set_preference('profile', profile_path)
options.set_preference('network.proxy.type', 1)
options.set_preference('network.proxy.socks', '127.0.0.1')
options.set_preference('network.proxy.socks_port', 9050)
options.set_preference('network.proxy.socks_remote_dns', False)
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.google.com")
driver.quit()
Potential Solution
A potential solution would be to use the tor browser as follows:
selenium4 compatible python code
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
import os
torexe = os.popen(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile_path = r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default'
firefox_options=Options()
firefox_options.set_preference('profile', profile_path)
firefox_options.set_preference('network.proxy.type', 1)
firefox_options.set_preference('network.proxy.socks', '127.0.0.1')
firefox_options.set_preference('network.proxy.socks_port', 9050)
firefox_options.set_preference("network.proxy.socks_remote_dns", False)
firefox_options.binary_location = r'C:\Users\username\Desktop\Tor Browser\Browser\firefox.exe'
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = webdriver.Firefox(service=service, options=firefox_options)
driver.get("https://www.tiktok.com/")
References
You can find a couple of relevant detailed discussions in
How to initiate a Tor Browser 9.5 which uses the default Firefox to 68.9.0esr using GeckoDriver and Selenium through Python
How to connect to Tor browser using Python
How to use Tor with Chrome browser through Selenium

BotD detects you because you do not override navigator.webdriver attribute.
I was able to override it with this code:
((JavascriptExecutor)driver).executeScript("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})");
Re-run your code with this line after driver.get("BotD url") and click on
'Start detect' on the BotD page.
It will no longer show that webdriver is detected.
I understand you are looking for a way to make it work before the initial page load.
But here are 2 things to consider:
Webdriver developers want their tool to be detected by browsers.
Gecko driver developers are not going to implement an option to disable navigator.webdriver attribute. (This is the official reply from gecko developer.)

How to handle Untrusted Certificate Selenium Webdriver in Python?

I'm using Selenium to access a site, but I constantly get Warning: Potential Security Risk Ahead. I searched all over the Internet for a solution to my problem, but the message to accept the certificate manually continues to appear.
I'm using:
Firefox: 85.0.1 (64-bit)
Geckodriver
Python language
I tested several solutions such as:
from selenium import webdriver
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities['acceptSslCerts'] = True
#capabilities['acceptInsecureCerts'] = True
driver = webdriver.Firefox(capabilities=capabilities)
driver.get('xxxxxxxxx')
And,
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('xxxxxxxxx')
I also tried solutions based on creating a new profile in Firefox.
The question is: How can I automate the acceptance of a website's certificate when I launch Firefox with Selenium (Python programming language)?

https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities/acceptInsecureCerts
from selenium import webdriver
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities['acceptInsecureCerts'] = True
capabilities['marionette'] = True
driver = webdriver.Firefox(desired_capabilities=capabilities)
driver.get("https://self-signed.badssl.com/")

I solved my problem creating a new Firefox Profile and executing:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
#Path to the Firefox Profile directory
options.profile = r'C:\Users\x\AppData\Roaming\Mozilla\Firefox\Profiles\x.x'
driver = webdriver.Firefox(options=options)
driver.get('https://xxxxx')

How to initiate a Tor Browser 9.5 which uses the default Firefox to 68.9.0esr using GeckoDriver and Selenium through Python

I'm trying to initiate a tor browsing session through Tor Browser 9.5 which uses the default Firefox v68.9.0esr using GeckoDriver and Selenium through Python on a windows-10 system. But I'm facing an error as:
Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = r'C:\Users\username\Desktop\Tor Browser\Browser\firefox.exe'
driver = webdriver.Firefox(firefox_profile= profile, options = firefox_options, executable_path=r'C:\WebDrivers\geckodriver.exe')
driver.get("https://www.tiktok.com/")
Where as the same code block works through Firefox and Firefox Nightly using the respective binaries.
Do I need any additional settings? Can someone help me out?
Firefox Snapshot:
Firefox Nightly Snapshot:

I managed to resolve this by updating to v9.5.1 and implementing the following changes:
Note that although the code is in C# the same changes to the Tor browser and how it is launched should be applied.
FirefoxProfile profile = new FirefoxProfile(profilePath);
profile.SetPreference("network.proxy.type", 1);
profile.SetPreference("network.proxy.socks", "127.0.0.1");
profile.SetPreference("network.proxy.socks_port", 9153);
profile.SetPreference("network.proxy.socks_remote_dns", false);
FirefoxDriverService firefoxDriverService = FirefoxDriverService.CreateDefaultService(geckoDriverDirectory);
firefoxDriverService.FirefoxBinaryPath = torPath;
firefoxDriverService.BrowserCommunicationPort = 2828;
var firefoxOptions = new FirefoxOptions
{
Profile = null,
LogLevel = FirefoxDriverLogLevel.Trace
};
firefoxOptions.AddArguments("-profile", profilePath);
FirefoxDriver driver = new FirefoxDriver(firefoxDriverService, firefoxOptions);
driver.Navigate().GoToUrl("https://www.google.com");
Important notes:
The following TOR configs need to be changed in about:config :
marionette.enabled: true
marionette.port: set to an unused port, and set this value to firefoxDriverService.BrowserCommunicationPort in your code. This was set to 2828 in my example.

note:
I am not sure whether this really is the definite answer (thus, I'd really appreciate feedback)
solution:
I've managed to send a get request to the check tor page (https://check.torproject.org/) and it displayed an unknown IP to me (additionally, IPs differ if you repeat the request after a time)
Essentially, I've set up the chrome driver to run TOR. Here's the code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
tor_proxy = "127.0.0.1:9150"
chrome_options = Options()
chrome_options.add_argument("--test-type")
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument("--incognito")
chrome_options.add_argument('--proxy-server=socks5://%s' % tor_proxy)
driver = webdriver.Chrome(options=chrome_options)
driver.get('https://check.torproject.org/')
Because the driver is not in headless mode you can inspect the resulting page yourself. It should read:
"Congratulations. This browser is configured to use Tor. [IP Info]. However, it does not appear to be Tor Browser. Click here to go to the download page"
Make sure that the chromedriver.exe file is linked on the path or provide the path to the file as an argument to the driver.Chrome() function.
Edit: make sure TOR browser is running in the background, thanks #Abhishek Rai for pointing that out

Python: Selenium with tor proxy wont allow connection

When i try to connect to the tor proxy via my firefox selenium bot i get an error message "The Proxy server is refusing connections"
Error Message: https://i.stack.imgur.com/nmZoK.png
My Code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
binary = FirefoxBinary(r"C:\Program Files\Mozilla Firefox\firefox.exe")
user_agent = "Firefox"
proxy_ip = "127.0.0.1"
proxy_port = 9050
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.socks_version", 5)
profile.set_preference("network.proxy.socks", proxy_ip)
profile.set_preference("network.proxy.socks_port", proxy_port)
profile.set_preference("network.http.use-cache", False)
profile.set_preference("general.useragent.override", user_agent)
profile.update_preferences()
driver = webdriver.Firefox(firefox_binary=binary,capabilities=firefox_capabilities,firefox_profile=profile)
def interactWithSite(driver):
driver.get("https://check.torproject.org/")
interactWithSite(driver)

Solution: You need to have Tor Browser running in the background to connect to the tor browser network/proxy with your FireFox user agent/browser.

How to capture network traffic with selenium

I'm starting a new Django project, I'm trying to capture the network traffic with Selenium.
I have already achieved this objective with Selenium-wire (MITM Proxy), but Django doesn't like to work with selenium-wire ( must start the server with "--nothreading --noreload", connection bug... ).
I'm looking for achieve this with modern solutions, like parsing the network devtools of firefox directly or with a firefox addons.
I'm using Firefox Geckodriver for my test.
for x in range(0, 10):
profile = profile_firefox()
options = options_firefox()
driver = webdriver.Firefox(firefox_profile=profile, options=options, executable_path='/Users/*****/Desktop/selenium-master/headless_browser/geckodriver')
try:
driver.set_window_position(0, 0)
driver.set_window_size(randint(1024, 2060), randint(1024, 4100))
time.sleep(randint(3,10))
driver.get(url)
wait = ui.WebDriverWait(driver, 10)
time.sleep(randint(8,10))
if driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button"):
driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
del driver.requests
time.sleep(randint(8,10))
driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
time.sleep(randint(10,20))
for request in driver.requests:
if request.path == "https://api.*********.**/*******/*******":
request_api = request
raw = str(request_api.body)
request_api = raw.split(('b\''))
payload_raw = request_api[1]
payload = payload_raw[:-1]
if payload:
header = request.headers
time.sleep(8)
break
except:
print("Houston on a eu un probleme")
firefox_closing(driver)
Edit :
def profile_firefox():
profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference("general.useragent.override", firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.set_preference("driver.privatebrowsing.autostart", True)
profile.update_preferences()
return profile
Test 2 with Socks,HTTP,SSL configuration :
server = Server('/Users/*****/Desktop/selenium-master/browsermob-proxy-2.1.4/bin/browsermob-proxy')
server.start()
proxy = server.create_proxy()
proxy.selenium_proxy()#Dont understand what it does ???
port = int(proxy.port)
profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference('general.useragent.override', firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference('network.proxy.ssl', 'localhost')
profile.set_preference('network.proxy.ssl_port', port)
profile.set_preference('network.proxy.http', 'localhost')
profile.set_preference('network.proxy.http_port', port)
profile.set_preference('network.proxy.socks_remote_dns', False)
profile.set_preference('driver.privatebrowsing.autostart', True)
profile.update_preferences()
It seems Http proxy override the socks configuration...
Thanks a lot if you have any clue or advice about my code or solutions.

You can use a proxy to catch the network traffic. browsermob-proxy works well with selenium in Python. You need to download browsermob executable before. This is the piece of code with Firefox :
from browsermobproxy import Server
from selenium import webdriver
server = Server('path_to_executable')
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("file_name", options={'captureHeaders': True, 'captureContent': True})
driver.get("your_url")
proxy.wait_for_traffic_to_stop(1, 60)
for ent in proxy.har['log']['entries']:
print(ent)
server.stop()
driver.quit()

Python has a package called selenium-wire. You can use that package to capture the network traffics and also validate them. selenium-wire is an extended version of selenium will all the capabilities of selenium along with extra API to capture the network and validate. following is a link of an article
https://sensoumya94.medium.com/validate-network-requests-using-selenium-and-python-3da5be112f7b
following is the repository of the package
https://github.com/wkeeling/selenium-wire
Sample code -
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# Go to the Google home page
driver.get('https://www.google.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.url,
request.response.status_code,
request.response.headers['Content-Type']
)

Browsermob is the right way.
I must understand how browsermob works and tor too.
For Tor you must enable the HTTPTunnelPort configuration like this.
tor --HTTPTunnelPort 8088
And configure browsermob to use it.
proxy_params = {'httpProxy': 'localhost:8088', 'httpsProxy': 'localhost:8088'}
proxy_b = server.create_proxy(params=proxy_params)
Thanks.