We Keep Coding

Connection to google.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely

While running my test scripts with selenium == 4.2.0 like this:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.common.proxy import Proxy, ProxyType
options = FirefoxOptions()
service = Service()
options.headless = True
options.accept_insecure_certs = True
proxy = Proxy({
'httpProxy': proxy_addr,
'sslProxy': proxy_addr,
'proxyType': ProxyType.MANUAL
})
options.proxy = proxy
wd = Firefox(service=service, options=options)
wd.execute("get", {'url': 'http://google.com'})
I'm getting the following error:
An error occurred during a connection to www.google.com has a security
policy called HTTP Strict Transport Security (HSTS), which means that
Firefox can only connect to it securely. You can’t add an exception to
visit this site. Please contact the website owners to inform them of
this problem. This website might not support the TLS 1.2 protocol,
which is the minimum version supported by Firefox. Enabling TLS 1.0
and TLS 1.1 might allow this connection to succeed.
I think the problem is that I'm using a proxy which is running on localhost. Since I use the browser in headless mode and configured accept_insecure_certs = True I don't see how to find a workaround for this error. I'll be grateful if someone advises what else I can try to do.

I'm assuming that you're using a MITM that allows you to intercept the TLS traffic. If so, then this is exactly the scenario that HSTS preload is intended to prevent ;)
Your MITM will be generating a fake certificate on the fly, but because it does not match the HSTS preload list that is baked into the browser, then this is why you get presented with an error (rather than a dialog that asks if you want to continue).
You may be able to get around this by configuring the proxy to strip the HSTS header on all responses (check the documentation for the particular MITM that you are using).

How to login to website which is detecting bot usage using Selenium [duplicate]

I am running the Chrome driver over Selenium on a Ubuntu server behind a residential proxy network. Yet, my Selenium is being detected. Is there a way to make the Chrome driver and Selenium 100% undetectable?
I have been trying for so long I lost track of the many things I have done including:
Trying different versions of Chrome
Adding several flags and removing some words from the Chrome driver file.
Running it behind a proxy (residential ones also) using incognito mode.
Loading profiles.
Random mouse movements.
Randomising everything.
I am looking for a true version of Selenium that is 100% undetectable. If that ever existed. Or another automation way that is not detectable by bot trackers.
This is part of the starting of the browser:
sx = random.randint(1000, 1500)
sn = random.randint(3000, 4500)
display = Display(visible=0, size=(sx,sn))
display.start()
randagent = random.randint(0,len(useragents_desktop)-1)
uag = useragents_desktop[randagent]
#this is to prevent ip leaking
preferences =
"webrtc.ip_handling_policy" : "disable_non_proxied_udp",
"webrtc.multiple_routes_enabled": False,
"webrtc.nonproxied_udp_enabled" : False
chrome_options.add_experimental_option("prefs", preferences)
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-impl-side-painting")
chrome_options.add_argument("--disable-setuid-sandbox")
chrome_options.add_argument("--disable-seccomp-filter-sandbox")
chrome_options.add_argument("--disable-breakpad")
chrome_options.add_argument("--disable-client-side-phishing-detection")
chrome_options.add_argument("--disable-cast")
chrome_options.add_argument("--disable-cast-streaming-hw-encoding")
chrome_options.add_argument("--disable-cloud-import")
chrome_options.add_argument("--disable-popup-blocking")
chrome_options.add_argument("--ignore-certificate-errors")
chrome_options.add_argument("--disable-session-crashed-bubble")
chrome_options.add_argument("--disable-ipv6")
chrome_options.add_argument("--allow-http-screen-capture")
chrome_options.add_argument("--start-maximized")
wsize = "--window-size=" + str(sx-10) + ',' + str(sn-10)
chrome_options.add_argument(str(wsize) )
prefs = {"profile.managed_default_content_settings.images": 2}
chrome_options.add_experimental_option("prefs", prefs)
chrome_options.add_argument("blink-settings=imagesEnabled=true")
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("user-agent="+uag)
chrome_options.add_extension(pluginfile)#this is for the residential proxy
driver = webdriver.Chrome(executable_path="/usr/bin/chromedriver", chrome_options=chrome_options)

The fact that selenium driven WebDriver gets detected doesn't depends on any specific Selenium, Chrome or ChromeDriver version. The Websites themselves can detect the network traffic and can identify the Browser Client i.e. Web Browser as WebDriver controled.
However some generic approaches to avoid getting detected while web-scraping are as follows:
The first and foremost attribute a website can determine your script/program is through your monitor size. So it is recommended not to use the conventional Viewport.
If you need to send multiple requests to a website, you need to keep on changing the user-agent on each request. You can find a detailed discussion in Way to change Google Chrome user agent in Selenium?
To simulate human like behavior you may require to slow down the script execution even beyond WebDriverWait and expected_conditions inducing time.sleep(secs). Here you can find a detailed discussion on How to sleep webdriver in python for milliseconds
#Antoine Vastel in his blog site Detecting Chrome Headless mentioned several approaches, which distinguish the Chrome browser from a headless Chrome browser.
User agent: The user agent attribute is commonly used to detect the OS as well as the browser of the user. With Chrome version 59 it has the following value:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/59.0.3071.115 Safari/537.36
A check for the presence of Chrome headless can be done through:
if (/HeadlessChrome/.test(window.navigator.userAgent)) {
console.log("Chrome headless detected");
}
Plugins: navigator.plugins returns an array of plugins present in the browser. Typically, on Chrome we find default plugins, such as Chrome PDF viewer or Google Native Client. On the opposite, in headless mode, the array returned contains no plugin.
A check for the presence of Plugins can be done through:
if(navigator.plugins.length == 0) {
console.log("It may be Chrome headless");
}
Languages: In Chrome two Javascript attributes enable to obtain languages used by the user: navigator.language and navigator.languages. The first one is the language of the browser UI, while the second one is an array of string representing the user’s preferred languages. However, in headless mode, navigator.languages returns an empty string.
A check for the presence of Languages can be done through:
if(navigator.languages == "") {
console.log("Chrome headless detected");
}
WebGL: WebGL is an API to perform 3D rendering in an HTML canvas. With this API, it is possible to query for the vendor of the graphic driver as well as the renderer of the graphic driver. With a vanilla Chrome and Linux, we can obtain the following values for renderer and vendor: Google SwiftShader and Google Inc.. In headless mode, we can obtain Mesa OffScreen, which is the technology used for rendering without using any sort of window system and Brian Paul, which is the program that started the open source Mesa graphics library.
A check for the presence of WebGL can be done through:
var canvas = document.createElement('canvas');
var gl = canvas.getContext('webgl');
var debugInfo = gl.getExtension('WEBGL_debug_renderer_info');
var vendor = gl.getParameter(debugInfo.UNMASKED_VENDOR_WEBGL);
var renderer = gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL);
if(vendor == "Brian Paul" && renderer == "Mesa OffScreen") {
console.log("Chrome headless detected");
}
Not all Chrome headless will have the same values for vendor and renderer. Others keep values that could also be found on non headless version. However, Mesa Offscreen and Brian Paul indicates the presence of the headless version.
Browser features: Modernizr library enables to test if a wide range of HTML and CSS features are present in a browser. The only difference we found between Chrome and headless Chrome was that the latter did not have the hairline feature, which detects support for hidpi/retina hairlines.
A check for the presence of hairline feature can be done through:
if(!Modernizr["hairline"]) {
console.log("It may be Chrome headless");
}
Missing image: The last on our list also seems to be the most robust, comes from the dimension of the image used by Chrome in case an image cannot be loaded. In case of a vanilla Chrome, the image has a width and height that depends on the zoom of the browser, but are different from zero. In a headless Chrome, the image has a width and an height equal to zero.
A check for the presence of Missing image can be done through:
var body = document.getElementsByTagName("body")[0];
var image = document.createElement("img");
image.src = "http://iloveponeydotcom32188.jg";
image.setAttribute("id", "fakeimage");
body.appendChild(image);
image.onerror = function(){
if(image.width == 0 && image.height == 0) {
console.log("Chrome headless detected");
}
}
References
You can find a couple of similar discussions in:
How to bypass Google captcha with Selenium and python?
How to make Selenium script undetectable using GeckoDriver and Firefox through Python?
tl; dr
Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
How does recaptcha 3 know I'm using selenium/chromedriver?
Selenium and non-headless browser keeps asking for Captcha

why not try undetected-chromedriver?
Optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io Automatically downloads the driver binary and patches it.
Tested until current chrome beta versions
Works also on Brave Browser and many other Chromium based browsers
Python 3.6++
you can install it with: pip install undetected-chromedriver
There are important things you should be ware of:
Due to the inner workings of the module, it is needed to browse programmatically (ie: using .get(url) ). Never use the gui to navigate. Using your keybord and mouse for navigation causes possible detection! New Tabs: same story. If you really need multi-tabs, then open the tab with the blank page (hint: url is data:, including comma, and yes, driver accepts it) and do your thing as usual. If you follow these "rules" (actually its default behaviour), then you will have a great time for now.
In [1]: import undetected_chromedriver as uc
In [2]: driver = uc.Chrome()
In [3]: driver.execute_script('return navigator.webdriver')
Out[3]: True # Detectable
In [4]: driver.get('https://distilnetworks.com') # starts magic
In [4]: driver.execute_script('return navigator.webdriver')
In [5]: None # Undetectable!

For Python with Chrome or Chromium-based browsers, there's Selenium-Profiles
It currently supports:
Overwrite device metrics with fake-profiles
Mobile and Desktop emulation
Undetected by Google, Cloudflare, ..
Modifying headers supported using Selenium-Interceptor
Touch Actions
proxies with authentication
making single POST, GET or other requests using driver.requests.fetch(url, options) (syntax)
Installation
pip install selenium-profiles
Example script
from selenium_profiles.driver import driver as mydriver
from selenium_profiles.profiles import profiles
mydriver = mydriver()
driver = mydriver.start(profiles.Windows()) # or .Android
# get url
driver.get('https://nowsecure.nl/#relax') # test undetectability
input("Press ENTER to exit: ")
driver.quit() # Execute on the End!
Notes:
The package is licenced under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , means, in case you want to use it for something commercial, you need to ask the author first.
headless support currently isn't guaranteed, but you can use pyvirtualdisplay

What about:
import random
from selenium import webdriver
import time
driver = webdriver.Chrome("C:\\Users\\DusEck\\Desktop\\chromedriver.exe")
username = "username" # data_user
password = "password" # data_pass
driver.get("https://www.depop.com/login/") # get URL
driver.find_element_by_xpath('/html/body/div[1]/div/div[3]/div[2]/button[2]').click() # Accept cookies
split_char_pw = [] # Empty lists
split_char = []
n = 1 # Splitter
for index in range(0, len(username), n):
split_char.append(username[index: index + n])
for user_letter in split_char:
time.sleep(random.uniform(0.1, 0.8))
driver.find_element_by_id("username").send_keys(user_letter)
for index in range(0, len(password), n):
split_char.append(password[index: index + n])
for pw_letter in split_char_pw:
time.sleep(random.uniform(0.1, 0.8))
driver.find_element_by_id("password").send_keys(pw_letter)

Redirect to Final url from selenium failing

Requirement
Get the Final Url after redirect chain like it happens on the normal browser using selenium .
The urls are article urls got from twitter.
Behavior on normal Desktop browser after viewing the redirect and headers:
The Url is a twitter URL which gets a 301 error -moved permanently
Then it follows the location tag to a shortened url which then again gets a 302 error .
It again follows the redirect chain and lands on the final page.
Behavior using Selenium
It redirects finally to the main website homepage/index rather than the actual article page .Final url is not the same as actual one.
Initial basic code
chrome_options = Options()
chrome_options.add_argument('--user-agent= Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0');
#above user agent is an example but multiple different user agents were tried
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
browser = webdriver.Chrome(driver_loaction,chrome_options=chrome_options)
browser.set_page_load_timeout(45)
browser.get(url)
final_url = browser.current_url
Various attempts to get the final url instead of the main website home/index page
With normal wait
browser.get(url)
time.sleep(25)
With WebDriverWait-
WebDriverWait(browser,20)
with expected_conditions
WebDriverWait(browser, 20).until(EC.url_contains("portion of the final url"))
ends up timing out everytime , even with different conditions like url_to_be etc.
Behavior on Trying with non-selenium options
1.Wget -
Below is response from a wget call edited for obscuring actual details -
Resolving t.co (t.co)...,
... Connecting to t.co (t.co)|:443... connected. HTTP
request sent, awaiting response... 301 Moved Permanently Location:
[following]
Resolving domain (domain)...
... Connecting to ... connected.
HTTP request sent, awaiting response... 302 Found Location:
[following]
--Date-- Resolving website (website)... ip ,
connected. HTTP request sent, awaiting response... 200 OK
As seen finally we got the homepage rather than the website page .
Request library -
response = requests.get(Url, allow_redirects=True, headers=self.headers, timeout=30)
(header contains user agents but tried with actual same request headers from browser that gets the proper final url response )- gets the homepage --
checking redirects by response.history we see that from t.co(twitter url ) - we redirect to the short url then redirect to website homepage and end .
urlib library -
same final response.
Test e.g url - t.co/Mg3IYF8ZLm?amp=1 (add the https:// - i removed for posting)
After days of different approaches , i am stuck -- i somehow think that selenium is key to resolve this because it works on normal desktop browsers then should work with selenium - right?
Edit: It seems it is happening with other versions of drivers and selenium too ,it would be great if we could at least find out the actual reasoning its happening with certain links like the example given .

I have used your code only.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--user-agent= Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0');
#above user agent is an example but multiple different user agents were tried
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
browser = webdriver.Chrome(chrome_options=chrome_options)
browser.set_page_load_timeout(120)
browser.get("url")
final_url = browser.current_url
print(final_url)
Snapshot:

The issue was after the intermediate url, the url cannot redirect to the final page due to the missing headers, the intermediate url (http://www.cumhuriyet.com.tr/) is blocked with 403 status code.
Once we add the headers for the request the url is navigating to the final url.
I have given the Java implementation below. Sorry, I am not familiar with this code in Python. Please convert this code to python and it may solve the issue.
#Test
public void RUN() {
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
proxy.addRequestFilter((request, contents, messageInfo)->{
request.headers().add("Accept-Language", "en-US,en;q=0.5");
request.headers().add("Upgrade-Insecure-Requests", "1");
return null;
});
String proxyOption = "--proxy-server=" + seleniumProxy.getHttpProxy();
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
options.addArguments(proxyOption);
System.setProperty("webdriver.chrome.driver","<driverLocation>");
WebDriver driver = new ChromeDriver(options);
driver.get("<url>");
System.out.println(driver.getCurrentUrl());
}

How to use existing login token for telegram web using selenium webdriver

I'm trying to read telegram messages from https://web.telegram.org with selenium.
When i open https://web.telegram.org in firefox i'm already logged in, but when opening the same page from selenium webdriver(firefox) i get the login page.
I saw that telegram web is not using cookies for the auth but rather saves values in local storage. I can access the local storage with selenium and have keys there such as: "dc2_auth_key", "dc2_server_salt", "dc4_auth_key", ... but I'm not sure what to do with them in order to login(and if i do need to do something with them then why? its the same browser why wont it work the same when opening without selenium?)
To reproduce:
open firefox and login to https://web.telegram.org then run this code:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://web.telegram.org")
# my code is here but is irrelevant since im at the login page.
driver.close()

When you open https://web.telegram.org manually using Firefox, the Default Firefox Profile is used. As you login and browse through the website, the websites stores Authentication Cookies within your system. As the cookies gets stored within the local storage of the Default Firefox Profile even on reopening the browsers you are automatically authenticated.
But when GeckoDriver initiates a new web browsing session for your tests everytime a temporary new mozprofile is created while launching Firefox which is evident from the following log:
mozrunner::runner INFO Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-profile" "C:\\Users\\ATECHM~1\\AppData\\Local\\Temp\\rust_mozprofile.fDJt0BIqNu0n"
You can find a detailed discussion in Is it Firefox or Geckodriver, which creates “rust_mozprofile” directory
Once the Test Execution completes and quit() is invoked the temporary mozprofile is deleted in the following process:
webdriver::server DEBUG -> DELETE /session/f84dbafc-4166-4a08-afd3-79b98bad1470
geckodriver::marionette TRACE -> 37:[0,3,"quit",{"flags":["eForceQuit"]}]
Marionette TRACE 0 -> [0,3,"quit",{"flags":["eForceQuit"]}]
Marionette DEBUG New connections will no longer be accepted
Marionette TRACE 0 <- [1,3,null,{"cause":"shutdown"}]
geckodriver::marionette TRACE <- [1,3,null,{"cause":"shutdown"}]
webdriver::server DEBUG Deleting session
geckodriver::marionette DEBUG Stopping browser process
So, when you open the same page using Selenium, GeckoDriver and Firefox the cookies which were stored within the local storage of the Default Firefox Profile aren't accessible and hence you are redirected to the Login Page.
To store and use the cookies within the local storage to get authenticated automatically you need to create and use a Custom Firefox Profile.
Here you can find a relevant discussion on webdriver.FirefoxProfile(): Is it possible to use a profile without making a copy of it?

You can auth using your current data from local storage.
Example:
driver.get(TELEGRAM_WEB_URL);
LocalStorage localStorage = ((ChromeDriver) DRIVER).getLocalStorage();
localStorage.clear();
localStorage.setItem("dc2_auth_key","<YOUR AUTH KEY>");
localStorage.setItem("user_auth","<YOUR USER INFO>");
driver.get(TELEGRAM_WEB_URL);

Error using webdriver with headless chrome to download file

I want to use headless chrome driver to download pdf. Everything works fine when I downloaded pdf without headless chrome. Here is part of my driver setting code:
options = webdriver.ChromeOptions()
prefs = {'profile.default_content_settings.popups': 0,
"plugins.plugins_list": [{"enabled": False, "name": "Chrome PDF Viewer"}], # Disable Chrome's PDF Viewer
'download.default_directory': 'download_dir' ,
"download.extensions_to_open": "applications/pdf"}
options.add_argument('--headless')
options.add_argument('--disable-gpu')
options.add_argument('"--no-sandbox"')
options.add_argument('--ignore-certificate-errors')
options.add_experimental_option('prefs', prefs)
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_dir}}
driver.execute("send_command", params)
driver.get(url)
try : WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, 'check-pdf')))
finally:
driver.find_element_by_class_name('check-pdf').click()
The error shows up when I run this file in cmd.
[0623/130628.966:INFO:CONSOLE(7)] "A parser-blocking, cross site (i.e. different eTLD+1) script, http://s11.cnzz.com/z_stat.php?id=1261865322, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.", source: http://utrack.hexun.com/dp/hexun_uweb.js (7)
[0623/130628.968:INFO:CONSOLE(7)] "A parser-blocking, cross site (i.e. different eTLD+1) script, http://s11.cnzz.com/z_stat.php?id=1261865322, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.", source: http://utrack.hexun.com/dp/hexun_uweb.js (7)
[0623/130628.974:INFO:CONSOLE(16)] "A parser-blocking, cross site (i.e. different eTLD+1) script, http://c.cnzz.com/core.php?web_id=1261865322&t=z, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.", source: https://s11.cnzz.com/z_stat.php?id=1261865322 (16)
[0623/130628.976:INFO:CONSOLE(16)] "A parser-blocking, cross site (i.e. different eTLD+1) script, http://c.cnzz.com/core.php?web_id=1261865322&t=z, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.", source: https://s11.cnzz.com/z_stat.php?id=1261865322 (16)
[0623/130629.038:INFO:CONSOLE(8)] "Uncaught ReferenceError: jQuery is not defined", source: http://img.hexun.com/zl/hx/index/js/appDplus.js (8)
[0623/130629.479:WARNING:render_frame_host_impl.cc(2750)] OnDidStopLoading was called twice
I am wondering what the error message means and how I can fix it ?
Any idea would be helpful !