So, this is the code I'm having troubles with:
def scrap():
options = webdriver.ChromeOptions();
options.add_argument('headless');
options.add_argument('--profile-directory=Profile 1')
options.add_argument("--user-data-dir=C:/Users/omarl/AppData/Local/Google/Chrome/User Data/")
options.add_argument("--remote-debugging-port=45447")
options.add_argument("--disable-gpu")
browser = webdriver.Chrome(executable_path=r"C:\Users\omarl\OneDrive\Escritorio\chromedriver.exe", options=options)
scrapURL = "https://es.wallapop.com/search?distance=30000&keywords=leggins&latitude=41.38804&longitude=2.17001&filters_source=quick_filters"
browser.get(scrapURL)
#...
And the error:
WebDriverException: unknown error: unable to discover open pages
I don't have any instances of chrome when I execute the script, and when I'm using it without the headless option it works fine. Any idea why this is happening? Please, note that I'm using the --remote-debuggin-port provided in similar questions.
I'm using ChromeDriver 86.0.4240.22
To invoke a Chrome Profile in Headless mode you can use the --user-data-dir argument only and you can safely remove --profile-directory argument as follows:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--window-size=1920,1080')
# options.add_argument('--profile-directory=Profile 1')
options.add_argument(r"--user-data-dir=C:\Users\Soma Bhattacharjee\AppData\Local\Google\Chrome\User Data\Default")
options.add_argument("--remote-debugging-port=9222")
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.google.com/')
print("Chrome Headless launched")
Console Output:
DevTools listening on ws://127.0.0.1:9222/devtools/browser/93c67c41-e125-4d12-abc0-fcf0f07a62f4
Chrome Headless launched
References
You can find a couple of relevant detailed discussions in:
How to open a Chrome Profile through --user-data-dir argument of Selenium
Selenium: Point towards default Chrome session
Additional Considerations
Ensure that:
Selenium is upgraded to current released Version 3.141.0.
ChromeDriver is updated to current ChromeDriver v86.0 level.
Chrome is updated to current Chrome Version 86.0 level. (as per ChromeDriver v86.0 release notes).
Execute your #Test as non-root user.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
tl; dr
ChromeDriver remote debug port reservation race conditions
Have you tried using arg --no-sandbox?
A lot of people on Chrome Driver Error using Selenium: Unable to Discover Open Pages have had success with that argument.
Related
I am trying to execute a basic program using Selenium 4.8.0 Python clients in headless mode:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.headless = True
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get('https://www.google.com/')
driver.quit()
With the following configuration:
Selenium 4.8.0 Python
Chrome _Version 109.0.5414.120 (Official Build) (64-bit)
ChromeDriver 109.0.5414.25
Though the program gets executed successfully there seems to a DeprecationWarning as:
DeprecationWarning: headless property is deprecated, instead use add_argument('--headless') or add_argument('--headless=new')
Can anyone explain the DeprecationWarning and the required changes?
The deprecation of the headless property was announced in the Selenium Blog post Headless is Going Away! (archive) on January 29, 2023. The summary and suggested changes are as follows:
Headless is Going Away!
Headless is an execution mode for Firefox and Chromium based browsers. It allows users to run automated scripts in headless mode, meaning that the browser window wouldn’t be visible. In most of Selenium’s bindings there is a convenience method to set this execution mode while setting the browser options. However, Selenium 4.8.0 will be deprecated [sic] this method and now users need to set it through arguments when setting the browser options.
[...]
How can I set headless mode from now on?
In short, users can add the headless mode they want to use through arguments in browser options.
Before
options = ChromeOptions()
options.headless = True
driver = webdriver.Chrome(options=options)
driver.get('http://selenium.dev')
driver.quit()
After
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get('http://selenium.dev')
driver.quit()
See the full blog post for additional background on why this change was implemented.
Test Automation developers had been using Headless Chrome and Firefox Headless for quite sometime now to execute the automated scripts in headless mode where the browser window wouldn't be visible. This was the traditional headless mode which now turns the old Headless mode.
The snippets being used were:
Java:
ChromeOptions options = new ChromeOptions();
options.setHeadless(true);
WebDriver driver = new ChromeDriver(options);
driver.get("https://selenium.dev");
driver.quit();
Python:
options = ChromeOptions()
options.headless = True
driver = webdriver.Chrome(options=options)
driver.get('http://selenium.dev')
driver.quit()
Javascript:
let driver = await env
.builder()
.setChromeOptions(new chrome.Options().headless())
.build();
await driver.get('https://selenium.dev');
await driver.quit();
According to this Selenium Blog this old headless mode will be still available by using the --headless switch with no value or with old value. This convenient yet deprecated method will be removed in Selenium 4.10.0
Renaming NativeHeadlessChrome to new Headless
Recently Chromium team have released the Native Headless mode which is now officially called the new Headless mode. This functionality have landed with:
Chromium v109.0.5400.0
ChromeDriver v109.0.5414.25
aptly supported through:
Selenium v4.8.0
The new syntax requires --headless=new to be passed as an argument, where as we passed only --headless while using Chrome since v96 till v108.
Sample Code snippets:
Java:
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
driver.get("https://selenium.dev);
driver.quit();
Python:
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get('http://selenium.dev')
driver.quit()
Javascript:
let driver = await env
.builder()
.setChromeOptions(options.addArguments('--headless=new'))
.build();
await driver.get('https://selenium.dev');
await driver.quit();
CSharp:
var options = new ChromeOptions();
options.AddArgument("--headless=new");
var driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("https://selenium.dev");
driver.Quit();
Ruby:
options = Selenium::WebDriver::Options.chrome(args: ['--headless=new'])
driver = Selenium::WebDriver.for :chrome, options: options
driver.get('https://selenium.dev')
driver.quit
Is it possible after setting selenium webdriver to a headless mode set it back to a normal mode?
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.get(http://stackoverflow.com)
# set driver back to normal mode
No, it won't be possible to make Chrome operate initially in headless mode and then switch back to normal mode within the same session.
When you configure an instance of a ChromeDriver with ChromeOptions() to span a new Chrome Browsing Context the configuration gets baked within the chromedriver executable which will persist for the lifetime of the WebDriver and being uneditable. So you can't modify/add any existing/new configuration through ChromeOptions() class to the WebDriver instance which is currently in execution.
Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. Session ID, Cookies, UserAgent and other session attributes from the already initiated ChromeDriver and Chrome Browsing Session still you won't be able to change the set of attributes of the ChromeDriver.
A cleaner way would be to call driver.quit() within tearDown(){} method to close and destroy the current ChromeDriver and Chrome Browser instances gracefully and then span a new set of ChromeDriver and Chrome Browser instance with the new set of configurations.
tl; dr
You can find a couple of relevant discussions in:
Change ChromeOptions in an existing webdriver
How do I make Chrome Headless after I login manually
On my Mac, I use the below selenium code to search the Python keyword in headless Chrome.
from selenium import webdriver
wd = webdriver.Chrome(r"/opt/webdrivers/chromedriver")
wd.implicitly_wait(5)
wd.get("https://www.google.com")
element = wd.find_element_by_id('kw')
element.send_keys('Python\n')
element = wd.find_element_by_class_name('c-abstract')
print(element.text)
but, however, the Chrome will open up a window automatically.
my understanding of headless browser will now open a window. In spite of this, is it possible to restrain the GUI get up, let it run in silence?
I don't know about headless chrome but you can run chrome with headless mode by following options
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('window-size=1920x1080')
options.add_argument("disable-gpu")
wd = webdriver.Chrome(r"/opt/webdrivers/chromedriver")
Is it possible after setting selenium webdriver to a headless mode set it back to a normal mode?
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.get(http://stackoverflow.com)
# set driver back to normal mode
No, it won't be possible to make Chrome operate initially in headless mode and then switch back to normal mode within the same session.
When you configure an instance of a ChromeDriver with ChromeOptions() to span a new Chrome Browsing Context the configuration gets baked within the chromedriver executable which will persist for the lifetime of the WebDriver and being uneditable. So you can't modify/add any existing/new configuration through ChromeOptions() class to the WebDriver instance which is currently in execution.
Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. Session ID, Cookies, UserAgent and other session attributes from the already initiated ChromeDriver and Chrome Browsing Session still you won't be able to change the set of attributes of the ChromeDriver.
A cleaner way would be to call driver.quit() within tearDown(){} method to close and destroy the current ChromeDriver and Chrome Browser instances gracefully and then span a new set of ChromeDriver and Chrome Browser instance with the new set of configurations.
tl; dr
You can find a couple of relevant discussions in:
Change ChromeOptions in an existing webdriver
How do I make Chrome Headless after I login manually
Is it possible after setting selenium webdriver to a headless mode set it back to a normal mode?
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.get(http://stackoverflow.com)
# set driver back to normal mode
No, it won't be possible to make Chrome operate initially in headless mode and then switch back to normal mode within the same session.
When you configure an instance of a ChromeDriver with ChromeOptions() to span a new Chrome Browsing Context the configuration gets baked within the chromedriver executable which will persist for the lifetime of the WebDriver and being uneditable. So you can't modify/add any existing/new configuration through ChromeOptions() class to the WebDriver instance which is currently in execution.
Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. Session ID, Cookies, UserAgent and other session attributes from the already initiated ChromeDriver and Chrome Browsing Session still you won't be able to change the set of attributes of the ChromeDriver.
A cleaner way would be to call driver.quit() within tearDown(){} method to close and destroy the current ChromeDriver and Chrome Browser instances gracefully and then span a new set of ChromeDriver and Chrome Browser instance with the new set of configurations.
tl; dr
You can find a couple of relevant discussions in:
Change ChromeOptions in an existing webdriver
How do I make Chrome Headless after I login manually