System setup:
I used Python 3.10 in my setup.
I used Selenium 4.
I used the Python webdriver manager in my test setup. (https://pypi.org/project/webdriver-manager/)
I used unittesting for some test pytest.
Details:
Currently, my test setup and the included test cases work under Chrome as a normal GUI call, and the test cases run through under those conditions.
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=chrome_options)
class OpenDevcrowd(unittest.TestCase):
def setUp(self):
#self.driver = webdriver.Firefox(service=Service(GeckoDriverManager().install()))
self.driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
self.driver.implicitly_wait(30)
self.base_url = "https://www.google.com/"
self.verificationErrors = []
self.accept_next_alert = True
I have used the options from 2020/2021 to change my setup to headless.
I tried the following information from stackoverflow in my build, but had no success with it at all:
webdriver-manager not using proxy in order to update
Running Selenium with Headless Chrome Webdriver
Issue:
It seems that the options (and I tried almost all the options from the information already found in stackoverflow) would not work anymore?
Only the GUI is still called, and the headless option is simply not used.
Questions:
Did the command range change somehow from 2020/21 to 2022?
Is it the webdriver manager that prevents the options from being called?
--headless should come with --window-size
Ex: "--window-size=1920,1080"
Related
I would like to use an existing installation of chrome (or firefox or brave browser) with selenium. Like that I could set prespecified settings / extensions (e.g. start nord-vpn when opening a new instance) that are active when the browser is opened with selenium.
I know there is selenium.webdriver.service with the "executeable-path" option, but it doesn't seem to work when you specify a specific chrome.exe, the usage seems to be for the chrome-driver only and then it still opens a "fresh" installation of chrome.
Starting selenium with extension-file I think is also not an option to use with the nord-vpn extension, as I have two-factor authentication active and login every single time would take too much time and effort, if possible at all.
Firefox profile
To use the existing installation of firefox you have to pass the profile path through set_preference() method using an instance of Option from selenium.webdriver.common.options as follows:
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
profile_path = r'C:\Users\Admin\AppData\Roaming\Mozilla\Firefox\Profiles\s8543x41.default-release'
options=Options()
options.set_preference('profile', profile_path)
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.google.com")
You can find a relevant detailed discussion in Error update preferences in Firefox profile: 'Options' object has no attribute 'update_preferences'
Chrome profile
Where as to use an existing installation of google-chrome you have to pass the user profile path through add_argument() using the user-data-dir key through an instance of Option from selenium.webdriver.common.options as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.add_argument("user-data-dir=C:\\Users\\username\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://www.google.com/")
You can find a relevant detailed discussion in How to open a Chrome Profile through Python
I have but one hurdle to overcome before I can truly call my first bot complete and that is to figure out where to put the options class(?) in order to run ChromeDriverManager in headless mode, and so it stops opening chrome instances! The way the driver is called is:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
options = Options()
options.headless = True
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
Since the old method of calling webdriver by path hasn't been entirely deprecated yet I don't think there have been very many questions pertaining to the new webdriver-manager. I've found only one or two methods that didn't work, like adding ,options=options after .install() or .options somewhere in the mix. In any case, any suggestions would be appreciated.
try this:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
I typed out this comment and never finished it, so my apologies. The correct code to run selenium 4 WebDriver-Manager in headless mode is indeed:
options = Options()
options.headless = True #
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
#as opposed to what I was trying: #
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install(),options=options))
I imagine that 'options' just needs to be a direct argument of webdriver.Chrome(), so I think this should also work:
driver = webdriver.Chrome(service=Service,options=options(ChromeDriverManager().install()))
Also, I think I have figured out that headless mode makes it easier for websites to mark you as a bot and prompt you to do captchas as after some time of running, because of either captchas or an error in a change of code, my bot couldn't use the search function while headless was true, but performed perfectly with it disabled.
I want to scrap some data with selenium python. I have this type of screen sometimes :
Do you know to proceed in order to remove this type of verification ? Here my code :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
options = Options()
options.add_argument("start-maximized")
options.add_argument("--disable-web-security")
options.add_argument("--disable-site-isolation-trials")
options.add_argument("--allow-running-insecure-content")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get('THE_WEBSITE_COM')
Selenium specifically and other automation tools have certain user agents and other identifiers which indicate that it's automated. So maybe have a play around with things like that. Some websites use anti bot tools to analyze browsing behaviours and patterns so try to slow it and randomize it eg. random time between page requests
Another trick is to look at the website and try to find if there are any alternative routes to get the information. For example: is there a public API you can use to bypass it? Is there a mobile version of the website? Sometimes mobile versions have less aggressive Captcha enforcement.
What I do most of the time is to launch the browser separately and connect to it using Dev port which is beautifully explained in this article.
To enable Chrome to open a port for remote debugging, we need to launch it with a custom flag –
chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenum\ChromeProfile"
Then connect to the browser using this
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
#Change chrome driver path accordingly
chrome_driver = "C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_driver, chrome_options=chrome_options)
print driver.title
I have checked online and it was mentioned that Selenium closes the browser after running unless you use the option module or the driver.quit() or driver.close() functions but I used the option as shown in the code below but Chrome still closes after 2-3 seconds.
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_driver_path = r"C:\Development\chromedriver.exe"
serv = Service(chrome_driver_path)
driver = webdriver.Chrome(service=serv, options=chrome_options)
driver.get("https://www.google.com")
I am working on a project with selenium to scrape the data, but I don't want the browser to open and pop up. I just wanted to hide the browser and also not to display it in the taskbar also...
Some also suggested to use phantomJS but I didn't get them. What to do now ...
If you're using Chrome you can just set the headless argument like so:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
driver_exe = 'chromedriver'
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(driver_exe, options=options)
For chrome you could pass in the --headless parameter.
Alternatively you could let selenium work on a virtual display like this:
from selenium import webdriver
from xvfbwrapper import Xvfb
display = Xvfb()
display.start()
driver = webdriver.Chrome()
driver.get('http://www.stackoverflow.com')
print(driver.title)
driver.quit()
display.stop()
The latter has worked for me quite well.
To hide the browser while executing tests using Selenium's python you can use the minimize_window() method which eventually minimizes/pushes the Chrome Browsing Context effectively to the background using the following solution:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://www.google.co.in')
driver.minimize_window()
Alternative
As an alternative you can use the headless attribute to configure ChromeDriver to initiate google-chrome browser in Headless mode using Selenium and you can find a couple of relevant discussions in:
How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
If you're using Firefox, try this:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
driver_exe = 'path/to/firefoxdriver'
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(driver_exe, options=options)
similar to what #Meshi answered in case of Chrome
if you want to hide chrome or selenium driver there is a library pyautogui
import pyautogui
window = [ x for x in pyautogui.getAllWindows()]
by this, you are getting all window title
now you need to find your window
for i in window:
if 'Google Chrome' in i.title:
i.hide()
or you can play with your driver title also