I've been trying to wrap my brain around this problem. I have a webscraping project with Selenium and my script works when headless mode is disabled. Once I add the argument ("--headless") to my chrome driver options (chrome driver version 88.0.4324) it isn't able to click the element. I have seen other people mentioning that maximizing the window or setting the window size of the headless driver will solve this, I have had no luck with such solutions.
This is the line that clicks the element:
browser.find_element_by_xpath('//a [#href][#class="button icon arrowdown"]').click()
This is what I was using before the above xpath:
browser.find_element_by_class('button icon arrowdown').click()
Thanks for Everyones Help!
I had same issue tried using javascript click
driver.execute_script("document.getElementsByClassName('button icon arrowdown')[0].click()")
Take a screenshot and snapshot of the headless window when exception occurs. That will help you to find out why this happens. You can try to use selene library https://github.com/yashaka/selene which already have these features, or write your own snap-shooter.
Depending on this quote
I have seen other people mentioning that maximizing the window or
setting the window size of the headless driver will solve this, I have
had no luck with such solutions.
you need to know, that maximizing in headless - doesn't work because it headless and doesn't know which screen size is to maximize.
You should take the dimensions of the browser in non-headless mode and set it up directly in selenium both for head and headless mode. This can solve your issue. And even if not - try to take screenshot or make a video to figure out what's going on in headless mode.
Related
New to Stack Overflow posting so apologies in advance.
I am trying to use selenium + headless chrome to perform browser automation, but in initial testing noticed some odd behavior with saving cookies and other local data when comparing headless vs non headless chrome instances. For reference, my chrome version is: Version 87.0.4280.66 (Official Build) (64-bit), my chromedriver version is 87.0.4280.88. And I'm using 64-bit Centos8 to run this on.
For example, take my two pieces of code I was testing, and see the folder structure as a result in the attached image (note the difference in headless=True):
self.driver_options = Options()
self.driver_options.headless = True
self.driver_options.add_argument("--user-data-dir=/home/<user>/Projects/.headless")
self.driver_options.add_argument("--window-size=1200,1200")
self.driver = webdriver.Chrome('../bin/chromedriver',options=self.driver_options)
self.driver_options = Options()
self.driver_options.add_argument("--user-data-dir=/home/<user>/Projects/.nonheadless")
self.driver_options.add_argument("--window-size=1200,1200")
self.driver = webdriver.Chrome('../bin/chromedriver',options=self.driver_options)
Here's a picture comparing directory structure of headless (left) and non-headless (right)
I read that this may have something to do with headless chrome instances using a different data directory, so maybe some of the files are 'missing' due to that (in another location) - but the biggest issue is that the [user-dir]/Default/Cookies file is populated in the non-headless version, but unpopulated in the headless version. I've also tried to remedy it by adding a "remote-debugging-port" option, but still got the same outcome.
I want to run a few different headless threads, where cookies are important for long running jobs. Has anyone seen anything like this, or know why I am experiencing this behavior? Is it to be expected?
Thanks all
I need a browser in Selenium to work as if it's maximized, so that website that I process perceived it as maximized, but I want it really to be minimized, so that I could work in parallel while the script executes. The simple: driver.maximize_window() just maximizes windows.
So, is there a way for a window of browser to be maximized for website, but in reality minimized?
Several approaches here, depending on what your needs are.
First, most browser allow setting screen size when they are fired up. See details here How do I set browser width and height in Selenium WebDriver? Chrome, for example, can be controlled from outside of the selenium script either with ini files where you set your screen size properties or via command line arguments. See this page for specific args for chromium for example: https://peter.sh/experiments/chromium-command-line-switches/
Secondly, you could start your browser in headless mode and restrict it a specific size. If you do so, your code will run in the headless browser, meaning you can't see it on a real screen. See the miriad of stackoverflow question addressing this, including this one: How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
Thirdly, you could start a virtual display of a certain size and have your driver maximize the browser in all the tests. Tools like xvfb are a god send here. How do I run Selenium in Xvfb?
Lastly, you could use a real display that's set in the resolution/dpi you need. See answers like this Resize display resolution using python with cross platform support
You can try just running it in a small window instead of Maximized or Minimized.
Don't tell the browser to do anything in terms of size.
So where you tell Selenium
browser.maximize_window()
Erase or comment it out, and run the script that way.
It's either that or use multiple screens.
If you want to work while WebDriver is executed, you can created a virtual instance (with e.g. VirtualBox or HyperV) an run test suite in VM.
Maximized is just a browser size from the site perspective. Set the browser size to the screen resolution of your desktop and minimize the browser.
I have a project to get some information from website.
I want to look at the process inside the chrome window, So I can't use headless browser. But sometimes I want to minimize the chrome window.
But I found the selenium would go wrong after I minimize the chrome window manually, but sometimes not. When go wrong, exception
element is not clickable at the point, other element will receive the click
will be raised, or sometimes selenium just stop.
I have searched for a long time that some people said that the chrome window should be focused on and can't be minimized by clicking '-' on the window title bar. And the alternative solution is:
web.set_window_position(-2000,-2000)
To make the window move out the screen.
And someone says by simulating shortcuts to minimize the window. But I think it's the same as click '-' manually, am I wrong?
My question is :
Does selenium really requires chrome window not minimized? Why sometimes it can run selenium normally after minimized but sometimes not?
If I use
set_window_position(-2000,-2000)
to move the window invisible, then I click the window icon on the bottom of os (I'm sorry that I don't know what it's called). Normally when clicked, the window will be minimized. So as for this chrome window, Will it be considered as minimized window and go wrong?
I am really sorry for my poor English. I hope I have a clear description of my problem.
Environment:
python 3.6
chrome 66.0
selenium 3.11.0
windows server 2012
Edit to add code:
wait.WebDriverWait(driver,100000).until(EC.visibility_of_element_located((By.ID,'commMgrCompositionMessage')))
textArea = driver.find_element_by_id('commMgrCompositionMessage')
driver.execute_script("arguments[0].value="+"'"+modelStr+"';",textArea)
time.sleep(1)
wait.WebDriverWait(driver,10000).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,'#sendemail_label')))
allSendMailLabel = driver.find_elements_by_css_selector('#sendemail')
allSendMailLabel = allSendMailLabel[1]
driver.execute_script("arguments[0].click();", allSendMailLabel)
If you see the question here Debugging "Element is not clickable at point" error, there is a bug in the chromedriver that causes this. The issue for it was created here. There is a workaround listed in the 27th comment, but what you can do is switch to the firefox driver and see if that works. Minimised windows should not cause a problem otherwise.
I'm developing a data scraping script using Selenium and Python and I've got the following line of code which works on my Ubuntu PC, my wife's Windows PC but not my colleague's Windows PC.
ActionChains(driver).context_click(inputElement).send_keys(Keys.ARROW_DOWN).click().perform()
We're in the UK and he is in Canada is the only difference I know of.
EDIT
What I've realised is happening, but I do not know how to resolve, is the Keys.ARROW_DOWN is being performed on the webpage and not on the context_click element. Any thoughts? Is this a Chrome issue?
That might be a timing issue related to the quality of the network connection to the target site.
I'd strengthen it by introducing an Explicit Wait for the inputElement, use the WebDriverWait with element_to_be_clickable Expected Condition before starting the action chain.
You may also add a "scroll into view" of the element to tackle potential browser/resolution/etc specific layout issues:
driver.execute_script("arguments[0].scrollIntoView();", inputElement)
Solution:
I rewrote using Firefox instead of Chrome and it is working fine.
So I have a website which has a combo that I need to select an item from, now the problem is it's a bit untraditional and doesn't have option's as elements but instead it has divs.
So I need my program to click combo box then wait (the best way to do this I found is via implicitly_wait(3)# 3 seconds) and then click the box element I need. Firefox is doing a great job with it but PhantomJS seem to throw:
Selenium.common.exceptions.ElementNotVisibleException:
Message: 'Error Message => \'Element is not currently visible and may not be manipulated\'
I'm not sure what's the cause of it, but I suspect that PhantomJS fails to correctly wait via implicitly_wait for some reason and tries to select non-visible element.
Any idea how to approach this without forced thread sleep?
Yup, your issue sounds exactly something I've just fixed in a UI test that was starting to anoy me. Quite a complex one, that passed on all browsers, except my favorite PhantomJs (which is the fastest).
It was quite anoying, when in the debugger I could clearly see that even the parent element was set to visible. Only the prime faces component I needed to click for whatever reason (not CSS or active Styles) was not visible.
After looking at Phantom JS screenshot, I realized the window was quite tiny, and indeed the UI element was not visually visible.
You see now where I am geting at?
Phantom JS is a headless browser, it does not render you the window, but it does use a window, and the window does have a size.
The exception is quite unexpected, because the engine behind is the same as the one used for chrome and safari ... things should just work.
Well, and they did in the end.
Try to tune your driver factory like this:
WebDriver driver = new PhantomJSDriver();
driver.manage().window().setSize(new Dimension(800, 600));
return driver;
Good luck.