New to Stack Overflow posting so apologies in advance.
I am trying to use selenium + headless chrome to perform browser automation, but in initial testing noticed some odd behavior with saving cookies and other local data when comparing headless vs non headless chrome instances. For reference, my chrome version is: Version 87.0.4280.66 (Official Build) (64-bit), my chromedriver version is 87.0.4280.88. And I'm using 64-bit Centos8 to run this on.
For example, take my two pieces of code I was testing, and see the folder structure as a result in the attached image (note the difference in headless=True):
self.driver_options = Options()
self.driver_options.headless = True
self.driver_options.add_argument("--user-data-dir=/home/<user>/Projects/.headless")
self.driver_options.add_argument("--window-size=1200,1200")
self.driver = webdriver.Chrome('../bin/chromedriver',options=self.driver_options)
self.driver_options = Options()
self.driver_options.add_argument("--user-data-dir=/home/<user>/Projects/.nonheadless")
self.driver_options.add_argument("--window-size=1200,1200")
self.driver = webdriver.Chrome('../bin/chromedriver',options=self.driver_options)
Here's a picture comparing directory structure of headless (left) and non-headless (right)
I read that this may have something to do with headless chrome instances using a different data directory, so maybe some of the files are 'missing' due to that (in another location) - but the biggest issue is that the [user-dir]/Default/Cookies file is populated in the non-headless version, but unpopulated in the headless version. I've also tried to remedy it by adding a "remote-debugging-port" option, but still got the same outcome.
I want to run a few different headless threads, where cookies are important for long running jobs. Has anyone seen anything like this, or know why I am experiencing this behavior? Is it to be expected?
Thanks all
Related
I am using python to interact with webpages via selenium. I am using the geckodriver Firefox driver (not interested in Chrome solutions).
I want to zoom the browser window, equivalent to pressing ctrl+- (or cmd+- on Mac OS).
Granted, there are lots of questions on this topic. I have tried to do my due diligence in digging through them before posting here. The following are solutions that do not work for me (let driver be a WebDriver instance running with geckodriver):
No effect:
driver.execute_script("document.body.style.zoom='zoom 90%'")
from selenium.webdriver.common.keys import Keys
body = driver.find_element_by_tag_name('body')
body.send_keys(Keys.COMMAND, Keys.SUBTRACT)
from selenium.webdriver.common.action_chains import ActionChains
actor = ActionChains(driver)
actor.key_down(Keys.COMMAND).key_down(Keys.SUBTRACT).perform()
Wrong effect:
driver.execute_script("document.body.style.transform='scale(0.9)'")
driver.execute_script("document.body.style.MozTransform='scale(0.9)'")
Error:
driver.set_context("chrome")
win = driver.find_element_by_tag_name("window")
win.send_keys(Keys.CONTROL + "-")
driver.set_context("content")
(raises NoSuchElementException)
Related questions:
stackoverflow.com/questions/28111539
stackoverflow.com/questions/60941346
stackoverflow.com/questions/67699434
stackoverflow.com/questions/56193465
stackoverflow.com/questions/32135085
...and many others.
The only way I have found that actually works is to use pyautogui, however I would rather not install additional 3rd-party libraries, & I don't believe I'll be able to run this in a headless state without a lot of hassle.
However, it seems there should be some way to do this using Selenium/geckodriver's native functionality, rather than installing additional 3rd-party packages for this purpose.
Is there a way??? The abundance of questions on this topic makes it clear that I am not the only one to have had trouble with this...
EDIT: Testing on Python 3.9, Selenium 3.141.0, geckodriver 0.29.1, Firefox 90.0, Mac OS
In Python 3.7.7, selenium 3.141.0, geckodriver 0.29.1, firefox 90.0, Ubuntu 20.04.2 LTS, I combined 2 of your items and made it work.
I could not find the window tag either, but I found body. After I got the url (and only after, otherwise I got an error), I had to do the set context to "chrome." I don't know why, but if not, as you observed, nothing happened. Then I was able to send control + or control - with the desired effect (I took two + to really see the difference, but the 110% and then 120% also shows).
driver.get(URL)#URL is a constant containing the url
driver.set_context("chrome")#needed to make keys work, must come after there is a context
win = driver.find_element_by_tag_name("body")#there is a body even if not a window
win.send_keys(Keys.CONTROL, "+")#zoom to 110%
win.send_keys(Keys.CONTROL, "+")#zoom to 120%
I've been trying to wrap my brain around this problem. I have a webscraping project with Selenium and my script works when headless mode is disabled. Once I add the argument ("--headless") to my chrome driver options (chrome driver version 88.0.4324) it isn't able to click the element. I have seen other people mentioning that maximizing the window or setting the window size of the headless driver will solve this, I have had no luck with such solutions.
This is the line that clicks the element:
browser.find_element_by_xpath('//a [#href][#class="button icon arrowdown"]').click()
This is what I was using before the above xpath:
browser.find_element_by_class('button icon arrowdown').click()
Thanks for Everyones Help!
I had same issue tried using javascript click
driver.execute_script("document.getElementsByClassName('button icon arrowdown')[0].click()")
Take a screenshot and snapshot of the headless window when exception occurs. That will help you to find out why this happens. You can try to use selene library https://github.com/yashaka/selene which already have these features, or write your own snap-shooter.
Depending on this quote
I have seen other people mentioning that maximizing the window or
setting the window size of the headless driver will solve this, I have
had no luck with such solutions.
you need to know, that maximizing in headless - doesn't work because it headless and doesn't know which screen size is to maximize.
You should take the dimensions of the browser in non-headless mode and set it up directly in selenium both for head and headless mode. This can solve your issue. And even if not - try to take screenshot or make a video to figure out what's going on in headless mode.
I need a browser in Selenium to work as if it's maximized, so that website that I process perceived it as maximized, but I want it really to be minimized, so that I could work in parallel while the script executes. The simple: driver.maximize_window() just maximizes windows.
So, is there a way for a window of browser to be maximized for website, but in reality minimized?
Several approaches here, depending on what your needs are.
First, most browser allow setting screen size when they are fired up. See details here How do I set browser width and height in Selenium WebDriver? Chrome, for example, can be controlled from outside of the selenium script either with ini files where you set your screen size properties or via command line arguments. See this page for specific args for chromium for example: https://peter.sh/experiments/chromium-command-line-switches/
Secondly, you could start your browser in headless mode and restrict it a specific size. If you do so, your code will run in the headless browser, meaning you can't see it on a real screen. See the miriad of stackoverflow question addressing this, including this one: How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
Thirdly, you could start a virtual display of a certain size and have your driver maximize the browser in all the tests. Tools like xvfb are a god send here. How do I run Selenium in Xvfb?
Lastly, you could use a real display that's set in the resolution/dpi you need. See answers like this Resize display resolution using python with cross platform support
You can try just running it in a small window instead of Maximized or Minimized.
Don't tell the browser to do anything in terms of size.
So where you tell Selenium
browser.maximize_window()
Erase or comment it out, and run the script that way.
It's either that or use multiple screens.
If you want to work while WebDriver is executed, you can created a virtual instance (with e.g. VirtualBox or HyperV) an run test suite in VM.
Maximized is just a browser size from the site perspective. Set the browser size to the screen resolution of your desktop and minimize the browser.
I'm developing a data scraping script using Selenium and Python and I've got the following line of code which works on my Ubuntu PC, my wife's Windows PC but not my colleague's Windows PC.
ActionChains(driver).context_click(inputElement).send_keys(Keys.ARROW_DOWN).click().perform()
We're in the UK and he is in Canada is the only difference I know of.
EDIT
What I've realised is happening, but I do not know how to resolve, is the Keys.ARROW_DOWN is being performed on the webpage and not on the context_click element. Any thoughts? Is this a Chrome issue?
That might be a timing issue related to the quality of the network connection to the target site.
I'd strengthen it by introducing an Explicit Wait for the inputElement, use the WebDriverWait with element_to_be_clickable Expected Condition before starting the action chain.
You may also add a "scroll into view" of the element to tackle potential browser/resolution/etc specific layout issues:
driver.execute_script("arguments[0].scrollIntoView();", inputElement)
Solution:
I rewrote using Firefox instead of Chrome and it is working fine.
Given this python code:
import webbrowser
webbrowser.open("http://slashdot.org",new=0)
webbrowser.open("http://cnn.com",new=0)
I would expect a browser to open up, load the first website, then load the second website in the same window. However, it opens up in a new window (or new tab depending on which browser I'm using).
Tried on Mac OSX with Safari, Firefox and Chrome and on Ubuntue with Firefox. I'm inclined to believe that new=0 isn't honored. Am I just missing something?
tia,
Note that the documentation specifically avoids guarantees with the language if possible: http://docs.python.org/library/webbrowser.html#webbrowser.open
Most browser settings by default specify tab behavior and will not allow Python to override it. I have seen it in the past using Firefox and tried your example on Chrome to the same effect.
On Windows, it is not possible to specify the tab behavior at all, as suggested by my comment below. The url opening code ignores new:
if sys.platform[:3] == "win":
class WindowsDefault(BaseBrowser):
def open(self, url, new=0, autoraise=True):
try:
os.startfile(url)
I added a delay between successive invocations of webbrowser.open(). Then each was opened in a new tab instead of a separate window (on my Windows 10 machine).
import time
...
time.sleep(0.5)