I'm making a web scraping program, but to keep from being blocking by anti-scraping software I gotta keep the driver out of headless mode. sometimes I also need to restart the driver to clear the cookies, so whenever it opens back up I minimize it immediately, but it still gets in the way of whatever Im doing for about a second, and this program runs for hours so its incredibly annoying
Im thinking theres probably something I can add like driver.add_option("start in minimized") or driver.add_option("start in off screen") (equivalent to driver.set_window_position(-2000,0) but before the driver opens
does anyone know of an options setting I can add for this?
heres what my current code looks like, though this is more of a feature question than a bug fixing problem
import undetected_chromedriver.v2 as uc
def start_uc():
'''opens a UC chrome driver with prefered settings'''
options = uc.ChromeOptions()
# just some options passing in to skip annoying popups
options.add_argument('--no-first-run --no-service-autorun --password-store=basic')
return uc.Chrome(options=options)
driver = start_uc()
driver.minimize_window()
for item in list:
#scrape item info
if blocked from page:
driver.close()
driver = start_uc()
driver.minimize_window()
Unfortunately there is no such option / capability.
You can set driver to start maximized with
options.add_argument("start-maximized")
But there is no such option as options.add_argument("start-minimized").
As you mentioned, you can minimize the driver screen immediately after it's creating with driver.set_window_position(-2000,0) but, again, this will be applied only after the driver is opened.
Related
This code currently used to manipulate Chrome..
option = Options()
option.add_argument("--incognito")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=option)
But this code makes to open a new chrome window.
What I want is to manipulate the chrome that's already open, is it possible?
Somebody say, Using Chrome in debug mode is answer, but I don't think that's a good choice.
I want to open chrome without any control.
I want to create an application which will take a keyword as input and search that on youtube and then scrape the links and save them and in a Notepad file, and all of this works in background. I am familiar about BeautifulSoup library, and selenium but I want it to work in background unlike Selenium which works in front of us. I hope the question is clear, if not you may ask.
I am familiar about Selenium, but I want to automate the search in background.
from selenium import webdriver
driver=webdriver.Chrome("C:\\Users\\MyPC\\Downloads\\chromedriver_win32\\chromedriver.exe")
driver.set_page_load_timeout(10)
driver.get("http://www.youtube.com")
driver.find_element_by_name("search_query").send_keys("Selenium Tutorial")
driver.find_element_by_id("search-icon-legacy").click()
time.sleep(4)
driver.quit()
This code opens the browser and then performs the search, but I want everything to happen in background and fast without and delay.
You can run browser with option --headless and it will not display its window. It works for Firefox and Chrome.
Firefox
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
#options.headless = True
options.add_argument('--headless')
driver = webdriver.Firefox(options=options)
driver.get('https://stackoverflow.com')
driver.save_screenshot('screenshot-firefox.png')
driver.close()
Chrome
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
#options.headless = True
options.add_argument('--headless')
#options.add_argument('--disable-gpu')
driver = webdriver.Chrome(options=options)
driver.get('https://stackoverflow.com')
driver.save_screenshot('screenshot-chrome.png')
driver.close()
There was webdriver PhantomJS which simulated headless web browser but it is not developed any more. This code still runs but it gives me empty page_sourceand empty file screenshot.png
from selenium import webdriver
driver = webdriver.PhantomJS()
driver.get('https://stackoverflow.com')
print(driver.page_source)
driver.save_screenshot('screenshot.png')
driver.close()
On Linux you could use Xvfb to create fake/virtual monitor and program can use it to display its window. This way you don't see this window on your screen.
All this methods has to render page so it may not run faster.
To scrape faster you would have to analyze requests/responses from/to web browser and do the same with python module requests - but it is not ease. But this way program doesn't have to render page and run JavaScript so it will run a lot faster.
But then you may have another problem - if you make request too offen (too fast) then server can block you and you need proxy servers to have different IPs.
theres a workaround using pyvirtualdisplay also if you want to hide the selenium browser, dont forget to close the browse when you're done
i think webhosts can detect headless browsers, so u may get different results
stop loading the page after u get what you're looking for and/or close the browser after you save the source, insert javascript
When my script opens the chromedriver (driver = webdriver.Chrome()) in Windows 10, the window maximizes to full size on my monitor. To minimize disruption, I would prefer it was barely noticeable. The closest I can get is the following:
driver = webdriver.Chrome()
driver.set_window_size(1, 1)
This works well in that the window size is minimal. Unfortunately, it still first pops open to the full monitor and then readjusts to the preferred minimal size. Is there any way to preset the window size before opening up the chromedriver?
You can place driver.set_window_size(1,1) in the setup method for example:
class ExampleTestSuite(unittest.TestCase):
def setUp(self):
self.driver= webdriver.Chrome()
self.driver.set_window_size(1,1)
def test_example_test(self):
#test code here
What I think you really are looking for is running your tests headless. One way to do it this describe in this post: http://fgimian.github.io/blog/2014/04/06/headless-browser-testing-with-chrome-and-firefox/.
Another option is to use a cloud based testing service like Sauce Labs, or Browserstack.
I actually came across a pretty good alternative to #user3277225's answer. It turns out you can set options for the Chrome webdriver and then open it as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("window-size=1,1")
driver = webdriver.Chrome(chrome_options=chrome_options)
While scraping a page using selenium webdriver, there is a "pop up" that appears .
On Opening the page, http://www.fanatics.com/search/red%20shoes - I see a popup window with xpath '//*[#id="mm_DesktopInterstitialImage"]' - but I don't want to be using the xpath to close this alert, and have something genric that can dismiss/close the alert.
Here's what I tried so far -:
from selenium import webdriver
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
chromedriver = "/usr/local/CHROMEDRIVER"
desired_capabilities=DesiredCapabilities.CHROME
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver,desired_capabilities=desired_capabilities)
url='http://www.fanatics.com/search/red%20shoes'
driver.get(url)
#driver.set_page_load_timeout(300)
driver.implicitly_wait(60)
alert = driver.switch_to_alert()
alert.dismiss
handle=driver.window_handles
print handle
#[u'CDwindow-B1E9C127-719D-ACAA-19DE-1A6FA265A4FF']
From what I understand from related examples, folks usually switch window handles, but my handle has a single element.i.e, driver.switch_to_window(driver.window_handles[1]) then driver.close() and finally shift again, using driver.switch_to_window(driver.window_handles[1]) I also used implicit wait, since I was not sure, if the alert window was being read at-all - but that did not work either. I do not wnat to hard-code for the xpath,if that is possible.
What am I doing wrong ?
Related, but does't work for me : Selenium python how to close a pop up window?
As I can see, it's not an alert box!!. It is just a Simple Pop-up that appears when you are entering the page and it is present in main window itself(no need of switching and closing it too). Use the below code to close it.
driver.find_element_by_xpath('//div[contains(#class,"ui-dialog") and #aria-describedby="dialogContent2"]//button[#title="Close"]').click()
It works perfectly, give it a try ;)
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--disable-notifications")
browser = webdriver.Chrome('C:\Python34\Lib\site-packages\selenium\webdriver\chromedriver.exe', chrome_options=options)
The popup you are trying to close is not a browser alert but a DOM popup. A browser alert is a dialog box that the browser creates at the OS level. (It is appears as an additional window on your desktop.) You can use .switch_to_alert() to close these alerts. A DOM popup is just an HTML element that happens to be used the the page as if it were a dialog box. This popup has existence only in your browser. (The OS does not know about it.) You cannot use .switch_to_alert() to close DOM popups.
To get this popup and close it, you have to inspect the DOM (like you've started doing), find the DOM element that represents the close button and have Selenium click it.
I ran into a similar problem, except that our pop-up's closebox webelement had a dynamic id tag, making it difficult to latch onto. We ended up solving it by using By.className to find it. In the code below, I use a List because I was also looking to see how many elements I was dealing with, but that's not required. Also, the production code has additional handling in case window closebox elements were found.
List<WebElement> closeboxes = driver.findElements(By.className("v-window-closebox"));
for (WebElement we : closeboxes) {
we.click();
}
You can try the following:
from selenium.webdriver import ActionChains, Chrome
driver = Chrome('/usr/local/CHROMEDRIVER')
ActionChains(driver).move_to_element_with_offset(
driver.find_element_by_xpath('//html'), 0, 2338
).click().perform()
This will click an area outside of the popup thus closing it. ðŸ¤
I'm currently writing a visual test that require adblock to be installed to stop some ads from ruining the test. However when I run the Chromedriver with the adblock extension installed it brings up the successfully installed adblock tab. This tab screws up my screenshots because the driver is working in a previous tab that opens. Is there any way to stop the adblock tab from opening or prevent the black screenshot from happening? Here is the initialization for the driver.
def setUp(self):
chrome_options = Options()
chrome_options.add_extension('AdBlock_v2.6.4.crx')
self.driver = webdriver.Chrome('chromedriver', port=0, chrome_options=chrome_options)
self.driver.implicitly_wait(15)
handles = self.driver.window_handles
self.driver.switch_to_window(handles[-1])
seems to be the best way to handle it. It simply tells the driver to switch to the tab which is considered a window. I feel silly for trying to find a more complex solution