I have a web scraper using Selenium in Python 3, I need to record a video of the session in order to do some debugging. I first create Xvfb virtual display:
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1000, 1000))
display.start()
then I create chromium session:
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get("myURL.com")
#
#some web scraping happening here....
#
driver.quit()
display.sendstop()
import sys
sys.exit()
My problem is, it sometimes doesn't stop the driver and display, I want to record a video in order to see what's happenin. Any ideas on how to achieve that?
Create a new display (you already do that)
Start video recording.
Start browser via webdriver.
Wait until browser completed you scenario.
Stop video recording.
Destroy display.
Related
I'm trying a simple browser operation where I locate a username element on a website and then try to login. I'm using selenium and python to do this. Here's some simple code that works on my own local machine. The code opens a browser on my computer and then navigates to the correct username box and enters the username.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get(my_url)
username_element = browser.find_elements_by_name("USERNAME")[0]
username_element.clear()
username_element.send_keys(my_username)
However, when I try to deploy the same code on an AWS server using pyvirtualdisplay so that Firefox doesn't need to pop up, it no longer works.
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
browser = webdriver.Firefox()
browser.get(my_url)
username_element = browser.find_elements_by_name("USERNAME")[0]
username_element.clear()
username_element.send_keys(my_username)
The element is definitely found, but I get the element not visible error:
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
which is confirmed by:
>> username_element.is_displayed()
False
I've tried various things I found on SO including:
making sure xvfb and xephyr are installed
adding a browser.implicitly_wait(30)
trying a WebDriverWait(browser,30).until(EC.visibility_of_element_located((By.NAME, "USERNAME"))) which times out
Any ideas on how to solve this?
You can scroll screen:
browser.execute_script("window.scrollTo(0, 600)")
Figured it out after taking a screenshot. Turns out my screen display wasn't set large enough. Changing the display size to (1600,900) solved the problem.
I want to automate the navigation of a page, scan its QR code and then interact with it.
I am doing this with selenium and python.
But I don't see any display to scan the QR code,
this is how my code begins:
from pyvirtualdisplay import Display
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
display = Display(visible=0, size=(800, 600))
display.start()
browser = webdriver.Chrome()
What shall I do to see the display and scan QR code and resume the program?
I read examples of using Selenium on Windows in which the browser shows up automatically.
The reason is that you are using a virtual display ( i.e. headless browser ), remove these two lines and you must be able to see the browser.
display = Display(visible=0, size=(800, 600))
display.start()
Since you don't need the virtual display, remove the first line as well.
I am running Selenium python on server where I need to hide chrome display. Python script runs most of time but sometimes it stucks when it creates new chromedriver session. Don't have any idea why it stucks sometimes.
Snippet Code:
from selenium import webdriver
from pyvirtualdisplay import Display
chromedriver = '/usr/local/bin/chromedriver'
os.environ['webdriver.chrome.driver'] = chromedriver
display = Display(visible=0, size=(800,600))
display.start()
driver = webdriver.Chrome("/usr/local/bin/chromedriver") => Stuck here
driver.get("example.com")
I just currently set up Selenium on my server. If you get your permission right, try to put this line.
options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
driver = webdriver.Chrome(desired_capabilities=options.to_capabilities())
to turn off the sandbox.
I expect to be able to save a page and then use a lxml.html.parse() but I was wondering if I could do it directly off a opened page?
I'm using Ubuntu if it makes any difference.
Edit: There's a method to use xpath directly(find_element_by_xpath), so I guess I don't need lxml. But to save the page all you have to do is call the page_source method.
To answer the 'use Selenium without spawning a visible window' question, yes you can use PyVirtualDisplay on Ubunutu easily.
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
Code is from this blog post
I'm doing remote web crawling and scraping, and hoping not to reload a new browser window for every link on one page.
The problem is that new tabs are not opening up with my Firefox web driver.
Here's what I've tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from pyvirtualdisplay import Display
# launch our headless display
display = Display(visible=0, size=(800, 600))
display.start()
# launch our web driver and get a page
browser = webdriver.Firefox()
browser.get("http://www.google.com/")
# try to open a new tab
ActionChains(browser).key_down(Keys.CONTROL).send_keys("t").key_up(Keys.CONTROL).perform()
# this should print 2, but it prints 1, because new tab not opened
print len(browser.window_handles)
# clean up everything
browser.quit()
display.stop()
Specifications:
Ubuntu 14.04.2
Python 2.7.6
Selenium 2.47.1
PyVirtualDisplay 0.1.3
Based on this response from a Selenium developer, new tabs in Firefox are not supported as of August 2015. He suggested exploring Marionette but currently its dependencies cause more trouble than it's worth, at least for my use case. His solution is to just use new windows (driver.execute_script("window.open()")), instead of new tabs.