I am trying to scrape job titles from a dynamic job listing. When I use the function find_elements_by_class_name, the function doesn't return anything. I'm new to selenium so i'm not sure if i'm simply doing something incorrectly or misunderstanding the functionality.
The page im trying to scrape is: https://recruit.hirebridge.com/v3/CareerCenter/v2/?cid=7724
from selenium import webdriver
import time
#define the path for the chrome webdriver
chrome_path = r"C:/web/jobListing/chromedriver.exe"
#create a instance of the webdriver
driver = webdriver.Chrome(chrome_path)
driver.get("https://recruit.hirebridge.com/v3/CareerCenter/v2/?cid=7724")
time.sleep(10)
jobs = driver.find_elements_by_class_name("col-md-8 jobtitle")
print("starting print")
for job in jobs:
print(job.text)
Root Cause:
col-md-8 and jobtitle are 2 different classes. When you use find_element_by_class_name it will internally convert the class name to a css selector and try to find the element.
Below is the evidence where find_element_by_class_name uses the css internally.
Solution:
As Selenium internally uses the css you have to make sure the classes are clubbed together meaning class1.class2.class3. In simple terms replace all white spaces with single dot in the class name from UI.
How to implement this to your situation:
You have to use the below syntax.
driver.find_element_by_class_name('col-md-8.jobtitle')
Try:
jobs = driver.find_elements_by_xpath("//div[#class='col-md-8 jobtitle']/a")
I've switched the find element by class for xpath, this way you have more flexibility and it generally works better, I suggest you look into it!
Seems like a bug?
This works:
jobs = driver.execute_script("""
return document.getElementsByClassName("col-md-8 jobtitle")
""")
if I launch chrome webdriver from a python function why does it automatically close the browser window after execution and how do I prevent this?
Here's the code:
from selenium import webdriver
def open_chrome_driver():
chrome_driver = webdriver.Chrome(executable_path=r'C:/Users/User/Documents/pythonfiles/chromedriver.exe')
return chrome_driver
open_chrome_driver()
Because the python runtime will clean up all the resources it allocated for its use when the script ends.
put a breakpoint in the last line of code you want to be executed and run it in debug mode (depends on your IDE). Once it pauses, you can do whatever you want with it.
Please try the following - it should detach chrome process from chromedriver and prevent from closing.
chrome_options.add_experimental_option("detach", True)
Hope this solves your issue.
You will need to import Options from selenium.webdriver.chrome.options import Options
And off course as Davis Jahn said - you may put a breakpoint
It worked for me to simply return the chromedriver object back to a variable with the same name
chrome_driver = open_chrome_driver()
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver= webdriver.Chrome()
driver.get("http://www.dsvv.ac.in/")
scroll_link= driver.find_element_by_link_text("Skill Development Workshops (CCAM) 2017-18")
scroll_link.click()
driver.close()
Actually want to get test results for this code in pycharm...But I can't found the option "Go to | Test" on the context menu for this code.
Disclaimer: Please note that this only answers the question "why won't PyCharm let me create tests for my code with the context menu".
PyCharm will offer the "Go to | Test" context menu option when you are clicking inside something it deems suitable to create tests for, or tests already exist.
That could be a class or a function for example.
Your code doesn't contain anything PyCharm will recognize as candidate to create a test for, so it doesn't show the option.
Try to modify your code:
from selenium import webdriver
def selfun():
driver= webdriver.Chrome()
driver.get("http://www.dsvv.ac.in/")
scroll_link= driver.find_element_by_link_text("Skill Development Workshops (CCAM) 2017-18")
scroll_link.click()
driver.close()
if __name__ == "__main__":
selfun()
Click on (or somewhere inside) the selfun function, and PyCharm will offer to create a test when you select "Go to | Test".
This question already has answers here:
How to select a drop-down menu value with Selenium using Python?
(18 answers)
Closed 8 years ago.
first time posting!
I'm brand new to Python and Selenium, but I'm trying to automate a basic test and I can't find the answer to this problem.
On the main ebay.com page, I'm trying to choose the "All Categories" dropdown menu and choose the "Dolls & Bears" option (option value = "237"). When I execute my script, the menu is accessed, but the "Dolls & Bears" option does not get selected. The test does not return any errors. I have also tried using select_by_visible_text.
Here is my code. I appreciate any help!
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.ebay.com")
assert "Electronics" in driver.title
elem = driver.find_element_by_id("gh-ac")
elem.send_keys("funny bear")
driver.find_element_by_id("gh-cat").click()
def select_a_value(select):
Select.select_by_value("237").click()
Welcome to Stack Overflow!
You are very close with this code. "Select" is a class that can be instantiated, not just a library of functions. Here's a working version of your script:
from selenium import webdriver
from selenium.webdriver.support.select import Select
import time
driver = webdriver.Firefox()
driver.get("http://www.ebay.com")
assert "Electronics" in driver.title
elem = driver.find_element_by_id("gh-ac")
elem.send_keys("funny bear")
dropdown_web_element = driver.find_element_by_id("gh-cat")
select_box = Select(dropdown_web_element)
time.sleep(1)
select_box.select_by_value("237")
Notice that the call to Select passes in a value (a WebElement in this case) to instantiate the object. You can look at the selenium source code to figure out how it's called (Python27/Lib/site-packages/selenium/webdriver/support/select.py on Windows).
Also, I added a time.sleep(1) in there. You have stumbled on one of the frustrating things with selenium. Asynchronous loading of data on websites can lead to tests failing because they run faster than humans typically click buttons. The data might not be there when you get to that point in the script! The right way to deal with this is to dynamically wait a reasonable amount of time until the element you are looking for is there. A bit outside the scope of what you asked, but you will be dealing with that problem soon enough, I'm sure.
Good luck!
When I execute multiple test simultaneously, i don't want to keep Firefox browser window visible.. I can minimize it using selenium.minimizeWindow() but I don't want to do it.
Is there any way to hide Firefox window? I am using FireFox WebDriver.
Python
The easiest way to hide the browser is to install PhantomJS. Then, change this line:
driver = webdriver.Firefox()
to:
driver = webdriver.PhantomJS()
The rest of your code won't need to be changed and no browser will open. For debugging purposes, use driver.save_screenshot('screen.png') at different steps of your code or just switch to the Firefox webdriver again.
On Windows, you will have to specify the path to phantomjs.exe:
driver = webdriver.PhantomJS('C:\phantomjs-1.9.7-windows\phantomjs.exe')
Java
Have a look at Ghost Driver: How to run ghostdriver with Selenium using java
C#
How to hide FirefoxDriver (using Selenium) without findElement function error in PhantomDriver(headless browser)?
Just add the following code.
import os
os.environ['MOZ_HEADLESS'] = '1'
driver = webdriver.Firefox()
Finally I found the solution for those who are using windows Machine for running the Tests using any method. Well, implementation is not in Java, but you can do it very easily.
Use AutoIt tool. It has all the capability to handle windows. It is a free tool.
Install AutoIt:
http://www.autoitscript.com/site/autoit/downloads/
Open the Editor and write below code
for Hiding any window.
AutoItSetOption("WinTitleMatchMode", 2)
WinSetState("Title Of Your Window", "", #SW_HIDE)
To Unhide it, you can use below line of code.
AutoItSetOption("WinTitleMatchMode", 2)
WinSetState("Title Of Your Window", "", #SW_SHOW)
WinTitleMatchMode has different options which can be used to match Windows title.
1 = Match the title from the start (default)`
2 = Match any substring in the title
3 = Exact title match
4 = Advanced mode, see Window Titles & Text (Advanced)
So, what I've done is: I have created an .exe file of a small program and passed a parameter as a command line argument as below.
Runtime.getRuntime().exec("C:/Diiinnovation/HideNSeek.exe 0 \"" + "Mozilla Firefox" + "\"");
in HideNSeek.exe - I have below AutoIt Code:
AutoItSetOption("WinTitleMatchMode", 1)
if $CmdLine[0] > 0 Then
if $CmdLine[1] == 0 Then
WinSetState($CmdLine[2], "", #SW_HIDE)
ElseIf $CmdLine[1] == 1 Then
WinSetState($CmdLine[2], "", #SW_SHOW)
Else
EndIf
EndIf
$CmdLine[] is an array, which will have all command line parameters...
$CmdLine[0] = number of Parameter
$CmdLine[1] = 1st Parameter after Exe Name
...
If there is any space in the Window Title, then you have to use double quotes to pass it as a command line parameter like above.
Below Line of code will execute AutoIt exe and if I pass '0' in 1st parameter then it will hide the window and if I will pass '1' then it will unhide windows matching the title.
Runtime.getRuntime().exec("C:/Diiinnovation/HideNSeek.exe 0 \"" + "Mozilla Firefox" + "\"");
I hope this will help you. Thanks!
Just do (Python):
opts = webdriver.FirefoxOptions()
opts.headless = True
firefox = webdriver.Firefox(options=opts)
I used xvfb to solve the problem like this.
First, install Xvfb:
# apt-get install xvfb
on Debian/Ubuntu; or
# yum install xorg-x11-Xvfb
on Fedora/RedHat. Then, choose a display number that is unlikely to ever clash (even if you add a real display later) – something high like 99 should do. Run Xvfb on this display, with access control off:
# Xvfb :99 -ac
Now you need to ensure that your display is set to 99 before running the Selenium server (which itself launches the browser). The easiest way to do this is to export DISPLAY=:99 into the environment for Selenium. First, make sure things are working from the command line like so:
$ export DISPLAY=:99
$ firefox
or just
$ DISPLAY=:99 firefox
Below there is a link that helped me
http://www.alittlemadness.com/2008/03/05/running-selenium-headless/
The default browser of PhantomJS is IE, though many browser features do not work there. If you want to open a headless(hidden) Firefox window, you can use the new feature of Firefox 56+.
With this feature you can get a headless driver like this:
System.setProperty("webdriver.gecko.driver", firefoxDriverExePath);
FirefoxOptions options = new FirefoxOptions();
options.addArguments("--headless");
FirefoxDriver driver = new FirefoxDriver(options);
New versions of Chrome also have the headless option.
just add these and it will work if you are using chrome, also useful in firefox
from selenium.webdriver.chrome.options import Options
'''option to make driver work background'''
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
If you're using Selenium RC or Remote WebDriver then you can run the browser instance on a remote, or virtual machine. This means that you shouldn't have to worry about hiding the browser windows as they won't be launching on your local machine.
Firefox has a headless mode. If you want to use it, you just have to set it on binary options like this:
binary = FirefoxBinary("C:/Program Files/Mozilla Firefox/firefox.exe")
options = webdriver.FirefoxOptions()
# set headless mode on
options.set_headless(True)
driver = webdriver.Firefox(firefox_binary=binary,options=options)
If you are using KDE Desktop, you can make Firefox Windows to be initially opened being minimized. That made my day to me regarding this problem. Just do the following:
Open Firefox
Click on the Firefox icon on the top left corner of the menu bar -> Advanced -> Special Application Settings...
Go to the "Size & Position" tab.
Click on "Minimized" and choose "Apply Initially" (YES).
These settings will apply for new Firefox windows from now on and you will not be bothered with pop-ups anymore when running tests with Webdriver.
I found the easiest way was to use PhantomJS, per Stéphane's suggestion. I downloaded the binary and put phantomjs in my PATH, in my case (Mac OS) in /usr/bin/. I like to retain the option of seeing what's going on so I wrapped it like this (in Python):
def new_driver():
if 'VISIBLE_WEBDRIVER' in os.environ:
return webdriver.Firefox()
else:
return webdriver.PhantomJS()
References:
http://blog.likewise.org/2013/04/webdriver-testing-with-python-and-ghostdriver/
http://www.realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/
Java
I had a similar problem with ChromeDriver (I needed to minimize the browser window while the tests are running). I could not find a better way to do it, so I ended up using the keyboard combination Alt+Space, N to do it. This should work only in Windows, the example uses the Java AWT Robot class to play the keyboard shortcuts:
//Alt + Space to open the window menu
Robot robot = new Robot();
robot.keyPress(KeyEvent.VK_ALT);
robot.keyPress(KeyEvent.VK_SPACE);
robot.keyRelease(KeyEvent.VK_SPACE);
robot.keyRelease(KeyEvent.VK_ALT);
Thread.sleep(200);
// miNimize
robot.keyPress(KeyEvent.VK_N);
robot.keyRelease(KeyEvent.VK_N);
In Java, you can use HtmlUnitDriver to launch a headless browser session which will not actually open the browser.
Add the following dependency to your pom.xml (or download and reference the following):
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.15</version>
</dependency>
... and test it it as you would a WebDriver driver instance:
driver = new HtmlUnitDriver();
driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);
driver.get("http://www.google.com");
// etc..
driver.quit();
Another similar question in SO: Avoid opening browser on remote server during selenium call
in options (Firefox options, chrome options )
set boolean headless to true by calling set_headless method.