I've been trying to scrape a site using a PhantomJS/Selenium setup in Python.
There's a dropdown in the page that, when selected, populates the options of another dropdown. When I'm automating this with Firefox, when I select_by_visible_text on the first dropdown, the second one gets populated.
However, the same code doesn't work with PhantomJS.
browser = webdriver.PhantomJS(executable_path=PHANTOMJS_PATH)
## browser = webdriver.Firefox()
wait = WebDriverWait(browser, WAIT_TIME)
browser.get(URL)
Select(browser.find_element_by_id('DropDown0')).select_by_visible_text('XXX')
def condition_wait_for_dropdown(driver):
ret = False
if driver.find_elements_by_xpath('//*[#id="DropDown1"]/option'):
ret = True
return ret
wait.until(condition_wait_for_dropdown)
I have tried using a hard-coded sleep instead of the custom waiting condition, and checked the DOM - the second dropdown never gets populated at all.
Can this be a problem with the webpage itself? If so, how do I work around this?
EDIT: Just to clarify, this works with Selenium+Firefox, so the possibility exists that this is a bug in the PJS driver itself.
I found a fix for this, so best to leave it here for posterity.
I had an input field in the page form, so after selecting the first dropdown I moved the focus to the input field, and sent an ENTER to it. The second dropdown got loaded immediately.
Still don't know wth is wrong with this web page.
Related
I have other examples, but for this specific example what the code does is locate a series of elements that each link to a specific page. Then I click on a specific link. Sometimes this code works, but sometimes it does not work and the page just stays the same without clicking on the link.
I have used a variety of methods like sleep, wait and ActionChains in an attempt to make things more consistent, but it still messes up really frequently.
print(f"No.{i} Opportunity found {optitle_list[i]}")
#click on tender
time.sleep(10)
opentender_button =WebDriverWait(driver,30).until(EC.visibility_of_all_elements_located((By.XPATH,"//img[#src = 'esources/theme/images/view.png']")))
ActionChains(driver).move_to_element(opentender_button[int((j-3)/10)]).double_click().perform()
Here is another example. In this code the keys 'health' are not always sent, resulting in the wrong search options being reviewed by my script.
#Enter into Search box ‘health' and hit "Enter" key
time.sleep(20)
search_box = WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,"//label//input[#type='text']")))
ActionChains(driver).move_to_element(search_box).double_click(search_box).send_keys('health').perform()
print("Search box filled")
I'm using Selenium in Python 3 to access webpages, and I want to click on a pop-up button, but I am unable to locate it with Selenium.
What I'm describing below applies to a number of sites with a pop-up, so I'll use a simple example.
url = "https://www.google.co.uk"
from selenium import webdriver
driver = webdriver.Firefox()
driver.implicitly_wait(10)
driver.get(url)
The page has a pop-up for agreeing to cookies.
I want the script to click on the "I agree" button, but I'm unable to locate it.
I've found a few questions and posts about this online (including on Stackoverflow), but all the suggestions I found seem to fall in one of the following categories and don't seem to work for me.
Wait longer for the pop-up to actually load.
I've tried adding delays, and in fact, I'm testing this interactively, so I can wait all I want for the page to load before I try to locate the button, but it doesn't make any difference.
Use something like driver.switch_to.alert
I get a NoAlertPresentException. The pop-up doesn't seem to be an alert.
Locate the element using driver.find_element.
This doesn't work either, regardless of which approach I use (xpath, class name, text etc.). I can find elements from the page under the pop-up, but nothing from the pop-up itself. For example,
# Elements in main page (under pop-up)
driver.find_element_by_partial_link_text("Sign in") # returns FirefoxWebElement
driver.find_element_by_class_name("gb_g") # returns FirefoxWebElement
# Elements on the pop-up
driver.find_element_by_partial_link_text("I agree") # NoSuchElementException
driver.find_element_by_class_name("RveJvd snByac") # NoSuchElementException
The popup just doesn't seem to be there in the page source. In fact, if I try looking at the loaded page source from the browser, I can't find anything related to the pop-up. I understand that many sites use client-side scripts to load elements dynamically, so many elements wouldn't show up in the raw source, but that was the point of using Selenium: to load the page, interpret the scripts and access the end result.
So, what am I doing wrong? Where is the pop-up coming from, and how can I access it?
I'm having trouble logging into a website using Selenium via Python.
I'm new to web scraping and as part of the learning process I'm attempting to web scrape my account activity from American Airlines with Python. This requires logging in, which is where my code (see below) fails. All the form fields are populated, however, when I submit, the page just seems to refresh and clears my entries.
Checks I've perform:
Login information is correct. I've manually logged in.
I've played around with different lengths for sleep time. No success
Cleared form prior to entry. No success
Run the code up to (but no including) the submit() line, and then manually click the "Log In" button. Login still fails. This makes me think that the fields are somehow populated incorrectly, but I can't see where the issue is.
Thanks in advance for any help!
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome()
driver.get("https://www.aa.com/homePage.do")
loginId = driver.find_element_by_name("loginId")
lastName = driver.find_element_by_name("lastName")
password = driver.find_element_by_name("password")
time.sleep(2)
loginId.send_keys("my-email-here")
lastName.send_keys("my-last-name-here")
password.send_keys("my-password-here")
time.sleep(2)
password.submit()
I believe that AA and other airlines have sophisticated bot detection that know you are using selenium to find and manipulate elements in the DOM, causing the page to navigate back to login. Even if I do a:
driver.find()
on the webpage and fill out the fields myself it fails, but if I do just a:
driver.get('https://www.aa.com')
and then fill out the fields it will allow me to continue. I have found a few posts on google and reddit that have gone over this if you do some searching. Also Can a website detect when you are using selenium with chromedriver? , goes over some of the ways they might be doing it.
I have web page, where the user needs to upload a list of documents and only after which the Submit button is enabled.
I am using python selenium for this automation. My program is able to upload the documents but is unable to click on the Submit button after it is enabled.
I tried this:
element = WebDriverWait(driver, 10000).until(EC.element_to_be_clickable("(//button[#type='submit'][2]"))
element.click()
but it is not working, as the jobs are not submitted in the front end
Change code to be:
element = WebDriverWait(driver, 100).until(
EC.element_to_be_clickable((By.XPATH, "//input[contains(#ng-click,'SubmitJob')]")))
element.click()
So we now are waiting for up to 100 seconds (instead of 3 hours), and we are passing a tuple argument (By.XPATH, "//input[contains(#ng-click,'SubmitJob')]") to EC.element_to_be_clickable, which is what it expects.
Cannot comment on the correctness of the xpath though, but please check it as well.
Edit: changed the xpath based on comment. There are many ways to express that xpath. I would prefer ng-click over class attribute, since classes may change; action, however, will likely stay the same. But if you choose using classes, I still suggest going with something like
//input[contains(#class,'btn') and contains(#class,'form-control')]
because you never know if the order of classes will change.
I am trying to login into a website using Selenium. The website is http://projecteuler.net/login.
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('http://projecteuler.net/login')
username = browser.find_element_by_id('username')
username.send_keys(USERNAME_HERE)
password = browser.find_element_by_name('password')
password.send_keys(PASSWORD_HERE)
browser.find_element_by_name("login").submit()
The program is working correctly upto the last statement. I tried omitting the last statement and manually logged in and it worked. But when I added the last statement and ran the program it just seemed to reload the same page minus the information that I had placed via the program.
So it is only the submission that is giving problem. I viewed the source and confirmed whether there is some other element by that name but there was no other element by name "login". So what am I getting wrong here? Do I need to take care of something else also?
There is a weird thing happening. When I have done form submission via code and try to view the source in Google Chrome 33.0.1750.154 m I am getting the below.
Try click() instead of submit()
Submit is particularly useful for forms without submit buttons, e.g. single-input “Search” forms.
Source:-
http://selenium.googlecode.com/git/docs/api/py/selenium/selenium.selenium.html?highlight=submit#selenium.selenium.selenium.submit
In your case there is a submit button, better to just click it.