I'm trying to scrape this with a combination of python 3.5/selenium/phantomjs.
There is a button which loads more offers
<button data-behavior="result-paging" class="button button-text--centered">
Mehr laden
</button>
This exact button is twice inside the HTML-Code and if all offers are listed the first button looks as follows, while the second button stays the same:
<button data-behavior="result-paging" class="button button-text--centered is-hidden">
Mehr laden
</button>
To click the first button I tried:
while True:
try:
time.sleep(4)
btnElements = driver.find_elements_by_xpath("//button[#data-behavior='result-paging']")
for btnElement in btnElements:
btnElement.click()
# btnElement.send_keys("\n")
if i==11:
break
else:
i=i+1
except:
break
and
while True:
try:
time.sleep(4)
elements= driver.find_elements_by_xpath("//button[#data-behavior='result-paging']")
driver.execute_script("arguments[0].click();", elements[0])
if i==11:
break
else:
i=i+1
except:
break
The loop was just there for quick testing purposes since the script would run eternally otherwise. With booth approaches no new content was loaded, so clicks werent working.
Does anybody have an idea how to solve this issus (perform clicks and load other offers)?
Edit:
Apparently this problem is caused by phantomjs, since I could run the same script with chromedriver. Unfortunately I need to run phantomjs. Did anybody experienced this behaviour before?
As per your code block the Locator Strategy which you have used as ...
find_elements_by_xpath("//button[#data-behavior='result-paging']")
... doesn't identifies the visible button with text as Mehr laden uniquely.
Solution
To identify the visible button with text as Mehr laden uniquely you can use eith of the following Locator Strategies :
CSS_SELECTOR :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# lines of code
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[data-psa-scope=tarifflist] button.button.button-text--centered"))).click()
XPATH :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
# lines of code
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#data-psa-scope='tarifflist']//button[#class='button button-text--centered']"))).click()
Related
I'm using Selenium Python to do the same action for different users and the flow is as follows,
1- the webpage show list of user ids
2- I search for 1 user-id (Iteration)
3- I click on the check box next to the user-id
4- I click on done
My issue is with the check box, it is clickable in the first row of iteration but then it becomes unclickable with the next row.
Checkbox Element:
<input type="checkbox" class="ant-checkbox-input" value="" xpath="1">
I have also tried ActionChains but I got the error: stale element reference: element is not attached to the page document
Thank you
the input checkbox might be in disable state or some javascript load is yet to happen before checkbox become availabe.
try this code:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.XPATH,"//tbody/tr[1]/td[1]/span[1]/label[1]/span[1]/input[1]"))).click()
another problem might be xpath changes depending on user you logged in.
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.XPATH,'//input [#type="checkbox"]'))).click()
above code will work only if you have 1 checkbox.
This code will try to click all the check boxes present on page, if this works we could narrow down XPath.
els=driver.find_elements_by_xpath('//input [#type="checkbox"]')
for el in els:
try:
el.click()
except:
pass
I'm trying to use Selenium w/ Python to click on answers to problems posted at a tutoring site, so I can take tests over the command line.
I enter the following:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver.get('https://www.varsitytutors.com/sat_critical_reading_diagnostic_1-problem-10821')
(an annoying popup comes up at this point -- we can ignore that for now)
Answers on the page are embedded in forms like this:
<div class="question_row">
<form class="button_to" method="post" action="/problem_question_answers/save_answer?answer_id=539461&problem_id=5065&qotd=false&question_id=10821">
<input id="answer_539461" class="test_button" type="submit" value="select" /><input type="hidden" name="authenticity_token" value="LE4B973DghoAF6Ja2qJUPIajNXhPRjy6fCDeELqemIl5vEuvxhHUbkbWeDeLHvBMtEUVIr7okC8Njp4eMIuU3Q==" /></form>
<div class="answer">
<p>English dramatists refused to employ slang in their work.</p>
</div>
<div style="clear:both"></div>
</div>
My goal is to click an answer such as this one in order to get to the next question using Selenium.
I thought it might be as easy as doing this:
answer_buttons=driver.find_elements_by_class_name('test_button')
answer_buttons[1].click()
But I get error messages saying that the element is out of the frame of the driver.
I've also tried submitting the form, which doesn't produce an error message:
answer_forms=driver.find_elements_by_class_name('button_to')
answer_forms[1].submit()
But it redirects to a different url that doesn't load:
http://www.varsitytutors.com/sat_critical_reading_diagnostic_1-problems-results-d9399f1a-4e00-42a0-8867-91b1c8c9057d
Is there any way to do this programmatically, or is the website's code going to prevent this?
Edit:
With some help I was able to click the button once initially. But an identical submit button (by xpath) for the next question remains unclickable. This is the code I'm presently using:
driver.get('https://www.varsitytutors.com/practice-tests')
# click subject
subject=driver.find_element_by_xpath('/html/body/div[3]/div[9]/div/div[2]/div[1]/div[1]')
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[3]/div[9]/div/div[2]/div[1]/div[1]')))
subject.click()
# select specialty
specialty=driver.find_element_by_xpath('/html/body/div[3]/div[9]/div/div[2]/div[2]/div[1]/div[2]/a[4]')
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[3]/div[9]/div/div[2]/div[2]/div[1]/div[2]/a[4]')))
specialty.click()
# select test
taketest=WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[3]/div[8]/div[3]/div[1]/div[1]/a[1]')))
driver.execute_script("arguments[0].click();", taketest)
# click away popup
button=WebDriverWait(driver,5).until(EC.element_to_be_clickable((By.XPATH,"//button[contains(.,'No Thanks')]")))
button.location_once_scrolled_into_view
button.click()
# select first choice
choice=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[3]/div[7]/div[1]/div[3]/div[1]/form/input[1]')))
driver.execute_script("arguments[0].click();", choice)
I repeat this code again over the next few lines. It has no effect, however; the drive stays on question two and the next few clicks don't work...
choice=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[3]/div[7]/div[1]/div[3]/div[1]/form/input[1]')))
driver.execute_script("arguments[0].click();", choice)
choice=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[3]/div[7]/div[1]/div[3]/div[1]/form/input[1]')))
driver.execute_script("arguments[0].click();", choice)
Try the following code.This will handle pop-up and click on the select button.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
driver=webdriver.Chrome()
driver.get('https://www.varsitytutors.com/sat_critical_reading_diagnostic_1-problem-10821')
driver.maximize_window()
button=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//button[contains(.,'No Thanks, Start The Test')]")))
button.location_once_scrolled_into_view
button.click()
eleQuestions=WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,'input.test_button')))
driver.execute_script("arguments[0].click();", eleQuestions[2])
button=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//button[contains(.,'No Thanks')]")))
button.location_once_scrolled_into_view
button.click()
Please note: you can change the indexes from 2 to 6.
Snapshot:
If you would like to select any particular Question as you mentioned then try below code.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
driver=webdriver.Chrome()
driver.get('https://www.varsitytutors.com/sat_critical_reading_diagnostic_1-problem-10821')
driver.maximize_window()
button=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//button[contains(.,'No Thanks, Start The Test')]")))
button.location_once_scrolled_into_view
button.click()
eleQuestions=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//div[./p[text()='English dramatists refused to employ slang in their work.']]/parent::div//input[1]")))
driver.execute_script("arguments[0].click();", eleQuestions)
button=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//button[contains(.,'No Thanks')]")))
button.location_once_scrolled_into_view
button.click()
I'm trying to develop a bot that automatically handles the deletion of various posts from a website. I have stumbled across a major problem which does not allow me to proceed any further.
The page I've achieved to open presents various checkboxes with the following input:
<input type="checkbox" name="ids[]" value="305664759" onclick="toggleDeleteButtons()">
What I have to do is check simultaneously each checkbox and then click on delete button. Then a popup will appear, where I have to click another "Delete" button with the following input:
<input id="btnDelAds" class="button" href="javascript:void(0)" onclick="document.manageads.cmd.value='del';if (submit_batch_delete()){document.manageads.submit();}else{closeDialogDelete();}">
And then another popup will appear for confirming, but that's another problem.
In fact, the troubles come when I try to find the checkboxes.
This is the code for handling the first part of the site, and findind the checkboxes:
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
#HANDLING ACCESS
email = "somemail"
password = "somepass"
driver = webdriver.Firefox()
driver.get("https://www.somesite.it/account/manageads")
login_field = driver.find_element_by_id("login_email")
login_field.clear()
login_field.send_keys(email)
login_field = driver.find_element_by_id("login_passwd")
login_field.clear()
login_field.send_keys(password)
login_field.send_keys(Keys.ENTER)
#HANDLING DELETE OF POSTS
while True:
try:
elem = driver.find_element_by_xpath("//input[#type='checkbox' and contains(#name, 'id')")
print("Found")
except NoSuchElementException:
print("End")
break
elem.click()
(I've censored site url and credentials)
print("Found") clause is obviously not executed. The idea was to check consecutively every checkbox, probably I've done this in the wrong way.
What I get instead is "END" in console.
Any help will be strongly appreciated. Thanks in advance.
To trigger the presence of the popup with text as Delete you have to induce WebDriverWait for the desired element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name^='ids'][onclick^='toggleDeleteButtons'][type='checkbox']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[starts-with(#name, 'ids') and starts-with(#onclick, 'toggleDeleteButtons')][#type='checkbox']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to locate a button with a dynamicID
I'm trying to press a button with selenium in python. I am not able to locate it with xpath or css selector.
For the rest of the things in the script I have been using xpath and it works fine, but this button's xpath seems to be relative each time I open it. Therefore I have tried to access is with the css selector.
The element I am trying to access looks like this:
<button type="button" data-dismiss="modal" class="btn btn-primary pull-right">Opret AnnonceAgent</button>
The css selector from inspect in chrome:
#\35 d167939-adc2-0901-34ea-406e6c26e1ab > div.modal-footer > div > button.btn.btn-primary.pull-right
Let me know if I should post more html.
I have tried many stack oveflow questions, and tried many variatiosn of the css selector, like putting it as:
driver.find_element_by_class_name('#\35 d167939-adc2-0901-34ea-406e6c26e1ab.div.modal-footer.div.button.btn.btn-primary.pull-rightdiv.button.btn.btn-primary-pull-right').click()
driver.find_element_by_class_name('div.button.btn.btn-primary-pull-right').click()
driver.find_element_by_class_name('button.btn.btn-primary-pull-right').click()
driver.find_element_by_class_name('btn.btn-primary-pull-right').click()
driver.find_element_by_class_name('btn-primary-pull-right').click()
I have also tried a sleep timer.
The button is in a window that opens when you press the previous button, with the background greyed out, if that helps. Picture.
# This opens up the window in which to press the next button (works fine)
button = driver.find_element_by_xpath('//*[#id="content"]/div[1]/section/div[2]/button')
button.click()
driver.implicitly_wait(15)
time.sleep(5)
# This is what doesn't work
driver.find_element_by_class_name('button.btn.btn-primary-pull-right').click()
I expect for the program to press the button and it doesn't, it just sits there.
The desired element is within a Modal Dialog Box, so you need to induce WebDriverWait for the desired element to be clickable and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.btn.btn-primary.pull-right[data-dismiss='modal']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='btn btn-primary pull-right' and #data-dismiss='modal'][text()='Opret AnnonceAgent']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
class_name recives one class as parameter, but you are using css syntax as locator.
With css_selector
driver.find_element_by_css_selector('button.btn.btn-primary-pull-right')
Which means <button> tag with 2 classes, btn and btn-primary-pull-right
With class_name
driver.find_element_by_class_name('btn-primary-pull-right')
My guess was right.There was an iframe which stopping to access the element.You have to switch to iframe first to access the button element.Try now with following code.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
#WebDriverWait(driver,20).until(expected_conditions.frame_to_be_available_and_switch_to_it((By.XPATH,'//iframe[starts-with(#id,"rdr_")]')))
WebDriverWait(driver,20).until(expected_conditions.frame_to_be_available_and_switch_to_it((By.XPATH,'//iframe[#id="qualaroo_dnt_frame"]')))
driver.find_element_by_css_selector('button.btn.btn-primary').click()
I'm trying to click on the following button using Selenium with python:
<button type="submit" tabindex="4" id="sbmt" name="_eventId_proceed">
Einloggen
</button>
This is just a simple button which looks like this:
Code:
driver.find_element_by_id('sbmt').click()
This results in the following exception:
selenium.common.exceptions.ElementNotInteractableException: Message:
Element <button id="sbmt" name="_eventId_proceed" type="submit">
could not be scrolledinto view
So, I tried scrolling to the element using ActionChains(driver).move_to_element(driver.find_elements_by_id('sbmt')[1]).perform() before clicking the button.
(Accessing the second element with [1] because the first would result in selenium.common.exceptions.WebDriverException: Message: TypeError: rect is undefined exception.).
Then I used
wait = WebDriverWait(driver, 5)
submit_btn = wait.until(EC.element_to_be_clickable((By.ID, 'sbmt')))
in order to wait for the button to be clickable. None of this helped.
I also used driver.find_element_by_xpath and others, I tested it with Firefox and Chrome.
How can I click on the button without getting an exception?
Any help would be greatly appreciated
To invoke click() on the element you need to first use WebDriverWait with expected_conditions for the element to be clickable and you can use the following solution:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='sbmt' and normalize-space()='Einloggen']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC