I'm new to python and selenium.
This is an HTML snippet of the page im trying to scrape a page this
I want to scrape on the basis of data-attr1 given in the html snippet.
Please share some code to find this link and click on it.
<a class="js-track-click challenge-list-item" data-analytics="ChallengeListChallengeName" data-js-track="Challenge-Title" data-attr1="grading" data-attr3="code" data-attr4="true" data-attr5="true" data-attr7="10" data-attr-10="0.9" data-attr11="false" href="/challenges/grading"><div class="single-item challenges-list-view-v2 first-challenge cursor"><div id="contest-challenges-problem" class="individual-challenge-card-v2 content--list-v2 track_content"><div class="content--list_body"><header class="content--list_header-v2"><div class="challenge-name-details "><div class="pull-left inline-block"><h4 class="challengecard-title">Grading Students<div class="card-details pmT"><span class="difficulty easy detail-item">Easy</span><span class="max-score detail-item">Max Score: 10</span><span class="success-ratio detail-item">Success Rate: 96.59%</span></div></h4></div></div><span class="bookmark-cta"><button class="ui-btn ui-btn-normal ui-btn-plain star-button" tabindex="0" aria-label="Add bookmark"><div class="ui-content align-icon-right"><span class="ui-text"><i class="js-bookmark star-icon ui-icon-star"></i></span></div></button></span><div class="cta-container"><div class="ctas"><div class="challenge-submit-btn"><button class="ui-btn ui-btn-normal primary-cta ui-btn-line-primary" tabindex="0"><div class="ui-content align-icon-right has-icon"><span class="ui-text">Solved</span><i class="ui-icon-check-circle ui-btn-icon"></i></div></button></div></div></div></header></div></div><div class="__react_component_tooltip place-top type-dark " data-id="tooltip"></div></div></a>
If the page is not loading as fast as expected you can try:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#data-attr1='grading']"))).click()
You can click on the element using the xpath:
element = driver.find_element_by_xpath("//a[#data-attr1='grading']")
element.click();
Since there is no name id or class available directly you can use xpath.
The element that you are looking for is in div with class challenges-list and you want to click on first link inside it. You can use this xpath
//a[#data-attr1='grading']
And for clicking you can do
driver.find_element_by_xpath("//a[#data-attr1='grading']").click()
Use double-click with Actionchains because click might not work in Python.
eg.
from selenium.webdriver import ActionChains
# Get the element however you want
element = driver.find_element_by_xpath("//a[#data-attr1='grading']")
ActionChains(driver).double_click(settings_icon).perform()
Related
I want to scrape links to news articles using scrapy + selenium. The website I am using uses a 'Load more' button, so I obviously want selenium to click on this button to load all articles.
I have looked for similar questions and tried various options already such as
element = driver.find_element(By.XPATH, value='//*[#id="fusion-app"]/main/div/div/div/div/div[4]/div/div/button')
driver.execute_script("arguments[0].click();", element)
and
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
ActionChains(driver).move_to_element(element).click().perform()
All to no result. I've also inserted some print statements in between to check whether it does run the code, and that seems to work fine; I think it's just a matter of the button not being located/clicked on.
This is the html of the button btw:
<button class="ais-InfiniteHits-loadMore">Load more </button>
And when I print element, this is what I get: <selenium.webdriver.remote.webelement.WebElement (session="545716eef622a12bdbeddef99e02bdef", element="551741ec-4616-4bd4-b8fd-57c2f4bffb00")>
Is someone able to help me out? Thank you in advance.
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get('https://www.businessoffashion.com/search/?q=Louis+Vuitton&f=Articles%2CFashion+Shows%2CNews')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'button.ab-close-button'))).click()
elem=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
driver.execute_script("arguments[0].click()", elem)
You hit two different errors with a pop up and an element click intereception when you can just use javascript to click that element.
Import:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am currently trying to automate a data entry process. Every time I refresh the page the ID tags change and make it impossible. I read that I can use CSS selectors or a possible xpath using tags that don't change. It seems that only the ID tags are the problem.
Below is the HTML code for the button. Every time i try and use the CSS selector i got a no such element exception. I think I am doing it wrong. Please help.
<button type="button" class="dark button secondary" data-automation-id=
"btn-footer-save" data-dojo-attach-event="onclick:saveAndStayPressed"
data-qbo-bind="visible:shouldShowSaveAndStayButton" style>Save</button>
Try this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = new WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(),'Save')]"))).click()
The element gets wrapped in a WebDriverWait until it is interactable, in this case clickable
The website I am trying to scrape is 'https://www.lamiastampante.it/cerca_codice_cartuccia.php?codice=D111L&lg=it', and I am using python with Selenium for that.
I want to click on the title of the first product came out of my search.
It is an a element within a div, but when I copy the parent's XPath of such element (the 's XPath), my python script thinks I am referring to another (incorrect) element which is a pane located on the right of the webpage.
I noticed that because if I print out the class of the element gotten from that XPath I get "panel-heading", while it should be "col-xs-12 col-sm-12 col-md-12".
This is my very short python script:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
driver = webdriver.Firefox()
driver.get("https://www.lamiastampante.it")
driver.find_element(By.ID, "form_oem_code").send_keys("D111L" + Keys.ENTER)
first_product = driver.find_element_by_xpath("""/html/body/div[6]/div/div[1]/div[4]/div[1]""")# XPath of the target's parent element.
# first_product.click() /Commented out because I should first get the <a> element within that contains the link that can be clicked.
You can go visit the web page and inspect its HTML structure. I had some hard times trying to copy paste it here in a comprehensive and useful manner.
To click on the link after search results induce WebDriverWait() and ait for element_to_be_clickable() and following xpath.
driver.get("https://www.lamiastampante.it")
driver.find_element(By.ID, "form_oem_code").send_keys("D111L" + Keys.ENTER)
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"(//a[starts-with(#id,'a_') and contains(.,'Toner')])[1]"))).click()
Or yon can use visibility_of_all_elements_located() and below xapth.
driver.get("https://www.lamiastampante.it")
driver.find_element(By.ID, "form_oem_code").send_keys("D111L" + Keys.ENTER)
elements=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.XPATH,"//a[starts-with(#id,'a_') and contains(.,'Toner')]")))
elements[0].click()
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
When I run the code, the website loads up fine but then it won't click on the button- an error appears saying the element is not interacterble. What do I need to do to click the button? I am relatively new to this and would be grateful for any help.
I have already tried finding it by id and tag.
page = driver.get("https://kenpreston.co.uk/author/")
element = driver.find_element_by_id('mk-button-31')
element.click()
SOLVED:
I used driver.find_element_by_link_text and this worked fine.
I have checked the website and noticed that mk-button-31 is an id for a div tag and inside it there is an a tag. Try getting the url from the a tag and do another driver.get instead of clicking on it.
Also the whole div tag is not clickable so that is why you are getting this error.
Use sleep from time library to be sure page fully loaded
from time import sleep
page = driver.get("https://kenpreston.co.uk/author/")
sleep(2)
element = driver.find_element_by_id('mk-button-31')
element.click()
Looks like your element is not clickable you need to replace this id selector with the css and need to wait for the element before click on it.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
page = driver.get("https://kenpreston.co.uk/author/")
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#mk-button-31 span"))
element.click();
Consider adding Explicit Wait to your script as it might be the case the DOM had finished loading and the button you're looking for is still not there.
The classes you're looking for are:
WebDriverWait
expected_conditions
Suggested code change:
#your other imports here
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
#your other code here
page = driver.get("https://kenpreston.co.uk/author/")
element = WebDriverWait(driver, 10).until(expected_conditions.element_to_be_clickable((By.ID, "mk-button-31")))
element.click()
More information: How to use Selenium to test web applications using AJAX technology
I'm very new to Selenium.
I'm crawling data from this page. I need to scroll down the page and click on "Load More Arguments" to get more text. This is the location to click on.
<a class="debate-more-btn" href="javascript:void(0);" onclick="loadMoreArguments('15F7E61D-89B8-443A-A21C-13FD5EAA6087');">
Load More Arguments
</a>
I have tried this code but it does not work. Should I need more code to locate to that (I think the 1 has already tell the location to click). Do you have any recommendation? Thank you in advance.
[1] btn_moreDebate = driver.find_elements_by_class_name("debate-more-btn")
[2] btn.click()
Find the link by link text, move to the element and click:
from selenium.webdriver.common.action_chains import ActionChains
link = driver.find_element_by_link_text('Load More Arguments')
ActionChains(browser).move_to_element(link).perform()
link.click()
If you get an exception while finding an element, you may need to use an Explicit Wait:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "Load More Arguments")))
ActionChains(browser).move_to_element(link).perform()
link.click()
If I understand your code correctly, I can see a few things wrong.
1. You're using find_elements_by_class_name. I'd recommend using find_element_by_class_name instead. elements returns a list, which isn't needed in a case where there is only one element.
2. You're using btn_moreDebate as the holder for the results of your find_elements, but then interacting with btn.
You should be able to perform the find and click in one action:
driver.find_element_by_class_name("debate-more-btn").click()