Selenium Web Scraping - python

I would like my web scraper to go to a web site and press the log in button, then put random credentials and submit, here is the html code, please help
<div class="container">
<div class="row header-box">
<div class="col-md-8">
<h1>
Quotes to Scrape
</h1>
</div>
<div class="col-md-4">
<p>
Login
</p>
</div>
</div>
Here is the code I have by far
# Start driver
driver_path = "chromedriver"
driver = webdriver.Chrome(driver_path)
# Navigate to the website
driver.get('http://quotes.toscrape.com/')
driver.maximize_window()
driver.find_element('//bento/orange[contains(#Class,"small")]').click()
To press login button and put random credentials in

Use WebDriverWait() and wait for element to be clickable. User following xpath options
driver.get("http://quotes.toscrape.com/")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//a[text()='Login']"))).click()
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//input[#id='username']"))).send_keys("testuser")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//input[#id='password']"))).send_keys("testuser")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//input[#value='Login']"))).click()
Import below libraries
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

To click on the Login link, fill in Username and Password and click on Login you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategies:
driver.get("https://quotes.toscrape.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href='/login']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#username"))).send_keys("Orkhan")
driver.find_element(By.CSS_SELECTOR, "input#password").send_keys("Karimov")
driver.find_element(By.CSS_SELECTOR, "input[value='Login']").click()

Related

Unable to access the button in Selenium Python using Xpath

Error Msg:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//input[text()="Cancel"]"}
I have tried with tagname as button too.
The HTML for this:
<form name="forgotPassword" id="forgotPassForm" action="https://login.salesforce.com/secur/forgotpassword.jsp"
autocomplete="off" method="post" target="_top" novalidate="novalidate">
<p class="username">To reset your password, enter your Salesforce username.</p>
<input type="hidden" name="locale" value="in" />
<div class="verifyform">
<label for="un" class="label">Username</label>
<input id="un" type="email" class="input wide mb12 mt8 username" onKeyPress="checkCaps(event)"
name="un" value="" autofocus />
<input type="submit" name="continue" id="continue" class="button primary fiftyfifty right focus" value="Continue" />
<input type="button" name="cancel" onclick="parent.location='/?locale=in'" class="secondary button fiftyfifty mb16" value="Cancel" >
</div>
<input type="hidden" name="lqs" value="locale=in" />
<div id='pwcaps' class='pwcaps' style='display:none'>Caps Lock is on. Please try to log in again and remember that passwords are case-sensitive.</div>
<a class="hiddenlink" href="javascript:document.forgotPassword.submit();" id="continueReset">Continue</a>
<p class="small">Video: Need Help Logging In?</p>
</form>
My code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(service=Service("C:\\Users\\Dell\\Desktop\\Selenium\\chromedriver.exe"))
driver.maximize_window()
driver.get("https://login.salesforce.com/")
driver.find_element(By.CSS_SELECTOR, '#username').send_keys('Tac')
driver.find_element(By.CSS_SELECTOR, '.password').send_keys('PASSWRd')
driver.find_element(By.CSS_SELECTOR, '.password').clear()
driver.find_element(By.LINK_TEXT, 'Forgot Your Password?').click()
driver.find_element(By.XPATH, '//button[text()="Cancel"]').click()
The element Cancel doesn't have a innerText rather the value attribute is set as Cancel
<input type="button" name="cancel" onclick="parent.location='/'" class="secondary button fiftyfifty mb16" value="Cancel">
Solution
To click element instead of presence_of_element_located() you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[value='Cancel']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#value='Cancel']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Have you tried the following code:
driver.find_element(By.XPATH, '//*[#id="forgotPassForm"]/div[1]/input[3]').click()
To get the basic XPath you can inspect element by right clicking on the web page and selecting 'Elements'. You can then right click the button and then 'Copy > Copy XPath'. This will allow you to get the very basics working before then refactoring and using a more robust piece of xcode.
See screenshot:
The element you are trying to access is input element NOT button element.
Use following xpath to identify the element.
driver.find_element(By.XPATH, "//input[#name='cancel' and #value='Cancel']").click()
To handle dynamic element use WebDriverWait() and wait for element to be clickable.
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='cancel' and #value='Cancel']"))).click()
You need to import below libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Click on item using Selenium and Python

I am trying to click on the "Training material statistics" by using the following code in Python, but it didn't work:
WebDriverWait(driver,20)\
.until(EC.element_to_be_clickable((By.XPATH,'//*[#id="report-navigation"]/div[2]')))\
.click()
HTML:
<div id="report-navigation">
<div class="report-nav-btn active" onclick="Report.changeGrid(this, 'report-users-grid')">
User statistics
<div class="report-nav-arrow active"></div>
</div>
<div class="report-nav-btn" onclick="Report.changeGrid(this, 'report-objects-grid')">
Training material statistics
<div class="report-nav-arrow"></div>
</div>
<div class="report-nav-btn" onclick="Report.changeGrid(this, 'report-deliverables-grid')">
Learner assignments
<div class="report-nav-arrow"></div>
</div>
</div>
HTML Snapshot:
To click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#report-navigation div[onclick*='report-objects-grid']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='report-navigation']//div[contains(#onclick, 'report-objects-grid')]"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

I'm trying to get information from the site, using the right css selectors, but it still returns me as if the element didn't exist

trying to build a bot to my own services, but while i'm trying to run its crashs
login_button = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "//div[#class='loginRegisterButtons sb-login-register-buttons clearfix ember-view']//a[#class='btn signin-btn sbLoginBtn buttons1571052392296']")))
login_button.click()
the html
<div class="loginRegisterButtons sb-login-register-buttons clearfix ember-view">
<div>
<a class="btn signin-btn sbLoginBtn buttons1571052392296">
<span class="btf-text">ENTRAR</span>
</a>
</div>
</div>
The desired element is a Ember.js enabled element, so to click on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.btn.signin-btn.sbLoginBtn > span.btf-text"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='btf-text' and text()='ENTRAR']"))).click()
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Python - Selenium Webdriver find an element where all the div nodes have similar names

I am trying to access a div where all divs have the same name. Let me explain. I am just starting out with selenium and python and I am trying to scrape a webpage just to learn. I ran into the following problem. I made the example html to show the buildup of the webpage. All the divs have the exact same class and title. Then there is the h1 tag for the item and the p tag for the color (which is a clickable link). I am trying to search a page when you give it certain instructions. Example: I am looking for a white racebike. I am able to find the bikes with the first line of code, but how do I find the right color within the racebike section? If I run the Python mentioned below I get an error message. Thanks in advance!
<!DOCTYPE html>
<html>
<body>
<div class=div title=example>
<h1>racebike</h1>
<p class='test'>black</p>
</div>
<div class=div title=example>
<h1>racebike</h1>
<p class='test'>white</p>
</div>
<div class=div title=example>
<h1>racebike</h1>
<p class='test'>yellow</p>
</div>
<div class=div title=example>
<h1>citybike</h1>
<p class='test'>yellow</p>
</div>
<div class=div title=example>
<h1>citybike</h1>
<p class='test'>green</p>
</div>
</body>
</html>
test = (self.driver.find_element_by_xpath("//*[contains(text(), racebike)]"))
test.self.driver.find_element_by_xpath(".//*[contains(text(), white)]").click
To locate/click() on the white racebike element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following xpath based Locator Strategies:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//h1[text()='racebike']//following-sibling::p[#class='test' and text()='white']"))).click()
Using XPATH considering the parent <div>:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='div' and #title='example']/h1[text()='racebike']//following-sibling::p[#class='test' and text()='white']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can use same xpath which you tried in your solution. It might be possible server is taking too long to repond.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
element = WebDriverWait(page, 10).until(EC.presence_of_element_located((By.XPATH, "//p[contains(#class, 'white')]")))
element.click()
for multiple bikes with whiite color
elements= WebDriverWait(driver, 30).until(EC.presence_of_all_elements_located((By.XPATH, "//p[contains(#class, 'white')]")))
for element in elements:
element.click()

How to click on item in navigation bar on top of page using selenium python?

I am trying to test a web page using selenium python. All is working well but issue is encountering when clicking on navbar item
I have used:
driver.find_element_by_xpath('./li/a[. = "Log in"]')
Also have used:
driver.find_element_by_link_text('Log in')
Nothing got luck !!
The code snippet:
<div class='container'>
<div class='navigationbar__header'>
<a class='navigationbar__header__logo tracking-link' data-link-name='logo' href='/' target='_self'>
<div id='hired-brand'>HIRED</div>
</a>
</div>
<div class='navigationbar__toggle'>
<div class='navigationbar__toggle__element'>
<img alt='Menu' class='icon icon--sandwich' src='data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7'>
</div>
<input class='navigationbar__toggle__helper' type='checkbox'>
<ul class='navigationbar__navigation'>
<li class="navigationbar__item "><a class="sm-ml0 tracking-link" data-link-name="employers_page" target="_self" href="/employers">For Employers</a></li>
<li class="navigationbar__item "><a class="sm-ml0 tracking-link" data-link-name="success_stories" target="_self" href="/success-stories">Success Stories</a></li>
<li class="navigationbar__item "><a class="sm-ml0 tracking-link" data-link-name="employers_resources" target="_self" href="/employers/resources">Resources</a></li>
<li class="navigationbar__item "><a class="text-medium sm-ml0 tracking-link" data-link-name="login" target="_self" href="/login">Log in</a></li>
<div class='xs-ptb1 xs-prl1 md-ptb0 md-inline-block'><li class="navigationbar__item "><a class="button button--primary tracking-link" data-link-name="signup" target="_self" href="/signup">Sign Up</a></li></div>
</ul>
</div>
</div>
</nav>
This code is visible on page inspect easily. Anyone know the better way to interact with it?
To click() on the link with text as Log in within the website you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "li.navigationbar__item a[data-link-name='login'][href='/login']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[#class='navigationbar__item ']/a[#data-link-name='login' and #href='/login']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Update
As an alternative you can use execute_script() as follows:
Using CSS_SELECTOR:
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "li.navigationbar__item a[data-link-name='login'][href='/login']"))))
Using XPATH:
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[#class='navigationbar__item ']/a[#data-link-name='login' and #href='/login']"))))
Ok so I checked the site.
The problem is that when the window size is small there is toggle for navigation that you need to click first.
try this
from selenium.common.exceptions import NoSuchElementException
try:
login_button = driver.find_element_by_link_text('Log in')
login_button.click()
except NoSuchElementException:
nav_bar_toggle = driver.find_element_by_class_name(
'navigationbar__toggle__helper'
)
nav_bar_toggle.click()
login_button = driver.find_element_by_link_text('Log in')
login_button.click()

Categories