UPDATE:
So thanks to the voted answer it displayed some information not the right information, it shows 0kb out of 100 and when in the inspect element console if doing console.log($0) then the item would be displayed in console how do I fetch this
I want to create a python 3.x programme that gets my stats off of netlify and easybase using selenium. The issue I have come across already is that the element does not have a specific class name and the text widget isn't just a tag nor a tag. Here is a screenshot of the html of netlify the screenshot, and this is the code that I used
element = driver.find_element_by_name("github")
element.click()
login = driver.find_element_by_name("login")
login.send_keys(email)
password = driver.find_element_by_name("password")
password.send_keys(passwordstr)
loginbtn = driver.find_element_by_name("commit")
loginbtn.click()
getbandwidth = driver.find_element_by_xpath('//*[#id="main"]/div/div[1]/div/section/div/div/div/dl/div/dd')
print(getbandwidth.text)
getbandwidth = driver.find_element_by_xpath("//dd[#class='tw-text-xl tw-mt-4px tw-leading-none']")
You can use this to grab the first xpath with that class. Below does the same but if you want to index other elements with similar classes.
(//dd[#class='tw-text-xl tw-mt-4px tw-leading-none'])[1]
Normally we use webdriver waits to allow for the element to become visible.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
getbandwidth = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,"//dd[#class='tw-text-xl tw-mt-4px tw-leading-none']")))
Related
I've looked around and I tried using different versions of WebElement userid = driver.findElement(By.xpath and WebElement email = driver.findElement(By.id("email")); but they return a syntax error when I try to use them.
For reference, this is the textbox I want to type in
and this is the html xpath I found from using inspect element on that box.
This is my first time using python & selenium so please be easy on me :)
EDIT: I'm now on the dropdown menu part of this script and I don't know how to use the select function
HTML
So, based on your HTML code, I think that the xpath that you need is this
//div[#data-section='customer-information']//input[#id='checkout_email']
Based on this xpath, you could write your code as such
sampleEmail = "MySampleEmail#Email.com"
driver.find_element(By.XPATH, "//div[#data-section='customer-information']//input[#id='checkout_email']").send_keys(sampleEmail)
if(driver.find_element(By.XPATH, "//div[#data-section='customer-information']//input[#id='checkout_email']").get_attribute('value').__len__() != sampleEmail.__len__()):
raise Exception(f'Failed to populate our e-mail textbox with our value: {sampleEmail}')
To wait for an input tag with class field__input to be able to load in and send keys to it
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.field__input"))).send_keys("ANYTHING")
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
When I run the code, the website loads up fine but then it won't click on the button- an error appears saying the element is not interacterble. What do I need to do to click the button? I am relatively new to this and would be grateful for any help.
I have already tried finding it by id and tag.
page = driver.get("https://kenpreston.co.uk/author/")
element = driver.find_element_by_id('mk-button-31')
element.click()
SOLVED:
I used driver.find_element_by_link_text and this worked fine.
I have checked the website and noticed that mk-button-31 is an id for a div tag and inside it there is an a tag. Try getting the url from the a tag and do another driver.get instead of clicking on it.
Also the whole div tag is not clickable so that is why you are getting this error.
Use sleep from time library to be sure page fully loaded
from time import sleep
page = driver.get("https://kenpreston.co.uk/author/")
sleep(2)
element = driver.find_element_by_id('mk-button-31')
element.click()
Looks like your element is not clickable you need to replace this id selector with the css and need to wait for the element before click on it.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
page = driver.get("https://kenpreston.co.uk/author/")
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#mk-button-31 span"))
element.click();
Consider adding Explicit Wait to your script as it might be the case the DOM had finished loading and the button you're looking for is still not there.
The classes you're looking for are:
WebDriverWait
expected_conditions
Suggested code change:
#your other imports here
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
#your other code here
page = driver.get("https://kenpreston.co.uk/author/")
element = WebDriverWait(driver, 10).until(expected_conditions.element_to_be_clickable((By.ID, "mk-button-31")))
element.click()
More information: How to use Selenium to test web applications using AJAX technology
I am trying to write a webscraping in python that will activate the "onclick" functionality of certain buttons on a webpage because the tables with the data I want are converted to csv, which makes it much easier to access. But the problem is that I am unable to locate elements by xpath at all when using PhantomJs. How can I click the element and access the csv content that I want?
This is my code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
url = "http://www.pro-football-reference.com/boxscores/201609180nwe.htm"
xpath = "//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button"
path_to_phantomjs = 'browser/phantomjs'
browser = webdriver.PhantomJS(executable_path = path_to_phantomjs)
browser.get(url)
delay=3
element_present = EC.presence_of_element_located((By.ID, 'all_player_offense'))
WebDriverWait(browser, delay).until(element_present)
browser.find_element_by_xpath(xpath).click()
And I get this error:
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"153","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50989","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"sessionId\": \"93ff24f0-9cbe-11e6-8711-bdfa3ff9cfb1\", \"value\": \"//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/93ff24f0-9cbe-11e6-8711-bdfa3ff9cfb1/element"}}
Screenshot: available via screen
IMPORTANT THING I FORGOT TO MENTION: As described in this this issue on GitHub, try putting set_window_size(width, height) or maximize_window()after setting the webdriver. You should also consider telling the webdriver to implicitly_wait(10) for the element to appear.
So there's a special maneuver you have to perform in order for the Selenium Webdriver to properly emulate what you're doing. In essence, to get the desired data, you'd have to:
A: Hover over the "Share & More" dropdown menu. Then
B: Click "Get table as CSV (Excel)".
For A, this involves having to place the emulated cursor on the element without clicking it. This idea of "mouse over" can be done with the move_to_element() function provided in the ActionChains class. So at the top you'd insert this:
from selenium.webdriver.common.action_chains import ActionChains
You want Selenium to find the specific element and move to it. You can achieve this with 2 lines of code:
dropdown = browser.find_element_by_xpath('//*[#id="all_player_offense"]/div[1]/div/ul/li[1]')
ActionChains(browser).move_to_element(dropdown).perform()
If you omit the above, you'll get an ElementNotVisibleException.
Now for B, you should be able to do browser.find_element_by_xpath(xpath).click().
I just want to write a simple log in code for one website. However, I think the log in page was written in JS. It's really hard to locate the elements with selenium.
The web page I am going to play with is:
"https://www.nike.com/snkrs/login?returnUrl=%2F"
This is how the page looks like and how the inspect element page look like:
I was trying to locate the element by following code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Firefox()
driver.get("https://www.nike.com/snkrs/login?returnUrl=%2Fthread%2Fe9680e08e7e3cd76b8832684037a58a369cad5ed")
time.sleep(5)
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
elem =driver.find_element_by_xpath("//*[#id='ce3feab5-6156-441a-970e-23544473a623']")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
driver.close()
This code return the error that the element could not find by [#id='ce3feab5-6156-441a-970e-23544473a623'.
I tried playing with frames, it seems does not work. If I went to "view web source" page, it is full of JS code.
Is there a good way to play with such a web page with selenium?
Try changing the code :
elem =driver.find_element_by_xpath("//*[#id='ce3feab5-6156-441a-970e-23544473a623']")
to
elem =driver.find_element_by_xpath("//*[#type='email']")
My guess (and observation) is that the id changes each time you visit the page. The id looks auto-generated, and when I go to the page multiple times, the id is different each time.
You'll need to search for something that doesn't change. For example, you can search for the name attribute, which has the seemingly static value "emailAddress"
element = driver.find_element_by_name("emailAddress")
You could also use an xpath expression to search for other attributes, such as data-componentname:
element = driver.find_element_by_xpath("//input[#data-componentname='emailAddress']")
Also, instead of a hard-coded sleep, you can simply wait for the element to be visible:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("https://www.nike.com/snkrs/login")
element = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.NAME, "emailAddress"))
)
Using Python27:
I am trying to automate a search and extract method using beautifulsoup to parse and input data on this database server. So far I have managed to get Python to login to it. But now when I try to search for the input element to make a search, I can't seem to get the identifier/code correct.
The highlighted in blue code says:
<input id="QUICKSEARCH_STRING" type="text" on focus="setTimeout('focusSearchElem()',100...
The highlighted portion in blue is where I believe I need to search for that element in order to input text then search. I think the rest, like inputting the results I get from the page might be a bit easier.
My code is as follows:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://somewebpage')
emailElem = browser.find_element_by_id('j_username')
emailElem.send_keys('blahblahblah')
passwordElem = browser.find_element_by_id('j_password')
passwordElem.send_keys('blahblahblah!')
login_form = browser.find_element_by_xpath("//a[#id='login']").click()
searchElem = browser.find_element_by_id('search_panel')
searchElem.send_keys('blahblahblah')
I'm not sure where I'm going wrong, but I think I am close.
browser.find_element_by_id('search_panel')
I don't see any elements with id="search_panel".
Here is how I would locate the desired input element:
browser.find_element_by_id("QUICKSEARCH_STRING")
browser.find_element_by_css_selector("div.search_panel input#QUICKSEARCH_STRING")
You may need to wait for it to become present after the logging in:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
search_input = wait.until(EC.presence_of_element_located((By.ID, "QUICKSEARCH_STRING")))
search_input.send_keys('blahblahblah')