We Keep Coding

Loop through the div tag containing paragraph tags

I am trying to scrape information from an automobile blog but i can't loop through the div tag containing the paragraph tags that contain the info.
driver.get("https://www.autocar.co.uk/car-news")
driver.maximize_window()
for i in range(3):
i+=1
info = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, f'//*[#id="page"]/div[2]/div[1]/div[1]/div[2]/div/div[1]/div/div[1]/div[1]/div[{i}]/div')))
heading = info.find_element_by_tag_name('h2')
clickable = heading.find_element_by_tag_name('a')
driver.execute_script("arguments[0].click();", clickable)
# the code starts to fail around here
try:
body_info = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, 'field-item even')))
main_text = []
for j in range(3):
j+=1
text = body_info.find_element_by_tag_name('p')
main_text.append(text)
for t in main_text:
t_info = t.text
print(f'{heading.text}\n{t_info}')
except:
print("couldn't find tag")
driver.back()

There's an issue with your code, (By.CLASS_NAME, 'field-item even').
Selenium does not have support for multiple classes or classes with space.
Simply replace space with . and that would be the CSS_SELECTOR
Try something like this:
try:
body_info = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, '.field-item.even')))

It must be '.field-item even' and not 'field-item even' if you are using By.CSS_SELECTOR for presence_of_element_located().
So Replace,
body_info = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, 'field-item even')))
with,
body_info = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, 'field-item even')))
Official docs. https://selenium-python.readthedocs.io/api.html#locate-elements-by

To select multiple classes, you must use the class selector. You can't select multiple classes through a space, like in CSS. You need to select multiple classes using the class selector. So you must put a dot before all of the classes and not give any space between them

Python Selenium can't click on next page button

I want to click the next page until no more page, but it does not click.
returns the error:raise exception_class(message, screen, stacktrace)
StaleElementReferenceException: stale element reference: element is not attached to the page document
my codes:
Thanks in advance!
driver.get('http://www.chinamoney.com.cn/chinese/zjfxzx/?tbnm=%E6%9C%80%E6%96%B0&tc=null&isNewTab=1')
driver.implicitly_wait(10)
driver.refresh()
driver.implicitly_wait(10)
wait = WebDriverWait(driver, 5)
datefield_st = wait.until(EC.element_to_be_clickable((By.ID, "pdbp-date-1")))
datefield_st.click()
select_st = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-year"))))
select_st.select_by_visible_text("2021")
select2 = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-month"))))
select2.select_by_value("1")
day=1
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[#data-handler='selectDay']/a[text()='{}']".format(str(day))))).click()
datefield_ed = wait.until(EC.element_to_be_clickable((By.ID, "pdbp-date-2")))
datefield_ed.click()
select_ed = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-year"))))
select_ed.select_by_visible_text("2021")
select2 = Select(wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ui-datepicker-month"))))
select2.select_by_value("1")
day=1
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[#data-handler='selectDay']/a[text()='{}']".format(str(day))))).click()
driver.find_element_by_link_text("查询").click()
while True:
driver.implicitly_wait(10)
links=[link.get_attribute('href') for link in driver.find_elements_by_xpath("//a[contains(#title,'同业存单') and not(contains(#title,'申购说明')) and not(contains(#title,'公告'))]")]
titles = [title.text for title in driver.find_elements_by_xpath("//a[contains(#title,'中期票据') and not(contains(#title,'申购说明')) and not(contains(#title,'公告'))]")]
dates = [date.text for date in driver.find_elements_by_xpath('//*[#class="san-grid-r text-date"]')]
driver.implicitly_wait(10)
for link, title,date in zip(links, titles,dates):
dataframe = pd.DataFrame({'col1':date,'col2':title,'col3':link},index=[0])
dataframe.to_csv('Chinamoney.csv',mode='a+',header=False,index=False,encoding='utf-8-sig')
print(link,title,date)
try:
driver.find_element_by_xpath('//*[contains(#class, "page-next")]').click()
except:
print('No more pages')

You passed two class names into selector while it's not allowed for search by class name. Either try
(By.CLASS_NAME, 'page-next')
or
(By.CSS_SELECTOR, '.page-btn.page-next')
Also your element and icon select the same element. So you don't need to define icon. Simply use element.click()

You are using:
driver.find_element_by_xpath('//*[contains(#class, "page-next")]').click()
Try:
element = driver.find_element_by_xpath('//*[contains(#class, "page-next")]')
driver.execute_script("arguments[0].click();", element)
If this doesnt work, you can try to obtain the url/link value and store it, and later you can go to the url or do what you want without click in it.

Selenium: Wait for element's height to be not equal to X

I am trying to take screenshot of webpage when canvas height changes. Its default height is 557. I want selenium to wait till height changes to any other value than 557. Is there any way to do that ?
<canvas id="faceCanvasPhoto" width="600" height="557" class="center-block reportCanvas">
Your browser does not support the canvas element.
</canvas>
I tried EC.visibility_of_element_located but couldnt catch it
try:
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "//*[#id='faceCanvasPhoto']"))
)
driver.find_element_by_tag_name('body').screenshot(facemap +'.png')
except TimeoutException:
driver.find_element_by_tag_name('body').screenshot(facemap +'.png')
driver.implicitly_wait(1)

Define your on wait class:
class element_height_changes(object):
"""An expectation for checking that an element has a particular css class.
locator - used to find the element
returns the WebElement once it has the particular css class
"""
def __init__(self, locator, curnt_val):
self.locator = locator
self.curnt_val = curnt_val
def __call__(self, driver):
# Finding the referenced element
element = driver.find_element(*self.locator)
if element.get_attribute("height") == str(self.curnt_val):
return False
else:
return True
And call it as :
print(WebDriverWait(driver, 10).until(
element_height_changes(
(By.XPATH, "//iframe"),59)
))
THis will print true or false , whether iframe length was changed from 59. If not wait till 10 second for it to change else time out.

Select an element of a drop down -selenium -python

I try of select the sport 'Football' in a drop down of sport but impossible of click on it.
I tried with the Select() method:
driver = webdriver.Chrome()
url = "https://www.flashscore.com/"
driver.get(url)
Team = 'Paris SG'
Type = 'Teams'
sport = 'Football'
buttonSearch = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".header__button--search"))).click()
fill_search_bar = driver.find_element(By.CSS_SELECTOR, ".input___1NGxU3-")
fill_search_bar.clear()
fill_search_bar.send_keys(Team)
driver.find_element(By.CSS_SELECTOR, ".dropDown").click()
select_sport = Select(driver.find_element(By.XPATH,"//div[contains(#class, 'dropDown__list')]"))
select_sport.select_by_visible_text(sport)
This code return this error : UnexpectedTagNameException: Message: Select only works on <select> elements, not on <div>.
Here is my second version:
fill_search_bar = driver.find_element(By.CSS_SELECTOR, ".input___1NGxU3-")
fill_search_bar.clear()
fill_search_bar.send_keys(Team)
driver.find_element(By.CSS_SELECTOR, ".dropDown").click()
select_sport = WebDriverWait(driver, timeout=10).until(EC.element_to_be_clickable((By.XPATH,"//[#class='dropDown__list']/[contains(text(),'"+ sport +"')]"))).click()
This code return this error : selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='dropDown__list']/div[contains(text(),'Football')]"}.
How can I solve this problem ?

I would suggest to break down the wait until class into two lines for simplicity. Its totally optional and wouldn't make much of a difference.
wait = WebDriverWait(driver, 300)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".header__button--search")))
element_to_be_clicked=driver.find_element_by_css_selector(".header__button--search")
element_to_be_clicked.click()
For the second part try using the values of the options in drop down list:
fill_search_bar.clear()
fill_search_bar.send_keys(Team)
driver.find_element_by_xpath("//div[#class='dropDown__selectedValue dropDownValueSelected___3msxRQS']").click()
select_sport=Select(driver.find_element_by_class("dropDown__list dropDownList___3V-ppVu"))
select_sport.select_by_value('1') #football has value 1 in the list

How to wait for presence of an element with non empty content?

How can I wait for a presence of an element which the content (text) is not empty? I tried select by xpath using //*[#id="test" and text() != ""], but the return of WebDriverWait#until does not retrieve the element.
My code:
selector = '//*[#id="test" and text() != ""]'
element = WebDriverWait(driver, timeout).until(
expected_conditions.presence_of_element_located((By.XPATH, selector))
)
I would like to get the text content of the element. I tried:
print element.text # prints 0
If i print only element, the output is <selenium.webdriver.remote.webelement.WebElement(session="xxx", element="xxx")>. What is wrong?
The div I'm tryin to get has this structure:
<div>
Test:
<div id="test"><b>this text here</b></div>
</div>

You can wait for an element by xpath with non empty text like this:
xpath = "//*[#id='test']"
WebDriverWait(driver, 30).until(lambda driver: driver.find_element_by_xpath(xpath).text.strip() != '')
source

The text() xpath function would not help in this case since it would not consider the texts of the child elements. You actually need to call the .text in your Expected Condition. You can create a custom one:
from selenium.webdriver.support import expected_conditions as EC
class wait_for_non_empty_text(object):
def __init__(self, locator):
self.locator = locator
def __call__(self, driver):
try:
element_text = EC._find_element(driver, self.locator).text.strip()
return element_text != ""
except StaleElementReferenceException:
return False
Usage:
wait = WebDriverWait(driver, timeout)
element = wait.until(
wait_for_non_empty_text((By.ID, "test"))
)
print(element.text)