We Keep Coding

Selenium: How to find a div that hiding in <a>

I want to use selenium to find a div:
My code is :
self.browser.find_element_by_xpath('//div[#class="bh-headerBar-nav-item"]').click()
But I got the error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class="bh-headerBar-nav-item"]"}
What should I do to find this div?

You can try changing the style value of the element to block using javascript.
self.driver.execute_script("document.getElementsByClassName("class-name")[0].style.display = 'block';")
Then try to access the element.

Short:
Your xpath looks correct for the exampled page html, but the class on screenshot ends with the space char. (it could be the reason).
Longer:
If you getting NoSuchElementException there can be 3 cases:
your div present in iframe, so you need to switch to iframe with driver.switch_to.iframe command before invoke find_element.
you looking for div with the exact class match. class attribute sometimes changed after some actions, some classes could be added or removed. So, try to use //div[contains(#class, 'your_class')]. I see, your class ends with the space char on screenshot, it may affect.
the element really not present on the page.
Try to take screenshot, or print pagesource and look for the element, may be this will bring more information regarding the real reason.

The DIV tag is a descendant of the A tag.
To click on the descendant DIV you can use either of the following Locator Strategies:
Using css_selector:
self.browser.find_element(By.CSS_SELECTOR, "a[href='#/sportVenueBookBC'] div.bh-headerBar-nav-item.").click()
Using xpath:
self.browser.find_element(By.XPATH, "//a[#href='#/sportVenueBookBC']//div[#class='bh-headerBar-nav-item ']").click()
However, the desired element is a dynamic element, so to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href='#/sportVenueBookBC'] div.bh-headerBar-nav-item."))).click()
Using XPATH:
WebDriverWait(self.driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#href='#/sportVenueBookBC']//div[#class='bh-headerBar-nav-item ']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

Locating an element with dynamic ID using Selenium and Python

I need to find an element by ID "start-ads", but the numbers at the end change every time. Is it possible to search for an element not by the whole ID, but only by its part?
Full element id: <div id="start-ads-202308">

To identify the following element:
<div id="start-ads-202308">
considering the static part of the value of id attribute you can use either of the following Locator Strategies:
Using css_selector:
element = driver.find_element(By.CSS_SELECTOR, "div[id^='start-ads-']")
where ^ denotes starts-with
Using xpath:
element = driver.find_element(By.XPATH, "//div[starts-with(#id, 'start-ads-')]")

You can use find_element_by_css_selector and use a CSS selector that matches using a prefix
driver.find_element_by_css_selector("div[id^='start-ads-']")

Is there a way to get the value within a <div> using selenium in python?

I'm new to python and webscraping so I'm not sure what the name of the value inbetween the <div>'s in an element is called. Sorry for not being able to specify.
<div class="syllable">value</div>
Is there a way to have the value inbetween the <div>'s get assigned to a string variable in python using selenium using XPath?
For example, the "value" in the element would be a string and it would print out:
value
I'm new to python and selenium so I can't figure it out.

To print out the text of the element.
elem=driver.find_element_by_class_name("syllable")
print(elem.text)
xpath:
elem=driver.find_element_by_xpath("//div[#class='syllable']/text()")
print(elem)

it is called html innerText
you can retrieve this value using text in selenium , or get_attribute.
This returns the rendered text (means displayed text)
elem=driver.find_element_by_class_name("syllable")
print(elem.text)
This return the text with out checking the style attribute meaning returns value even if its not displayed in UI
elem=driver.find_element_by_class_name("syllable")
print(elem.get_attribute("textContent")
you can find elem using this text also:
// partial match
elem=driver.find_element_by_xpath("//div[contains(text(),'value')])
print(elem.text)
// exact match
elem=driver.find_element_by_xpath("//div[text()='value')])
print(elem.text)
// exact match of the elements text if there is any child element like span it won't return the element
elem=driver.find_element_by_xpath("//div[.='value')])
print(elem.text)
Also note:
Other things you could read about outerHTML , innerHTML

To print the text value you can use either of the following Locator Strategies:
Using class_name and get_attribute("textContent"):
print(driver.find_element_by_class_name("syllable").get_attribute("textContent"))
Using css_selector and get_attribute("innerHTML"):
print(driver.find_element_by_css_selector("div.syllable").get_attribute("innerHTML"))
Using xpath and text attribute:
print(driver.find_element_by_xpath("//div[#class='syllable']").text)
Ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CLASS_NAME and get_attribute("textContent"):
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CLASS_NAME, "syllable"))).get_attribute("textContent"))
Using CSS_SELECTOR and text attribute:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.syllable"))).text)
Using XPATH and get_attribute():
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='syllable']"))).get_attribute("innerHTML"))
Console Output:
value
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
References
Link to useful documentation:
get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

List not printing all .text values using Selenium and Python

So I've been using selenium recently and decided to pull some values into a list using the following:
Values = browser.find_elements_by_xpath("//div[#class='main-col']//li[#class='test-item test-item--favourites']")
The values are all stored as expected. Though when I use for example:
for i in range (len(Values)):
print(Values[i].text)
i+=1
The code runs up to about the 50th index in the list and outputs the details, but after this its blank. I can run:
print(values[50])
It will output but .text gives me nothing.
Im using pYcharm for the development and looked into the variables at this stage. I can see that once I click on the variable it states 'collecting variable data' and after this I can then output some more of the variables in the list.
Is there a way I can force selenium to collect all the variable data so I can loop through each index in the list and store it in the .text format or is there a better alternative/method I can use?

If the .text is blank that means that there is no text between CSS Selector. To get an attribute out of a CSS Selector you have to use the get_attribute() function. Inside the () you place a string which is the name of an attribute which value you are looking for. for example
print(i.get_attribute('class'))

You are able to extract the text till the 50th index as find_elements_by_xpath() was able to identify 50 odd elements as per your Locator Strategy.
Solution
To collect all the desired elements using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.main-col li.test-item.test-item--favourites")))])
Using XPATH and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#class='main-col']//li[#class='test-item test-item--favourites']")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Web scraping with selenium and python - xpath with contains text

I will try to make it really short. I am trying to click on a product that came out of a search from a website. Basically there is a list of matching products, and I want to click on the first one which contains the product name I searched in its title.
I will post the link of the website so you can inspect its DOM structure: https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and
In this case, many contain my query string, and I would simply like to click on the first one.
Here is the snippet of code I wrote for this:
def click_on_first_matching_product(self):
first_product = WebDriverWait(self.driver, 6).until(
EC.visibility_of_all_elements_located((By.XPATH, f"//a[#class='df-card__main']/div/div[#class=df-card__title] and contains(text(), '{self.product_code}')"))
)[0]
first_product.click()
The problem is that 6 seconds go by and it cant find an element that satisfies the xPath condition i wrote, but I cant figure out how to make it work.
I am trying to get a search result a element and check if the title it has down its structure contains the query string I searched.
Can I have some help and an explanation please? I am quite new to selenium and XPaths...
Can I please also have a link to a reliable selenium documentation? I am having some hard times trying to find a good one. Maybe one that also explains how to make conditions for xPaths please.

You need to consider a couple of things. Your use-case would be either to click on the first search result or to click on the item with respect to the card title. In case of clicking on a definite WebElement inducing WebDriverWait for visibility_of_all_elements_located() will be too expensive.
To click on the item with respect to the card title you have to induce WebDriverWait for the element_to_be_clickable() and you can use the following xpath based Locator Strategies:
Using the text CE285A Toner Compatibile Per Hp LaserJet P1102 directly:
driver.get('https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='CE285A Toner Compatibile Per Hp LaserJet P1102']"))).click()
Using a variable for the text through format():
driver.get('https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and')
text = "CE285A Toner Compatibile Per Hp LaserJet P1102"
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='{}']".format(text)))).click()
Using a variable for the text through %s:
driver.get('https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and')
text = "CE285A Toner Compatibile Per Hp LaserJet P1102"
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='%s']"% str(text)))).click()
To click on first search product you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
CSS_SELECTOR:
driver.get('https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.df-card>a"))).click()
XPATH:
driver.get('https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='df-card']/a"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Your xpath seems incorrect.Try following xpath to click on product.
driver.get("https://www.tonercartuccestampanti.it/#/dfclassic/query=CE285A&query_name=match_and")
def click_on_first_matching_product(product_code):
first_product = WebDriverWait(driver, 6).until(EC.visibility_of_all_elements_located((By.XPATH,"//div[#class='df-card__title' and contains(text(), '{}')]".format(product_code))))[0]
first_product.click()
click_on_first_matching_product("CE285A")