We Keep Coding

Python | Selenium Issue with scrolling down and find by class name

For one study research I would like to scrape some links from webpages which located out of viewport (to see this links you need to scroll down the page).
Page example (https://www.twitch.tv/lirik)
Link example: https://www.amazon.com/dp/B09FVR22R2
Link located in div class='Layout-sc-nxg1ff-0 itdjvg default-panel' (in total 16 links on the page).
I have write the script but I get empty list:
from selenium import webdriver
import time
browser = webdriver.Firefox()
browser.get('https://www.twitch.tv/lirik')
time.sleep(3)
browser.execute_script("window.scrollBy(0,document.body.scrollHeight)")
time.sleep(3)
panel_blocks = browser.find_elements(by='class name', value='Layout-sc-nxg1ff-0 itdjvg default-panel')
browser.close()
print(panel_blocks)
print(type(panel_blocks))
I just get empty list after page was loaded. Here is output from the script above:
/usr/local/bin/python /Users/greg.fetisov/PycharmProjects/baltazar_platform/Twitch_parser.py
[]
<class 'list'>
Process finished with exit code 0
p.s.
when webdriver opens the page, I see there is no scroll down action. It just open a page and then close it after time.sleep cooldown.
How I can change the script to get the links properly?
Any help or advice would be appreciated!

You are using a wrong locator.
You should use expected conditions explicit waits instead of hardcoded pauses.
find_elements method returns a list of web elements while you want to the link inside the element(s).
This should work better:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
browser = webdriver.Firefox()
browser.get('https://www.twitch.tv/lirik')
wait = WebDriverWait(browser, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='channel-panels-container']//a")))
time.sleep(0.5)
link_blocks = browser.find_elements_by_xpath("//div[#class='channel-panels-container']//a")
for link_block in link_blocks:
link = link_block.get_attribute("href")
print(link)
browser.close()

To print the values of the href attribute you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://www.twitch.tv/lirik")
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.Layout-sc-nxg1ff-0.itdjvg.default-panel > a")))])
Console Output:
['https://www.amazon.com/dp/B09FVR22R2', 'http://bs.serving-sys.com/Serving/adServer.bs?cn=trd&pli=1077437714&gdpr=$%7BGDPR%7D&gdpr_consent=$%7BGDPR_CONSENT_68%7D&adid=1085757156&ord=[timestamp]', 'https://store.epicgames.com/lirik/rumbleverse', 'https://bitly/3GP0cM0', 'https://lirik.com/', 'https://streamlabs.com/lirik', 'https://twitch.amazon.com/tp', 'https://www.twitch.tv/subs/lirik', 'https://www.youtube.com/lirik?sub_confirmation=1', 'http://www.twitter.com/lirik', 'http://www.instagram.com/lirik', 'http://gfuel.ly/lirik', 'http://www.cyberpowerpc.com/', 'https://www.cyberpowerpc.com/page/Intel/LIRIK/', 'https://discord.gg/lirik', 'http://www.amazon.com/?_encoding=UTF8&camp=1789&creative=390957&linkCode=ur2&tag=l0e6d-20&linkId=YNM2SXSSG3KWGYZ7']
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium WebDriver find_element_by_xpath not working for text

I'm trying to click on a link on a webpage that has no ID and no individual class. The only thing to lock it down to is the text 'Sessions'.
I have tried:
driver.find_element_by_xpath("//*[contains(text(),'Sessions')]");
driver.find_element_by_xpath("//*[text()='Sessions']");
Both come back with "No such element".
Edit: I have also tried driver.find_element_by_link_text which also didn't work.
I've tried using the full xpath:
/html/body/div/div/div[1]/div/nav/a[3]
To no avail.

That is a link_Text cause it's between anchor tag, use this :
driver.find_element_by_link_text('Sessions').click()
or
A way more good approach is to use ExplicitWaits :
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, 'Sessions')))
element.click()
If you want explicit wait you would need to import the below :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
If the above gives you NoSuchElementException, I would probably suspect this it is in iframe (See the screenshot first tag - I can see body), if it happens to be then in that case you would need to switch to iframe first and continute with this web element.
Code
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "iframe xpath here")))
wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Sessions"))).click()
Imports :

selenium chromedriver different values of xpath between terminal and actual driver

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://www.msha.gov/mine-data-retrieval-system'
driver = webdriver.Chrome(executable_path='chromedriver')
driver.get(url)
#driver.find_element_by_xpath('//*[#id="mstr90"]/div[1]/div/div') error
#driver.find_elements_by_xpath('//input') gives 3 while in driver gives 10
I am unable to find element where the input "Search by Mine ID by typing here.." is, the document is fully loaded but it can't locate it. What I want to do is simply pass in an input "0100003" then submit

Iframe is present on your page. Before you interact with inputbox you need to switch on to iframee. Refer below code to resolve your issue.
wait = WebDriverWait(driver, 10)
driver.get("https://www.msha.gov/mine-data-retrieval-system")
driver.switch_to.frame("iframe1")
wait = WebDriverWait(driver, 10)
inputBox = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='mstrmojo-SimpleObjectInputBox-empty']"))).click()
inputBox1 = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='mstrmojo-SimpleObjectInputBox-container mstrmojo-scrollNode']//input")))
inputBox1.send_keys("0100003")
Updated Code to handle dropdown
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#mstr100,mstrmojo-Popup.mstrmojo.SearchBoxSelector-suggest"))).click()
Note: please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
Output:

The element you are trying to find is inside an iframe, so you will need to switch to that iframe first and then do your find element. Also, it's a best practice to use waits to give pages/elements time to load before a find element timeouts and throws an error.
iframe = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#iframe1')))
driver.switch_to.frame(iframe)
mine_id = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, '//*[#id="mstr90"]/div[1]/div/div')))
Then you need to click this element to make it interactable.
mine_id.click()
Once you click then you need to re-find the input box before sending keys.
mine_id_input = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#mstr90 input')))
mine_id_input.send_keys('0100003')
To select the suggestion displayed:
suggestion = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#mstr100')))
suggestion.click()
if you wanted to continue on interacting outside the iframe after this is done, you will want to switch back out of the iframe like this:
driver.switch_to.default_content()

Selenium Webdriver (Python) - Unable to locate element inside nested iframes

I am trying to scrape some data from an iframe located within a webpage. The URL of the webpage is https://www.nissanoflithiasprings.com/schedule-service. I am trying to access the button shown in the image below:
When I right-click on the button (located inside the iframe) to view the source code, I am able to see the HTML id and name (see screenshot below):
The "id" for the button is "new_customer_button". However, when I use selenium webdriver's driver.find_element_by_id("new_customer_button") to access the button, the code is not able to locate the button inside the iframe and throws the following error:
NoSuchElementException: no such element: Unable to locate element: {"method":"id","selector":"new_customer_button"}
Below is the code that I have tried so far:
from selenium import webdriver
chrome_path = r"C:\Users\gh455\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
dest_iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(dest_iframe)
driver.find_element_by_id("new_customer_button")
Not sure why this is happening. Any help will be appreciated. Thanks!

The element is inside multiple <iframe> tags, you need to switch to them one by one. You should also maximize the window and use explicit wait as it take some time to load
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chrome_path = r"C:\Users\gh455\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button")

To click() on the element with text as Make · Year · Model as the the desired element is within an nested <iframe>s so you have to:
Induce WebDriverWait for the desired parent frame to be available and switch to it.
Induce WebDriverWait for the desired child frame to be available and switch to it.
Induce WebDriverWait for the desired element_to_be_clickable().
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument('disable-infobars')
driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='com/scheduling']")))
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='consumerportal']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.button.button--action.btn.btn-secondary#new_customer_button"))).click()
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe

Selenium webdriver python find element by xpath - Could not find element

I m trying to write a script with selenium webdriver python.
When I try to do a
find_element_by_xpath("//*[#id='posted_1']/div[3]")
it says
NoElementFoundException.
Can someone please help me here?
Regards
Bala

If you are getting NoSuchElementException as your provided exception, There may be following reasons :-
May be you are locating with incorrect locator, So you need to share HTML for better locator solution.
May be when you are going to find element, it would not be present on the DOM, So you should implement WebDriverWait to wait until element visible as below :-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.visibility_of_element_located((By.XPATH, "//*[#id='posted_1']/div[3]")))
May be this element is inside any frame or iframe. If it is, you need to switch that frame or iframe before finding the element as below :-
driver.switch_to_frame("frame/iframe I'd or name")
wait = WebDriverWait(driver, 10)
element = wait.until(EC.visibility_of_element_located((By.XPATH, "//*[#id='posted_1']/div[3]")))
#Once all your stuff done with this frame need to switch back to default
driver.switch_to_default_content();

that exception, unsurprisingly, means that that element wasn't available on the DOM. There are a couple of options here:
driver.implicitly_wait(10)
will tell the driver to wait 10 seconds (or any amount of time) after an element is not found/not clickable etc., and tries again after. Sometimes elements don't load right away, so an implicit wait fixes those types of problems.
The other option here is to do an explicit wait. This will wait until the element appears, and until the existence of that element is confirmed, the script will not move on to the next line:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(ff, 10).until(EC.presence_of_element_located((By.XPATH, "//*[#id='posted_1']/div[3]")))
In my experience, an implicit wait is usually fine, but imprecise.