We Keep Coding

How can Selenium (Python, Chrome) find web elements visible in dev tools, but not visible in page source?

I need to click the first item in the menu in a webpage with Python3 using Selenium.
I manage to log-in and navigate to the required page using Selenium, but there I get stuck: it looks like Selenium can't find any element in the page beyond the very first div in body.
I tried to find the element by ID, class, xpath, selector... The problem is probably not about that. I thought it could be about an iframe, but the content I need does not seem to be in one.
I guess that the problem is that the element I need to find is visible in the devtools, but not in the page source, so Selenium just can't see it - does this make sense? If so, can this be fixed?
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
self.driver.get("my site")
# log-in website and navigate to needed page
# [...]
# find element in page
# this works
first_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper")
# this does not work
second_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper > div.layout.flex.flex-col.overflow-x-hidden.h-display-flex.h-flex-direction-column.h-screen")
Edit
The problem is most likely due to a dynamic webpage with parts of the DOM tree attached later on by a script. I downloaded a local version of page.html, removed scripts, and successfully found the sought-after element in the local page with
from selenium import webdriver
from selenium.webdriver.common.by import By
from pathlib import Path
driver = webdriver.Chrome()
html_file = Path.cwd() / "page.html"
driver.get(html_file.as_uri())
my_element = driver.find_element(By.CSS_SELECTOR, "[title='my-title']")
The exact same driver.find_element query won't work on the online page. I'm trying to implement a waiting condition as suggested in Misc08's answer.

I guess that the problem is that the element I need to find is visible
in the devtools, but not in the page source, so Selenium just can't
see it - does this make sense? If so, can this be fixed?
No, this does not make sense, since Selenium is executing a full browser in the background like you are using when you investigate the page source with the devtools.
But you have some options to narrow your problem. The first thing you can do, is to print the source the webdriver is "seeing" in this moment:
print(driver.page_source)
If you see the elements you are looking for in the page source, than you should try to improve your selector. It is helpful to go down the DOM step by step. Look for an upper element in the page tree first. If this works, try to find the next child element, then the next child, and so on. You can check if selenium found the element like this:
try:
myelement = driver.find_element(By.CSS_SELECTOR, 'p.content')
print("Found :)")
except NoSuchElementException:
print("No found :(")
By the way, i think your CSS selector is by far to complex, just use on CSS class, not all of them:
second_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper > div.layout")
But there might be the case, where the elements you are looking for, are not present in the page source from the beginning on. Dynamic webpages getting more and more popular. In this case parts of the DOM tree are attached later on by a script. So you have to wait for the execution of the scripts, before you can find this "dynamic" elements. One dirty and unreliable option is to just add a sleep() here. Much better is to to use an explicit waiting condition, see https://selenium-python.readthedocs.io/waits.html

Accepting cookies popups using selenium in python

I'm trying to put together a small scraper for public trademark data. I have a database available that i'm using selenium and python to access.
I can do just about anything I need to be able to, but for some reason i can't actually click the "accept cookies" button on the website. The following code i use highlights the button, but it does not get rid of the popup.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://data.inpi.fr/recherche_avancee/marques')
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "tarteaucitronPersonalize2"))
).click()
I have looked up similar threads on this forum, and I have tried multiple things :
- adding a waiting period, that ended up highlighting the button, so at least i know it does something
- using javascript code to do the actual click, did not work any better
- i tried calling the button via its ID, its XPATH, its CSS selector, anything i could find really
I even downloaded Selenium IDE to record my clicks to see exactly how I could replicate it, but it still only recorded a click.
I tried my best, does anyone know where my mistake lie ? I am open to using other languages, or another platform

Well it looks like I managed to solve it just minutes after posting my question !
For some reason you need to resize the window. I just added the following line of code after opening the URL and it worked first time.
driver.maximize_window()
I added this answer in case anyone stumbles upon this post and wants to avoid pulling their hair out over this !

Selenium Xpath, can someone help me with this find.element

I have been really banging my head on this, I can't figure out my problem, if someone can help me it would be appreciated, thank you :).
I'm trying to select with Xpath the twitter search box, from what I know, my input is correct, since on google when I press Ctrl+F and search for it in the "inspect mode" it's highlighting in yellow the search box input tag. But in selenium, I wish it was.
(Using google chrome)
import csv
from getpass import getpass
import time
from time import sleep
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from selenium import webdriver
#other selenium code, working.
twitterS = driver.find_element_by_xpath('//input[contains(#data-testid,"SearchBox")]')
Return
selenium.common.exceptions.NoSuchElementException:
Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//input[contains(#data-testid,"SearchBox")]"}
I tried the relative path, not working either
/html/body/div[1]/div/div/div[2]/main/div/div/div/div[2]/div/div[2]/div/div/div/div[1]/div/div/div/form/div[1]/div/div/div[2]/input

Your XPath seems to be fine (I've checked it). If Selenium tells you the elements is not there it means that it's not there in the moment it tries to find it. I don't see any frame or shadow DOM there so I would say you need to wait for this element before getting it. I would also make sure if at that point you are really on the page you're expecting to see the element at. Without the code I can't really help better.

selenium.common.exceptions.ElementNotVisibleException: Message: element not interactable and explicit wait not working with Selenium and Python

I am trying to access PenFed in order to get my current outstanding amount. I have done quite a bit of research and unfortunately, I am still stumped. I am using Python Selenium and I am trying to click on the initial login button on the side in order to see the username field. This is the element's HTML code:
Login
When I try to run the following code:
driver.find_element_by_id("mobile-login").click()
I get the following error:
selenium.common.exceptions.ElementNotVisibleException: Message: element not interactable
Even when I try to use WebDriver Wait functions such as these:
try:
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.ID, "mobile-login"))).click()
except ElementNotVisibleException:
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "mobile-login"))).click()
No matter how long I make them wait, I get a timeout message:
raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:
All of my research says that invoking a wait function should fix it but it doesn't work for me. I also read that there might be an image overlay on top of the element that I would have to invoke before clicking on the button, but I didn't see anything in the website code as well. If I am testing it out, the only way that I am able to click on the button through code is if I physically click on it first, so I am unaware of anything else I can use. Thank you in advance for the help!
UPDATE: I have discovered that the following code works for me:
element = driver.find_element_by_id("mobile-login")
driver.execute_script("$(arguments[0]).click();", element)
But I do not know what the execute_script actually does. Can someone explain that piece of code works or if any other alternatives work for them?

The code you specified is JQuery. execute_script(p1, p2) runs a js script, where p1 is the script (in your case a JQuery line that clicks the element) and p2 is the desired element. It seems like you shouldn't need p2 if arguments[0] is equal to "element," but I'm not totally sure.
One potential fix is to use a counter for the number of times you clicked the element. If the counter reaches a certain number and the page doesn't change (you can check by finding a unique element/value on your current page), then you know it's not clickable.
Good luck!

The desired element is a dynamic element so to locate the element you have to induce WebDriverWait for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument('start-maximized')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.penfed.org/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.pfui-button.login-slide-button.pfui-button-login.dtm-global-nav[data-id='Open Log In Drawer']"))).click()
Browser Snapshot:

The link you are attempting to click is for the mobile site and is not visible if you are viewing the site at desktop resolutions. If you shrink your browser down until it changes layout, you will see the LOGIN button appear that corresponds to that link. That's why you are getting the ElementNotVisibleException.
To your second question, the reason that using .execute_script() works is that it executes JS directly and can click on anything... hidden or not. Selenium was designed to interact with the page as a user would so it won't let you click invisible elements, etc.
If you are intending for your script to act like a user, you will want to avoid using .execute_script() because it allows you to do things on the page that a user cannot do.
If you want to log in like a desktop user would, you need to click the "LOG IN" button using the CSS selector below
button[data-id='Open Log In Drawer']
That will open a side panel where you can enter your username, etc. and log in. FYI... you will likely need a wait to give the panel a chance to open before continuing the log in process.

Select Radio Button using Python/Selenium

Have searched through some of the previous questions and have not been able to rectify - I'm a complete newbie to this, so please forgive the ignorance... Trying to select the third 'Radio' button on the page using the following:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
import smtplib
import time
date = time.strftime("%d %b %Y" + ", " + "%H:%M%p")
print (time.strftime("%d %b %Y" + ", " + "%H:%M%p"))
driver=webdriver.Chrome()
driver.maximize_window()
driver.get("http://www.water.nsw.gov.au/water-licensing/registers")
driver.implicitly_wait(10)
driver.find_element_by_xpath('// [#id="wizard_tabs"]/div/div[1]/div/ul/li[3]/input').click()
The result is:
"Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//[#id="wizard_tabs"]/div/div[1]/div/ul/li[3]/input"}"
Have tried waiting for longer (500), but it doesn't make a difference. When I 'inspect' the page that has opened, the xpath is still the same. I know there's a simple answer and I'm hoping you internet legends can help!

A few points of order:
There is no point in implicitly waiting when you can just perform an explicit wait (WebdriverWait.until) on a unique element on that page.
If the webdriver says the element isn't there, then the locator you are looking for is not there, at least not where it is actually looking. iframes can make you think you are looking in one place when really the webdriver is looking in another.
Always confirm your locators in the browser the moment you suspect something is off. Or just every time, it will save all that time you spend running your script, launching your browser, and loading the page every time as you tinker. Go to your browser's dev tools, click on the console tab, and type $$("css selector here") or $x("xpath here") and hit enter. If an element is returned, click the arrow to expand it and hover over what displays. Is it highlighting the desired element on the page? Then and ONLY then are you using a good locator.
If the dev tools console says your locator is good, but python still won't find it - even after a WebdriverWait - find the iframe that is stealing your focus and use webdriver.switch_to(iframe_name) to steal your focus back to the iframe with your element.
Practice less-brittle XPATHs. They can be quite surgical and readable with minimal effort. I would be willing to bet your input has something we could use to find it directly. If not, an element right above or below it in the hierarchy likely does. Post the HTML of your desired element and we'll know for sure.
To handle an iframe issue:
iframe_element = driver.find_element_by_css_selector("iframe[name='name_of_the_iframe']")
driver.switch_to.frame(iframe_element)
# Now find your element