We Keep Coding

How to pass the RGPD popup / iframe with Selenium (Python)?

Ok, attempt to use Selenium on a website : lefigaro.fr, but no class related to the RGPD popup to be found by Selenium, even after a switch to frame. :/
I'm juste looking a reliable way to close it.
It goes this way :
from selenium import webdriver
WINDOW_SIZE = "1920,1080"
ADRESSE = 'https://www.lefigaro.fr/'
driver = webdriver.Firefox() #the chrome version was even worse, geckodriver a the root
driver.get(ADRESSE)
elt = driver.find_element_by_class_name("sc-18sn7k8-1") #error
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
elt = driver.find_element_by_class_name("sc-18sn7k8-1") #error

You can switch to iframe via index.
It seems that popup is the only iframe on the page, so perhaps this will work:
driver.switch_to.frame(0) # or perhaps (1)
EDIT:
It seems the page already is in an iframe when it loads.
Do driver.switch_to.default_content() first.
Then it seems the iframe you want is (3):
driver.switch_to.frame(3)
Hope this works.
Hmm, actually, not quite... I'll spend some more time on this.

Selenium Error: Element not interactable (coockie and other pop up)

I am trying to press a button with selenium because afterwards I need to inspect the full html of the website. This is the code that I am using:
driver = webdriver.Chrome()
driver.get('https://www.quattroruote.it/listino/audi/a4-allroad')
time.sleep(10)
html = driver.find_element_by_id('btnallestimenti')
html.click()
But I get this error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
when the page is open there are cookies and other things that shows up, is there a way to block all of them so that I can work on the html?
Thanks a lot!

As you can see the "cookies" banner is an HTML element itself and it contains a "Close" ("Chiudi") button that can be clicked.
If you inspect the page source you will find this code that relates to that button:
<button type="button" class="iubenda-cs-close-btn" tabindex="0" role="button" aria-pressed="false" style="font-size:16px!important;">Chiudi</button>
Your script needs to be modified to search the element by visible text (using the XPath) and click it to close the banner:
close_button = driver.find_element_by_xpath("//*[text()='Chiudi']")
close_button.click()
I can see that this kind of banner appears 2 times (one for cookies one for "Informativa") but once you click this one away you are redirected to the right page.
Of course you will need to test your script and adjust it to the page's behavior.
Also, be aware that every time the pages change because the devs change it your script will break and you will need to re-adjust it.
EDIT
Posting here the full code, try to use it and continue from here:
import time
from selenium.webdriver import Chrome
driver = Chrome()
driver.get("https://www.quattroruote.it/listino/audi/a4-allroad")
time.sleep(6)
driver.find_element_by_xpath("//button[text()='Accetta']").click()
time.sleep(6)
driver.switch_to.frame("promo-premium-iframe")
driver.find_element_by_xpath("//a[normalize-space()='Non sono interessato']").click()
time.sleep(6)
driver.switch_to.default_content()
driver.find_element_by_id("btnallestimenti").click()
input()

You could try to accept the cookie and proceed further, check the below lines of code.
options = Options()
options.add_argument("--disable-notifications")
driver = webdriver.Chrome(options=options,"ChromeDriver_Path")
driver.maximize_window()
driver.get('https://www.quattroruote.it/listino/audi/a4-allroad')
sleep(10)
cookie_btn = driver.find_element_by_xpath("//button[text()='Accetta']")
cookie_btn.click()
sleep(3)
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"promo-premium-iframe")))
register_btn = driver.find_element_by_xpath("//a[normalize-space()='Accedi o Registrati']")
register_btn.click()
Iframe available so just switched to the iframe, tried the to perform the registration.

Scraping only the portion that loads - Without Scrolling

I have written a simple web scraping code using Selenium but I want to scrape only the portion that is present 'before scroll'
Say, if it is this page I want to scrape - https://en.wikipedia.org/wiki/Pandas_(software) - Selenium reads information till the absolute last element/text which for me is the 'Powered by Media Wiki' button on the far bottom-right of the page.
What I want Selenium to do is stop after DataFrames (see screenshot) and not scroll down to the bottom.
And I also want to know where on the page it stops. I have checked multiple sources and most of them ask for infinite scroll websites. No one asks for just the 'visible' half of a page.
This is my code now:
from selenium import webdriver
EXECUTABLE = r"chromedriver.exe"
# get the URL
url = "https://en.wikipedia.org/wiki/Pandas_(software)"
# open the chromedriver
driver = webdriver.Chrome(executable_path = EXECUTABLE)
# google window is maximized so that all webpages are rendered in the same size
driver.maximize_window()
# make the driver wait for 30 seconds before throwing a time-out exception
driver.implicitly_wait(30)
# get URL
driver.get(url)
for element in driver.find_elements_by_xpath("//*"):
try:
#stuff
except:
continue
driver.close()
Absolutely any direction is appreciated. I have tried to be as clear as possible here but let me know if any more details are required.

I don't think that is possible. Observe the DOM, all the informational elements are under one section I mean one tag div[#id='content'], which is already visible to Selenium. Even if you try with //*, div[#id='content'] is visible.
And trying to check whether the element is visible though not scrolled, will also return True. (If someone knows to do what you are asking for, even I would like to know.)
from selenium import webdriver
from selenium.webdriver.support.expected_conditions import _element_if_visible
driver = webdriver.Chrome(executable_path = 'path to chromedriver.exe')
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://en.wikipedia.org/wiki/Pandas_(software)")
elements = driver.find_elements_by_xpath("//div[#id='content']//*")
for element in elements:
try:
if _element_if_visible(element):
print(element.get_attribute("innerText"))
except:
break
driver.quit()

Python Selenium "Unable to locate element"

I am trying to use Selenium to sign up an email account automatically whenever I need to. It's just a fun learning project for me. For the life of me I don't understand why it can't find the element. This code works fine on the sign-in page but not the sign-up page. I have tried all different Selenium commands and even tried using the ID and class name. Either is says it can't locate the element or that it is not reachable by keyboard.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time
options = Options()
driver = webdriver.Firefox(options=options, executable_path=r'geckodriver.exe')
driver.get("https://mail.protonmail.com/create/new?language=en")
time.sleep(10)
username_input = driver.find_element_by_id("username").send_keys("testusername")
Also here is the HTML code: https://i.imgur.com/ZaBMTzG.png

The username field is in iframe, you need to switch to iframe to make this work.
Below is the code that works fine :
driver.get("https://mail.protonmail.com/create/new?language=en")
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[title='Registration form'][class='top']"))
driver.find_element_by_id("username").send_keys("some string")
read more about iframe here
learn more about how to switch to iframe/frame/framset using Python
selenium Bindings here
Update :
wait = WebDriverWait(driver, 30)
driver.get("https://mail.protonmail.com/create/new?language=en")
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[title='Registration form'][class='top']"))
driver.find_element_by_id("username").send_keys("some string")
driver.switch_to.default_content()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
sleep(5)
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[title='Registration form'][class='bottom']"))
wait.until(EC.element_to_be_clickable((By.NAME, "submitBtn"))).click()

I'm not sure if I've seen enough code to diagnose, but I think the way you are defining username_input seems problematic. driver.find_element_by_id("username").send_keys("testusername") doesn't actually return anything so it seems like you are setting username_input = null.

Is there a way around cookie and privacy popus in selenium browser?

So I'm trying to make a python script using selenium and bs4 to automatically buy shoes for from adidas.com. It's just that whenever selenium browser starts the adidas site it shows a popup concerning cookies and privacy. I can't click on the accept button using selenium(can't find the element to click on) and I've tried starting the selenium browser with my firefox profile containing cookies and what not. But it still shows that damn popup and it's stopping the script.
Is there anyway to fix this?
Tried starting selenium with firefox cookies like this:
ffprofile = webdriver.FirefoxProfile(r'C:\Users\chico\AppData\Local\Mozilla\Firefox\Profiles\e5108gza.default')
driver = webdriver.Firefox(firefox_profile=ffprofile)
driver.get('https://www.adidas.nl/on/demandware.store/Sites-adidas-NL-Site/nl_NL/MyAccount-CreateOrLogin')
driver.find_element_by_xpath("//input[#id='dwfrm_login_username']").send_keys('email')
driver.find_element_by_xpath("//input[#id='dwfrm_login_password']").send_keys('password')
driver.find_element_by_xpath("//button[#value='Inloggen']").click()
The popup from the adidas site just keeps poping up and stopping my script from continuing. Sometimes it will fill in email and password before being stopped, sometimes it will be stopped before that.

I ran into this same issue recently, and it was SO frustrating, because it can't be handled with a "switch to alert" and "accept" it approach sometimes.
Here's how I got around my issue. First, and this was most important, you need to "inspect" the page you are writing selenium code for in a way that the cookie acceptance popup will show up.
For me, I was using the Chrome driver, and so for inspecting the page, I had to run incognito, so that the page would not know I had already accepted cookies. Use the method for Firefox that will achieve the same effect.
Then you can find the element ID (or class name) for the acceptance button and add a click to the end of your command. My code to do this looks as follows:
from selenium import webdriver
import time
URL = "http:some_site.com"
driver = webdriver.Chrome()
driver.get(URL)
time.sleep(5)
driver.find_element_by_id(the_id_name_for_cookie_acceptance_as_a_string).click()
# Do more stuff ...
Now, if you run into a cookie dialog that won't act as an alert, you can handle it.

I tried the code you posted and didn't receive any alert pop up.But if there is a Alert windows it needs to be handled by switching to it and accepting it
from selenium import webdriver
driver = webdriver.Firefox()
driver.set_page_load_timeout(30)
driver.get('https://www.adidas.nl/on/demandware.store/Sites-adidas-NL-Site/nl_NL/MyAccount-CreateOrLogin')
try:
WebDriverWait(browser, 3).until(EC.alert_is_present(),'Timed out waiting for alert popup')
alert = driver.switch_to.alert
alert.accept()
except:
print "no alert"
driver.find_element_by_xpath("//input[#id='dwfrm_login_username']").send_keys('email')
driver.find_element_by_xpath("//input[#id='dwfrm_login_password']").send_keys('password')
driver.find_element_by_xpath("//button[#value='Inloggen']").click()