We Keep Coding

Selenium element not interactable error on headless mode but works without headless

I'm trying to scrape the webpage ted.europa.eu using Python with Selenium to retrieve information from the tenders. The script is supposed to be executed once a day with the new publications. The problem I have is that navigating to the new tenders I need Selenium to apply a filter to get only the ones from the same day the script it's executed. I already have the script for this and works perfectly, the problem is that when I activate the headless mode I get the following error selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable: [object HTMLInputElement] has no size and location
This is the code I have that applies the filter I need:
import sys
import time
import re
from datetime import datetime
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from dotenv import load_dotenv
load_dotenv("../../../../.env")
sys.path.append("../src")
sys.path.append("../../../../utils")
from driver import *
from lted import LTED
from runnable import *
# start
print('start...')
counter = 0
start = datetime.now()
# get driver
driver = get_driver_from_url("https://ted.europa.eu/TED/browse/browseByMap.do%22)
actions = ActionChains(driver)
# change language to spanish
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "lgId")))
driver.find_element(By.ID, "lgId").click()
driver.find_element(By.XPATH, "//select[#id='lgId']/option[text()='español (es)']").click()
# click on "Busqueda avanzada"
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "goToSearch")))
driver.find_element(By.ID, "goToSearch").click()
# accept cookies and close tab
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "cookie-consent-banner")))
driver.find_element(By.XPATH, "//div[#id='cookie-consent-banner']/div[1]/div[1]/div[2]/a[1]").click()
driver.find_element(By.XPATH, "//div[#id='cookie-consent-banner']/div[1]/div[1]/div[2]/a[1]").click()
# click on specific date and set to today
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "publicationDateSpecific")))
element = driver.find_element(By.ID, "publicationDateSpecific")
actions.move_to_element(element).perform()
driver.find_element(By.ID, "publicationDateSpecific").click()
driver.find_element(By.CLASS_NAME, "ui-state-highlight").click()
# click on search
driver.find_element(By.ID, "search").click()
From the imports the only think I need to explain is that from the line from dirver import * comes the method get_driver_from_url() that is used later in the code. This method looks like this:
def get_driver_from_url(url):
options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--start-maximized")
options.add_argument("--headless")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get(url)
return driver
As I said this code works perfectly without the headless mode, but when activated I get the error.
At first got another error and searching on the Internet found out that it could be because the element is not on screen, so I added the argument "--start-maximized" to make sure the Chrome tab is as big as possible and added the ActionChains to use actions.move_to_element(element).perform(), but I get this error on this exact code line.
Also tried changing the line WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "publicationDateSpecific"))) to WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "publicationDateSpecific"))) but it just didn't work.
Update: Also tried changing to EC.visibility_of_element_located as mentioned in this post but didn't work either
What am I doing wrong?

This is probably because of the window size.
Try adding this:
chrome_options = Options()
chrome_options.add_argument("--window-size=1920,1080")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--headless")

So, after a long time of try and error, I found that adding
element = driver.find_element(By.ID, "publicationDateSpecific")
driver.execute_script("window.scrollTo(0,"+str(element.location["y"])+")")
makes the script work both in headless mode and normal mode

Python - Using Selenium to Login to Web

I cannot login to this site with Selenium.
This is the url.
https://www.burn-cycle.com/my-account/pearl-district
What I tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import yaml
import time
conf = yaml.full_load(open("login_details.yml"))
my_burn_email = conf["user"]["email"]
my_burn_password = conf["user"]["password"]
driver = webdriver.Chrome()
driver.get("https://www.burn-cycle.com/my-account/pearl-district")
time.sleep(1)
username = driver.find_element(By.XPATH, "//*[#id='USERNAME']")
username.send_keys(my_burn_email)
pw = driver.find_element(By.XPATH, "//*[#id='PASSWORD']")
pw.send_keys(my_burn_password)
login_button = driver.find_element(By.XPATH("//*[#id='liFormWrap']/form[1]/button")).click()
The website loads (slowly) but nothing populates. This is the output:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id='USERNAME']"}
(Session info: chrome=103.0.5060.134)
What am I doing wrong?

You need to wait for the website to load completely so that you can fetch those elements from the webpage, you can achieve this by using implicitly.wait(#amount of second) command right after initializing the web driver.
driver = webdriver.Chrome()
driver.implicitly_wait(15) # gives an implicit wait for 15 seconds

the element is under iframe, you need try this way, first switch into iframe,
see this link
driver.get("https://www.burn-cycle.com/my-account/pearl-district")
time.sleep(5)
iframe = driver.find_element(By.XPATH, "//*[#id='sf-frame']")
# switch to selected iframe
driver.switch_to.frame(iframe)
username = driver.find_element(By.XPATH, "//input[#data-val-required='Username is required']")
username.send_keys("test")

Capture elements with Selenium which are loaded lazy in Python

Trying to click a button with Selenium, but I keep getting an error:
NoSuchElementException: Message: no such element: Unable to locate
element: {"method":"link text","selector":"AGREE"}
Here's the button I am trying to click.
I assume, the popup is loaded lazy. I found some sources, but could not make it work. Here's my code
import pandas as pd
import bs4
import selenium as sel
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import time
import os
driver = sel.webdriver.Chrome() #uses Chrome driver in usr/bin/ from https://chromedriver.chromium.org/downloads
url = 'https://fbref.com/en/squads/0cdc4311/Augsburg'
driver.get(url)
time.sleep(5)
html = driver.page_source
soup = bs4.BeautifulSoup(html, 'html.parser')
#click button -> accept cookies
element = driver.find_element(By.LINK_TEXT, "AGREE")
element.click()
>>> NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"AGREE"}
I also tried
[...]
driver = sel.webdriver.Chrome() #uses Chrome driver in usr/bin/ from https://chromedriver.chromium.org/downloads
driver.implicitly_wait(10) # seconds'
[...]
and the popup is definitively there. But still get the same error.

What happens?
You try to select a <button> via .LINK_TEXT, "AGREE", what won't work, cause it is a <button> not a link.
How to fix?
Wait for the <button> selected by xpath to be clickable:
#click button -> accept cookies
element = WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.XPATH, '//button[text()="AGREE"]')))
element.click()

Python: Selenium "no such element" XPath or ID

I am attempting to log into Tessco.com
I have learned this site uses JavaScript, which is why I was unable to locate a form using RoboBrowser.
I am now using Selenium. I have used two methods to enter information into a field. One, using the driver.find_element_by_xpath()as well as driver.find_element_by_id()
Both attempts yield an error.
The code is as follows:
import time
from selenium import webdriver
chrome_path = r"C:\Users\James\Documents\Python Scripts\jupyterNoteBooks\ScrapingData\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.tessco.com/login")
userName = "FirstName.SurName321123#gmail.com"
password = "PasswordForThis123"
elem = driver.find_element_by_xpath("""//*[#id="userID"]""")
elem = send_keys(userName)
elem = driver.find_element_by_xpath("""//*[#id="password"]""")
elem = send_keys(password)
driver.close()
The error is:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="userID"]"}
When I call the element by using ID as such:
elem = driver.find_element_by_id("userID")
elem = send_keys(userName)
elem = driver.find_element_by_id("password")
elem = send_keys(password)
I get:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="userID"]"}
I was under the impression I inspected the element and used the appropriate names.
Any hints or ideas what I am not doing correctly?
Solution provided below in the comments section.
Code modified to:
import time
from selenium import webdriver
#Webdriver wait functions being introduced to add a delay.
#I was running into problems, with the ID not being found
#add wait for the element to be clickable before trying send keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = r"C:\Users\James\Documents\Python Scripts\jupyterNoteBooks\ScrapingData\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.tessco.com/login")
userName = "FirstName.SurName321123#gmail.com"
password = "PasswordForThis123"
wait = WebDriverWait(driver, 10)
elem = wait.until(EC.element_to_be_clickable((By.ID, "userID")))
elem.send_keys(userName)
elem = wait.until(EC.element_to_be_clickable((By.ID, "password")))
elem.send_keys(password)
driver.close()

It's likely that the element is not visible at first and that causes the failure. Wait until the element is visible/clickable and then use send keys.
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "userID"))).send_keys(userName)

I guess the element search happens before its loaded. My suggestion would try putting some wait and it should work.

Selenium iframe Data issue - python

I have been trying to access the widget data from https://www.dukascopy.com/trading-tools/widgets/quotes/historical_data_feed so that my program can scrape the website and select the search bar to enter the data and return the result in the iframe. However, even when I convert my driver to the iframe it still can't find the iframe data elements. the error:
I tried to add the time delay for everything to load but still no luck.
elenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"d-e-Xg"}

There are actually two nested iframes in play here. That, combined with some WebDriverWaits allowed me to arrive at the following solution. Here I locate the "Instrument" field and input text into it:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(
options=options, executable_path=r"C:\chromedriver\chromedriver.exe"
)
driver.implicitly_wait(10)
try:
driver.get(
"https://www.dukascopy.com/trading-tools/widgets/quotes/historical_data_feed"
)
driver.switch_to.frame(driver.find_element(By.ID, "widget-container"))
driver.switch_to.frame(driver.find_element(By.TAG_NAME, "iframe"))
# Wait for "Loading" overlay to disappear
WebDriverWait(driver, 10).until(
ec.invisibility_of_element_located((By.CLASS_NAME, "d-e-r-k-n"))
)
# Click "Instrument"
driver.find_element(By.CLASS_NAME, "d-oh-i-ph-qh-rh-m").click()
# Enter text into now-visible instrument field
driver.find_element(By.CLASS_NAME, "d-e-Xg").send_keys("metlife")
finally:
driver.quit()