We Keep Coding

Cannot locate form-control object to send_keys using python Selenium

I am trying to navigate a scheduling website to eventually auto populate a schedule using the following script:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
# Create a Chrome webdriver
driver = webdriver.Chrome(r'C:\Users\chromedriver_win32\chromedriver.exe')
# Navigate to https://www.qgenda.com/
driver.get('https://www.qgenda.com/')
# Wait for the page to load
driver.implicitly_wait(5) # 5 seconds
# You can now interact with the page using the webdriver
# Locate the sign in button
sign_in_button = driver.find_element(By.XPATH,'/html/body/div[1]/div/header[3]/div/div[3]/div/div/div/div/a')
# Click the sign in button
sign_in_button.click()
# Find the input element
input_email = driver.find_element(By.XPATH,'//*[#id="Input_Email"]')
# Send text
input_email.send_keys('Josh')
However, I cannot seem to find the Input_Email object. I've tried all the Xpaths and Id's that make sense and also tried waiting until the object is clickable with no luck. Would really appreciate some guidance on this.
I was expecting Selenium to find the html object form box and pass in text but instead I get an error:
NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="Input_Email"]"}
even though the Xpath definitely exists.

The XPath seems fine. I am guessing you need to do some explicit wait or implicit wait to ensure the page is fully loaded before allocating the element.
Another thing I would like to point out is that given the login URL is available. Locating the sign in button seems to be redundant. You can access it directly via driver.get('https://login.qgenda.com/')
For instance,
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://login.qgenda.com/')
input_email = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="Input_Email"]'))
)
input_email.send_keys('Josh')
You can read more about it here.

click on web Scraping without getting an error with selenium

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome import options
import unittest
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
#link to website
website = 'https://www.ncei.noaa.gov/access/monitoring/climate-at-a-glance'
path = ('../chromedriver') #Folder location where is the chromedriver
driver = webdriver.Chrome(path)
driver.get(website)
driver.maximize_window()
#Selection of the sections where the information I am looking for is located
state_click = driver.find_element(by=By.XPATH, value='//*[#id="show-statewide"]/a').click()
time_series_click = driver.find_element(by=By.XPATH, value='.//*[#id="time-series"]/div[3]/button').click()
#selection of the years (for all files the same range of 1950 - 2021)
star_year_dropdown =Select(driver.find_element(by=By.ID, value='begyear'))
star_year_dropdown.select_by_visible_text('1950')
end_year_dropdown = Select(driver.find_element(by=By.ID, value='endyear'))
end_year_dropdown.select_by_visible_text('2021')
#selection of the parameter to download: Average temperature
parameter_dropdown = Select(driver.find_element(by=By.ID, value='parameter'))
parameter_dropdown.select_by_visible_text('Average Temperature')
#Creating a loop to loop through all the states and all the months:
#state selection
select_state = driver.find_element(by=By.XPATH, value='.//*[#id="state"]')
opcion_state = select_state.find_elements(by=By.TAG_NAME, value='option')
#month selection
select_month = driver.find_element(by=By.XPATH, value = '//*[#id="month"]')
opcion_month = select_month.find_elements(by = By.TAG_NAME, value='option')
for option in opcion_month:
option.click()
for option in opcion_state:
option.click()
time.sleep(3)
plot = driver.find_element(by=By.XPATH, value='.//input[#id="submit"]').click()
dowload = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="csv-download"]'))).click()
time.sleep(3)
The code works fine, but in the plot and download functions created (at the end of the whole) when trying to click it gives an error and cannot be solved. I think it is because the web page at the time of executing that command does not display the button in the screen to plot the graph and download the csv. I have tried modifying the waiting time, but not work. Let's see if someone can help me.
Thanks in advance!!

So the problem that you are having with downloading the CSV file is not that the command is wrong (it does work), however, the issue is that the download CSV button is not visible on the page, which prevents you from clicking it.
A way to get around having to visibly see the element on the page and still click it you can do the following:
driver.execute_script("arguments[0].click();", driver.find_element(By.XPATH, '//*[#id="csv-download"]'))
This would be the preferred method, otherwise you would have to scroll to where the button is visible on the page and then click the button. The page has a scrolling effect if trying to click on a button which you can do the following (but the previous method is preferred as it is cleaner and does not take any additional time to do):
from selenium.common.exceptions import ElementClickInterceptedException
download_btn = driver.find_element(By.XPATH, '//*[#id="csv-download"]')
try:
download_btn.click() # the first time you try to click it will throw an error, while navigating the button into the page view
except ElementClickInterceptedException: # catch the exception and proceed as if clicking again
download_btn.click()
Other comments on your code
you have option.click() twice in your code, and it is unclear if you want to click the month or state option - there may be a confusion about which options are clicked. I would suggest renaming your iterator variables appropriately so that you know which buttons are being clicked

selenium - wait for scrolldown

I am trying to automate downloads from a webpage using selenium.
So far my strategy is to instantiate a firefox driver that loads the page and clicks the download button. However, to be clickable, the button needs to be visible, i.e. not covered by any banners on the page (at least in my understanding). Therefore I need to scroll down (I use scrollIntoView()). If I run button.click() immediately after scrolling down, the download doesn't start, if I hard code a sufficient timeout in between it works out fine. Can somebody help me to set a timeout conditioned on the scroll down?
Here is my code:
from selenium import webdriver
profilePath = '/path/to/my/firefox/profile'
profile = webdriver.FirefoxProfile(profilePath)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/164-andrew-huberman-sleep-dreams-creativity-the-limits-of-the-human-mind")
button = driver.find_element_by_id('btn-download')
target=driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
time.sleep(10)
button.click()
and this is the html code of the button:
<button class="hs-btn-secondary small" id="btn-download" type="button">Download</button>
I would be very thankful for any direct help or suggestions on how to tackle the problem from a different angle.

It sounds like you may need to induce WebDriverWait for the element to by clickable. You could do so like this. It will wait for the element to be both visible and enabled before clicking.
from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
profilePath = '/path/to/my/firefox/profile'
profile = webdriver.FirefoxProfile(profilePath)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/164-andrew-huberman-sleep-dreams-creativity-the-limits-of-the-human-mind")
button = driver.find_element_by_id('btn-download')
target=driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
WebDriverWait(driver, 10).until(EC.element_to_be_clickable(button))
button.click()

Unable to select a menu item with selenium

I am making a screen scraper using the Selenium Python library and I have already made some code so that I can log in. For some reason I am now stuck on the main menu and can't select any of the options. I have tried using CSS Selector, Class Name, and XPATH and none have been able to select any of the possible choices. No matter what happens, I always get a TimeoutException even with a long delay.
The portion of the page I am trying to scrape from is here.
The relevant code is as follows:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
# Open browser and go to the Webex login page
driver = webdriver.Chrome()
driver.get('https://admin.webex.com')
delay = 10 # seconds
long_delay = 20
# Login portion removed
# Menu selection goes here.
# I have tried the following with no luck
# The following lines produce a TimeoutException error
# Selecting menu item
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, "span[#class='left-nav-item__link']"))).click()
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, '//mch-left-nav-item-group[4]/ul/mch-left-nav-item[3]/li/span'))).click()
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, "//webex-root/webex-main[#class='control-hub-container']//webex-sidebar/mch-left-nav/nav/mch-left-nav-item-group[4]/ul/mch-left-nav-item[3]//span[#class='left-nav-item__link']"))).click()
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "span[class='left-nav-item__link']"))).click()
WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "[aria-label] mch-left-nav-item-group:nth-of-type(4) mch-left-nav-item:nth-of-type(3) .left-nav-item__link"))).click()
# Selecting group of items
WebDriverWait(driver, delay).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'left-nav-item')))
WebDriverWait(driver, delay).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'li.left-nav-item')))
# Selecting parent
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "li[data-test-name='calling']")))
Does anyone have an idea why I'm not able to select any element?

Try xpath for the third one:
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, "span[#class='left-nav-item__link']"))).click()

To anyone having this issue.
When a new tab is opened while you are using Selenium, even though you are viewing the new page, Selenium is not. You have to switch to the correct window. Something such as the following can help you with that.
driver.switch_to.window(driver.window_handles[NUMBER])

Python with selenium webscraping unable to find elements

I am trying to write a webscraping in python that will activate the "onclick" functionality of certain buttons on a webpage because the tables with the data I want are converted to csv, which makes it much easier to access. But the problem is that I am unable to locate elements by xpath at all when using PhantomJs. How can I click the element and access the csv content that I want?
This is my code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
url = "http://www.pro-football-reference.com/boxscores/201609180nwe.htm"
xpath = "//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button"
path_to_phantomjs = 'browser/phantomjs'
browser = webdriver.PhantomJS(executable_path = path_to_phantomjs)
browser.get(url)
delay=3
element_present = EC.presence_of_element_located((By.ID, 'all_player_offense'))
WebDriverWait(browser, delay).until(element_present)
browser.find_element_by_xpath(xpath).click()
And I get this error:
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"153","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50989","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"sessionId\": \"93ff24f0-9cbe-11e6-8711-bdfa3ff9cfb1\", \"value\": \"//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/93ff24f0-9cbe-11e6-8711-bdfa3ff9cfb1/element"}}
Screenshot: available via screen

IMPORTANT THING I FORGOT TO MENTION: As described in this this issue on GitHub, try putting set_window_size(width, height) or maximize_window()after setting the webdriver. You should also consider telling the webdriver to implicitly_wait(10) for the element to appear.
So there's a special maneuver you have to perform in order for the Selenium Webdriver to properly emulate what you're doing. In essence, to get the desired data, you'd have to:
A: Hover over the "Share & More" dropdown menu. Then
B: Click "Get table as CSV (Excel)".
For A, this involves having to place the emulated cursor on the element without clicking it. This idea of "mouse over" can be done with the move_to_element() function provided in the ActionChains class. So at the top you'd insert this:
from selenium.webdriver.common.action_chains import ActionChains
You want Selenium to find the specific element and move to it. You can achieve this with 2 lines of code:
dropdown = browser.find_element_by_xpath('//*[#id="all_player_offense"]/div[1]/div/ul/li[1]')
ActionChains(browser).move_to_element(dropdown).perform()
If you omit the above, you'll get an ElementNotVisibleException.
Now for B, you should be able to do browser.find_element_by_xpath(xpath).click().