We Keep Coding

Selenium can't find element, while I can find it in the HTML code

I am using selenium to type in a word in a searchbar and then select a random suggested searchword and click on it. However, I am stuck since selenium doesn't seem to find the list of suggested searchwords. The code that I am using looks like this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
search = driver.find_element(By.XPATH,"//*[#id='searchfor']")
search.send_keys("test")
suggestions = WebDriverWait(driver,30).until(EC.visibility_of_all_elements_located((By.XPATH,"//*[#id='siteSearch']/wsp-search-suggestions/ul[2]/li")))
However, when I use the XPATH to find the elements in the HTML code I do find them:
as you can see in this image
I am getting a TimeoutException error. Does anyone have an idea of what I am missing?
UPDATE: When I opened the inspect in Chrome, the searchbar moved from its full-screen position to another position. Apparently the class of the search suggestions also changes then. So by just putting the inspect pane on the bottom of the page I was able to find the correct class and now it works!

Unable to select Radio button with Selenium (python)

I've been trying to select a radio button using selenium and I'm having no luck. All other selectors (Login, drop down, etc) have all worked fine.
driver.find_element_by_xpath('//*[#id="content_grid"]/div[1]/div[2]/div[4]/div[2]/div[3]/label/div[1]/input').click()
This is the radio button I'm trying to select...
[1]: https://i.stack.imgur.com/X64k8.png
Here is the webpage - https://stathead.com/basketball/pgl_finder.cgi
Appreciate any help! First time poster and noobie coder :)

Few options to solve the problem.
The problem occur as upon retrieving the element and scrolling down to click on it, the site adds a banner that may obstruct the element.
One option is to always open the browser in fullscreen.
driver = webdriver.Chrome()
driver.maximize_window()
This should help avoiding the banner to be in the way.. less scrolling and less chance for the banner to overlap with the radio button
Selenium also offer a library to return an element upon a condition being met using WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
el = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '[value="E"]')))
Basically, the above will return the element once it is reported as clickable. The clickability of the element might not be in cause here. This way only ensure that the element found can be clicked but does not validation if there is an overlap (which I suspect is the issue here).
Third option, you may go with javaScript to click on the element
driver.execute_script("arguments[0].click();", el)

Using selenium driver.find_element().click() on a subdivided webpage

First of all, I'm completely new in web scraping, html and selenium, so my question might not seem meaningful to you.
What I'm trying to do: automated click on an image on a webpage. Manual click on this image allows to display new information on a part of the webpage.
How I tried to do it: I found something specific to this image, for example:
<img src="images/btn_qmj.gif" border="0">
So I just entered in python:
xpath = '//img[#src="images/btn_qmj.gif"]'
driver.find_elements_by_xpath(xpath)
Problem: this returns me an empty list. I partially understood why, but I don't have any solution.
image of the html code inspector
I included here an image of the html tree I obtain on the web inspector. As one can see, the tree to reach my line of interest gives "/html/frameset/frame1/#document/html/body/center/table/tbody/tr[2]/td/a/img".
The issue is that I cannot access -by using an xpath- anything that comes after the #document. If I try to copy the xpath of my line of interest, it tracks it back only up to the 2nd "/html". It is like the page is subdivided, compartmentalized, with inner parts I cannot access. I suppose there is a way to access the #document content, maybe it refers to another webpage, but I'm stuck at that point.
I would be very grateful if anyone could help me on this.

Have you tried switching to the frame first?
driver.switch_to.frame('gauche')

You need to switch control to frame first to handle image on it. Please see below code for your reference
driver.switch_to_frame(frame)
WebDriverWait(driver, 20).until(
EC.visibility_of_element_located((By.XPATH, "'//img[#src="images/btn_qmj.gif"]'"))).click()
driver.switch_to_default_content()
Note :: You need to add below imports
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

The element is present inside an iframe.In order to access the element you need to switch it first.
Induce WebDriverWait() and frame_to_be_available_and_switch_to_it()
Induce WebDriverWait() and visibility_of_element_located()
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"gauche")))
WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//img[#src='images/btn_qmj.gif']"))).click()
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Scrape data not visible in the source file Python

I want to scrape the data on the website https://www.climatechangecommunication.org/climate-change-opinion-map/. I am somewhat familiar with selenium. But the data I need which is below the map and the tooltip on the map is not visible in the source file. I have read some posts about using PhantomJS and others. However, I am not sure where and how to start. Can someone please help get me started.
Thanks,
Rexon

You can use this sample code:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://www.climatechangecommunication.org/climate-change-opinion-map/")
# switch to iframe
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[#src = 'https://environment.yale.edu/ycom/factsheets/MapPage/2017Rev/?est=happening&type=value&geo=county']")))
# do your stuff
united_states = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[#id='document']/div[4]//*[name()='svg']")))
print(united_states.text)
# switch back to default content
driver.switch_to.default_content()
Output:
50%
No
12%
Yes
70%
United States
Screenshot of the element:
Explanantion: first of all, to be able to interact with elements below the map you have to switch to iframe content, otherwise it is not possible to interact with this elements. Then the data below the map is in svg tags, which are also not trivial. To be able to do this, you the sample I have provided.
PS: I have used WebDriverWait in my code. With WebDriverWait your code becomes quickier and stable, since Selenium waits for particular conditions like visibility or clickable of particular element. In the sample code the driver wait at least 10 seconds until expected condition will be satisfied.

How do I click on links that are contained in a div container fluid using python selenium

I'm writing a piece of python code to test whether users are able to click on dynamic tabs.
I'm using find_elements_by_xpath() to search for attributes of the tab. Now, this is a dynamic set of tabs, so they will never be viewable in the page source. But they do exist when I click on inspect element.
I'm using something like this to identify the tabs:
elem = self.browser.find_elements_by_xpath("//a[contains(#role,'tab')]")
for i in elem:
print(i.text)
I tried the get_attribute feature but that did not work. Is there any way I can use python selenium to click on a dynamic tab (that does not appear on page source)?

You can wait some time until required element present in DOM and clickable:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
wait(self.browser, 10).until(EC.element_to_be_clickable((By.XPATH, '//a[contains(#role, "tab")]'))).click()