I've been trying to select a radio button using selenium and I'm having no luck. All other selectors (Login, drop down, etc) have all worked fine.
driver.find_element_by_xpath('//*[#id="content_grid"]/div[1]/div[2]/div[4]/div[2]/div[3]/label/div[1]/input').click()
This is the radio button I'm trying to select...
[1]: https://i.stack.imgur.com/X64k8.png
Here is the webpage - https://stathead.com/basketball/pgl_finder.cgi
Appreciate any help! First time poster and noobie coder :)
Few options to solve the problem.
The problem occur as upon retrieving the element and scrolling down to click on it, the site adds a banner that may obstruct the element.
One option is to always open the browser in fullscreen.
driver = webdriver.Chrome()
driver.maximize_window()
This should help avoiding the banner to be in the way.. less scrolling and less chance for the banner to overlap with the radio button
Selenium also offer a library to return an element upon a condition being met using WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
el = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, '[value="E"]')))
Basically, the above will return the element once it is reported as clickable. The clickability of the element might not be in cause here. This way only ensure that the element found can be clicked but does not validation if there is an overlap (which I suspect is the issue here).
Third option, you may go with javaScript to click on the element
driver.execute_script("arguments[0].click();", el)
Related
I have gone through existing questions and google results of a similar nature, every solution posed has not worked for me within the particular website I am currently scraping.
https://dutchie.com/embedded-menu/revolutionary-clinics-somerville/menu
I am sending page down keys to the body element, which loads each item to be scraped. I have two issues with this, first I am unable to detect when the scrolling has stopped. Second, I have to manually click the browser window as it opens to allow the keys to be sent. I am not sure how to mimic this same focus giving behavior via code.
elem = driver.find_element_by_tag_name("body")
elem.send_keys(Keys.PAGE_DOWN)
I have tried the following, in many different iterations and the number printed never charged regardless of how far down the page I am or if I used innerHeight, or body instead of documentElement.
height = driver.execute_script("return document.documentElement.scrollHeight")
If I attempt to scroll down the page using a similar approach, this page does not move.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
I am unsure if this has to do with iframes or if I am simply misunderstanding the best approach.
Still have been unable to find a way to reliably detect the end of the page.
Thank you!
After importing the required imports
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
you can validate the page button is reached when the following element is visible:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//p[contains(text(),'License')]"))
As about the second issue, try clicking on the following element with Selenium:
driver.find_element_by_id("products-container").click
I have no environment to debug this, but I guess this will work
I'm having a really hard time locating elements on this website. My ultimate aim here is scrape the score card data for each country. The way I've envisioned doing that is to have selenium click list view (as opposed to the default country view) and loop going in and out of all the country profiles, scraping the data for each country along the way. But methods to locate elements on the page that worked for me in the past, have been fruitless with this site.
Here's some sample code outlining my issue.
from selenium import webdriver
driver = webdriver.Chrome(executable_path= "C:/work/chromedriver.exe")
driver.get('https://www.weforum.org/reports/global-gender-gap-report-2021/in-full/economy-profiles#economy-profiles')
# click the `list` view option
driver.find_element_by_xpath('//*[#id="root"]/div/div[1]/div[2]/div[2]/svg[2]')
As you can see, I've only gotten as far as step 1 of my plan. I've tried what other questions have suggested as far as adding waits, but to no avail. I see the site fully loaded in my DOM, but no xpaths are working for any element on there I can find. I apologize if this question is posted too often, but do know that any and all help is immensely appreciated. Thank you!
The element is inside an iframe you need to switch it to access the element.
Use WebDriverWait() wait for frame_to_be_available_and_switch_to_it()
driver.get("https://www.weforum.org/reports/global-gender-gap-report-2021/in-full/economy-profiles#economy-profiles")
wait=WebDriverWait(driver,10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "iFrameResizer0")))
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[name()='svg' and #class='sc-gxMtzJ bRxjeC']"))).click()
You need following imports.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
Another update:
driver.get("https://www.weforum.org/reports/global-gender-gap-report-2021/in-full/economy-profiles#economy-profiles")
wait=WebDriverWait(driver,10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "iFrameResizer0")))
driver.find_element_by_xpath("//*[name()='svg' and #class='sc-gxMtzJ bRxjeC']").click()
You incorrectly click List view. Your locator has to be stable. I checked it did not work.
So, in order to click the icon, use WebDriverWait:
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
wait = WebDriverWait(driver, timeout=30)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".sc-gzOgki.ftxBlu>.background")))
list_view = driver.find_element_by_css_selector(".sc-gzOgki.ftxBlu>.background")
list_view.click()
Next, to get the unique row locator, use the following css selector:
.sc-jbKcbu.kynSUT
It will give use the list of all countries.
I have been trying to find a way to select a dropdown option from google forms through selenium and python but have thus far been unsuccessful.
I tried several ways including the Select class (which does not work due to the dropdown form not using the select tag). As well as XPATH. But have only got so far as to click on the dropdown but not being able to select the options within the said dropdown list. Any help would be greatly appreciated!
I need to be able to specify which dropdown option to choose based on the text/value in the various options.
CODE:
browser.find_element_by_xpath("//div[#class='quantumWizMenuPaperselectOptionList']").click()
browser.find_element_by_xpath("//div[#class='freebirdThemedSelectOptionDarkerDisabled']/div[#class='quantumWizMenuPaperselectOption'][#data-value='1.05pm - 3.55pm']").click()
The error I get is that no such element is found even though this is the XPATH I have found from inspecting the dropdown menu accordingly.
I have created an example form here for reference: https://forms.gle/prBMqgVVFNv5KWQQA
These did not help as it uses the Select class, please do not flag this question as a duplicate as such
How to select/get drop down option in Selenium 2
How to select a drop-down menu value with Selenium using Python?
Using a webdriverwait also did not seem to do the trick as detailed in this post:
Automate Dropdown Menu in Selenium Without Select
The problem is the selection box wont popup even if your xpath is correct until you hover on that element, hovering on that element makes it clickable. You can use below code to hover on that selection box element and than try to click and select element
selectBox = WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.XPATH, "//div[#role='listbox']")))
action = ActionChains(browser);
action.move_to_element(selectBox).perform()
Full code here:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser = webdriver.Chrome()
url = 'https://docs.google.com/forms/d/e/1FAIpQLScosZjmDrvgUvh77tXsaAb24hKVgaBnjJfJz2BX1PvoqIO1Ow/viewform'
text = '1.05pm-3.55pm'
browser.maximize_window()
browser.get(url)
selectBox = WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.XPATH, "//div[#role='listbox']")))
action = ActionChains(browser);
action.move_to_element(selectBox).perform()
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#role='listbox']")))
selectBox.click()
selectionXpath = "//div[#class='exportSelectPopup quantumWizMenuPaperselectPopup appsMaterialWizMenuPaperselectPopup']//span[#class='quantumWizMenuPaperselectContent exportContent' and text()='"+text+"']"
selection = WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.XPATH, selectionXpath)))
selection.click()
Note that I have not verified that this way of selection preserves the information in the form as I have not submitted the form. You can test it verify it.
I am new to python and i am trying to automate click of load more button present in comment section of instagram using selenium and python. I am able to click only once. After first click the button disappears, I have waited for 10 minutes but it did not appear again. And also the status of the request changed to 302 when click in performed using automation. The status code remain 200 when clicked manually. Please help me guys how to perform click until all comment have been loaded. Any help will be appreciated. here is my code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
options = webdriver.FirefoxOptions()
options.add_argument('start-maximized')
browser = webdriver.Firefox(executable_path='./drivers/geckodriver')
browser.maximize_window()
url = 'https://www.instagram.com/p/CKCVIu2gDgn'
browser.get(url)
browser.implicitly_wait(10) #wait for 10 sec
load_more = browser.find_element(By.XPATH,'/html/body/div[1]/section/main/div/div[1]/article/div[3]/div[1]/ul/li/div/button/span').click()
For something like this, you might want to use Selenium's excute_script feature, which basically lets you execute Javascript on the webpage.
The following Javascript code will find the button in the comments and click it.
document.querySelector("button.dCJp8.afkep").click()
For such a feat, you would not want to use Selenium to click it because Selenium tries to mimic human behaviour as much as possible. In this case, Selenium would try to find the button, and if it is not visible on the page (ie. you need to scroll to find it), it would raise an Exception.
The following is a line of Python code you can use in your program to click the load more button.
driver.execute_script("document.querySelector('button.dCJp8.afkep').click()")
By the way, I am assuming that this round circular plus sign button is the click target.
Use instaloader library, it handles such stuff very easily.
here is the full documentation.
I want to scrape the data on the website https://www.climatechangecommunication.org/climate-change-opinion-map/. I am somewhat familiar with selenium. But the data I need which is below the map and the tooltip on the map is not visible in the source file. I have read some posts about using PhantomJS and others. However, I am not sure where and how to start. Can someone please help get me started.
Thanks,
Rexon
You can use this sample code:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://www.climatechangecommunication.org/climate-change-opinion-map/")
# switch to iframe
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[#src = 'https://environment.yale.edu/ycom/factsheets/MapPage/2017Rev/?est=happening&type=value&geo=county']")))
# do your stuff
united_states = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//*[#id='document']/div[4]//*[name()='svg']")))
print(united_states.text)
# switch back to default content
driver.switch_to.default_content()
Output:
50%
No
12%
Yes
70%
United States
Screenshot of the element:
Explanantion: first of all, to be able to interact with elements below the map you have to switch to iframe content, otherwise it is not possible to interact with this elements. Then the data below the map is in svg tags, which are also not trivial. To be able to do this, you the sample I have provided.
PS: I have used WebDriverWait in my code. With WebDriverWait your code becomes quickier and stable, since Selenium waits for particular conditions like visibility or clickable of particular element. In the sample code the driver wait at least 10 seconds until expected condition will be satisfied.