Inspect Youtube Page Element
I am new to Python and I am learning how to automate webpages. I under the basics around using the different locators under the inspect element tab to drive my code.
I have written some basic code to skip youtube ads however I am stuck on finding the correct page element to agree to the privacy policy pop up box in Youtube. I have used ChroPath to try and find the xpath of the page however there doesn't appear to be one. I was unable to locate any other page elements and I was wondering if anyone has any ideas on how I can automate the click of the 'I Agree' button?
Python Code:
from msedge.selenium_tools import Edge, EdgeOptions
options = EdgeOptions()
options.use_chromium = True
driver = Edge(options=options)
driver.get('http://www.youtube.com')
def agree():
while True:
try:
driver.find_element_by_xpath('/html/body/ytd-app/ytd-popup-container/paper-dialog/yt-upsell-dialog-renderer/div/div[3]/div[1]/yt-button-renderer/a/paper-button').click()
driver.find_elements_by_xpath('.<span class="RveJvd snByac">I agree</span>').click()
except:
continue
if __name__ == '__main__':
agree()
Youtube Inspect Element Screeshot is below:
I don't know if the xpath in your code is right as I can't see the whole html structure of the page. But you can use F12 dev tools in Edge to find the xpath and to check if the xpath you find is right:
Open the page you want to automate and open F12 dev tools in Edge.
Use Ctrl+Shift+C and click the element you want to locate and find the html code of the element.
Right click the html code and select Copy -> Copy XPath.
Then you can try to use the xpath you copy.
Besides, find_elements_by_xpath(xpath) will return a list with elements if any was found. I think you need to specify which one element of the list to click with. You need to pass in the value number of the elements list like this [x], for example:
driver.find_elements_by_xpath('.<span class="RveJvd snByac">I agree</span>')[0].click()
When inspecting the page elements I overlooked the element of iframe. After doing some digging I came across the fact I had to tell the Selenium Driver to switch from the main page to the iframe. I added the following code and now the click to the 'I Agree' button is automated:
frame_element = driver.find_element_by_id('iframe')
driver.switch_to.frame(frame_element)
agree2 = driver.find_element_by_xpath("/html/body/div/c-wiz/div[2]/div/div/div/div/div[2]/form/div/span/span").click()
driver.switch_to.default_content()
Related
I am trying to get data from a Power Bi table. There are some elements that appear when hovering over a table. When I right click on ... I don't see Inspect Element. However, when I left click on this element, I can see a menu, and if I right click on any items, I can see Inspect element.
My first question, is why I don't see Inspect Element in the right click menu for all elements in the browser. Am I somehow able to open this ... menu programmatically in Selenium?
the Export Data element only appears in HTML after the first left click. I'm assuming this is created using Javascript and in order to export data with Selenium I would have to programmatically instantiate this by clicking on the ... menu. Is selenium capable of triggering javascript functions that generate more html code in a dynamic webpage? Or do I need to somehow click on the ... element.
If I can execute a javascript function, how can I find out in Edge the javascript function that gets executed and how can I replicate this function in Selenium
Essentially, if I try to find the Export data element in Selenium, it is not able to find it, unless I set a breakpoint before search, then in EdgeDriver I open this menu, and then I can find it and click it through Python
If all else fails, can I programmatically open the left click menu by automating a mouse click at certain coordinates in Selenium?
1.1 why I don't see Inspect Element in the right click menu for all elements:
PowerBi has its own context menu so they suppress the browsers context menu. If the element is tricky to find the dev tools, you can press Ctrl + Shift + C (while dev tools is open) and then click the desired element. Your mouse needs to be already over the element before pressing the key combination.
1.2 Am I somehow able to open this ... menu programmatically in Selenium?
Seems a little tricky, but could work if you first find the title of that area and move the mouse there, like described here: https://stackoverflow.com/a/8261754/12914172
Then your element should be in the html and you can find it hopefully by its class name vcMenuBtn that seems to be unique on that page. But you need to verify that.
2. Is selenium capable of triggering javascript functions that generate more html code in a dynamic webpage? Or do I need to somehow click on the ... element.
Selenium is able to execute javascript like desribed here: https://stackoverflow.com/a/70544802/12914172
However in your sample, and I was quickly checking the PowerBI online page, this looks like a whole lot of reverse engineering to understand and can sometimes be dangerous as well. I would go for hoover over the area find the ... and click it.
3. How can I find out in Edge the javascript function that gets executed
In dev tools you can set breakpoints to debug the steps the pages does after an action. But again, I would not invest to much time in that.
4. Can I programmatically open the left click menu by automating a mouse click at certain coordinates in Selenium?
Yes but this never works as good as the way described above. If you still want to give it a try, maybe that answer helps: https://stackoverflow.com/a/26385456/12914172
Many thanks to r000bin, this solution works for me, downloading data from PowerBI using Selenium for Python:
import selenium, mouse, time
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
url = 'https://dataport.gasunie.nl/Vulling-Gasopslagen-Nederland'
driver = selenium.webdriver.Chrome(service=Service())
driver.get(url)
time.sleep(4)
#driver.fullscreen_window()
#driver.switch_to.window(driver.current_window_handle)
time.sleep(4)
iframe = driver.find_elements(By.TAG_NAME, 'iframe')
assert len(iframe)==1
driver.switch_to.frame(iframe[0])
time.sleep(4)
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
actions.move_to_element_with_offset(driver.find_element(By.TAG_NAME, 'html'), 0,0)
actions.move_by_offset('5', '5').click().perform()
time.sleep(4)
button = driver.find_element(By.CLASS_NAME, 'vcMenuBtn')
button.click()
button = driver.find_element(By.ID, '0')
button.click()
# 4 tabs and 1 enter
time.sleep(4)
for n in range(4):
element = driver.switch_to.active_element
time.sleep(2)
element.send_keys(Keys.TAB)
time.sleep(2)
element = driver.switch_to.active_element
time.sleep(2)
element.send_keys(Keys.ENTER)
driver.close()
I need to click the first item in the menu in a webpage with Python3 using Selenium.
I manage to log-in and navigate to the required page using Selenium, but there I get stuck: it looks like Selenium can't find any element in the page beyond the very first div in body.
I tried to find the element by ID, class, xpath, selector... The problem is probably not about that. I thought it could be about an iframe, but the content I need does not seem to be in one.
I guess that the problem is that the element I need to find is visible in the devtools, but not in the page source, so Selenium just can't see it - does this make sense? If so, can this be fixed?
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
self.driver.get("my site")
# log-in website and navigate to needed page
# [...]
# find element in page
# this works
first_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper")
# this does not work
second_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper > div.layout.flex.flex-col.overflow-x-hidden.h-display-flex.h-flex-direction-column.h-screen")
Edit
The problem is most likely due to a dynamic webpage with parts of the DOM tree attached later on by a script. I downloaded a local version of page.html, removed scripts, and successfully found the sought-after element in the local page with
from selenium import webdriver
from selenium.webdriver.common.by import By
from pathlib import Path
driver = webdriver.Chrome()
html_file = Path.cwd() / "page.html"
driver.get(html_file.as_uri())
my_element = driver.find_element(By.CSS_SELECTOR, "[title='my-title']")
The exact same driver.find_element query won't work on the online page. I'm trying to implement a waiting condition as suggested in Misc08's answer.
I guess that the problem is that the element I need to find is visible
in the devtools, but not in the page source, so Selenium just can't
see it - does this make sense? If so, can this be fixed?
No, this does not make sense, since Selenium is executing a full browser in the background like you are using when you investigate the page source with the devtools.
But you have some options to narrow your problem. The first thing you can do, is to print the source the webdriver is "seeing" in this moment:
print(driver.page_source)
If you see the elements you are looking for in the page source, than you should try to improve your selector. It is helpful to go down the DOM step by step. Look for an upper element in the page tree first. If this works, try to find the next child element, then the next child, and so on. You can check if selenium found the element like this:
try:
myelement = driver.find_element(By.CSS_SELECTOR, 'p.content')
print("Found :)")
except NoSuchElementException:
print("No found :(")
By the way, i think your CSS selector is by far to complex, just use on CSS class, not all of them:
second_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper > div.layout")
But there might be the case, where the elements you are looking for, are not present in the page source from the beginning on. Dynamic webpages getting more and more popular. In this case parts of the DOM tree are attached later on by a script. So you have to wait for the execution of the scripts, before you can find this "dynamic" elements. One dirty and unreliable option is to just add a sleep() here. Much better is to to use an explicit waiting condition, see https://selenium-python.readthedocs.io/waits.html
I am trying to click the search button in this link here
I would like to click the search button and then download all URL's on the next page but currently it is finding monthly list
The link below takes you to a screenshot of my code where it outputs the monthly list button instead of searcj/
Python code selenium
Welcome to Stack Overflow!
You can use the .click() method to click an element object, and the .find_element_by_xpath() method to find the element. I've located that the element's full XPATH is /html/body/div/div/div[3]/div[3]/div/form/fieldset/div[5]/input[2].
You can implement all the pieces together like so:
driver.find_element_by_xpath("/html/body/div/div/div[3]/div[3]/div/form/fieldset/div[5]/input[2]").click()
if you are starting I recommend you using some extension to help you find the xPath of the elements.
I used Xpath Helper from Chrome, but you can use any other.
The Xpath of the Search button is:
/html[#class='js']/body/div[#id='idox']/div[#id='pa']/div[#class='container'][2]/div[#class='content']/div[#class='tabcontainer']/form[#id='weeklyListForm']/fieldset/div[#class='buttons']/input[#class='button primary']
After you have the XPath you can use it in the selenium driver to find the elements.
This can be enough, but I recommend you to learn in depth how this works to know exactly what it is doing.
I'm trying to click on the webpage "https://2018.navalny.com/hq/arkhangelsk/" from the website's main page. However, I get this error
selenium.common.exceptions.ElementNotInteractableException: Message:
There's nothing after "Message:"
My code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
browser = webdriver.Firefox()
browser.get('https://2018.navalny.com/')
time.sleep(5)
linkElem = browser.find_element_by_xpath("//a[contains(#href,'arkhangelsk')]")
type(linkElem)
linkElem.click()
I think xpath is necessary for me because, ultimately, my goal is to click not on a single link but on 80 links on this webpage. I've already managed to print all the relevant links using this :
driver.find_elements_by_xpath("//a[contains(#href,'hq')]")
However, for starters, I'm trying to make it click at least a single link.
Thanks for your help,
The best way to figure out issues like this, is to look at the page source using developer tools of your preferred browser. For instance, when I go to this page and look at HTML tab of the Firebug, and look for //a[contains(#href,'arkhangelsk')] I see this:
So the link is located within div, which is currently not visible (in fact entire sub-section starting from div with id="hqList" is hidden). Selenium will not allow you to click on invisible elements, although it will allow you to inspect them. Hence getting element works, clicking on it - does not.
What you do with it depends on what your expectations are. In this particular case it looks like you need to click on <label class="branches-map__toggle-label" for="branchesToggle">Список</label> to get that link visible. So add this:
browser.find_element_by_link_text("Список").click();
after that you can click on any links in the list.
I'm writing test scripts for a web page in python using Selenium's remote control interface.
I'm writing it like this:
elem = browser.find_element_by_link_text("foo")
elem.click()
elem = browser.find_element_by_name("goo")
elem.send_keys("asdf")
elem = browser.find_element_by_link_text("foo2")
elem.click()
It then needs to select an item in a list. The list becomes visible when the mouse hovers over it, but selenium cannot find the element if it's hidden. The list also shows options based on who is logged in. The list is implemented in CSS, so trying to run it in javascript and using gettext() does not work.
I've tried searching for the link based on name, class and xpath, but it always reports that it is not visible I've verified from browser.page_source() that the link is in the source code, so it's reading the correct page.
How do I select the link inside the list? Any help is appreciated.
Selenium and :hover css suggests that this can't be done using the Selenium RC interface, but instead must be done using the WebDriver API
Try move_to_element(). Check out the API http://readthedocs.org/docs/selenium-python/en/latest/api.html