I'm pretty new to selenium and i searched a lot, but can't find an answer to my problem.
I want to open firefox, go on google and search for something. Store everthing in a list and print it to my console.
But everytime i open firefox with selenium, a pop-up windows opens with a confirmation to the privacy regulations of google. I dont know how to close it.
Here is what i tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
# create a new Firefox session
driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.maximize_window()
# Navigate to the application home page
driver.get("http://www.google.com")
main_page = driver.current_window_handle
time.sleep(3)
#Tried to find out in which browers window I am right now -> terminal says '19'
print('The actual browers window is: {}'.format(main_page))
driver.find_element_by_xpath("//*[#id='introAgreeButton']/span/span").click()
# get the search textbox
search_field = driver.find_element_by_id("lst-ib")
search_field.clear()
# enter search keyword and submit
search_field.send_keys("Flowers")
search_field.submit()
# get the list of elements which are displayed after the search
# currently on result page using find_elements_by_class_name method
lists= driver.find_elements_by_class_name("_Rm")
# get the number of elements found
print ("Found " + str(len(lists)) + " searches:")
# iterate through each element and print the text that is
# name of the search
i=0
for listitem in lists:
print (listitem.get_attribute("innerHTML"))
i=i+1
if(i>10):
break
# close the browser window
driver.quit()
Google privacy popup is contained in iframe. You must switch to iframe and then look for accept (or deny) button. Like so
wait = WebDriverWait(driver, 5)
iframe = wait.until(EC.element_to_be_clickable([By.CSS_SELECTOR, '#cnsw > iframe']))
driver.switch_to.frame(iframe)
And then when in iframe, look for Accept button a click it.
Related
I am trying to press a button with selenium because afterwards I need to inspect the full html of the website. This is the code that I am using:
driver = webdriver.Chrome()
driver.get('https://www.quattroruote.it/listino/audi/a4-allroad')
time.sleep(10)
html = driver.find_element_by_id('btnallestimenti')
html.click()
But I get this error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
when the page is open there are cookies and other things that shows up, is there a way to block all of them so that I can work on the html?
Thanks a lot!
As you can see the "cookies" banner is an HTML element itself and it contains a "Close" ("Chiudi") button that can be clicked.
If you inspect the page source you will find this code that relates to that button:
<button type="button" class="iubenda-cs-close-btn" tabindex="0" role="button" aria-pressed="false" style="font-size:16px!important;">Chiudi</button>
Your script needs to be modified to search the element by visible text (using the XPath) and click it to close the banner:
close_button = driver.find_element_by_xpath("//*[text()='Chiudi']")
close_button.click()
I can see that this kind of banner appears 2 times (one for cookies one for "Informativa") but once you click this one away you are redirected to the right page.
Of course you will need to test your script and adjust it to the page's behavior.
Also, be aware that every time the pages change because the devs change it your script will break and you will need to re-adjust it.
EDIT
Posting here the full code, try to use it and continue from here:
import time
from selenium.webdriver import Chrome
driver = Chrome()
driver.get("https://www.quattroruote.it/listino/audi/a4-allroad")
time.sleep(6)
driver.find_element_by_xpath("//button[text()='Accetta']").click()
time.sleep(6)
driver.switch_to.frame("promo-premium-iframe")
driver.find_element_by_xpath("//a[normalize-space()='Non sono interessato']").click()
time.sleep(6)
driver.switch_to.default_content()
driver.find_element_by_id("btnallestimenti").click()
input()
You could try to accept the cookie and proceed further, check the below lines of code.
options = Options()
options.add_argument("--disable-notifications")
driver = webdriver.Chrome(options=options,"ChromeDriver_Path")
driver.maximize_window()
driver.get('https://www.quattroruote.it/listino/audi/a4-allroad')
sleep(10)
cookie_btn = driver.find_element_by_xpath("//button[text()='Accetta']")
cookie_btn.click()
sleep(3)
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"promo-premium-iframe")))
register_btn = driver.find_element_by_xpath("//a[normalize-space()='Accedi o Registrati']")
register_btn.click()
Iframe available so just switched to the iframe, tried the to perform the registration.
I have written a simple web scraping code using Selenium but I want to scrape only the portion that is present 'before scroll'
Say, if it is this page I want to scrape - https://en.wikipedia.org/wiki/Pandas_(software) - Selenium reads information till the absolute last element/text which for me is the 'Powered by Media Wiki' button on the far bottom-right of the page.
What I want Selenium to do is stop after DataFrames (see screenshot) and not scroll down to the bottom.
And I also want to know where on the page it stops. I have checked multiple sources and most of them ask for infinite scroll websites. No one asks for just the 'visible' half of a page.
This is my code now:
from selenium import webdriver
EXECUTABLE = r"chromedriver.exe"
# get the URL
url = "https://en.wikipedia.org/wiki/Pandas_(software)"
# open the chromedriver
driver = webdriver.Chrome(executable_path = EXECUTABLE)
# google window is maximized so that all webpages are rendered in the same size
driver.maximize_window()
# make the driver wait for 30 seconds before throwing a time-out exception
driver.implicitly_wait(30)
# get URL
driver.get(url)
for element in driver.find_elements_by_xpath("//*"):
try:
#stuff
except:
continue
driver.close()
Absolutely any direction is appreciated. I have tried to be as clear as possible here but let me know if any more details are required.
I don't think that is possible. Observe the DOM, all the informational elements are under one section I mean one tag div[#id='content'], which is already visible to Selenium. Even if you try with //*, div[#id='content'] is visible.
And trying to check whether the element is visible though not scrolled, will also return True. (If someone knows to do what you are asking for, even I would like to know.)
from selenium import webdriver
from selenium.webdriver.support.expected_conditions import _element_if_visible
driver = webdriver.Chrome(executable_path = 'path to chromedriver.exe')
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://en.wikipedia.org/wiki/Pandas_(software)")
elements = driver.find_elements_by_xpath("//div[#id='content']//*")
for element in elements:
try:
if _element_if_visible(element):
print(element.get_attribute("innerText"))
except:
break
driver.quit()
I want to open links that I find on a website in a new tab. I have tried to open a new tab and pass the url of the link to the driver as suggested here, however, the new tab simply will not open. (There are a couple of other suggestion for how to open a new tab, but none of them seem to work for me.)
So my latest attempt was to right-click the link and press "t" to open the link in a new tab, like so:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
# Using Firefox to access web
driver = webdriver.Firefox()
# Open the website
driver.get('https://registers.esma.europa.eu/publication/searchRegister?core=esma_registers_firds')
# search for information
elem = driver.find_element_by_id('keywordField')
elem.clear()
elem.send_keys('XS1114155283')
button = driver.find_element_by_id('searchSolrButton')
button.click()
table_body = driver.find_element_by_xpath("//table[#id='T01']/tbody")
for link in table_body.find_elements_by_tag_name('a'):
act = ActionChains(driver)
act.context_click(link)
act.send_keys("t")
act.perform()
# ... do something in the new tab, close tab, and open next link ...
However, I get an error message on act.perform() which reads
MoveTargetOutOfBoundsException: (974, 695) is out of bounds of viewport width (1366) and height (654)
I managed a work around by opening the link in a new window, but I would really prefer the tab-version, as it would take longer to open a new browser window rather than a new tab.
You can use driver.execute_script() function to open link in a new tab
from selenium import webdriver
# Using Firefox to access web
driver = webdriver.Firefox()
# Open the website
driver.get('https://registers.esma.europa.eu/publication/searchRegister?core=esma_registers_firds')
# search for information
elem = driver.find_element_by_id('keywordField')
elem.clear()
elem.send_keys('XS1114155283')
button = driver.find_element_by_id('searchSolrButton')
button.click()
table_body = driver.find_element_by_xpath("//table[#id='T01']/tbody")
for link in table_body.find_elements_by_tag_name('a'):
href = link.get_attribute('href')
# open in new tab
driver.execute_script("window.open('%s', '_blank')" % href)
# Switch to new tab
driver.switch_to.window(driver.window_handles[-1])
# Continuous your code
I am trying to run a script in selenium webdriver python. Where I am trying to click on search field, but its always showing exception of "An element could not be located on the page using the given search parameters."
Here is script:
from selenium import webdriver
from selenium.webdriver.common.by import By
class Exercise:
def safari(self):
class Exercise:
def safari(self):
driver = webdriver.Safari()
driver.maximize_window()
url= "https://www.airbnb.com"
driver.implicitly_wait(15)
Title = driver.title
driver.get(url)
CurrentURL = driver.current_url
print("Current URL is "+CurrentURL)
SearchButton =driver.find_element(By.XPATH, "//*[#id='GeocompleteController-via-SearchBarV2-SearchBarV2']")
SearchButton.click()
note= Exercise()
note.safari()
Please Tell me, where I am wrong?
There appears to be two matching cases:
The one that matches the search bar is actually the second one. So you'd edit your XPath as follows:
SearchButton = driver.find_element(By.XPATH, "(//*[#id='GeocompleteController-via-SearchBarV2-SearchBarV2'])[2]")
Or simply:
SearchButton = driver.find_element_by_xpath("(//*[#id='GeocompleteController-via-SearchBarV2-SearchBarV2'])[2]")
You can paste your XPath in Chrome's Inspector tool (as seen above) by loading the same website in Google Chrome and hitting F12 (or just right click anywhere and click "Inspect"). This gives you the matching elements. If you scroll to 2 of 2 it highlights the search bar. Therefore, we want the second result. XPath indices start at 1 unlike most languages (which usually have indices start at 0), so to get the second index, encapsulate the entire original XPath in parentheses and then add [2] next to it.
I am trying to send some shortcut keys to open a new tab in chrome, using selenium in python. I open up facebook, then I log in, then I want to open a new tab where I can pass the url of one of my friend so that I can view his profile. I wrote the following code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import getpass
driver = webdriver.Chrome()
#opening browser and loging into the account
def login():
driver.get("http://www.facebook.com")
elem = driver.find_element_by_xpath("//*[#id=\"email\"]")
elem.send_keys("myUsername")
password = driver.find_element_by_name("pass")
password.send_keys("myPassword")
elem.send_keys(Keys.RETURN)
driver.implicitly_wait(5)
def scout():
scroll = driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + 't')
driver.get("http://www.facebook.com/some.friend")
driver.implicitly_wait(8)
login()
scout()
But it gives me the error:
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
I made the browser wait implicitly but still it's unable to find the tag. Please help.
First of all, try calling WebDriverWait instead of find_element_by_css_selector in order to find elements. It calls find_element_by_<method> every half a second until given timeout. That way you don't have to gamble with implicitly_wait or time.sleep().
Regarding your open new tab problem - you cannot send ctrl + t to chromedriver, it simply wouldn't do anything (it would work on firefox though). I've encountered this problem before, and the best solution I found is to search for a link in the website itself, so if you're on Facebook you can use that F icon at the top left. Then you can send ctrl + shift + click to open that in a new tab.
from selenium.webdriver import ActionChains
actions = ActionChains(driver);
actions.move_to_element(link)
actions.send_keys(Keys.CONTROL+ Keys.SHIFT)
actions.click(link)
actions.perform()
From there you should just switch tabs using
tabs = driver.window_handles
driver.switch_to.window(tabs[<index>])