Selenium webdriver screenshot is different when using '--headless' - python

Using selenium and python, I'm opening a document by making click on a link which opens in the same tab. When I make a screenshot without using '--headless' it takes the document screenshot, but when I activate '--headless' the screenshot is from the previous page (the one where I make click to the document to be opened)
I have tried different browsers and ways (opening in new tab and switching but not working when using '--headless') but not working...
Any idea why '--headless' is behaving like this?

While taking a screenshot ideally you need to induce WebDriverWait for the visibility_of_element_located() of some static and visible element e.g. some <h1> / <h2> element on the desired page and then take the screenshot as follows:
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "css_visible_element")))
driver.save_screenshot("Federico Albrieu.png")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

I solved the problem.
I was using the action
ActionChains(driver).move_by_offset('xcoordinate', 'ycoordinate).click().perform()
to open the desired document. Apparently, while using headless, selenium can't make click in the document coordinate (?) but it can when not using headless.
I tried instead making click with element_to_be_clickable((By.XPATH, 'XPATH'))).click() and it works when using headless.
Thanks!

Related

Sending slash on send_keys on Selenium Chrome Driver on Linux

I have been trying to send a URL link to a text area. I am using python3 and Chrome Version 109.0.5414.119. It worked fine on my local OSX machine but when I tried to automate it on a Linux Machine. It started behaving weirdly. So I have this feeling that It can be a Chrome Driver-related issue.
When I send a URL like this:
l.send_keys("https://google.com")
But becomes:
/google.comhttps:
Then I tried to debug this behavior by sending the following:
>>> l.send_keys("/") # /|
>>> l.send_keys("/") # |/
So the position of the cursor is going ahead of the line for the second /. I was not expecting this. I wonder if you can shed light on how to solve this?
Possibly you are trying to invoke send_keys() too early within the <input> field even before the element have rendered properly.
Solution
Ideally to send a character sequence to the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "element_css_locator"))).send_keys("https://google.com")
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Selenium - Issue Getting/Clicking Button Element

I have been messing around with python and selenium and was trying to make a program that could automatically checkout on bestbuy.ca . I have gotten to nearly the last stage of the checkout process but am having difficulty getting the element/clicking the buttons to continue. The only button I have been able to get to work is the add to cart button on the product page.
Here is the site as well as the HTML:
Click here for image
Some things I have tried with no success include:
driver.find_element_by_class_name().click()
For this, I used the class in button and span. I have tried using each individual class in separately and together with little success. It would usually give me a "No Element Found" error or just not click anything afterwards.
driver.driver.find_element_by_xpath().click()
For this, I tried using the element locator extension for the xpath as well as "//*[#type='submit']" with no success.
driver.driver.find_element_by_css_selector().click()
For this, I tried the element locator extension again as well as copying it from the firefox inspect with no success.
Not really sure if there are any other ways to go about doing this but any help would be appreciated.
Did you try to copy the entire class attribute from the html and use :
wait.until(EC.presence_of_element_located((By.XPATH, "//button[#class='asd asd sad asd']"))).click()
copy the class and paste it.
Try either of the following to click on the button tag.
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Continue']/parent::button"))).click()
Or by attribute data automation
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-automation='review-order']"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

How to handle load more button in selenium python

I am new to python and i am trying to automate click of load more button present in comment section of instagram using selenium and python. I am able to click only once. After first click the button disappears, I have waited for 10 minutes but it did not appear again. And also the status of the request changed to 302 when click in performed using automation. The status code remain 200 when clicked manually. Please help me guys how to perform click until all comment have been loaded. Any help will be appreciated. here is my code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
options = webdriver.FirefoxOptions()
options.add_argument('start-maximized')
browser = webdriver.Firefox(executable_path='./drivers/geckodriver')
browser.maximize_window()
url = 'https://www.instagram.com/p/CKCVIu2gDgn'
browser.get(url)
browser.implicitly_wait(10) #wait for 10 sec
load_more = browser.find_element(By.XPATH,'/html/body/div[1]/section/main/div/div[1]/article/div[3]/div[1]/ul/li/div/button/span').click()
For something like this, you might want to use Selenium's excute_script feature, which basically lets you execute Javascript on the webpage.
The following Javascript code will find the button in the comments and click it.
document.querySelector("button.dCJp8.afkep").click()
For such a feat, you would not want to use Selenium to click it because Selenium tries to mimic human behaviour as much as possible. In this case, Selenium would try to find the button, and if it is not visible on the page (ie. you need to scroll to find it), it would raise an Exception.
The following is a line of Python code you can use in your program to click the load more button.
driver.execute_script("document.querySelector('button.dCJp8.afkep').click()")
By the way, I am assuming that this round circular plus sign button is the click target.
Use instaloader library, it handles such stuff very easily.
here is the full documentation.

Using selenium driver.find_element().click() on a subdivided webpage

First of all, I'm completely new in web scraping, html and selenium, so my question might not seem meaningful to you.
What I'm trying to do: automated click on an image on a webpage. Manual click on this image allows to display new information on a part of the webpage.
How I tried to do it: I found something specific to this image, for example:
<img src="images/btn_qmj.gif" border="0">
So I just entered in python:
xpath = '//img[#src="images/btn_qmj.gif"]'
driver.find_elements_by_xpath(xpath)
Problem: this returns me an empty list. I partially understood why, but I don't have any solution.
image of the html code inspector
I included here an image of the html tree I obtain on the web inspector. As one can see, the tree to reach my line of interest gives "/html/frameset/frame1/#document/html/body/center/table/tbody/tr[2]/td/a/img".
The issue is that I cannot access -by using an xpath- anything that comes after the #document. If I try to copy the xpath of my line of interest, it tracks it back only up to the 2nd "/html". It is like the page is subdivided, compartmentalized, with inner parts I cannot access. I suppose there is a way to access the #document content, maybe it refers to another webpage, but I'm stuck at that point.
I would be very grateful if anyone could help me on this.
Have you tried switching to the frame first?
driver.switch_to.frame('gauche')
You need to switch control to frame first to handle image on it. Please see below code for your reference
driver.switch_to_frame(frame)
WebDriverWait(driver, 20).until(
EC.visibility_of_element_located((By.XPATH, "'//img[#src="images/btn_qmj.gif"]'"))).click()
driver.switch_to_default_content()
Note :: You need to add below imports
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
The element is present inside an iframe.In order to access the element you need to switch it first.
Induce WebDriverWait() and frame_to_be_available_and_switch_to_it()
Induce WebDriverWait() and visibility_of_element_located()
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"gauche")))
WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//img[#src='images/btn_qmj.gif']"))).click()
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

How do I click on links that are contained in a div container fluid using python selenium

I'm writing a piece of python code to test whether users are able to click on dynamic tabs.
I'm using find_elements_by_xpath() to search for attributes of the tab. Now, this is a dynamic set of tabs, so they will never be viewable in the page source. But they do exist when I click on inspect element.
I'm using something like this to identify the tabs:
elem = self.browser.find_elements_by_xpath("//a[contains(#role,'tab')]")
for i in elem:
print(i.text)
I tried the get_attribute feature but that did not work. Is there any way I can use python selenium to click on a dynamic tab (that does not appear on page source)?
You can wait some time until required element present in DOM and clickable:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
wait(self.browser, 10).until(EC.element_to_be_clickable((By.XPATH, '//a[contains(#role, "tab")]'))).click()

Categories