I am writing a bot using Python with Selenium module.When I open a webpage with my bot, since the webpage contains too many external sources than dom, it takes a lot to get all of the page loaded. I used the explicit and implicit waits to eliminate this problem since I just wanted a specific element to be loaded and not all of the webpage, it didn't work. The problem is If i run the following statement:
driver = webdriver.Firefox()
driver.get('somewebpage')
elm = WebDriverWait(driver, 5).until(ExpectedConditions.presence_of_element_located((By.ID, 'someelementID'))
elm.click()
It doesn't work since the Selenium waits for the driver.get() to fully retrieve the webpage and then, it proceeds further. Now I want to write a code that sets a timeout for driver.get(), Like:
driver.get('somewebpage').timeout(5)
Where the driver.get() stops loading the page after 5 secs and the program flow proceeds, whether the driver.get() fully loaded webpage or not.
By the way, I have searched about the feature that I said above, and came across there:
Selenium WebDriver go to page without waiting for page load
But the problem is that the answer in the above link does not say anything about the Python equivalent code.
How do I accomplish the future that I am searching for?
python equivalent code for the question mentioned in the current question (Selenium WebDriver go to page without waiting for page load):
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference('webdriver.load.strategy', 'unstable')
driver = webdriver.Firefox(profile)
and:
driver.set_page_load_timeout(5)
There is a ton of questions on this, here is an example. Here is an example that waits until all jquery ajax calls have completed or a 5 second timeout.
from selenium.webdriver.support.ui import WebDriverWait
WebDriverWait(driver, 5).until(lambda s: s.execute_script("return jQuery.active == 0"))
It was a really tedious issue to solve. I just did the following and the problem got resolved:
driver= webdriver.Firefox()
driver.set_page_load_timeout(5)
driver.get('somewebpage')
It worked for me using Firefox driver (and Chrome driver as well).
Related
I was trying to scrape a website with Selenium (as it is a website with a dynamically loaded content).
However, to wait for such a dynamic content to be loaded, while I usually use time.sleep(), I have just known about (and tried) driver.implicitly_wait() but it seems I do not get the expected content.
Here is the code:
from selenium import webdriver
import os
import time
os.environ['MOZ_HEADLESS'] = '1'
baseSite = 'https://bair.berkeley.edu/students.html'
driver = webdriver.Firefox()
#driver.implicitly_wait(5) --> full content is not retrieved
driver.get(baseSite)
time.sleep(5) # full content is retrieved
source = driver.page_source
print(source)
The setting of an implicit wait timeout (i.e., using implicitly_wait() does not affect how the browser loads a page. What that method does is poll the DOM for the desired element when using find_element or find_elements. In the code you posted, setting the implicit wait timeout has no effect because you are not attempting to find any elements on the page.
If you were to provide more details about what you’re expecting (aside from saying, “I want the page to be ‘fully loaded’”, because that phrase is so vague as to be meaningless), it might be easier to provide more guidance.
You could use an explicit wait, which would wait until a certain condition is met. For example,
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "myDynamicElement")))
would wait until the WebDriver located that element (until 10 seconds have passed, at which point, it would time out). This link is helpful for learning more about different types of waits in Selenium. https://selenium-python.readthedocs.io/waits.html
I am trying to make Python play a video which is embedded in a website (https://harsh10822.wixsite.com/intro).
I tried using ID,xpath etc etc but It didn't workout.
There is a similar Question to mine here (How to click on the play button of a youtube video embedded within smtebook through selenium and python) But I couldn't figure out how to apply the code.
If You could help me out by providing the code,I will be really happy.
Thankyou
This should work.
driver = webdriver.Firefox()
driver.get('https://harsh10822.wixsite.com/intro')
time.sleep(5)
video = driver.find_element_by_id('video-player-comp-ka1067od')
video.click()
Waiting is important in this case, because embedded video doesn't load instantly with the page so selenium needs to wait. You can change 5 seconds to any number that works for you.
See: How to click on the play button of a youtube video embedded within smtebook through selenium and python
But this uses a slightly different technique instead of WebDriverWait or sleeping for a fixed number of seconds, which can be wasteful if you are sleeping for longer than necessary.
Instead, use a call to driver.implicitly_wait(10). Then any calls that attempt to locate elements will implicitly wait for up to 10 seconds for the element to appear before timing out but will wait no longer than necessary. It's just a question of knowing which elements to look for and where:
driver = webdriver.Firefox()
driver.implicitly_wait(10) # wait for up to 10 seconds before timeout for any find operation
driver.get('https://harsh10822.wixsite.com/intro')
driver.switch_to.frame(driver.find_element_by_xpath('//iframe[starts-with(#src, "https://www.youtube.com/embed")]')) # switch to iframe
button = driver.find_element_by_xpath('//button[#aria-label="Play"]') # look for button
button.click()
Make sure that particular web driver is downloaded and placed in the directory wherein your code is saved. And for reference you can refer to Tech With Tim selenium tutorials.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
video=input()
driver=webdriver.Chrome()
driver.get("https://youtube.com")
search=driver.find_element_by_id("search")
time.sleep(2)
search.send_keys(video,Keys.RETURN)
time.sleep(2)
play=driver.find_element_by_id("video-title")
play.send_keys(Keys.RETURN)
This question already has answers here:
Do we have any generic function to check if page has completely loaded in Selenium
(7 answers)
Selenium IE WebDriver only works while debugging
(1 answer)
Closed 3 years ago.
I'm trying to get selenium to wait for a page to fully load using .readyState but I can't find the correct way to get selenium to test the .readyState of the web page before proceeding.
The two hashed out lines are my best attempts to get it to work including the example from the link below; in which ".equals" doesn't seem to have any effect.
Is there a way with python-selenium to wait until all elements of a page has loaded?
When running the code with either hashed out line, the output will print "loading" and raise an error because the element postceding the print function is not yet loaded.
note: this problem can easily be solved using waits and Expected Conditions but the question is about using .readyState.
ps: my understanding is that when pageLoadStrategy is set to "normal" then selenium defaults to waiting until .readyState is "complete" however pageLoadStrategy can't be switched between "none" and "normal" once the driver is initiated.
If anyone knows otherwise then this could be a possible solution.
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities().CHROME.copy()
caps["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(
desired_capabilities=caps,
executable_path=r'C:\chromedriver_win32\chromedriver.exe'
)
driver.get("https://www.google.com/maps")
# driver.execute_script("return document.readyState").equals("complete"))
# driver.execute_script("return document.readyState == 'complete'")
print(driver.execute_script("return document.readyState"))
driver.find_element_by_xpath('//*[#id="searchboxinput"]').send_keys("somewhere")
You need to use Explicit Wait and read readyState property value, something like:
WebDriverWait(driver, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
you will also need the following import:
from selenium.webdriver.support.ui import WebDriverWait
More information: How to use Selenium to test web applications using AJAX technology
driver = webdriver.Chrome()
#driver.set_page_load_timeout(10)
driver.get("sitename.com")
driver.find_element_by_id("usernameId").send_keys("myusername")
Setting a page load time proved counterproductive as the page load was killed even before the elements were actually loaded!
Currently, when I try to login on a site, the find_element_by_id() waits for the complete page to load, then gets me the element. I've read about implicit/explicit waits used along with ExpectedConditions, but as far as I understand they are used for waiting for an element to appear(dynamically) after the complete page has loaded.
Is there a way I can find an element as soon(polling is good enough) as it is visible(without waiting for the complete page to load)? It would be great to do so, some pages take quite a while to load(heavy traffic/low availability/poor internet connection could be reasons though).
I am using Selenium with Python, and a Chrome Driver. Thanks.
Take a look at selenium python documentation.
It has visibility_of_element_located.
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.visibility_of_element_located((By.ID,'someid')))
It is a best practice to wait for entire page to load before you take any further action. However, if you want to stop the page load in between(or load the page only for a specified time and carry on), you can change this in the browser's profile setting.
In case of Firefox :
profile = webdriver.FirefoxProfile()
profile.set_preference("http.response.timeout", 10)
profile.set_preference("dom.max_script_run_time", 10)
driver = webdriver.Firefox(firefox_profile=profile)
Hope it helps, cheers.
I am getting some very odd behavior on a project. Essentially this automation creates an article on a web page using selenium and verifies it's existence by clicking/opening the article. However when i run the program without debug it is failing to "click()" on the web element. So I investigate and start to debug. I notice that the web element ("target_element") is "None". I continue past for just debugging and it fails to click "target_element" as expected.
When i rerun the program and instead of continuing on i open the class in the debug view. I see that "target_element" exists, so i continue on and target_element.click() successfully loads the article on the webpage. Does anyone have any suggestions on how to fix this problem?
TLDR; article.target_element does not exist when running the program UNLESS i set a breakpoint and look at "article" in the debugger. Then if i continue in the program article.target_element suddenly exists.
Possible reason is the element load time.
When you are running the test Selenium runs faster than the time needed by the element to load properly and eventually throws the exception. On the other hand, during debug you are stepping in and giving enough time for the element to load and Selenium finds the element without any issue.
If, the problem is exactly what it is mentioned above the use of explicit wait should be the way to go.
Example taken from the Selenium Python doc
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()