Chromedriver is launched correctly, then start loading the page, the URL changes to 'http://app1.nmpa.gov.cn/?CbSlDlH0=qGk8rqrP3bxP3bxP39Exb4.QiGYTsZaK4uvCaZ_lRVZqqDE' however the page remains blank ( white).
import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome(executable_path="D:\Python\chromedriver.exe")
driver.maximize_window()
url = 'http://app1.nmpa.gov.cn/'
driver.get(url)
wait = WebDriverWait(driver, 60)
time.sleep(5)
driver.quit()
I tried it, I get redirected to the page "http://app1.nmpa.gov.cn/data_nmpa/face3/dir.html" which looks like this:
I did it with the firefox browser and geckodriver. Can you try and see if the problem still exists?
Related
I'm trying to scrap the list of services we have for us from this site but not able to click to the next page.
This is what I've tried so far using selenium & bs4,
#attempt1
next_pg_btn = browser.find_elements(By.CLASS_NAME, 'ui-lib-pagination_item_nav')
next_pg_btn.click() # nothing happens
#attemp2
browser.find_element(By.XPATH, "//div[#role = 'button']").click() # nothing happens
#attempt3 - saw in some stackoverflow post that sometimes we need to scroll to the
#bottom of page to have the button clickable, so tried that
browser.execute_script("window.scrollTo(0,2500)")
browser.find_element(By.XPATH, "//div[#role = 'button']").click() # nothing happens
I'm not so experienced with scrapping, pls advice how to handle this and where I'm going wrong.
Thanks
Several issues with your code:
You tried wrong locators.
You probably need to wait for the element to be loaded before clicking it. But if before clicking the pagination you performing some actions on the page this is not needed since during you scraping the page content web elements are already got loaded.
Pagination button is on the buttom of the page, so you need to scroll the page to bring the pagination button into the visible screen.
After scrolling some delay should be added, as you can see in the code below.
Now pagination element can be clicked.
The following code works
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.tamm.abudhabi/en/life-events/individual/HousingProperties"
driver.get(url)
pagination = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "ui-lib-pagination__item_nav")))
pagination.location_once_scrolled_into_view
time.sleep(0.5)
pagination.click()
I'm a beginner in web scrapping and I've followed a few YouTube videos about how to do this, but regardless to what I try I can't have my code accept the cookies.
This is the code I have so far:
from selenium import webdriver
import time
driver = webdriver.Safari()
URL = "https://www.zoopla.co.uk/new-homes/property/london/?q=London&results_sort=newest_listings&search_source=new-homes&page_size=25&pn=1&view_type=list"
driver.get(URL)
time.sleep(2) # Wait a couple of seconds, so the website doesn't suspect you are a bot
try:
driver.switch_to_frame('gdpr-consent-notice') # This is the id of the frame
accept_cookies_button = driver.find_element_by_xpath('//*[#id="save"]')
accept_cookies_button.click()
except AttributeError: # If you have the latest version of selenium, the code above won't run because the "switch_to_frame" is deprecated
driver.switch_to.frame('gdpr-consent-notice') # This is the id of the frame
accept_cookies_button = driver.find_element_by_xpath('//*[#id="save"]')
accept_cookies_button.click()
except:
pass # If there is no cookies button, we won't find it, so we can pass
I don't have safari webdriver but chrome webdriver, but I think they works similar. On chrome you close the cookie banner with this code
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get(URL)
# wait no more than 20 seconds for the `iframe` with id `gdpr-consent-notice` to appear, then switch to it
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "gdpr-consent-notice")))
# click accept cookies button
driver.find_element(By.CSS_SELECTOR, '#save').click()
I want to send a string to the web page whose text field name is "inputfield". Actually, I can send the word to the page, but when I run the program, a new "chrome" page opens, which is used for testing purposes. However, I want to send a string to the field on a chrome page that is already open.
Here my code:
from selenium import webdriver
import time
url = "https://10fastfingers.com/typing-test/turkish"
options = webdriver.ChromeOptions()
options.binary_location = r"C://Program Files//Google//Chrome//Application//chrome.exe"
chrome_driver_binary = 'chromedriver.exe'
options.add_argument('headless')
driver = webdriver.Chrome(chrome_driver_binary, options=options)
driver.get(url)
driver.implicitly_wait(10)
text_area = driver.find_element_by_id('inputfield')
text_area.send_keys("Hello")
Nothing happens when I run this code. Can you please help? Can you run it by putting a sample web page in the url part?
Thank you.
EDIT: It is working when I deleted options. But still opening a new page when I run it. Is there a way use a page which already open on background.
chrome_driver_binary = 'chromedriver.exe'
driver = webdriver.Chrome(chrome_driver_binary)
driver.get('https://10fastfingers.com/typing-test/turkish')
text_area = driver.find_element_by_id('inputfield')
text_area.send_keys("Hello")
Click the popup prior to sending keys.
driver.get('https://10fastfingers.com/typing-test/turkish')
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.ID, "CybotCookiebotDialogBodyLevelButtonLevelOptinAllowallSelectionWrapper"))).click()
text_area = wait.until(EC.element_to_be_clickable((By.ID, "inputfield")))
text_area.send_keys("Hello")
Imports
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am not sure what is your question but if the issue is multiple tabs or windows being opened then:
you can switch between the windows as:
// you can move to specific handle
chwd = driver.window_handles
print(chwd)
driver.switch_to.window(chwd[-1])
you should shoul switch to the correct window before you can interact with elements on that window
just switch to the window that was already opened bypassing the index
If the problem is that you want to interact with an already opened chrome then you should follow below steps:
Start chrome with debug port:
<path>\chrome.exe" --remote-debugging-port=1559
Python :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:1559")
driver = webdriver.Chrome(options=chrome_options)
I've searched several solutions but it didn't work.
That's my code
driver = webdriver.PhantomJS()
driver.get('https://baijia.baidu.com')
wait = WebDriverWait(driver,10)
wait.until(EC.element_to_be_clickable((By.ID, 'getMoreArticles'))).click()
content = driver.page_source
page = open('test.html','wb')
page.write(content)
I've tried to debug the code, It successfully returns the clicked page.
when I run the code, it also returns successfully, however it don't returns the clicked page, just the source page.
I tried to search the solutions, take the page down to the bottom:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)",element)
But it's the same result, only debug successfully.
Thanks
It seems, that your button initiate an AJAX request. Driver doesn't wait it finished, because there is no page reload. So you should add explicit wait. Something like that:
expected_number_of_articles = 10 # enter your number
article_locator = (By.CSS_SELECTOR, 'div#article') # enter your locator
wait.until(lambda driver: len(driver.find_elements(*article_locator)) >= expected_number_of_articles)
Before accessing page source wait for a small interval to wait for page load
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
driver = webdriver.Firefox()
driver.get('https://baijia.baidu.com')
wait = WebDriverWait(driver,10)
wait.until(EC.element_to_be_clickable((By.ID, 'getMoreArticles'))).click()
time.sleep(4)
content = driver.page_source
page = open('test3.html','w')
page.write(content)
I want to run tests with selenium. IE gives me a modal error after bringing up IE 8 with this text "This is the initial start page for the WebDriver server" :
from selenium import webdriver
import time
browser = webdriver.Ie() # Get local session of IE
browser.get("http://www.google.com") # Load page
time.sleep(5)
browser.close()
So I tried Chrome.
from selenium import webdriver
browser = webdriver.Chrome()
browser.get("http://www.google.com")
time.sleep(5)
browser.close()
and Selenium errors for not having the right path to the chrome.exe application. Chrome is installed as expected... C:\Users\%USERNAME%\AppData\Local\Google\Chrome\Application\chrome.exe
A little help here would be greatly appreciated.
Have u downloaded the Chrome Driver?
To get set up, first download the appropriate prebuilt server. Make sure the server can be located on your PATH or specify its location via the webdriver.chrome.driver system property.
Then when u run
from selenium import webdriver
browser = webdriver.Chrome()
browser.get("http://www.google.com")
time.sleep(5)
browser.close()
It should work.