I am trying to scrape data from this File-tuning dynamic website which is loading data through javascript (ajax) requests.
what is want to do is that It selects cars from type and then selects make, model, engine iteratively, and then I want to scrape data for each make, model, and engine.
Here is the code I write to select cars from type
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.select import Select
PATH = "C:\SeleniumDrivers\geckodriver.exe"
driver = webdriver.Firefox(executable_path=PATH)
driver.get("https://file-tuning.com/chiptuning")
type_element_select = driver.find_element_by_id("type")
action = ActionChains(driver)
action.move_to_element(type_element_select)
action.click(type_element_select)
action.perform()
action.move_to_element(Select(type_element_select).select_by_value("cars"))
action.click(Select(type_element_select).select_by_value("cars"))
action.perform()
The error I get:
Traceback (most recent call last):
File "D:\Python\selenium\test.py", line 27, in <module>
action.move_to_element(Select(type_element_select).select_by_value("cars"))
File "C:\Users\Umair\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\support\select.py", line 82, in select_by_value
self._setSelected(opt)
File "C:\Users\Umair\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\support\select.py", line 212, in _setSelected
option.click()
File "C:\Users\Umair\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\Umair\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "C:\Users\Umair\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Umair\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: Element <option> could not be scrolled into view
I also try this
types_element = driver.find_element_by_id("type")
types_object = Select(types_element)
types_object.select_by_visible_text("Cars")
but it also gives me the same exception.
how I can scrape through this site?
already have seen and tried other StackOverflow related question but didn't work out for me.
You need to use browser in full screen mode :
driver.maximize_window()
so that all web elements would be visible to Selenium.
Sample code :
PATH = "C:\SeleniumDrivers\geckodriver.exe"
driver = webdriver.Firefox(executable_path=PATH)
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://file-tuning.com/chiptuning")
wait = WebDriverWait(driver, 10)
type_element_select = driver.find_element_by_id("type")
action = ActionChains(driver)
action.move_to_element(type_element_select)
action.click(type_element_select)
action.perform()
action.move_to_element(Select(type_element_select).select_by_value("cars"))
action.click(Select(type_element_select).select_by_value("cars"))
action.perform()
Related
I've error in Python Selenium. I'm trying to download all songs with Selenium, but there is some error. Here is code:
from selenium import webdriver
import time
driver = webdriver.Chrome('/home/tigran/Documents/chromedriver/chromedriver')
url = 'https://sefon.pro/genres/shanson/top/'
driver.get(url)
songs = driver.find_elements_by_xpath('/html/body/div[2]/div[2]/div[1]/div[3]/div/div[3]/div[2]/a')
for song in songs:
song.click()
time.sleep(5)
driver.find_element_by_xpath('/html/body/div[2]/div[2]/div[1]/div[1]/div[2]/div/div[3]/div[1]/a[2]').click()
time.sleep(8)
driver.get(url)
time.sleep(5)
And here is error:
Traceback (most recent call last):
File "test.py", line 13, in <module>
song.click()
File "/home/tigran/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/tigran/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/home/tigran/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/tigran/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=90.0.4430.72)
Any ideas why error comes?
Before your loop try to add try:
Sometimes the document is not attached, so it gives an error
Here's link selenium while loop not working. You need to locate elements in loop.
Final code:
from selenium import webdriver
import time
driver = webdriver.Chrome('/home/tigran/Documents/chromedriver/chromedriver')
url = 'https://sefon.pro/genres/shanson/top/'
driver.get(url)
size = len(driver.find_elements_by_xpath('/html/body/div[2]/div[2]/div[1]/div[3]/div/div[3]/div[2]/a'))
for i in range(0, size):
songs = driver.find_elements_by_xpath('/html/body/div[2]/div[2]/div[1]/div[3]/div/div[3]/div[2]/a')
songs[i].click()
time.sleep(5)
driver.find_element_by_xpath('/html/body/div[2]/div[2]/div[1]/div[1]/div[2]/div/div[3]/div[1]/a[2]').click()
time.sleep(8)
driver.execute_script("window.history.go(-1)")
time.sleep(5)
My objective is to take a value (in my case a shipment tracking #) and input it into a tracking field on a website using selenium. I am unable to input the value and get the following error message:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chromedriver = "/Users/GUVA/Downloads/chromedriver"
driver = webdriver.Chrome(chromedriver)
driver.get("https://www.17track.net/en")
#data = df.values[1] # grabbing one tracking number from an excel file
# click "agree" to close window
python_button = driver.find_elements_by_xpath('//*[#id="modal-gdpr"]/div/div/div[3]/button')[0]
python_button.click()
# enter tracking number into text box
que=driver.find_element_by_id('//input[#id="jsTrackBox"]')
que.send_keys("data")
Traceback (most recent call last):
File "/Users/GUVA/PycharmProjects/Shipment_Tracking/Track Shipment.py", line 26, in <module>
que=driver.find_element_by_id('//input[#id="jsTrackBox"]')
File "/Users/GUVA/PycharmProjects/Shipment_Tracking/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/Users/GUVA/PycharmProjects/Shipment_Tracking/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/Users/GUVA/PycharmProjects/Shipment_Tracking/venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Users/GUVA/PycharmProjects/Shipment_Tracking/venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified
(Session info: chrome=83.0.4103.61)
Any suggestions please?
//input[#id="jsTrackBox"] is an xpath. So you need to fetch the element by using find_element_by_xpath method(where you are currently using find_element_by_id method).
Your code should be like:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chromedriver = "/Users/GUVA/Downloads/chromedriver"
driver = webdriver.Chrome(chromedriver)
driver.get("https://www.17track.net/en")
#data = df.values[1] # grabbing one tracking number from an excel file
# click "agree" to close window
python_button = driver.find_elements_by_xpath('//*[#id="modal-gdpr"]/div/div/div[3]/button')[0]
python_button.click()
# enter tracking number into text box
que=driver.find_element_by_xpath("//div[#id='jsTrackBox']//div[#class='CodeMirror-scroll']")
que.click()
que.send_keys("data")
from selenium import webdriver
from info import user_name, pass_word
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
driver = webdriver.Firefox() #opening web browser
driver.get('https://tuportal.temple.edu/') #going to temple wbsite
username = driver.find_element_by_id("username") #finds space where to enteruser
password = driver.find_element_by_id("password") #same for password
username.send_keys(user_name()) #enters my username
password.send_keys(pass_word()) #enters my password
driver.find_element_by_name("_eventId_proceed").click()#click login
#waits until the element for student tools tab is found
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "layout_8")))
element.click()
time.sleep(3)
driver.switch_to_frame("iFrame_AppTupChannelsStudentRegistration")#accessing iframe
#scrolls
element = driver.find_element_by_id('StudentRegistration_3')
driver.execute_script("return arguments[0].scrollIntoView();", element)
#end of scroll
element.click() #click look up classes
All of the code works up until the last line, "element.click()"
I am getting an error -->
Traceback (most recent call last):
File "C:\Users\Karl\Desktop\project\signin.py", line 31, in
element.click() #click look up classes
File "C:\Users\Karl\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\Karl\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "C:\Users\Karl\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "C:\Users\Karl\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message: Element could not be scrolled into view
The confusing part is the 2 lines right before the line that causes the error are
element = driver.find_element_by_id('StudentRegistration_3')
driver.execute_script("return arguments[0].scrollIntoView();", element)
which do exactly what the error says hasn't happened, they scroll the the button I am trying to click on with the last line into view.
Can anyone identify a solution?
I am using python 3, selenium and Firefox are updated to current versions.
I'd like to scroll to the bottom of a page using Selenium with the Firefox driver.
Here is what I have so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser=webdriver.Firefox()
browser.get('http://nostarch.com')
htmlElem=browser.find_element_by_tag_name('html')
htmlElem.send_keys(Keys.END) #scrolls to bottom
Unfortunately, my code fails with the following exception:
Traceback (most recent call last):
File "<pyshell#26>", line 1, in <module>
htmlElem.send_keys(Keys.END) #scrolls to bottom
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\webelement.py", line 352, in send_keys
'value': keys_to_typing(value)})
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\webelement.py", line 501, in _execute
return self._parent.execute(command, params)
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\webdriver.py", line 308, in execute
self.error_handler.check_response(response)
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\errorhandler.py", line 194, in
check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message:
Element is not visible
Use WebDriverWait (explicit wait) till element to be visible i dont know too much about python so give just general syntax
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser=webdriver.Firefox()
browser.get('http://nostarch.com')
htmlElem=WebDriverWait(browser, 5).until(
EC.presence_of_element_located((By.find_element_by_tag_name, "html"))
htmlElem.send_keys(Keys.END)
To scroll to the bottom of a page using Selenium you can use :
Python
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Java
((JavascriptExecutor)driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
I'm attempting to create an automated login script to netflix website: https://www.netflix.com/it/
That's the code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
while True:
driver.get("https://www.netflix.com/it/")
login = driver.find_element_by_css_selector(".authLinks.signupBasicHeader")
login.click()
element = driver.find_element_by_name("email")
element.send_keys("test1#email.com")
submit = driver.find_element_by_css_selector(".btn.login-button.btn-submit.btn-small")
submit.click()
element2 = driver.find_element_by_name("password")
element2.send_keys("test1")
submit.click()
But sometimes it works and sometimes it doesn't and it raises this exception:
Traceback (most recent call last):
File "netflix.py", line 35, in <module>
submit.click()
File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 72, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webelement.py", line 461, in _execute
return self._parent.execute(command, params)
File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/home/user/.local/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
Stacktrace:
at fxdriver.preconditions.visible (file:///tmp/tmp7otgskq8/extensions/fxdriver#googlecode.com/components/command-processor.js:10092)
at DelayedCommand.prototype.checkPreconditions_ (file:///tmp/tmp7otgskq8/extensions/fxdriver#googlecode.com/components/command-processor.js:12644)
at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmp7otgskq8/extensions/fxdriver#googlecode.com/components/command-processor.js:12661)
at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmp7otgskq8/extensions/fxdriver#googlecode.com/components/command-processor.js:12666)
at DelayedCommand.prototype.execute/< (file:///tmp/tmp7otgskq8/extensions/fxdriver#googlecode.com/components/command-processor.js:12608)
The exception says that a part of the web page is invisible (even if that part IS in the page actually)... It's a sort of bug.
How can I bypass this?
No need to put the test code inside while loop, since login form should be submitted once. I guess when it works successfully, next iteration gives error since there is no form.
You are also clicking the submit button before and after password entry. It is better to click after form is fully filled, if possible and not testing empty form.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://www.netflix.com/it/")
login = driver.find_element_by_css_selector(".authLinks.signupBasicHeader")
login.click()
element = driver.find_element_by_name("email")
element.send_keys("test1#email.com")
submit = driver.find_element_by_css_selector(".btn.login-button.btn-submit.btn-small")
submit.click()
element2 = driver.find_element_by_name("password")
element2.send_keys("test1")
submit = driver.find_element_by_css_selector(".btn.login-button.btn-submit.btn-small")
submit.click()