Trying to scrap the website using selenium but getting this error - python

type hefrom selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
s=Service('I:\chromedriver_win32\chromedriver.exe')
path='I:\chromedriver_win32\chromedriver.exe'
#Website to scrap
website='https://www.adamchoi.co.uk/overs/detailed'
driver=webdriver.Chrome(service=s,options=chrome_options)
driver.get(website)
#Locating and clicking an element
all_matches_button=driver.find_element(by='xpath',value="//label[normalize-space()='All matches']").click()
matches=driver.find_elements(by="xpath",value='tr')
for match in matches:
print(match.text)
Error:"USB: usb_device_handle_win.cc:1045 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)" and "Bluetooth: bluetooth_adapter_winrt.cc:1074 Getting Default Adapter failed."
A soultion to my problem


how did you get usb error and bluetooth? whats going on?
type hefrom selenium import webdriver where from you copy pasted?
variable all_matches_button never used in your code and there is no need to save anything to variable when click on element.
#Locating and clicking an element
all_matches_button=driver.find_element(by='xpath',value="//label[normalize-space()='All matches']").click()
here is working code to start dig deeper
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
website = 'https://www.adamchoi.co.uk/overs/detailed'
driver = webdriver.Chrome(options=chrome_options)
driver.get(website)
sleep(4)
tr_elements = driver.find_elements(By.XPATH, "//tr")
for tr in tr_elements:
print(tr.tag_name, tr.get_attribute('textContent'))


however, this message shouldn't prevent you from getting scraping result. try to correct your xpath in the last line by adding slashes
matches=driver.find_elements(by="xpath",value='//tr')

Related

Using Selenium with Python not working in google chrome

i have code it opens website in chrome for this code but it does not go further
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome("C:\Drivers\chromedriver_win32 (2)\chromedriver.exe")
driver.get('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login')
driver.find_element("username").send_keys("Admin")
driver.find_element("password").send_keys("admin123")
driver.close()
Result this one
i need answer how to solve this problem

The way you are initializing your webdriver and selecting the input elements seem to be the problem.
Try doing it this way instead.
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
import time
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login')
time.sleep(3)
driver.find_element(By.XPATH, "//input[#name='username']").send_keys("Admin")
driver.find_element(By.XPATH, "//input[#name='password']").send_keys("admin123")
time.sleep(3)
driver.close()
You may also check out selenium's quick start guide here

Can anyone send keys to this input field with selenium ? (Selenium Detected)

I am just trying to send keys to #inputPlaylist text field in this website https://youtubemultidownloader.net/playlists.html, but selenium is detected in someway.
I tried the most famous driver arguments and JS scripts but in vain.
Can anyone handle this website ?

#Othman Alkhatib, I tried the following and it is working. Let me know if this is what you were looking for. The last line time.sleep(5) is kept just so that anyone running this can see what is happening before the browser gets closed:
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
chrome_options = Options()
s = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(options=chrome_options, service=s)
driver.get("https://youtubemultidownloader.net/playlists.html")
driver.find_element(By.ID, "inputPlaylist").send_keys("ABC")
time.sleep(5)

Module selenium.webdriver has no attribute "get"

I've been trying to run a script in Python to make Chrome open up to a specific page. Here is my code so far
Code part 1
Code part 2
The code is
from selenium import webdriver as wd
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
Then
driver = wd.Chrome()
driver.implicitly_wait(5)
Then
driver.get("https://www.facebook.com")
Despite the error message in the screenshot after the second cell, Chrome opens when I run the script. It just opens to a blank page. I've tried changing the name of driver and wd and webdriver and I get the "module selenium.webdriver has no attribute "get"" every time. This post from yesterday is similar to what I'm having trouble with
but the solution isn't working for me.

Not sure, but try one of the following:
remove wd as it causing misleading
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver = webdriver.Chrome()
driver.implicitly_wait(5)
or this
from selenium import webdriver as wd
from webdriver_manager.chrome import ChromeDriverManager
driver = wd.Chrome(ChromeDriverManager().install())
driver = wd.Chrome()
driver.implicitly_wait(5)
But please don't mix between them.
I prefer the first option.

I can't locate the element but after inspect element manually the code works

I get this error when I run the code :
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#data-test-id="birth-date__day"]/select"
If I inspect Element manually in browser the code works. I was thinking it's because of some frame but can't find it. Also I tried to first click and then try to use "Select"but still not works.
This is the code :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
import requests
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
options = webdriver.ChromeOptions()
options.add_experimental_option("useAutomationExtension", False)
options.add_experimental_option("excludeSwitches",["enable-automation"])
driver_path = 'C:\Program Files (x86)\chromedriver.exe'
driver = webdriver.Chrome(executable_path=driver_path, options=options)
url = "https://account.mail.ru/signup?from=main&rf=auth.mail.ru"
Fname = "John"
Lname = "Micheals"
driver.get(url)
time.sleep(4)
day = driver.find_element_by_xpath('//div[#data-test-id="birth-date__day"]/select')
Select(day).select_by_value("11")
Please if someone can help.

Headless webdriver returns error but non-headless works

I am doing a simple experiment with Amazon and Webdriver. However, using Webdriver Headless cannot find elements and errors out, but non-headless works.
Any suggestions how to get it working headless?
There is a comment right above the --headless flag.
from selenium import webdriver
import sys
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def get_inventory(url):
chrome_options = Options()
# Making it headless breaks. Commenting
# this line, making it non-headless works.
chrome_options.add_argument("--headless")
chrome_options.add_experimental_option(
"prefs", {'profile.managed_default_content_settings.javascript': 2})
chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'
driver = webdriver.Chrome(executable_path=os.path.abspath(
'chromedriver'), chrome_options=chrome_options)
driver.set_window_size(1200, 1000)
try:
driver.get(url)
add_to_cart_button_xp = '//*[#id="add-to-cart-button"]'
add_to_cart_button = driver.find_element_by_xpath(add_to_cart_button_xp)
add_to_cart_button.click()
driver.get('https://www.amazon.com/gp/cart/view.html/ref=lh_cart')
qty_field_xp = '//div/input[starts-with(#name, "quantity.") and #type="text"]'
qty_field = driver.find_element_by_xpath(qty_field_xp)
qty_field.clear()
qty_field.send_keys("2")
update_link_xp = f'//input[#value="Update" and #type="submit"]'
update_link = driver.find_element_by_xpath(update_link_xp)
update_link.click()
url = 'https://www.amazon.com/Pexio-Professional-Stainless-Food-Safe-Dishwasher/dp/B07BGBSY9F'
get_inventory(url)

I think you just had some selector issues. I checked the elements and updated the quantity setting; everything else should be pretty much the same, aside from the binary locations.
from selenium import webdriver
import sys
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def get_inventory(url):
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(
executable_path='/usr/bin/chromedriver',
chrome_options=chrome_options,
)
chrome_options.add_experimental_option(
"prefs",
{'profile.managed_default_content_settings.javascript': 2},
)
chrome_options.binary_location = '/usr/bin'
driver.set_window_size(1200, 1000)
try:
driver.get(url)
driver.save_screenshot("/tmp/x1.png")
driver.find_element_by_xpath('//*[#id="add-to-cart-button"]').click()
driver.get('https://www.amazon.com/gp/cart/view.html/ref=lh_cart')
driver.find_element_by_xpath("//span[#data-action='a-dropdown-button']").click()
driver.find_element_by_xpath("//*[#class='a-dropdown-link'][text()[contains(., '2')]]").click()
driver.find_element_by_class_name("nav-logo-base").click()
driver.save_screenshot("/tmp/confirm.png")
driver.close()
except Exception as e:
print(e)
url = 'https://www.amazon.com/Pexio-Professional-Stainless-Food-Safe-Dishwasher/dp/B07BGBSY9F'
get_inventory(url)
I've run this with and without --headless and it's working fine for me. I navigated to the homepage at the end so you can confirm the quantity change worked (hence the screenshot).

What is the behavior you see?
When I enabled headless, scripts started failing because running headless slows execution down.
I currently run chrome with these options:
'--no-sandbox', '--headless', '--window-size=1920,1080', '--proxy-server="direct://"', '--proxy-bypass-list=*'
The last two options supposedly help with the slowness, but I didn't see any difference.
Hope this helps.

I verified your claim on my Mac (using /Applications/Google Chrome.app/Contents/MacOS/Google Chrome).
My guess is that, since you are moving from an item page to the cart page of Amazon, the cookies are lost, so that the cart page won't show any item, and therefore won't contain any text input with a name starting with “quantity”, which is what the exception is about.
Googling for headless chrome cookies yields this page, which in turn points to this page, the content of which could also be about your problem. Be it this, or be it a particularly smart behavior of the Amazon website, the fact remains: the cookie that stores the cart (or a key thereof, but the result is the same) is not read by the cart page when in headless mode.

Categories