Python Selenium for Web Scraping not working [duplicate] - python

This question already has answers here:
Error "name 'by' is not defined" using Python Selenium WebDriver
(3 answers)
Closed 3 months ago.
from selenium import webdriver
import os
os.environ['PATH'] += "C:\\Users\\czoca\\PycharmProjects\\pythonProject4\\chromedriver.exe"
driver = webdriver.Chrome()
driver.get("https://www.teintes.fr/p/recapitulatif.html")
driver.implicitly_wait(10)
myelement1 = driver.find_element(By.ID, "Autoriser")
myelement = driver.find_element(By.ID, "Carens III")
myelement1.click()
myelement.click()
This is giving me this error:
""Traceback (most recent call last):
File "C:\Users\czoca\PycharmProjects\imagesscraping\main.py", line 9, in
myelement1 = driver.find_element(By.ID, "Autoriser")
^^
NameError: name 'By' is not definedenter image description here

You are missing the import of 'By' from the package selenium. You can import it like so:
from selenium.webdriver.common.by import By
You can look up more details about selenium on the docs

Add the following import to your file:
from selenium.webdriver.common.by import By
And it should be good.

You need to import the By module from Selenium in order to use it ( I agree that it is not obvious at first time), by simply adding this line
from selenium.webdriver.common.by import By
You will also probably need thoses packages :
# Import required packages
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver import EdgeOptions
Hope I helped you
Have a good day

Related

How to enter file path?

How can I do to type something in the field of the image below?
I've tried without success:
from threading import local
import pandas as pd
import pyautogui
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait.until(EC.presence_of_element_located((By.XPATH,"//input[#type='file']"))send_keys("C:/Users/my_user/Downloads/doch.jpeg")
for index, row in df.iterrows():
actions.send_keys((row["message"]))
actions.perform()
The only palliative solution was:
pyautogui.write((row["photo"]))
pyautogui.press("enter")
I don't want to use pyautogui as it uses the keyboard command and I can't do anything on the computer while the code is running.
Selenium can't upload files using the Windows select file option, so you'll have to do something else - you might be able to use the send_keys function, i.e.:
elem = driver.find_element(By.XPATH, "//input[#type='file']")
elem.send_keys('C:\\Path\\To\\File')
Note that this may not work, depending on the type of input, and you may be able to instead simulate a drag-and-drop operation if the website supports this.
See How to upload file ( picture ) with selenium, python for more info
For windows path you need double backslashes. Try this:
wait.until(EC.presence_of_element_located((By.XPATH,"//input[#type='file']"))send_keys("C:\\Users\\my_user\\Downloads\\doch.jpeg")

AttributeError: type object 'By' has no attribute 'id' Selenium [duplicate]

This question already has an answer here:
Selenium Webdriver Python AttributeError type object has no attribute
(1 answer)
Closed last year.
I am trying to fill dropdown form using Selenium. I am trying select the dropdown element by its ID but getting this error: AttributeError: type object 'By' has no attribute 'id' I am getting this error for this line: element = driver.find_element(By.id('State'))
from lib2to3.pgen2 import driver
from sre_parse import State
from tkinter.tix import Select
from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
import time
# web = webdriver.Chrome(ChromeDriverManager().install())
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://ssg2021.in/citizenfeedback")
time.sleep(2)
element = driver.find_element(By.id('State'))
drp = Select(element)
drp.select_by_visible_text('Chhattisgarh')
wait=WebDriverWait(driver,30)
driver.get("https://ssg2021.in/citizenfeedback")
element = wait.until(EC.presence_of_element_located((By.ID,"State")))
drp = Select(element)
drp.select_by_visible_text('Chhattisgarh')
Just wait for presence of element and then select by text.
Outputs:
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
You can try as following
select = Select(driver.find_element_by_id('State'))
# select by visible text
select.select_by_visible_text('Chhattisgarh')

Unable to import selenium webdriver from within a function

I have a few scripts, wherein I am required to import a few modules everytime. To avoid using the import statements everytime I write a new script, I tried to write a function as follows so that I can import the function instead. Here's how wrote the code for that :
def mylibs():
import selenium
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
return
mylibs()
But when I am running the next line of the code- which should use the webdriver imported above to launch an instance of the chrome browser :
browser = webdriver.Chrome(r"c:\users\nila9\drivers\chromedriver.exe")
I am getting an error like "webdriver not defined", so the browser fails to launch.
I am not able to understand what I am getting wrong here...I also tried to do it without the return also, but same result.
If this works, I can then import the module into any other script, whenever I need to use the block of import codes.
Any help appreciated.
try
driver = webdriver.Chrome(executable_path="C:\Users\nila9\drivers\chromedriver.exe")
or
driver = webdriver.Chrome("C:\Users\nila9\drivers\chromedriver.exe")

Scrape google maps review text data for one company

I wanted to scrape text review data from google maps review for one company in order to perform sentiment analysis. However, my code is not running! I am getting error. I was wondering if you could guide me to fix this. Thanks!
!pip install selenium
!apt-get update
!apt install chromium-chromedriver
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver =webdriver.Chrome('chromedriver',chrome_options=chrome_options)
#add your google map link whose data you want to scrape
from selenium import webdriver
from bs4 import BeautifulSoup
import time
import io
import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup
import time
import io
import pandas as pd
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver.get('https://www.google.com/maps/place/Embassy+of+Bangladesh/#38.9418017,-77.0679642,15z/data=!4m7!3m6!1s0x0:0x5621455e7625f36e!8m2!3d38.9418017!4d-77.0679642!9m1!1b1')
wait = WebDriverWait(driver, 10)
menu_bt = wait.until(EC.element_to_be_clickable(
(By.XPATH, '//button[#data-value=\'Sort\']'))
)
menu_bt.click()
recent_rating_bt = driver.find_elements_by_xpath(
'//div[#role=\'menuitem\']')[50]
recent_rating_bt.click()
time.sleep(5)
Error message:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-35-94b4c6e89470> in <module>()
5 menu_bt.click()
6 recent_rating_bt = driver.find_elements_by_xpath(
----> 7 '//div[#role=\'menuitem\']')[50]
8 recent_rating_bt.click()
9 time.sleep(5)
IndexError: list index out of range
You're accessing the item indexed by 50 on the list returned by find_elements_by_xpath(). The error message indicates that this index does not exist, i.e. the returned list is smaller than that.
You should check the length of the returned list before accessing it.

How do I fill out a form using Python and Selenium?

I am new to Python and wanted to use it for automatic login. I found https://automatetheboringstuff.com/chapter11/ and tried:
#! python3
from selenium import webdriver
browser = webdriver.Firefox()
type(browser)
browser.get('https://forum-studienstiftung.de/')
emailEl = browser.find_element_by_id(username)
Unfortunately, this leads to:
Traceback (most recent call last): File "", line 1, in
emailEl = browser.find_element_by_id(username) NameError:
name 'username' is not defined
According to the Firefox Developer Tools the correct ID is "username".
Wrap username in quotation marks. Right now you are passing in a variable called username which selenium is trying to match with an id on the page with the same value. Since the value is none, Selenium cannot find it hence the error.
The page you are trying to access takes time to load. You have to wait for the element to be visible before accessing it.
Try this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Firefox()
type(browser)
browser.get('https://forum-studienstiftung.de/')
emailEl = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.id, "username")))

Categories