We Keep Coding

Use Python Selenium to extract span text

Hi i'm new at selenium and webscraping and i need some help.
i try to scrape one site and i need and i dont know how to get span class.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATCH = "/Users/bobo/Downloads/chromedriver"
driver = webdriver.Chrome(PATCH)
driver.get("https://neonet.pl")
print(driver.title)
search = driver.find_element_by_class_name("inputCss-input__label-263")
search.send_keys(Keys.RETURN)
time.sleep(5)
i try to extract this span
<span class="inputCss-input__label-263">Szukaj produktu</span>

I can see that you are trying to search something in the search bar.
First I recommend you to use the xpath instead of the class name, here is a simple technique to get the xpath of every element on a webpage:
right-click/ inspect element/ select the mouse in a box element on the upper left/ click on the element on the webpage/ it will directly show you the corresponding html/ then right click on the selected html/ copy options and then xpath.
Here is a code example that searches an element on the webpage, I also included the 'Webdriver-wait' option because sometimes the code runs to fast and can't find the next element so this function make the code wait till the element is visible:
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path="/Users/bobo/Downloads/chromedriver")
driver.get("https://neonet.pl") #loading page
wait = WebDriverWait(driver, 20) #defining webdriver wait
search_word = 'iphone\n' # \n is going to act as an enter key
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/div[1]/div[4]/div/div/div[2]/div/button[1]'))).click() #clicking on cookies popup
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/header/div[2]/button'))).click() #clicking on search button
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/aside[2]/section/form/label/input'))).send_keys(search_word) #searching on input button
print('done!')
sleep(10)
Hope this helped you!

wait=WebDriverWait(driver,10)
driver.get('https://neonet.pl')
elem=wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(#class,'inputCss-input')]"))).text
print(elem)
To output the value of the search bar you use .text on the Selenium Webelement.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Click “Accept Cookies” popup with Selenium in Python

I've been trying to scrape some information of this E-commerce website with selenium. However when I access the website I need to accept cookies to continue. This only happens when the bot accesses the website, not when I do it manually. When I try to find the corresponding element either by xpath, as I find it when I inspect the page manually I always get this error message:
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
My code is mentined below.
import time
import pandas
pip install selenium
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup #pip install beautifulsoup4
PATH = "/Users/Ziye/Desktop/Python/chromedriver"
delay = 15
driver = webdriver.Chrome(PATH)
driver.implicitly_wait(10)
driver.get("https://en.zalando.de/women/?q=michael+michael+kors+taschen")
driver.find_element_by_xpath('//*[#id="uc-btn-accept-banner"]').click()
This is the HTML corresponding to the button "That's ok". The XPATH is as above.
<button aria-label="" id="uc-btn-accept-banner" class="uc-btn uc-btn-primary">
That’s OK <span id="uc-optin-timer-display"></span>
</button>
Does anyone know where my mistake lies?

You should add explicit wait for this button:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
driver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
driver.implicitly_wait(10)
driver.get("https://en.zalando.de/women/?q=michael+michael+kors+taschen")
wait = WebDriverWait(driver, 15)
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="uc-btn-accept-banner"]')))
driver.find_element_by_xpath('//*[#id="uc-btn-accept-banner"]').click()
Your locator is correct.
As css selector, you can use .uc-btn-footer-container .uc-btn.uc-btn-primary

Selenium find_element_by_xpath not working for instagram

I am trying to web scrape Instagram using Python and Selenium. I have had many issues regarding locating the elements but somehow managed to pull through when I tried enough xpaths. But when I try to web scrape Donald Trump's following list (I want this to work for ANY USER'S following list/page), it just doesn't work. Here is the error it's throwing:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="f3b066159b38864"]/div/div/a"}
I get the xpaths by right clicking on the element using Google Chrome's inspect feature. If anyone needs me to post the full code I'd be happy to do so.

Try below xpath::
wait = WebDriverWait(driver, 20)
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(text(),'laraleatrump')]")))
Note : please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
Working solution :
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
driver.get("https://www.instagram.com/realdonaldtrump/")
driver.maximize_window()
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(.,'following')]"))).click()
peoples = wait.until(
EC.visibility_of_all_elements_located((By.XPATH, "//div[#role='dialog']//div[contains(#class,'PZuss')]//a")))
for peoplename in peoples:
print peoplename.text

selenium using python: how to correctly click() an element?

while learning how to use selenium, Im trying to click an element but nothing happens and Im unable to reach the next page. this is the relevant page: http://buyme.co.il and Im trying to click: הרשמה
I managed to print the desired element (הרשמה) so I guess Im reaching the correct place in the page. but 'click()' doesnt work.
the second span <span>הרשמה</span> is what i want to click:
<li data-ember-action="636">
<a>
<span class="seperator-link">כניסה</span>
<span>הרשמה</span>
</a>
</li>
for elem in driver.find_elements_by_xpath('//* [#id="ember591"]/div/ul[1]/li[3]/a/span[2]'):
print (elem.text)
elem.click()
also tried this:
driver.find_element_by_xpath('//*[#id="ember591"]/div/ul[1]/li[3]/a').click()
I expected to get to the "lightbox" which contain the registration fields.
Any thoughts on the best way to accomplish this?

Explicit Waits - An explicit wait is a code you define to wait for a certain condition to occur before proceeding further in the code.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome()
browser.get("https://buyme.co.il/")
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.ID, 'ember591')))
elm = browser.find_elements_by_xpath('//div[#id="ember591"]/div/ul[1]/li[3]/a')
elm[0].click()
Update:
WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, 'login')))
email = browser.find_elements_by_xpath("//form[#id='ember1005']/div[1]/label/input")
email[0].send_keys("abc#gmail.com")
password = browser.find_elements_by_xpath("//form[#id='ember1005']/div[2]/label/input")
password[0].send_keys("test1234567")
login = browser.find_elements_by_xpath('//form[#id="ember1005"]/button')
login[0].click()

The desired element is an Ember.js enabled element so to locate the element you have to induce WebDriverWait for the element to be clickable and you can use the following Locator Strategy:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='הרשמה']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Python 2.7 Selenium No Such Element on Website

I'm trying to do some webscraping from a betting website:
As part of the process, I have to click on the different buttons under the "Favourites" section on the left side to select different competitions.
Let's take the ENG Premier League button as example. I identified the button as:
(source: 666kb.com)
The XPath is: //*[#id="SportMenuF"]/div[3] and the ID is 91.
My code for clicking on the button is as follows:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = "C:\Python27\Scripts\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("URL Removed")
content = driver.find_element_by_xpath('//*[#id="SportMenuF"]/div[3]')
content.click()
Unfortunately, I always get this error message when I run the script:
"no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="SportMenuF"]/div[3]"}"
I have tried different identifiers such as CCS Selector, ID and, as shown in the example above, the Xpath. I tried using waits and explicit conditions, too. None of this has worked.
I also attempted scraping some values from the website without any success:
from selenium import webdriver
from selenium.webdriver.common.by import By
chrome_path = "C:\Python27\Scripts\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("URL removed")
content = driver.find_elements_by_class_name('price-val')
for entry in content:
print entry.text
Same problem, nothing shows up.
The website embeddes an iframe from a different website. Could this be the cause of my problems? I tried scraping directly from the iframe URL, too, which didn't work, either.
I would appreciate any suggestions.

Sometimes elements are either hiding behind an iframe, or they haven't loaded yet
For the iframe check, try:
driver.switch_to.frame(0)
For the wait check, try:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '-put the x-path here-')))