We Keep Coding

Changing country while scraping Aliexpress using python selenium

I'm working on a scraping project of Aliexpress, and I want to change the ship to country using selenium,for example change spain to Australia and click Save button and then scrap the page, I already found an answer it worked just I don't know how can I save it by clicking the button save using selenium, any help is highly appreciated. This is my code using for this task :
country_button = driver.find_element_by_class_name('ship-to')
country_button.click()
country_buttonn = driver.find_element_by_class_name('shipping-text')
country_buttonn.click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[#class='address-select-item ']//span[#class='shipping-text' and text()='Australia']"))).click()

Well, there are 2 pop-ups there you need to close first in order to access any other elements. Then You can select the desired shipment destination. I used WebDriverWait for all those commands to make the code stable. Also, I used scrolling to scroll the desired destination button before clicking on it and finally clicked the save button.
The code below works.
Just pay attention that after selecting a new destination pop-ups can appear again.
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
options = Options()
options.add_argument("--start-maximized")
s = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=s)
url = 'https://www.aliexpress.com/'
wait = WebDriverWait(driver, 10)
actions = ActionChains(driver)
driver.get(url)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#style,'display: block')]//img[contains(#src,'TB1')]"))).click()
except:
pass
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//img[#class='_24EHh']"))).click()
except:
pass
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "ship-to"))).click()
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "shipping-text"))).click()
ship_to_australia_element = driver.find_element(By.XPATH, "//li[#class='address-select-item ']//span[#class='shipping-text' and text()='Australia']")
actions.move_to_element(ship_to_australia_element).perform()
time.sleep(0.5)
ship_to_australia_element.click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-role='save']"))).click()
I mostly used XPath locators here. CSS Selectors could be used as well

selenium chromedriver different values of xpath between terminal and actual driver

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://www.msha.gov/mine-data-retrieval-system'
driver = webdriver.Chrome(executable_path='chromedriver')
driver.get(url)
#driver.find_element_by_xpath('//*[#id="mstr90"]/div[1]/div/div') error
#driver.find_elements_by_xpath('//input') gives 3 while in driver gives 10
I am unable to find element where the input "Search by Mine ID by typing here.." is, the document is fully loaded but it can't locate it. What I want to do is simply pass in an input "0100003" then submit

Iframe is present on your page. Before you interact with inputbox you need to switch on to iframee. Refer below code to resolve your issue.
wait = WebDriverWait(driver, 10)
driver.get("https://www.msha.gov/mine-data-retrieval-system")
driver.switch_to.frame("iframe1")
wait = WebDriverWait(driver, 10)
inputBox = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='mstrmojo-SimpleObjectInputBox-empty']"))).click()
inputBox1 = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='mstrmojo-SimpleObjectInputBox-container mstrmojo-scrollNode']//input")))
inputBox1.send_keys("0100003")
Updated Code to handle dropdown
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#mstr100,mstrmojo-Popup.mstrmojo.SearchBoxSelector-suggest"))).click()
Note: please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
Output:

The element you are trying to find is inside an iframe, so you will need to switch to that iframe first and then do your find element. Also, it's a best practice to use waits to give pages/elements time to load before a find element timeouts and throws an error.
iframe = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#iframe1')))
driver.switch_to.frame(iframe)
mine_id = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, '//*[#id="mstr90"]/div[1]/div/div')))
Then you need to click this element to make it interactable.
mine_id.click()
Once you click then you need to re-find the input box before sending keys.
mine_id_input = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#mstr90 input')))
mine_id_input.send_keys('0100003')
To select the suggestion displayed:
suggestion = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#mstr100')))
suggestion.click()
if you wanted to continue on interacting outside the iframe after this is done, you will want to switch back out of the iframe like this:
driver.switch_to.default_content()

Selenium Webdriver (Python) - Click on a div element (checkbox)

I am trying to click on a non-button element (checkbox) inside an iframe but am not able to click on it using the selenium webdriver. The URL is https://www.nissanoflithiasprings.com/schedule-service and the box that I want to click is shown in the screenshot below:
NOTE: Select the following options to reach the screen shown in the screenshot below: Click on New Customer "MAKE. MODEL. YEAR". Select Make as "NISSAN" -> Year as "2018" -> Model as "ALTIMA" -> Trim as "SL" -> Engine Type as "I4" -> Enter Mileage as "35000" -> Click on "CONTINUE" at the bottom. On the following page, the goal is to click on the checkbox Maintenance Package Standard / Schedule 1 (the checkbox needs to be clicked but I am not able to find the correct code line for Selenium to be able to click on it)
Below is the working Selenium Python script that I wrote to successfully navigate until the page that shows the checkbox that I wish to click:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chrome_path = r"C:\Users\gh456\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button").click()
driver.find_element_by_id("NISSAN").click()
wait.until(ec.visibility_of_element_located((By.ID, "2018"))).click()
wait.until(ec.visibility_of_element_located((By.ID, "ALTIMA"))).click()
wait.until(ec.visibility_of_element_located((By.ID, "SL"))).click()
wait.until(ec.visibility_of_element_located((By.ID, "I4"))).click()
wait.until(ec.visibility_of_element_located((By.NAME, "mileage_input"))).send_keys("35000")
wait.until(ec.visibility_of_element_located((By.ID, "continue_button"))).click()
# Click on the checkbox
# ---??????????????????
Can someone help me with the correct code to make selenium webdriver to click that checkbox?

The element is covered by another element...
Also, I changed the expected_conditions as element_to_be_clickable().
So you can use ActionChains:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException, NoSuchElementException
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.action_chains import ActionChains
chrome_path = r"C:\Users\gh456\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button").click()
driver.find_element_by_id("NISSAN").click()
wait.until(ec.element_to_be_clickable((By.ID, "2018"))).click()
wait.until(ec.element_to_be_clickable((By.ID, "ALTIMA"))).click()
wait.until(ec.element_to_be_clickable((By.ID, "SL"))).click()
wait.until(ec.element_to_be_clickable((By.ID, "I4"))).click()
wait.until(ec.visibility_of_element_located((By.NAME, "mileage_input"))).send_keys("35000")
wait.until(ec.element_to_be_clickable((By.ID, "continue_button"))).click()
check_box_el = wait.until(ec.visibility_of_element_located((By.XPATH, '//div[#id="maintenance_package_section"]//label[#class="checkbox"]')))
ActionChains(driver).move_to_element(check_box_el).click().perform()
Screenshot:
You might want to use CSS_SELECTOR and not XPATH:
check_box_el = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '#maintenance_package_section > div label')))
ActionChains(driver).move_to_element(check_box_el).click().perform()

Try this locator (//*[#class="checkbox"])[1] and use ActionChains.
chk = wait.until(ec.element_to_be_clickable((By.XPATH, '(//*[#class="checkbox"])[1]')))
ActionChains(driver).move_to_element(chk).click().perform()
You can change [1] to [2] or [3] if you want

It seems to be your check box is not enabled. Please, have some conditions following is_selected()...And check whether it is enabled now.....
http://selenium-interview-questions.blogspot.com/2014/03/working-with-checkboxes-in-selenium.html

Selenium Webdriver (Python) - Unable to locate element inside nested iframes

I am trying to scrape some data from an iframe located within a webpage. The URL of the webpage is https://www.nissanoflithiasprings.com/schedule-service. I am trying to access the button shown in the image below:
When I right-click on the button (located inside the iframe) to view the source code, I am able to see the HTML id and name (see screenshot below):
The "id" for the button is "new_customer_button". However, when I use selenium webdriver's driver.find_element_by_id("new_customer_button") to access the button, the code is not able to locate the button inside the iframe and throws the following error:
NoSuchElementException: no such element: Unable to locate element: {"method":"id","selector":"new_customer_button"}
Below is the code that I have tried so far:
from selenium import webdriver
chrome_path = r"C:\Users\gh455\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
dest_iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(dest_iframe)
driver.find_element_by_id("new_customer_button")
Not sure why this is happening. Any help will be appreciated. Thanks!

The element is inside multiple <iframe> tags, you need to switch to them one by one. You should also maximize the window and use explicit wait as it take some time to load
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chrome_path = r"C:\Users\gh455\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button")

To click() on the element with text as Make · Year · Model as the the desired element is within an nested <iframe>s so you have to:
Induce WebDriverWait for the desired parent frame to be available and switch to it.
Induce WebDriverWait for the desired child frame to be available and switch to it.
Induce WebDriverWait for the desired element_to_be_clickable().
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument('disable-infobars')
driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='com/scheduling']")))
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='consumerportal']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.button.button--action.btn.btn-secondary#new_customer_button"))).click()
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe

How to click same button in another section while scraping using selenium

So I'm scraping using selenium and I want to click 'next' button in 'Defensive' section but the code I wrote clicks 'next' on 'Summary'.
Here's the url for you to try :
https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019
So it's selecting 'Defensive' and I can see it selected in the window but the next page doesnt appear. On clicking 'Summary' I found out next function is actually happening there.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser= webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
browser.find_element_by_xpath("""//*[#id="stage-top-player-stats-options"]/li[2]/a""").click()
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, """//*[#id="next"]""")))
browser.execute_script("arguments[0].click();", element)

The xpath for next button is not unique for this page. try this,
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
browser.execute_script("arguments[0].click();", element)
or
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
element.click()

For each tab (Summary, Defensive, ..) new next button with same id=next added to the DOM.
Select Defensive and you will see there will be two next buttons with same id=next, select Offensive and there will be three next buttons.
With basic id=next selector you always click to the first next button from Summary tab. Because you're using JavaScript and nothing happen, try to click with Selenium click method and you will get an error.
To solve the problem adjust your selector to be more specific to the dom - #statistics-paging-defensive #next.
Also when you first time open the page there's cookies acceptance screen appears and block the page, you can use method like below to skip it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import selenium.common.exceptions as EX
def accept_cookies():
try:
WebDriverWait(browser, 20)\
.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.qc-cmp-button")))\
.click()
except EX.NoSuchElementException or EX.TimeoutException:
pass
#...
browser = webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
wait = WebDriverWait(browser, 20)
browser.get(baseUrl)
accept_cookies()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[href='#stage-top-player-stats-defensive']"))).click()
next_button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "#statistics-paging-defensive #next")))
next_button.click()

Your elements locators must be unique
Avoid using XPath wildcards - * as it will cause performance degradation and prolonged elements lookup timings
Avoid using JavaScriptExecutor for clicking, well-behaved Selenium test must do what real user does and I doubt that real user will be opening browser console and typing something like document.getElementById('next').click(), he will use the mouse
Assuming all above you should come up with a selector which uniquely identifies next button on Defensive tab which would be something like:
//div[#id='statistics-paging-defensive']/descendant::a[#id='next']
References:
XPath Tutorial
XPath Axes
XPath Operators & Functions