I want to make a bot using selenium, but I'm having trouble with my bot going to a different part of a website. In my code, my driver successfully goes to nike.com (1), then successfully clicks and loads a different link within Nike (clicks circled area in (1) and goes to (2)). Then, my problems begin here, I try to click and load a different link (2) but my driver does nothing. I know my driver found the second link because if I print out 'second.text' then I get the correct text (3)...
I am still new to selenium and I pretty much don't know what I am doing. Any help would be helpful.
Thank you.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
if __name__ == '__main__':
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get("https://www.nike.com/men")
driver.implicitly_wait(5)
first = driver.find_element(by=By.CLASS_NAME, value="prl3-sm")
first.click()
driver.implicitly_wait(5)
second = driver.find_element(by=By.CSS_SELECTOR, value='a[class="JSftBPEZ"]')
#print(second.text)
second.click()
I have tested it.
Through Javascript click its getting clicked.
here is the code to click on second link.
second = driver.find_element(by=By.CSS_SELECTOR, value='a[class="JSftBPEZ"]')
driver.execute_script("arguments[0].click();",second)
BTW you may need to define xpath properly. Example the second link pointing to 6 elements. But anyway through Javascript sclick it would click the 1st option
Related
I’ve run into a problem trying to use Selenium ChromeDriver to scroll down the sidebar of a google maps results page. I am trying to get to the 6th result down but the result does not fully load until you scroll down. Using the find_element_by_xpath method, I am successfully able to access results 1-5 and click into them individually, but when trying to use the actions.move_to_element(link).perform() method to scroll to the 6th element, it does not work and throws an error message.
The error that I get is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:
However, I know this element exists because when I manually scroll and more results are loaded, the Xpath works correctly. What am I doing wrong? I’ve spent many hours trying to solve this and I haven’t been able to solve with the available content out there. I appreciate any help or insights you can offer, thank you!
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup as soup
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.google.com/maps")
time.sleep(7)
page = soup(driver.page_source, 'html.parser')
#find the searchbar, enter search, and hit return
search = driver.find_element_by_id('searchboxinput')
search.send_keys("dentists in Austin Texas")
search.send_keys(Keys.RETURN)
driver.maximize_window()
time.sleep(7)
#I want to get the 6th result down but it requires a sidebar scroll to load
link = driver.find_element_by_xpath("//*[#id='pane']/div/div[1]/div/div/div[4]/div[1]/div[13]/div/a")
actions.move_to_element(link).perform()
link.click()
time.sleep(5)
driver.back()```
I found a solution that works, it is to target the element in XPATH from the javascript interface of selenium. You must then execute two commands on an instruction (targeting and scroll)
driver.executeScript("var el = document.evaluate('/html/body/jsl/div[3]/div[10]/div[8]/div/div[1]/div/div/div[4]/div[1]', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue; el.scroll(0, 5000);");
this is the only solution that worked for me
The search results in the google map are located with //div[contains(#aria-label,'dentists in Austin Texas')]//div[contains(#jsaction,'mouseover')] XPath.
So, to select 6-th element there you can do the following
from selenium.webdriver.common.action_chains import ActionChains
results = driver.find_elements_by_xpath('//div[contains(#aria-label,"dentists in Austin Texas")]//div[contains(#jsaction,"mouseover")]')
ActionChains(driver).move_to_element(results[6]).click(button).perform()
I was just implementing scrolling on google map sidebar, it's working on my side. check this code please
# selecting scroll body
driver.find_element_by_xpath('/html/body/div[3]/div[9]/div[9]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]').click()
#start scrolling your sidebar
html = driver.find_element_by_xpath('/html/body/div[3]/div[9]/div[9]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]')
html.send_keys(Keys.END)
also add the "KEYS" library
from selenium.webdriver.common.keys import Keys
I hope it would help you.
by the way I have implemented scrapping of google map with its available data and used above code to scroll. check if you have any problem, let me know then
Im trying to make a bot that visits my adfly link using the chrome webdriver. Every time I try to use the code below though, the webdriver tells me that there were too many redirects and doesn't follow through. The code below is just being used for testing at the moment:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server="+"http://102.129.249.120:8080")
browser = webdriver.Chrome(options=options)
browser.get("http://raboninco.com/18Whc")
Image of error here
Okay so i figured it out. I can use tor with selenium to get access to adfly. Works great btw. Thanks for the help and time guys. If you want to see the code I used, here it is:
from selenium import webdriver
import os
os.popen(r'C:\Users\joldb\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server="+"socks5://127.0.0.1:9050")
browser = webdriver.Chrome(options=options)
browser.get("http://raboninco.com/18Whc")
I'm testing some web scraping on Instagram with Selenium and Python.
In this case I want to upload a picture.
Normally you have to click on the upload icon and choose the file from a window. How can I manage it with Selenium?
I tried:
driver.find_element_by_class_name("coreSpriteFeedCreation").send_keys('C:\\path-to-file\\file.jpg')
and also with find_element_by_xpath but I get an exception:
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot focus element
I tried also only with click() but nothing happens.
Any Idea?
EDIT
Thanks to #homersimpson comment I tried this:
actions = ActionChains(driver)
element = driver.find_element_by_class_name("coreSpriteFeedCreation")
actions.move_to_element(element)
actions.click()
actions.send_keys('C:\\path-to-file\\file.jpg')
actions.perform()
Now the window to choose the file appears. The problem is that I would like to avoid this window and give directly the path of my file.
If right understand, you are trying to avoid handling with a native window. You can try this:
# get all inputs
inputs = driver.find_elements_by_xpath("//input[#accept = 'image/jpeg']").send_keys(os.getcwd() + "/image.png")
Now you can try all of them. I don't know which of them will work.
More about os.getcwd() is here
To be able to perform this code you have to have an element like this:
<input type="file" name="fileToUpload" id="fileToUpload2" class="fileToUpload">
EDIT:
It looks like instagram turned of input fields interaction for posts. For Account image it still works, but not for posting. I assume it is was done to prevent bots to post images. Anyway, there is a solution for this problem. You can use AutoIt like this:
import autoit
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
ActionChains(driver).move_to_element( driver.find_element_by_xpath("//path/to/upload/button")).click().perform()
handle = "[CLASS:#32770; TITLE:Open]"
autoit.win_wait(handle, 60)
autoit.control_set_text(handle, "Edit1", "\\file\\path")
autoit.control_click(handle, "Button1")
I think I may have found a solution that works for me. I found if you first have the bot click the plus icon while the browser is in the mobile view.
self.driver.find_element_by_xpath("/html/body/div[1]/section/nav[2]/div/div/div[2]/div/div/div[3]")\
.click()
after that, I would immediately send my file to an input tag in the HTML and I find you may need to play around as to which one works but I find the last input tag worked for me.
self.driver.find_element_by_xpath("/html/body/div[1]/section/nav[2]/div/div/form/input")\
.send_keys("/image/path.jpg")
The one weird thing about this is you will have a popup menu on top of the page but your code will still function with this window displayed over the window you are working on.
Addition to HumbleFox's answer. To solve his problem regarding the pop-up box not closing or the file pop-up box not closing (bug)
The solution to this is to make the browser headless here's a part of my code for example:
mobile_emulation = { "deviceName": "Pixel 2" }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
chrome_options.binary_location = self.opt.binary_location
self.driver = webdriver.Chrome(executable_path=self.path, options=chrome_options)
I'm new to selenium and trying to automate the download of some government data. When using the code below. I manage to navigate to the right page and enter the right parmeter in the form, but then can't find a way to click the 'submit' button. I've tried find_element_by_partial_link_text("Subm").click() and I've tried find_element_by_class_name on a number of class names. Nothing works. Any ideas?
import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
main_url="http://data.stats.gov.cn/english/easyquery.htm?cn=E0101"
driver = webdriver.Firefox()
driver.get(main_url)
time.sleep(8)
driver.find_element_by_partial_link_text("Industry").click()
time.sleep(8)
driver.find_element_by_partial_link_text("Main Economic Indicat").click()
time.sleep(8)
driver.find_element_by_id("mySelect_sj").click()
time.sleep(3)
driver.find_element_by_class_name("dtText").send_keys("last72")
time.sleep(4)
try:
driver.find_element_by_class_name("dtFoot").click()
except:
driver.find_element_by_class_name("dtFoot").submit()
Solved my own problem, the key was using
driver.find_element_by_class_name(`dtTextBtn`)
instead of
driver.find_element_by_class_name(`dtTextBtn f10`)
The latter was what I saw in the source code, but the f10 blocked selenium.
All I am trying to do is: go to "http://news.google.com" and click on the "Technology" link on the side menu. Here's my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import unittest, time, re
class GoogleTech(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.base_url = "https://www.google.com/"
self.verificationErrors = []
self.accept_next_alert = True
def test_googletech(self):
driver = self.driver
driver.get("http://news.google.com")
#driver.find_element_by_xpath("//div[#id='main-wrapper']/div[#id='main-pane']/div[#class='background']/div[#class='centered']/div[#id='nav-menu-wrapper']/div[#class='browse-sidebar']/ul[#id='anchorman-two-browse-nav']/li[#class='nav-item nv-en_us:tc']/a[contains(text(),'Technology')]").click()
#driver.find_element_by_xpath("//ul[#id='anchorman-two-browse-nav']/li/a[contains(text(),'Technology')]").click()
#driver.find_element_by_xpath("//ul[#id='anchorman-two-browse-nav']/li[#class='nav-item nv-en_us:tc']/a[contains(text(),'Technology')]").click()
#driver.find_element_by_xpath("html/body/div[3]/div[1]/div/div/div[2]/div/ul/li[6]/a").click()
#driver.find_element_by_link_text("Technology").click()
wait = WebDriverWait(driver, 10)
link = wait.until(EC.presence_of_element_located((By.XPATH, "//ul[#id='anchorman-two-browse-nav']/li/a[contains(text(),'Technology')]")))
link.click()
time.sleep(10)
...
I've tried many combinations (all the commented out lines plus more!), but still cannot get it to work. I've check to make sure that the element I am using does exists (using Element Inspector) before entering it into the code. But even the "find_element_by_link_text" is giving me a NoSuchElementException exception. Can someone please tell me what I am doing wrong?
UPDATE: After doing further tests, I now have a better understanding of when this error is occuring for the news.google.com page. So, basically, the side menu, that contains the "Technology" link, is set to scrollable (I believe this error is only happening because the link is in scrollable area). The error is show up if the browser (that the Selenium script opens up when you run it), does not show the "Technology" link ie. You have to maximize the browser or scroll down to see the link. You can test this error out yourself - when the test opens up the browser, quickly resize the browser window so that the "Technology" link is not showing, and the test will fail. If your initial browser window doesn't show "Technology" link when it first opens, it will fail unless you maximize the browser window or scroll down really quickly until the "Technology" link is displayed. The error can take two forms - if you are using the wait.until method, you will get a Timeoutexception. If you are not using the wait.until method, you will see the ElementNotVisibleException. I've tested in both Chrome and Firefox and for both, I am seeing this problem. I am new to Selenium, but I believe this is not normal behavior, can someone please confirm? If this normal, then can someone please tell me how I can make sure my test runs correctly each time?
I think you want to use element_to_be_clickable as your ExpectedCondition.
That means that your driver will keep polling Selenium until it finds that the element is visible and enabled source
Try the below solution
link = wait.until(EC.element_to_be_clickable((By.XPATH, "//ul[#id='anchorman-two-browse-nav']/li/a[contains(text(),'Technology')]")))
link.click()
time.sleep(10)