We Keep Coding

Selenium find_element_by_xpath not working for instagram

I am trying to web scrape Instagram using Python and Selenium. I have had many issues regarding locating the elements but somehow managed to pull through when I tried enough xpaths. But when I try to web scrape Donald Trump's following list (I want this to work for ANY USER'S following list/page), it just doesn't work. Here is the error it's throwing:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="f3b066159b38864"]/div/div/a"}
I get the xpaths by right clicking on the element using Google Chrome's inspect feature. If anyone needs me to post the full code I'd be happy to do so.

Try below xpath::
wait = WebDriverWait(driver, 20)
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(text(),'laraleatrump')]")))
Note : please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
Working solution :
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
driver.get("https://www.instagram.com/realdonaldtrump/")
driver.maximize_window()
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(.,'following')]"))).click()
peoples = wait.until(
EC.visibility_of_all_elements_located((By.XPATH, "//div[#role='dialog']//div[contains(#class,'PZuss')]//a")))
for peoplename in peoples:
print peoplename.text

unable to select an element using xpath using selenium

I'm trying to automate a process using selenium and have been able to open a webpage and click on links, however I stumbled upon a table in which a link needs to be clicked but I'm unable to select that link and getting an error. need help to select that particular element
right now this is what I've done
elem2=browser.find_elements_by_xpath('/html/body/div[3]/table/tbody/tr[1]/td[2]/div[2]/table/tbody/tr[7]/td[3]/a::text')
elem2.click()
you can see in the picture that I'm trying to access the findhtml.org link.
the error that I get is
InvalidSelectorException: Message: invalid selector: Unable to locate an element with the xpath expression /html/body/div[3]/table/tbody/tr[1]/td[2]/div[2]/table/tbody/tr[7]/td[3]/a::text because of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string '/html/body/div[3]/table/tbody/tr[1]/td[2]/div[2]/table/tbody/tr[7]/td[3]/a::text' is not a valid XPath expression.
(Session info: chrome=81.0.4044.113)

First you have to switch to the iframe
Example:
frame = browser.find_elements_by_xpath('//iframe[contains(#src, \'hbx.media.net\')]')
browser.switch_to.frame(frame)
Now you can click
link = browser.find_elements_by_xpath('//a[contains(#href, \'http://www.findhtml.org\')]')
link.click()

To click on the specific link try below code.
Induce WebDriverWait() and presence_of_element_located() and following xpath.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("https://publicrecords.netronline.com/state/IL/county/dupage")
element=WebDriverWait(driver,10).until(EC.presence_of_element_located((By.XPATH,"//a[#href='http://www.dupageco.org/PropertyInfo/PropertyLookUp.aspx' and contains(.,'Go to Data')]")))
element.location_once_scrolled_into_view
element.click()
Please note the element is not inside any iframe

browser.get('https://publicrecords.netronline.com/state/IL/county/dupage')
wait = WebDriverWait(browser, 20)
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[contains(text(),'DuPage Supervisor of Assessments')]//following-sibling::td[2]//a"))).click()
output:
Note : please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

Executing event and scrape using selenium

I am pretty new in selenium and I was looking forward to execute a click() button that expands a table, and after that I was looking forward to scrape the content of the expanded table.
This is what I tried:
url = 'https://www.telegraph.co.uk/markets-hub/assets/shares/'
phantomjs_path = '/usr/local/bin/phantomjs'
driver = webdriver.PhantomJS(executable_path=phantomjs_path,service_args=['--load-images=no'])
driver.get(url)
element = driver.find_element_by_xpath("//div[#class='tmg-show-more']")
html_source = driver.page_source
element.click()
driver.quit()
soup = BeautifulSoup(html_source,"html5lib")
tables=soup.find('table',class_="table-static kurser-table")
link=pd.read_html(str(tables),flavor='html5lib',thousands='.',decimal=',',header=0)
print(link)
The current output is as it follows:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: {"errorMessage":"Element is not currently visible and may not be manipulated",
"request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"81","Content-
Type":"application/json;charset=UTF-8","Host":"127.0.0.1:58447","User-Agent":"Python http auth"},
"httpVersion":"1.1","method":"POST","post":"{\"id\": \":wdc:1556893378248\", \"sessionId\": \"f223cb80-6dae-11e9-b00b-4bb158952171\"}",
"url":"/click","urlParsed":{"anchor":"","query":"","file":"click","directory":"/","path":"/click","relative":"/click","port":"","host":"","password":"","user":"",
"userInfo":"","authority":"","protocol":"","source":"/click","queryKey":{},"chunks":["click"]},"urlOriginal":"/session/f223cb80-6dae-11e9-b00b-4bb158952171/element/:wdc:1556893378248/click"}}
Screenshot: available via screen

You would need to scroll down to let the selenium web driver knows where the element is actually present.
Code :
driver.get("https://www.telegraph.co.uk/markets-hub/assets/shares/")
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.section-box.kurser-table-container')))
element = driver.find_element_by_xpath("//a[#ng-click='stocksShowMore()']")
actions = ActionChains(driver)
actions.move_to_element(element).perform()
element.click()
Imports :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

You are getting ElementNotVisibleException means you are trying to click the element before it is visible.
Also, you are getting the page source before clicking the button. You need to get the source after clicking and loading the new data.
Try the following code before getting the page source.
wait = WebDriverWait(driver, 20)
element = wait.until(EC.element_to_be_clickable(driver.find_element_by_xpath("//div[#class='tmg-show-more']")))
element.click()
wait.until(EC.invisibility_of_element(element))
To use the WebDriverWait you need to import these
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Python 2.7 Selenium No Such Element on Website

I'm trying to do some webscraping from a betting website:
As part of the process, I have to click on the different buttons under the "Favourites" section on the left side to select different competitions.
Let's take the ENG Premier League button as example. I identified the button as:
(source: 666kb.com)
The XPath is: //*[#id="SportMenuF"]/div[3] and the ID is 91.
My code for clicking on the button is as follows:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = "C:\Python27\Scripts\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("URL Removed")
content = driver.find_element_by_xpath('//*[#id="SportMenuF"]/div[3]')
content.click()
Unfortunately, I always get this error message when I run the script:
"no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="SportMenuF"]/div[3]"}"
I have tried different identifiers such as CCS Selector, ID and, as shown in the example above, the Xpath. I tried using waits and explicit conditions, too. None of this has worked.
I also attempted scraping some values from the website without any success:
from selenium import webdriver
from selenium.webdriver.common.by import By
chrome_path = "C:\Python27\Scripts\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("URL removed")
content = driver.find_elements_by_class_name('price-val')
for entry in content:
print entry.text
Same problem, nothing shows up.
The website embeddes an iframe from a different website. Could this be the cause of my problems? I tried scraping directly from the iframe URL, too, which didn't work, either.
I would appreciate any suggestions.

Sometimes elements are either hiding behind an iframe, or they haven't loaded yet
For the iframe check, try:
driver.switch_to.frame(0)
For the wait check, try:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '-put the x-path here-')))

Selenium + Python + onclick + NoSuchElementException

I have an "onclick" element in a webpage whose HTML reads as follows:
Text 1
I am trying to use selenium to click on this element. I tried the following code (including an implicit wait)
import time
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(webpage);
driver.implicitly_wait(5)
driver.find_element_by_xpath("//a[#onclick='blah.submit()']").click()
However, it returned a NoSuchElementException
NoSuchElementException: Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//a[#onclick='blah.submit()']"}
I then tried using WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get(webpage);
button = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH,
"//a[#onclick='blah.submit()']")))
button.click()
However, this still didn't work and returned the following error:
TimeoutException: Message:
Does anyone know what's going wrong? Thanks in advance.