We Keep Coding

Unable to identify what to 'click' for next page using selenium

I am trying to get search results from yahoo search using python - selenium and bs4. I have been able to get the links successfuly but I am not able to click the button at the bottom to go to the next page. I tried one way, but it could't identify after the second page.
Here is the link:
https://in.search.yahoo.com/search;_ylt=AwrwSY6ratRgKEcA0Bm6HAx.;_ylc=X1MDMjExNDcyMzAwMgRfcgMyBGZyAwRmcjIDc2ItdG9wLXNlYXJjaARncHJpZANidkhMeWFsMlJuLnZFX1ZVRk15LlBBBG5fcnNsdAMwBG5fc3VnZwMxMARvcmlnaW4DaW4uc2VhcmNoLnlhaG9vLmNvbQRwb3MDMARwcXN0cgMEcHFzdHJsAzAEcXN0cmwDMTQEcXVlcnkDc3RhY2slMjBvdmVyZmxvdwR0X3N0bXADMTYyNDUzMzY3OA--?p=stack+overflow&fr=sfp&iscqry=&fr2=sb-top-search
This is what im doing to get data from page but need to put in a loop which changes pages:
page = BeautifulSoup(driver.page_source, 'lxml')
lnks = page.find('div', {'id': 'web'}).find_all('a', href = True)
for i in lnks:
print(i['href'])

You don't need to scroll down to the bottom. The next button is accessible without scrolling. Suppose you want to navigate 10 pages. The python script can be like this:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
driver=webdriver.Chrome()
driver.get('Yahoo Search URL')
# Let's create a loop containing the XPath for next button
# As well as waiting for the next button to be clickable.
for i in range(10):
WebDriverWait(driver, 5).until(EC.element_to_be_clickable(By.XPATH, '//a[#class="next"]'))
navigate = driver.find_element_by_xpath('//a[#class="next"]').click()

The next page button is on the bottom of the page so you first need to scroll to that element and then click it. Like this:
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
next_page_btn = driver.find_element_by_css_selector("a.next")
actions.move_to_element(next_page_btn).build().perform()
time.sleep(0.5)
next_page_btn.click()

How to handle Google Privacy Popup in selenium (python)

I'm pretty new to selenium and i searched a lot, but can't find an answer to my problem.
I want to open firefox, go on google and search for something. Store everthing in a list and print it to my console.
But everytime i open firefox with selenium, a pop-up windows opens with a confirmation to the privacy regulations of google. I dont know how to close it.
Here is what i tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
# create a new Firefox session
driver = webdriver.Firefox()
driver.implicitly_wait(30)
driver.maximize_window()
# Navigate to the application home page
driver.get("http://www.google.com")
main_page = driver.current_window_handle
time.sleep(3)
#Tried to find out in which browers window I am right now -> terminal says '19'
print('The actual browers window is: {}'.format(main_page))
driver.find_element_by_xpath("//*[#id='introAgreeButton']/span/span").click()
# get the search textbox
search_field = driver.find_element_by_id("lst-ib")
search_field.clear()
# enter search keyword and submit
search_field.send_keys("Flowers")
search_field.submit()
# get the list of elements which are displayed after the search
# currently on result page using find_elements_by_class_name method
lists= driver.find_elements_by_class_name("_Rm")
# get the number of elements found
print ("Found " + str(len(lists)) + " searches:")
# iterate through each element and print the text that is
# name of the search
i=0
for listitem in lists:
print (listitem.get_attribute("innerHTML"))
i=i+1
if(i>10):
break
# close the browser window
driver.quit()

Google privacy popup is contained in iframe. You must switch to iframe and then look for accept (or deny) button. Like so
wait = WebDriverWait(driver, 5)
iframe = wait.until(EC.element_to_be_clickable([By.CSS_SELECTOR, '#cnsw > iframe']))
driver.switch_to.frame(iframe)
And then when in iframe, look for Accept button a click it.

How To Loop Through Multiple Pages And Open Links At The Same Time

I'm currently trying to figure out how to loop through a set of studios on a fitness class website.
On the search results page of this website, it lists 50 studios on each page and there are about 26 pages. https://classpass.com/search if you want to take a look.
My code parses the search result page, and selenium gets the link for each studio on the page(In my full code selenium opens goes to the link and scrapes data on the page).
After looping through all the results on page 1, I want to click the next page button and repeat on results page 2. I get the error Message: no such element: Unable to locate element: but I know the element is definitely on the results page and can be clicked. I tested this with a simplified script to confirm.
What could I be doing wrong? I've tried many suggestions but none have worked so far.
from selenium import webdriver
from bs4 import BeautifulSoup as soup
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as browser_wait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
import time
import re
import csv
# initialize the chrome browser
browser = webdriver.Chrome(executable_path=r'./chromedriver')
# URL
class_pass_url = 'https://www.classpass.com'
# Create file and writes the first row, added encoding type as write was giving errors
#f = open('ClassPass.csv', 'w', encoding='utf-8')
#headers = 'URL, Studio, Class Name, Description, Image, Address, Phone, Website, instagram, facebook, twitter\n'
#f.write(headers)
# classpass results page
page = "https://classpass.com/search"
browser.get(page)
# Browser waits
browser_wait(browser, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, "line")))
# Scrolls to bottom of page to reveal all classes
# browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# Extract page source and parse
search_source = browser.page_source
search_soup = soup(search_source, "html.parser")
pageCounter = 0
maxpagecount = 27
# Looks through results and gets link to class page
studios = search_soup.findAll('li', {'class': '_3vk1F9nlSJQIGcIG420bsK'})
while (pageCounter < maxpagecount):
search_source = browser.page_source
search_soup = soup(search_source, "html.parser")
studios = search_soup.findAll('li', {'class': '_3vk1F9nlSJQIGcIG420bsK'})
for studio in studios:
studio_link = class_pass_url + studio.a['href']
browser.get(studio_link)
browser_wait(browser, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, "line")))
element = browser.find_element_by_xpath('//*[#id="Search_Results"]/div[1]/div/div/nav/button[2]')
browser.execute_script("arguments[0].click();", element)

You have to return to the main page before finding the next page button. You could solve the problem by the replacing the following code. This code will initially collect all page's studio url.
studios = search_soup.findAll('li', {'class': '_3vk1F9nlSJQIGcIG420bsK'})
to
studios = []
for page in range(num_pages):
studios.append(search_soup.findAll('li', {'class': '_3vk1F9nlSJQIGcIG420bsK'}))
element = browser.find_element_by_xpath('//*[#id="Search_Results"]/div[1]/div/div/nav/button[2]')
browser.execute_script("arguments[0].click();", element)
and remove the code clicking the next page button element.

Selenium click() method on angular.js site

I am scraping an angular.js site. My initial link has a search button. I find by xpath and click with no issues. After I click search, I want to be able to click each of the athletes in the table to go to their info pages, but I am not having success with the click method. The links are attached to their names.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
TIMEOUT = 5
driver = webdriver.Firefox()
driver.set_page_load_timeout(TIMEOUT)
url = 'https://n.rivals.com/search#?formValues=%7B%22sport%22:%22Football%22,%22recruit_year%22:2021,%22offer_and_visit_type%22:%5B%22Offer%22%5D,%22prospect_profiles.prospect_colleges.offer%22:true,%22page_number%22:1,%22page_size%22:50%7D'
try:
driver.get(url)
except TimeoutException:
pass
search_button = driver.find_element_by_xpath('//*[#id="articles"]/div/div[2]/div/div/div[1]/form/div[2]/div[5]/button')
search_button.click();
#below is where I tried, but could not get to click
first_athlete = driver.find_element_by_xpath('//*[#id="content_"]/td[1]/div[2]/a')
first_athlete.click();

Works if you remove the last /a in the xpath:
first_athlete = driver.find_element_by_xpath('//*[#id="content_"]/td[1]/div[2]')
first_athlete.click()

If you want to search for all athletes and you have the name of athletes with you, you can use CSS selector as well.
athelete = driver.find_elements_by_css_selector(`#content_ > td > div > a[href *="donovan-jackson"]);
athelete.click();
This code will give you a unique web element for each player.
Thanks

Select an input element using Selenium

Inspect
Im trying to click on this button to move to the login page.
my code is :
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('http://moodle.tau.ac.il/')
thats work fine but i can only find the form by using
loginform = driver.find_element_by_xpath("//form[#id='login']/")
I don't know how to get to the button, it's very basic stuff but I didn't find any good example.

This will click on the login button on moodle.tau.ac.il page.
The line driver.find_element_by_xpath(".//*[#id='login']/div/input").click() finds the login button on the page and clicks it. Xpath is just a selector type that you can use with selenium to find web elements on a page. You can also use ID, classname, and CSSselectors.
from selenium import webdriver
driver = new webdriver.Chrome()
driver.get('moodle.tau.ac.il')
# This will take you to the login page.
driver.find_element_by_xpath(".//*[#id='login']/div/input").click()
# Fills out the login page
elem = driver.find_element_by_xpath("html/body/form/table/tbody/tr[2]/td/table/tbody/tr[1]/td[2]/input")
elem.send_keys('Your Username')
elem = driver.find_element_by_xpath("html/body/form/table/tbody/tr[2]/td/table/tbody/tr[3]/td[2]/input")
elem.send_keys('Your ID Number')
elem = driver.find_element_by_xpath("html/body/form/table/tbody/tr[2]/td/table/tbody/tr[1]/td[2]/input")
elem.send_keys('Your Password')
driver.find_element_by_xpath("html/body/form/table/tbody/tr[2]/td/table/tbody/tr[7]/td[2]/input").click()

The page has two identical login forms and your XPath returns the hidden one.
So with the visible one:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(r"http://moodle.tau.ac.il/")
driver.find_element_by_css_selector("#page-content #login input[type=submit]").click()
Or with an XPath:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get(r"http://moodle.tau.ac.il/")
driver.find_element_by_xpath("id('page-content')//form[#id='login']//input[#type='submit']").click()

You could find it using XPath as mentioned by #ChrisP
You could find it by CSS selector: "#login input[type='text']"
Or you could also just submit the form... loginForm.submit()
Ideally, you'd have a unique id for that button which would make it very easy to find.