We Keep Coding

How to loop through indeed job pages using selenium

I am trying to make a selenium python script to collect data from each job in an indeed job search. I can easily get the data from the first and second page. The problem I am running into is while looping through the pages, the script only clicks the next page and the previous page, in that order. Going from page 1 -> 2 -> 1 -> 2 -> ect. I know it is doing this because both the next and previous button have the same class name. So when I redeclare the webelement variable when the page uploads, it hits the previous button because that is the first location of the class in the stack. I tried making it always click the next button by using the xpath, but I still run into the same errors. I would inspect the next button element, and copy the full xpath. my code is below, I am using python 3.7.9 and pip version 21.2.4
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
HTTPS = "https://"
# hard coded data to test
siteDomain = "indeed.com"
jobSearch = "Software Developer"
locationSearch = "Richmond, VA"
listOfJobs = []
def if_exists_by_id(id):
try:
driver.find_element_by_id(id)
except NoSuchElementException:
return False
return True
def if_exists_by_class_name(class_name):
try:
driver.find_element_by_class_name(class_name)
except NoSuchElementException:
return False
return True
def if_exists_by_xpath(xpath):
try:
driver.find_element_by_xpath(xpath)
except NoSuchElementException:
return False
return True
def removeSpaces(strArray):
newjobCounter = 0
jobCounter = 0
for i, word in enumerate(strArray):
jobCounter += 1
if strArray[i].__contains__("\n"):
strArray[i] = strArray[i].replace("\n", " ")
if strArray[i].__contains__("new"):
newjobCounter += 1
print(strArray[i] + "\n")
if newjobCounter == 0:
print("Unfortunately, there are no new jobs for this search")
else:
print("With " + str(newjobCounter) + " out of " + str(jobCounter) + " new jobs!")
return strArray
try:
# Goes to Site
driver.get(HTTPS + siteDomain)
# obtains access to elements from website
searchJob = driver.find_element_by_name("q")
searchLocation = driver.find_element_by_name("l")
# clear text field
searchJob.send_keys(Keys.CONTROL, "a", Keys.BACK_SPACE)
searchLocation.send_keys(Keys.CONTROL, "a", Keys.BACK_SPACE)
# inputs values into website elements
searchJob.send_keys(jobSearch)
searchLocation.send_keys(locationSearch)
# presses button to search
searchLocation.send_keys(Keys.RETURN)
# Begin looping through pages
pageList = driver.find_element_by_class_name("pagination")
page = pageList.find_elements_by_tag_name("li")
numPages = 0
for i,x in enumerate(page):
time.sleep(1)
# checks for popup, if there is popup, exit out and sleep
if if_exists_by_id("popover-x"):
driver.find_element_by_id("popover-x").click()
time.sleep(1)
# increment page counter variabke
numPages += 1
# obtains data in class name value
jobCards = driver.find_elements_by_class_name("jobCard_mainContent")
# prints number of jobs returned
print(str(len(jobCards)) + " jobs in: " + locationSearch)
# inserts each job into list of jobs array
# commented out to make debugging easier
# for jobCard in jobCards:
# listOfJobs.append(jobCard.text)
# supposed to click the next page, but keeps alternating
# between next page and previous page
driver.find_element_by_class_name("np").click()
print("On page number: " + str(numPages))
# print(removeSpaces(listOfJobs))
except ValueError:
print(ValueError)
finally:
driver.quit()
Any help will be greatly appreciated, also if I am implementing bad coding practices in the structure of the script please let me know as I am trying to learn as much as possible! :)

I have tested your code.. the thing is there are 2 'np' class elements when we go to the 2nd page.. what you can do is for first time use find_element_by_class_name('np') and for all the other time use find_elements_by_class_name('np')[1] that will select the next button.. and you can use find_elements_by_class_name('np')[0] for the previous button if needed. Here is the code!
if i == 0:
driver.find_element_by_class_name("np").click()
else:
driver.find_elements_by_class_name("np")[1].click()
Just replace the line driver.find_element_by_class_name("np").click() with the code snippet above.. I have tested it and it worked like a charm.
Also i am not as experienced as the other devs here.. But i am glad if i could help you. (This is my first answer ever on stackoverflow)

looping through several columns and rows from csv to fill a form

Have been trying to emulate examples posted earlier, yet got stuck.
I have a simple web form: Last name, name, email, password, confirm password.
Also a .csv with 4 columns that corresponds to the form
Last name Name Email Password
0 Brown Stan brown#stan.com 12345678
1 White Eagle white#eagle.com 123456789
2 Dante Aligr adant#mail.au 98765432
So, all I want is to feed the 3 entries to the form and click "Sent" after each entry.
I copycated a code from here that seemed passing but I keep getting this
File "C:\Users\untitled2.py", line 43, in <module>
last.send_keys(lname)
AttributeError: 'list' object has no attribute 'send_keys'
the code I tried
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
options = Options()
options.binary_location = FirefoxBinary(r"C:\Program Files\Mozilla Firefox\firefox.exe")
driver = webdriver.Firefox(executable_path=r'C:\WebDriver\bin\geckodriver.exe', firefox_options=options)
import time
import pandas as pd
reg = pd.read_csv('Form.csv', header=0, delimiter=';', sep=r'\s*;\s*')
print(reg)
lname = reg['Last name'].tolist()
name = reg['Name'].tolist()
mail = reg['Email'].tolist()
password = reg['Password'].tolist()
password_c = reg['Password'].tolist()
driver.get('url')
first = driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div/div[2]/div[2]/div/div/div[2]/div/div[1]/div/div[1]/input")
last = driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input")
mail = driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div/div[2]/div[3]/div/div/div[2]/div/div[1]/div/div[1]/input")
password = driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div/div[2]/div[4]/div/div/div[2]/div/div[1]/div/div[1]/input")
password_confirm = driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div/div[2]/div[4]/div/div/div[2]/div/div[1]/div/div[1]/input")
submit = driver.find_elements_by_xpath('/html/body/div/div[3]/form/div[2]/div/div[3]/div[1]/div/div/span/span')
for lname, name, mail, password, password_c in zip(lname, name, mail, password, password_c):
last.send_keys(lname)
time.sleep(1)
first.send_keys(name)
time.sleep(1)
mail.send_keys(mail)
time.sleep(1)
password.send_keys(password)
time.sleep(1)
password_confirm.send_keys(password_c)
time.sleep(1)
submit.click()
time.sleep(3)
Any nudge into the right direction will be highly appreciated since I have seen plenty of examples of using lists with send_keys()
Thanks!

The error message indicates that you are using send_keys() with plain python lists.
According to Selenium docs, find_elements_by_xpath does indeed return a list.
It's possible that you meant to use find_element_by_xpath (without the 's' after element)?

Anyhow,
the execution part should have look like this
for lnames in lname:
count = 0
for names in name:
count_name = 0
for mails in mail_:
count_mail = 0
for value in last:
value.send_keys(lname[count])
count +=1
for x in mail:
x.send_keys(mail_[count_mail])
count_mail +=1
for y in first:
y.send_keys(name[count_name])
count_name +=1
submit = driver.find_element_by_xpath('//*[#id="mG61Hd"]/div[2]/div/div[3]/div[1]/div/div/span/span')
submit.click()

Python Selenium accepting the "Before you continue" form on google maps in GoogleChrome

Yesterday I started to change my code a little.
One part apparently was very tricky.
Its only use was to click on the continue button so the next click on the car symbol won't raise an error for. As I learned the error is raised because the form is in front of the button. (Apparently it sometimes works nevertheless)
This code snipped worked perfectly,
but changing it resulted in some errors I didn't see coming.
This code should work for itself if anyone wants to test it.
The variables *_strasse, (street name '+' separated)
*_hausnummer, (House number)
*_plz, (Post Code)
*_stadt (city) worked for me.
try:
#Getting the HTML
link = f"https://www.google.de/maps/dir/{start_strasse}+{start_hausnummer},+{start_plz}+{start_stadt},+Deutschland/{end_strasse}+{end_hausnummer},+D-{end_plz}+{end_stadt},+Deutschland"
driver.get(link)
driver.implicitly_wait(6)
#Wait for the button to appear
try:
elem = driver.find_element_by_xpath('//*[#id="introAgreeButton"]').click()
except :
continue
#Find the button for routes by car
list(filter(lambda x: x.get_attribute('jstcache') == '508', driver.find_elements_by_tag_name('button')))[0].click()
#Parse the fastest times
times = [ re.findall(r'[0-9]+', x.text[:-3]) for x in list(filter(lambda x: x.get_attribute('jstcache') == '265', driver.find_elements_by_tag_name('span')))]
print('Elemente:', times )
#Select the fastest time
length = times[0][0]
#Convert hh:mm format to minutes
length = int(length[0]) if len(length) == 1 else 60*int(length[0]) + int(length[1])
print(length)
except:
#used for debugging
#print(str(runde) + f'/{len(adresses.keys())}', link)
raise
My new code looks like this, but it's not able to find the continue button by XPATH.
try:
link = f"https://www.google.de/maps/dir/{start_strasse}+{start_hausnummer},+{start_plz}+{start_stadt},+Deutschland/{end_strasse}+{end_hausnummer},+D-{end_plz}+{end_stadt},+Deutschland"
driver.get(link)
driver.implicitly_wait(6)
elem = driver.find_element_by_xpath('//*[#id="introAgreeButton"]').click()
#element = driver.find_element_by_css('div[class*="U26fgb"]')
#driver.execute_script("arguments[0].click();", element)
#list(filter(lambda x: x.get_attribute('jstcache') == '508', driver.find_elements_by_tag_name('button')))[0].click()
driver.find_element_by_xpath("//button[#jstcache='508']").click()
#times = [ re.findall(r'[0-9]+', x.text[:-3]) for x in list(filter(lambda x: x.get_attribute('jstcache') == '265', driver.find_elements_by_tag_name('span')))]
#print('Elemente:', times )
#length = re.findall(r'[0-9]+', list(filter(lambda x: x.get_attribute('jstcache') == '265', driver.find_elements_by_tag_name('span')))[0].text[:-3])
#length = times[0][0]
elem = driver.find_element_by_xpath("//*[#jstcache='265']")
zeit = re.findall(r'[0-9]+', elem.text[:-3])
length = int(zeit[0]) if len(zeit) == 1 else 60*int(zeit[0]) + int(zeit[1])
verbindung = ','.join([start, end, str(length)]) + '\n'
datei.write(verbindung)
print(verbindung)
except:
print(link)
raise
I tested countless ideas from the web including invoking some java script functions instead of clicking, switching to an iframe, or to the active element or the active alert.
I think one of the ideas was actually right but I implemented it wrong.
Apparently implicit waits worked way better for me than any wait until what I found strange.
So I thought that my code could already have the problem inside.
I appreciate all help and comments!
It's my first post here so if I missed something pls tell me!
How the Browser looks like
The HTML of the button

Selenium Python driver.find.elements get attribute

def n_seguidores(self, username):
driver = self.driver
driver.get('https://www.instagram.com/'+ username +'/')
time.sleep(3)
user_botao = driver.find_elements_by_class_name('g47SY ')
print_us = user_botao.get_attribute('title')
print(print_us)
please help me to find numbers of following from html

.find_elements_* return a list, so you need access by index.
There are 3 numbers with the same class name in the page, and the numbers of following you mean is the third.
And to get the number you can use .text, not .get_attribute('title')
Try following code:
user_botao = driver.find_elements_by_class_name('g47SY ')
#second index
print_us = user_botao[2].text
print(print_us)

If Statment in Python while Scraping

I want to add if statement there:
if website_link and phone is not found than write N,N
This Script is typing if website link is not found it type N and if phone is not found it type N
I Want Also Add This:
If Website And Phone is not found Type N,N
Here is my Code:
from selenium import webdriver
import csv
import pandas
import itertools
with open("sans.csv",'r') as s:
s.read()
driver = webdriver.Firefox()
url = 'https://www.yelp.com/biz/konomama-san-francisco?osq=Restaurants'
driver.get(url)
website_link = driver.find_elements_by_css_selector('.text--offscreen__373c0__1SeFX+ .link-size--default__373c0__1skgq')
phone = driver.find_elements_by_css_selector('.text--offscreen__373c0__1SeFX+ .text-align--left__373c0__2pnx_')
items = len(website_link)
with open("sans.csv", 'a',encoding="utf-8") as s:
for combination in itertools.zip_longest(website_link, phone):
s.write(f'{combination[0].text if combination[0] else "N"}, {combination[1].text if combination[1] else "N"}\n')
driver.close()
print("Done")
Thanks!

You can add else to the for loop. It will run in case the loop doesn't when website_link and phone are empty
with open("sans.csv", 'a', encoding="utf-8") as s:
for combination in itertools.zip_longest(website_link, phone):
s.write(f'{combination[0].text if combination[0] else "N"}, {combination[1].text if combination[1] else "N"}\n')
else:
s.write('N, N')
For more info about for - else structure see this answer.

may be he wants the iterator to iterate and check for None in website_link and phone
with open("sans.csv", 'a',encoding="utf-8") as s:
for combination in itertools.zip_longest(website_link, phone):
s.write(f'{combination[0].text if combination[0] else "N"}, {combination[1].text if combination[1] else "N"}\n')
if(website_link == None) and (Phone == None):
s.write('N,N')