We Keep Coding

Getting error while fetching details with web scraping in python

I am getting Error while scraping data from a site please if anyone could help me with that
my Code
html = requests.get('https://www.cryptocompare.com/coins/btc/influence/USDT').text
soup = BeautifulSoup(html, 'html.parser')
total_commit = soup.select_one(' # col-body > div > social-influence > div.row.row-zero.influence-others > div:nth-child(2) > div > div > div > div.col-md-3.td-col.brd-right > div > div.repo-tag > span > span > a').text
print(total_commit)
error
soupsieve.util.SelectorSyntaxError: Malformed id selector at position 2
line 1:
# col-body > div > social-influence > div.row.row-zero.influence-others > div:nth-child(2) > div > div > div > div.col-md-3.td-col.brd-right > div > div.repo-tag > span > span > a
^
and also if anyone can tell me how to use the Css selectors which we copy directly from inspect element in bs4.

As mentioned by David Miró removing whitespace will fix the error but to get your goal you have to deal with selenium
Selenium will render the website and you can inspect the page_source and select your Element with bs4:
soup.select_one('div.repo-tag a')['href']
Example
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome('YOUR PATH TO DRIVER')
driver.get('https://www.cryptocompare.com/coins/btc/influence/USDT')
soup=BeautifulSoup(driver.page_source, 'html.parser')
soup.select_one('div.repo-tag a')['href']
Output
https://github.com/bitcoin/bitcoin

Try removing space between # and col-body.
html = requests.get('https://www.cryptocompare.com/coins/btc/influence/USDT').text
soup = BeautifulSoup(html, 'html.parser')
total_commit = soup.select_one('#col-body > div > social-influence > div.row.row-zero.influence-others > div:nth-child(2) > div > div > div > div.col-md-3.td-col.brd-right > div > div.repo-tag > span > span > a').text
print(total_commit)
But it doesn't work because a part of the html is generated by javascript. So, you need to simulate that you are a web browser (for example with Selenium):
<div class="col-body col-body-new" id="col-body" ui-view>
<div class="loader-ccc">
<div class="loader-ccc-logo"></div>
<div class="loader-ccc-sides"></div>
</div>
In the web browser information exists:

Selenium - switch to div class that is a window

I need help selecting an element on a webpage with Selenium. I have been using Selenium on this website for about 3 weeks and so far, I can usually find an element by css selector or XPath. However, this specific section of the website is giving me a very hard time.
After I click on “reset office 365 password” a window comes up and I want to programmably put in the new password but it can’t find anything in the popup window.
Here is what the page looks like:
(I am too low of score to post pictures here) https://cdn.discordapp.com/attachments/768594779344470022/845811910577881098/unknown.png
Here is the whole element’s information:
<input type="password" tabindex="1" name="password" class="m-third pass ng-pristine ng-empty ng-invalid ng-invalid-required ng-touched" ng-model="password.value" ng-blur="password.check = false" ng-focus="password.check = true" required="" autofocus="" ng-disabled="!active">
Here is what I tried: (I tried a lot of things)
Tried clicking on the password box by using css selector – failed: Invalid selector
im_blacklistaddbutton = browser_options.browser.find_element_by_css_selector('#ng-app > div.page-container > div > div > div.vertical-tabs.j-vertical-tabs.ng-scope > div.vertical-tabs-panes.p0 > div > div > div.page-content.ng-scope > div > div > form > div > div > div.ng-isolate-scope > div.modal > div.modal-body.ng-transclude > div > reset:password > ng-form > div:nth-child(1) > div > div.validation-input > input')
im_blacklistaddbutton.send_keys(email_pd.pd)
selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified
Tried clicking on the password box by using xpath selector – failed: Namespace Error
im_blacklistaddbutton = browser_options.browser.find_element_by_xpath('//*[#id="ng-app"]/div[2]/div/div/div[3]/div[2]/div/div/div[2]/div/div/form/div/div/div[3]/div[1]/div[2]/div/reset:password/ng-form/div[1]/div/div[1]/input')
im_blacklistaddbutton.send_keys(email_pd.pd)
NamespaceError: Failed to execute 'evaluate' on 'Document': The string '//*[#id="ng-app"]/div[2]/div/div/div[3]/div[2]/div/div/div[2]/div/div/form/div/div/div[3]/div[1]/div[2]/div/reset:password/ng-form/div[1]/div/div[1]/input' contains unresolvable namespaces.
Tried waiting for the element by partial link text: It timed out
wait.until(EC.visibility_of_element_located((By.PARTIAL_LINK_TEXT, 'Generate password')))
selenium.common.exceptions.TimeoutException: Message:
Tried waiting for the element by ID name text value: It timed out
wait.until(EC.text_to_be_present_in_element((By.CLASS_NAME, 'btn m-link'), "Generate Password"))
selenium.common.exceptions.TimeoutException: Message:
Tried to switch to a window or iframe but it said that the div class of "model" is not a window or an iframe.
From here I am completely lost as to why this stupid window is not accessible. Text window - why are you so mean to me?
Here is my specific function in total:
def reset_im_oa_password():
browser_options.browser.get('https://cpx.intermedia.net/ControlPanel/Menu/AccountMenu/?frameUrl=https://cpx.intermedia.net/aspx/Office365/Home/licenses#/installed/users')
wait = WebDriverWait(browser_options.browser, 10)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, 'player')))
except exceptions.TimeoutException as e:
pass
browser_options.browser.switch_to_frame('mainFrame')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#ng-app > div.page-container > div > div > div.vertical-tabs.j-vertical-tabs.ng-scope > div.vertical-tabs-panes.p0 > div > div > div.page-content.ng-scope > div > div > form > div > div > div:nth-child(2) > div.table-wrap.table-fixed.j-table-wrap.s-wide.ng-isolate-scope > div.table-filter > div.table-filter-search.searchbox.ng-isolate-scope > div > span:nth-child(3) > input')))
im_blacklistaddbutton = browser_options.browser.find_element_by_css_selector('#ng-app > div.page-container > div > div > div.vertical-tabs.j-vertical-tabs.ng-scope > div.vertical-tabs-panes.p0 > div > div > div.page-content.ng-scope > div > div > form > div > div > div:nth-child(2) > div.table-wrap.table-fixed.j-table-wrap.s-wide.ng-isolate-scope > div.table-filter > div.table-filter-search.searchbox.ng-isolate-scope > div > span:nth-child(3) > input')
im_blacklistaddbutton.send_keys(email_or_user_selection.email_select)
im_blacklistaddbutton = browser_options.browser.find_element_by_css_selector('#ng-app > div.page-container > div > div > div.vertical-tabs.j-vertical-tabs.ng-scope > div.vertical-tabs-panes.p0 > div > div > div.page-content.ng-scope > div > div > form > div > div > div:nth-child(2) > div.table-wrap.table-fixed.j-table-wrap.s-wide.ng-isolate-scope > div.table-filter > div.table-filter-search.searchbox.ng-isolate-scope > div > span:nth-child(3) > button')
im_blacklistaddbutton.send_keys(Keys.ENTER)
wait.until(EC.element_to_be_clickable((By.XPATH, ("//*[starts-with(#id, 'btnResetPassword')]"))))
im_blacklistaddbutton = browser_options.browser.find_element_by_xpath(("//*[starts-with(#id, 'btnResetPassword')]"))
im_blacklistaddbutton.send_keys(Keys.ENTER)
try:
wait.until(EC.visibility_of_element_located((By.PARTIAL_LINK_TEXT, 'Generate password')))
except exceptions.TimeoutException as e:
pass
browser_options.browser.switch_to_window('model') # anything past this section will fail
wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'model')))
im_blacklistaddbutton = browser_options.browser.find_element_by_xpath('//*[#id="ng-app"]/div[2]/div/div/div[3]/div[2]/div/div/div[2]/div/div/form/div/div/div[3]/div[1]/div[2]/div/reset:password/ng-form/div[1]/div/div[1]/input')
im_blacklistaddbutton.send_keys(email_pd.pd)
return
if anyone needs the full code from the webpage let me know. Thanks

If this element is not really inside an iframe as you write, then, wait for it to become clickable, like this:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[type='password']")))
im_blacklistaddbutton = browser.find_element_by_css_selector("input[type='password']")
im_blacklistaddbutton.send_keys("new_password")
But make sure that css selector input[type='password'] is unique.
If not, try this one: .validation-input>input[type='password']
(Check validation-input class name si correct as it is cut on your screenshot)
If the input frame is inside iframe nothing will work until you switch to this iframe.

Because of no webpage code, right now I can't say why the element is not detectable by Selenium but you can try one thing. Right click on the element(input tag in dom shown in picture) and go to "Copy to" option and select "Copy JS Path". Then go to console tab in dev tools and paste it. Then try to set it's value to some dummy text and see if it sets the password.
jsPath.value="some password" //this should set the password
If this works, then you can set the value by using JavaScript executor of Selenium in the same way.

I want to make a python script that scrapes (copies) all of the usernames from a person's following list

I tried to follow along with some youtube tutorials in order to make my code do what I want it to do, but I still haven't found any answer on the entire internet...
Here I tried to make the script using BeautifulSoup:
import bs4
import requests
resoult = requests.get("https://www.instagram.com/kyliejenner/following/")
src = resoult.content
Soup = bs4.BeautifulSoup(src, "lxml")
links = Soup.find_all("a")
print(links)
print("/n")
for link in links:
if "FPmhX notranslate _0imsa " in link.text:
print(link)
And here I tried to do the same thing with Selenium, but the problem is that I don't know the next steps in order to make my code copy the usernames a user is following
import selenium
from selenium import webdriver
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.instagram.com/")
time.sleep(2)
username = driver.find_element_by_css_selector ("#loginForm > div > div:nth-child(1) > div > label >
input")
username.send_keys ("my_username")
password = driver.find_element_by_css_selector ("#loginForm > div > div:nth-child(2) > div > label >
input")
password.send_keys("password")
loginButton = driver.find_element_by_css_selector ("#loginForm > div > div:nth-child(3)")
loginButton.click()
time.sleep(3)
saveinfoButton = driver.find_element_by_css_selector ("#react-root > section > main > div > div > div
>
section > div > button")
saveinfoButton.click()
time.sleep(3)
notnowButton = driver.find_element_by_css_selector("body > div.RnEpo.Yx5HN > div > div > div >
div.mt3GC
> button.aOOlW.HoLwm")
notnowButton.click()
I would really appreciate it if someone could solve this problem. Again, all that I want my script to do is to copy the usernames from the "following" section of someones profile.

Remember information about the previous page?

I'm trying to scrape Bet365 for it's live soccer odds. I'm iterating over a list of live matches available. For each match I need to click on the match and I'll be directed to some new content where I have all the detailed oddsinfo. From here it crashes when I go back to keep on my iterating.
It throws this error:
Traceback (most recent call last):
File "/Users/christian/Google Drev/Data Science/Bet365/main.py", line 32, in <module>
getScoreH = game.find_element_by_css_selector(scoreH).text
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 430, in find_element_by_css_selector
return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 654, in find_element
{"using": by, "value": value})['value']
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=69.0.3497.100)
(Driver info: chromedriver=2.42.591059 (a3d9684d10d61aa0c45f6723b327283be1ebaad8),platform=Mac OS X 10.14.0 x86_64)
The error comes from the last code in my main.py:
# HERE IT BREAKS!:
# Redirects to a games detailed odds page
game.find_element_by_css_selector(oddsBtn).click()
time.sleep(5)
# Go back and keep choose the click the next games details.
obj.find_element(overview).click()
time.sleep(5)
Below is my program. But as I said, the problem comes from the main.py's last few lines of code where I need to go back to an iteration. It seems like it doesn't remember where I left off.
cls_scraper.py:
"""
Class to find element(s) by css selector
"""
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import StaleElementReferenceException
import platform
import time
import os
class Scraper():
def __init__(self, driver):
self.driver = driver
def wait(self, element):
return WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, element)))
def element_exist_css(self, element):
try:
self.driver.find_element_by_css_selector(element)
except NoSuchElementException:
print('Element doesnt exist')
return False
return True
def element_css(self, element):
try:
time.sleep(2)
return WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable(
(By.CSS_SELECTOR, element)))
except StaleElementReferenceException:
print("XB: StaleElementReferenceException")
except WebDriverException:
print("XB: WebDriverException")
def find_elements(self, element):
time.sleep(2)
return self.driver.find_elements_by_css_selector(element)
def find_element(self, element):
time.sleep(2)
return self.driver.find_element_by_css_selector(element)
str_elements.py:
"""
String library to have relevant css selector elements in one place.
"""
""" BET 365 - Overview-page """
# Enter the page
enterPage = '#TopPromotionMainArea'
# Page with live odds
inPlay = 'body > div:nth-child(1) > div > div:nth-child(1) > div > div.hm-HeaderModule_Primary > div.hm-BigButtons > nav > a:nth-child(2)'
# Element containing relevent games and info about time, score etc, se below.
games = 'div.ipo-FixtureRenderer.ipo-Competition_Container > div'
# For each game in games, these elements can be found:
teamH = 'div.ipo-TeamStack > div:nth-child(1)'
teamA = 'div.ipo-TeamStack > div:nth-child(2)'
scoreH = 'div.ipo-TeamPoints_TeamScore.ipo-TeamPoints_TeamScore-teamone'
scoreA = 'div.ipo-TeamPoints_TeamScore.ipo-TeamPoints_TeamScore-teamtwo'
gameTime = 'div.ipo-InPlayTimer'
# The redirection in order to get all kinds of odds from a match
# Main overview-page only show a part of it.
oddsBtn = 'div.ipo-FixtureEventCountButton_EventCountWrapper'
# The overview tab to see all the live games
overview = 'div.ip-ControlBar > span.ip-ControlBar_ButtonBar > div:nth-child(1)'
# Choose english language
langTab = 'body > div:nth-child(1) > div > div:nth-child(1) > div > div.hm-HeaderModule_Secondary > div.hm-HeaderModule_Menus > div.hm-LanguageDropDownSelections.hm-DropDownSelections > a'
pickEng = 'body > div:nth-child(1) > div > div:nth-child(1) > div > div.hm-HeaderModule_Secondary > div.hm-HeaderModule_Menus > div.hm-LanguageDropDownSelections.hm-DropDownSelections.hm-DropDownSelections_Selected > div > div > a:nth-child(1)'
# Get a better overview
allMarkets = 'body > div:nth-child(1) > div > div.wc-PageView > div.wc-PageView_Main.wc-InPlayPage_MainContainer > div > div > div.ipo-OverViewView > div > div > div > div.ipo-OverViewDetail > div.ipo-OverViewDetail_Container.ipo-Classification > div.ipo-ClassificationHeader_Header.ipo-ClassificationHeader_Header-1.ipo-ClassificationHeader_Header-lightgreenborder.ipo-ClassificationHeader_Header-moremarkets > div.ipo-ClassificationHeader_MarketsButtonOuterWrapper > div > div.ipo-ClassificationHeader_MarketsButton.ipo-ClassificationHeader_MarketsButton-transparent'
""" BET 365 - Odds-page """
# Collect all the odds from the redirection page.
main.py:
""" Run program from here """
from str_elements import *
from cls_scraper import *
from browser.path import *
import time
if __name__ == '__main__':
print("Welcome \n")
# Open website
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(driver_path, chrome_options=options)
driver.get('https://www.bet365.dk/#/HO/')
""" Click relevant elements """
print("Bet365: Pressing buttons ...")
obj = Scraper(driver)
obj.element_css(enterPage).click() # Enters the bet365 main page
obj.element_css(inPlay).click() # Presses the in-play tab
obj.element_css(langTab).click() # Choose languages
obj.element_css(pickEng).click() # Choose english
obj.element_css(overview).click() # Shows all live games
obj.element_css(allMarkets).click() # Better overview
print("Bet365: Collecting game data ...")
# All live games
liveContainer = obj.find_elements(games) # Contains a list of running games
for game in liveContainer:
getScoreH = game.find_element_by_css_selector(scoreH).text
getScoreA = game.find_element_by_css_selector(scoreA).text
getTeamH = game.find_element_by_css_selector(teamH).text
getTeamA = game.find_element_by_css_selector(teamA).text
getTime = game.find_element_by_css_selector(gameTime).text
print("Score: ", getScoreH, "-", getScoreA)
print("GameTime:", getTime)
print("HomeTeam:", getTeamH)
print("AwayTeam:", getTeamA)
print("")
## HERE IT BREAKS!:
## Redirects to a games detailed odds page
# game.find_element_by_css_selector(oddsBtn).click()
# time.sleep(5)
## Go back and keep choose the click the next games details.
# obj.find_element(overview).click()
# time.sleep(5)

Clicking elements that are not visible (where there is a slider)

Is there a way to click elements outside line of vision in selenium?
I am trying to click all elements on the slider so I can scrape each page. However, the job gives me an error with:
Traceback (most recent call last):
File "C:/Users/Bain3/PycharmProjects/untitled4/TOPBETTA.py", line 1329, in <module>
clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '("//*[#class="name"]//span")[%s]' % str(index + 1))))
File "C:\Users\Bain3\Anaconda3\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
The Href appears to be the default page as https://www.topbetta.com.au/sports/ as seen in the image so extracting the Href and navigating to each page individually is not possible.
I was able to scrape this with Winautomation. Any ideas on how I can do this with Selenium?
driver.execute_script('document.getElementByxpath("//[#class="name"]//span").style.visibility = "visible";')
The above code unfortunately did not help and gave:
Traceback (most recent call last):
File "C:/Users/Bain3/PycharmProjects/untitled4/TOPBETTA.py", line 1329, in <module>
driver.execute_script('document.getElementByxpath("//*[#class="name"]//span").style.visibility = "visible";')
File "C:\Users\Bain3\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 532, in execute_script
'args': converted_args})['value']
File "C:\Users\Bain3\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "C:\Users\Bain3\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Runtime.evaluate threw exception: SyntaxError: missing ) after argument list
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.31.488763 (092de99f48a300323ecf8c2a4e2e7cab51de5ba8),platform=Windows NT 10.0.16299 x86_64)
Code:
try:
os.remove('vtg121.csv')
except OSError:
pass
driver.get('https://www.topbetta.com.au/sports/football/')
#SCROLL_PAUSE_TIME = 0.5
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#clickMe = wait(driver, 3).until(EC.element_to_be_clickable((By.XPATH, ('//*[#id="TopPromotionBetNow"]'))))
#if driver.find_element_by_css_selector('#TopPromotionBetNow'):
#driver.find_element_by_css_selector('#TopPromotionBetNow').click()
#last_height = driver.execute_script("return document.body.scrollHeight")
#while True:
#driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
#time.sleep(SCROLL_PAUSE_TIME)
#new_height = driver.execute_script("return document.body.scrollHeight")
#if new_height == last_height:
#break
#last_height = new_height
time.sleep(1)
#clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, ('//div[text()="Soccer"][contains(#class, "wn-Classification")]'))))
#clickMe.click()
#time.sleep(0)
options = driver.find_elements_by_xpath('//*[#class="name"]//span')
indexes = [index for index in range(len(options))]
shuffle(indexes)
for index in indexes:
time.sleep(0)
clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '("//*[#class="name"]//span")[%s]' % str(index + 1))))
clickMe.click()
time.sleep(0)
# Team
clickMe = wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,("#js_body-region > div > div:nth-child(1) > div.sports-body-region > div.js_event-list-region > div > div.js_events-container.events-container > div > div.js_event-region.event-region > div > a > div > div.team-container.home > div"))))
langs3 = driver.find_elements_by_css_selector("#js_body-region > div > div:nth-child(1) > div.sports-body-region > div.js_event-list-region > div > div.js_events-container.events-container > div > div.js_event-region.event-region > div > a > div > div.team-container.home > div")
langs3_text = []
for lang in langs3:
#print(lang.text)
langs3_text.append(lang.text)
time.sleep(0)
# Team ODDS
langs = driver.find_elements_by_css_selector(" #js_body-region > div > div:nth-child(1) > div.sports-body-region > div.js_event-list-region > div > div.js_events-container.events-container > div > div.js_event-region.event-region > div > div > div > div.js_teams-container.market-items > div.head-to-head-item.home > div > div > button.js_price-button.price")
langs_text = []
for lang in langs:
#print(lang.text)
langs_text.append(lang.text)
time.sleep(0)
# Draw odds
#langs1 = driver.find_elements_by_xpath("//ul[#class='runners']//li[2]")
langs1 = driver.find_elements_by_css_selector("#js_body-region > div > div:nth-child(1) > div.sports-body-region > div.js_event-list-region > div > div.js_events-container.events-container > div > div.js_event-region.event-region > div > a > div > div.team-container.away > div")
langs1_text = []
for lang in langs1:
#print(lang.text)
langs1_text.append(lang.text)
time.sleep(0)
# HREF
#langs2 = driver.find_elements_by_xpath("//ul[#class='runners']//li[1]")
url1 = driver.current_url
print(("NEW LINE BREAK"))
import sys
import io
with open('vtg121.csv', 'a', newline='', encoding="utf-8") as outfile:
writer = csv.writer(outfile)
for row in zip(langs_text, langs1_text, langs3_text):
writer.writerow(row + (url1,))
print(row + (url1,))

An element is not clickable when it is not visible in Selenium. You will have to use the scroll buttons to click on all the elements.

Try below solution to scrape required pages:
url = "https://www.topbetta.com.au/sports/football/"
driver.get(url)
counter = 0
for link in range(len(wait(driver, 15).until(EC.presence_of_all_elements_located((By.XPATH, '//a[#href="/sports" and ./div[#class="name"]]'))))):
wait(driver, 15).until_not(EC.visibility_of_element_located((By.CLASS_NAME, "mask")))
link = wait(driver, 15).until(EC.presence_of_all_elements_located((By.XPATH, '//a[#href="/sports" and ./div[#class="name"]]')))[counter]
link.location_once_scrolled_into_view
link = wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '(//a[#href="/sports" and ./div[#class="name"]])[%s]' % str(counter + 1))))
wait(driver, 15).until_not(EC.visibility_of_element_located((By.CLASS_NAME, "mask")))
link.click()
print(driver.current_url)
wait(driver, 10).until(EC.staleness_of(driver.find_element(By.XPATH, '//div[#class="competition-events-module"]')))
counter += 1
driver.get(url)
Just replace print(driver.current_url) with the code you want to apply to each page