We Keep Coding

Selenium python : my current_url doesnt update after click

Im scraping a website where I need to retrieve values from the url when i click on a button providing different form values.
I have a problem: when i click the button and retrieve the current_url, the provided values in the forms doesnt reflect in the url which should be updated (it's a search button). There is no new tab created.
My code to retrieve the url value is :
driver = webdriver.Firefox()
driver.get(url)
arrlist = []
idlist = []
service=value
for i in key_list:
form = driver.find_elements(by=By.XPATH, value='//input[#id="geo_nav"]')
form[0].send_keys(i)
form2=driver.find_elements(by=By.XPATH, value='//input[#id="sev_nav"]')
form2[0].send_keys(service)
button=driver.find_elements(by=By.XPATH, value='//button[#data-role="filter-apply"]')
button[0].click()
time.sleep(5)
url=driver.current_url
print(dept)
print(i)
id=re.findall(r"(?<=\[population\]=)(\d{9})",url)[0]
arrlist.append(i)
idlist.append(id)
the button html code is :
<button class="filter-apply cta-navigate relative hide-mobile flex withNumber" data-role="filter-apply">
<p class="hide-mobile m-r-4">Appliquer</p>
<div class="svg relative">
<span class="filters-apply-length">2</span>
<svg height="18" viewBox="0 0 16 18" width="16" xmlns="http://www.w3.org/2000/svg"><path d="m10.877 17.457 2.026 1.533v-4.553c0-.166.042-.329.12-.475l4.3-7.962h-10.68l4.122 7.978c.074.142.112.3.112.459zm3.026 4.543c-.213 0-.426-.068-.603-.203l-4.026-3.045c-.25-.189-.397-.484-.397-.797v-3.274l-4.765-9.222c-.161-.31-.148-.681.034-.979.181-.298.505-.48.854-.48h14c.352 0 .678.185.859.488.18.302.188.677.021.987l-4.977 9.215v6.31c0 .379-.214.726-.554.895-.141.07-.294.105-.446.105z" fill="#0579c7" fill-rule="evenodd" transform="translate(-4 -4)"></path></svg> </div>
</button>
I've tried to use
driver.switch_to.window(driver.window_handles[-1]);
following this post : Python Selenium Chromedriver - Can't Get current_url of new opened tab after click()
But I dont have tab or new windows issues.
I tried to click autocompletion lists in the 2 forms in inputand one of the form produces a modification of the url but not the other (the one of which effects on the url i need to monitor).
The form code that works is :
<form data-component="sev_nav_input" data-no-results="Sans résultats" data-default-pho="Services" data-selected-name="Achat compulsif" data-selected-id="5928" class="filter-input filter-services relative">
<input type="text" placeholder="Services" autocomplete="off" name="sev_nav" id="sev_nav" data-role="js_filter" data-id="5928" class="autocomplete-with-result">
<span id="clear-sev-input" class="clear-sev-input" style="display: none;">
<img src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjI0IiB2aWV3Qm94PSIwIDAgMjQgMjQiIHdpZHRoPSIyNCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48ZyBmaWxsPSJub25lIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiPjxwYXRoIGQ9Im0wIDBoMjR2MjRoLTI0eiIvPjxwYXRoIGQ9Im0xMS43NSAyMC41YzIuNDIyIDAgNC40ODYtLjg1MyA2LjE5MS0yLjU1OSAxLjcwNi0xLjcwNSAyLjU1OS0zLjc3IDIuNTU5LTYuMTkxIDAtMi40MjItLjg1My00LjQ4Ni0yLjU1OS02LjE5MS0xLjcwNS0xLjcwNi0zLjc3LTIuNTU5LTYuMTkxLTIuNTU5LTIuNDIyIDAtNC40OTIuODYtNi4yMSAyLjU3OC0xLjY5NSAxLjY5My0yLjU0IDMuNzUtMi41NCA2LjE3MnMuODUzIDQuNDg2IDIuNTU5IDYuMTkxYzEuNzA1IDEuNzA2IDMuNzcgMi41NTkgNi4xOTEgMi41NTl6bTMuMTY0LTQuNDE0Yy0uMDUyIDAtLjExNy0uMDQtLjE5NS0uMTE3bC0yLjk2OS0yLjkzLTIuOTMgMi45NjljLS4wNTIuMDUyLS4xMy4wNzgtLjIzNC4wNzhzLS4xODItLjAyNi0uMjM0LS4wNzhsLS44Mi0uODZjLS4wNTMtLjA1Mi0uMDc5LS4xMy0uMDc5LS4yMzRzLjAyNi0uMTcuMDc4LS4xOTVsMi45NjktMi45NjktMi45NjktMi45M2MtLjE1Ni0uMTU2LS4xNTYtLjMxMiAwLS40NjhsLjgyLS44MmMuMDc5LS4wNzkuMTU3LS4xMTguMjM1LS4xMTguMDUyIDAgLjExNy4wNC4xOTUuMTE3bDIuOTY5IDIuODkgMi45NjktMi44OWMuMDc4LS4wNzguMTQzLS4xMTcuMTk1LS4xMTcuMDc4IDAgLjE1Ni4wNC4yMzQuMTE3bC44Ni44MmMuMTU2LjE1Ny4xNTYuMzEzIDAgLjQ3bC0yLjk2OSAyLjkyOSAyLjkzIDIuOTNjLjA3OC4wNzguMTE3LjE1Ni4xMTcuMjM0IDAgLjEwNC0uMDQuMTgyLS4xMTcuMjM0bC0uODIuODJjLS4wNzkuMDc5LS4xNTcuMTE4LS4yMzUuMTE4eiIgZmlsbD0iIzE0OWM5NyIvPjwvZz48L3N2Zz4K"></span>
<span class="gradient"></span>
<span class="gradient" style="display: none;"></span>
<div class="spinner" style="display: none;"></div>
<div id="services-list" class="services-list" style="display: none;"><ul data-role="autocomplete-list" class="autocomplete-list"> </ul></div></form>
The form code that doesnt work is :
<form data-component="geo_nav_input" data-selected-name="" data-selected-id="" data-selected-neighborhood-id="0" data-selected-type="" data-no-results="Sans résultats" data-pho="Localité" data-default-pho="Localité" class="filter-input relative">
<div class="hide">Chercher des professionnels en/à...</div>
<span class="icon-x toggle_geo_nav hide"></span>
<label for="geo_nav" class="hidden-label">Localité</label>
<input type="text" placeholder="Localité" autocomplete="off" name="geo_nav" id="geo_nav" data-role="js_filter" data-id="" data-neighborhoodid="0" data-type="" class="autocomplete-with-result"> <span id="clear-geo-input" class="clear-geo-input" style="display: none;">
<img src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjI0IiB2aWV3Qm94PSIwIDAgMjQgMjQiIHdpZHRoPSIyNCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48ZyBmaWxsPSJub25lIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiPjxwYXRoIGQ9Im0wIDBoMjR2MjRoLTI0eiIvPjxwYXRoIGQ9Im0xMS43NSAyMC41YzIuNDIyIDAgNC40ODYtLjg1MyA2LjE5MS0yLjU1OSAxLjcwNi0xLjcwNSAyLjU1OS0zLjc3IDIuNTU5LTYuMTkxIDAtMi40MjItLjg1My00LjQ4Ni0yLjU1OS02LjE5MS0xLjcwNS0xLjcwNi0zLjc3LTIuNTU5LTYuMTkxLTIuNTU5LTIuNDIyIDAtNC40OTIuODYtNi4yMSAyLjU3OC0xLjY5NSAxLjY5My0yLjU0IDMuNzUtMi41NCA2LjE3MnMuODUzIDQuNDg2IDIuNTU5IDYuMTkxYzEuNzA1IDEuNzA2IDMuNzcgMi41NTkgNi4xOTEgMi41NTl6bTMuMTY0LTQuNDE0Yy0uMDUyIDAtLjExNy0uMDQtLjE5NS0uMTE3bC0yLjk2OS0yLjkzLTIuOTMgMi45NjljLS4wNTIuMDUyLS4xMy4wNzgtLjIzNC4wNzhzLS4xODItLjAyNi0uMjM0LS4wNzhsLS44Mi0uODZjLS4wNTMtLjA1Mi0uMDc5LS4xMy0uMDc5LS4yMzRzLjAyNi0uMTcuMDc4LS4xOTVsMi45NjktMi45NjktMi45NjktMi45M2MtLjE1Ni0uMTU2LS4xNTYtLjMxMiAwLS40NjhsLjgyLS44MmMuMDc5LS4wNzkuMTU3LS4xMTguMjM1LS4xMTguMDUyIDAgLjExNy4wNC4xOTUuMTE3bDIuOTY5IDIuODkgMi45NjktMi44OWMuMDc4LS4wNzguMTQzLS4xMTcuMTk1LS4xMTcuMDc4IDAgLjE1Ni4wNC4yMzQuMTE3bC44Ni44MmMuMTU2LjE1Ny4xNTYuMzEzIDAgLjQ3bC0yLjk2OSAyLjkyOSAyLjkzIDIuOTNjLjA3OC4wNzguMTE3LjE1Ni4xMTcuMjM0IDAgLjEwNC0uMDQuMTgyLS4xMTcuMjM0bC0uODIuODJjLS4wNzkuMDc5LS4xNTcuMTE4LS4yMzUuMTE4eiIgZmlsbD0iIzE0OWM5NyIvPjwvZz48L3N2Zz4K"></span>
<span class="gradient"></span>
<span class="gradient" style="display: none;"></span>
<div class="spinner" style="display: none;"></div>
<div id="location-list" class="location-list" style="display: none;"><ul data-role="autocomplete-list" class="autocomplete-list"> </ul></div>
</form>

Can you make a function to navigate pages, and on each page do the actions you require. And with each call of the function use driver.switch_to.window to ensure you are on the latest page.
Although based on your edits, it now seems the issue is that you are having trouble locating and following one of the links on the pages.
def navigate(n):
""" Move through the pages. Select the relevant buttons on each page"""
window_after = driver.window_handles[0]
driver.switch_to.window(window_after)
if n == 0:
form = driver.find_elements(by=By.XPATH, value='//input[#id="geo_nav"]')
button = driver.find_elements(by=By.XPATH, value='//button[#data-role="filter-apply"]').click()
elif n == 1:
pass
# Do something
else:
pass
# Do something else
for i in range(3):
navigate(i)
time.sleep(3)

The solution was in fact linked to the autocompletion forms. They require you to click on the autocompletion suggestions so the button is actually working.
FYI, here is the full code to autocomplete with clicking the form, deleting the content, adding the content, clicking the list and clicking the button.
def get_city_locations(service):
url='url'
#options = Options()
#options.headless = True
driver = webdriver.Firefox()#options=options)
driver.get(url)
time.sleep(2)
buttoncookie = driver.find_elements(by=By.XPATH, value='//button[#class="cf2Lf6"]')
buttoncookie[0].click()
time.sleep(1)
form2 = driver.find_elements(by=By.XPATH, value='//input[#id="sev_nav"]')
form2[0].click()
time.sleep(1)
Static.clear_text(driver)
form2[0].send_keys(service)
time.sleep(1)
autocompleteservice = driver.find_elements(by=By.XPATH, value='//li[not(#class)]')
for f in autocompleteservice:
if f.text == service:
f.click()
df_pref=pd.read_csv('arrondissement_2022.csv',sep=',')
deptlist = []
arrlist = []
idlist = []
for i in df_pref['LIBELLE']:
df_dep=df_pref[df_pref['LIBELLE']==i]
dept = df_dep.loc[df_dep.index.values[0], 'DEP']
form = driver.find_elements(by=By.XPATH, value='//input[#id="geo_nav"]')
form[0].click()
time.sleep(1)
Static.clear_text(driver)
form[0].send_keys(i)
time.sleep(3)
autocompletelocation=driver.find_elements(by=By.XPATH, value='//li[not(#class)]')
cond=0
for a in autocompletelocation:
if a.text==i:
print ('condition ok')
cond=1
a.click()
break
time.sleep(3)
button=driver.find_elements(by=By.XPATH, value='//button[#data-role="filter-apply"]')
button[0].click()
time.sleep(3)
driver.switch_to.window(driver.window_handles[-1]);
url=driver.current_url
print(dept)
print(i)
print(url)
if cond==0:
id=0
else:
id=re.findall(r"(?<=\[population\]=)(\d{7,10})",url)[0]
print(f'id = {id}')
print('\n')
deptlist.append(dept)
arrlist.append(i)
idlist.append(id)
df0 = pd.DataFrame({"dept": deptlist, "arrondissement":arrlist,"id":idlist})
df0.to_csv('arr_id.csv',sep=';',index=False)

Web scrape second number between tags

I am new to Python, and never done HTML. So any help would be appreciated.
I need to extract two numbers: '1062' and '348', from a website's inspect element.
This is my code:
page = requests.get("https://www.traderscockpit.com/?pageView=live-nse-advance-decline-ratio-chart")
soup = BeautifulSoup(page.content, 'html.parser')
Adv = soup.select_one ('.col-sm-6 .advDec:nth-child(1)').text[10:]
Dec = soup.select_two ('.col-sm-6 .advDec:nth-child(2)').text[10:]
The website element looks like below:
<div class="nifty-header-shade1 col-xs-12 col-sm-6 col-md-3">
<div class="row">
<div class="col-sm-12">
<h4>Stocks</h4>
</div>
<div class="col-sm-6">
<p class="advDec">Advanced: 1062</p>
</div>
<div class="col-sm-6">
<p class="advDec">Declined: 348</p>
</div>
</div>
</div>
Using my code, am able to extract first number (1062). But unable to extract the second number (348). Can you please help.

Assuming the Pattern is always the same, you can select your elements by text and get its next_sibling:
adv = soup.select_one('a:-soup-contains("Advanced:")').next_sibling.strip()
dec = soup.select_one('a:-soup-contains("Declined:")').next_sibling.strip()
Example
import requests
from bs4 import BeautifulSoup
page = requests.get("https://www.traderscockpit.com/?pageView=live-nse-advance-decline-ratio-chart")
soup = BeautifulSoup(page.content)
adv = soup.select_one('a:-soup-contains("Advanced:")').next_sibling.strip()
dec = soup.select_one('a:-soup-contains("Declined:")').next_sibling.strip()
print(adv, dec)

If there are always 2 elements, then the simplest way would probably be to destructure the array of selected elements.
import requests
from bs4 import BeautifulSoup
page = requests.get("https://www.traderscockpit.com/?pageView=live-nse-advance-decline-ratio-chart")
soup = BeautifulSoup(page.content, "html.parser")
adv, dec = [elm.next_sibling.strip() for elm in soup.select(".advDec a") ]
print("Advanced:", adv)
print("Declined", dec)

Selenium: Can't find an element in HTML

Hi I am working on the script to automate downloads of videos from this side https://pixabay.com/videos/
I can find a class with href(href is an attribute with URL) but after that Selenium gives me a bug with any error only a result of print(xy.get_atribute("href)) is None:
my code:
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from webdriver_manager.firefox import GeckoDriverManager
from selenium.webdriver.common.by import By
from mutagen.mp3 import MP3
import requests
import time
tag = "city "
while True:
s = Service(GeckoDriverManager().install())
driver = webdriver.Firefox(service=s)
driver.minimize_window()
tester = tag.split()
print(tester)
print(len(tester))
if len(tester) == 2:
tag = tester[0] + "%20" + tester[1]
print(tag)
print("2 " + tag)
driver.get("https://pixabay.com/cs/videos/search/" + tag )
images = driver.find_elements(By.CLASS_NAME, 'item' )
print(images)
n = 0
lenght = 0
for image in images:
image = image.get_attribute("href")
print(image)
break
HTML on the side
<html lang="cs" prefix="og: http://ogp.me/ns#">
<head>.</head> <body class="" data-new-gr-C-5-check-loaded="14.1050." data-gr-ext-installed>
<noscript>_</noscript> <div id="wrapper"> > <div id="header">...</div> <div id="content" class="clearfix">
::before <div id="search-term" style="display:none">city</div> <div class="media_list">
<div style="border-bottom:1px solid #f0f1f4"> </div> <div style="background:#e8eaec" class="external-media">.</div> <div style="background: #f6f5fa"> <div style="max-width: 1824px;padding: 10px 3px 20px;margin: auto">
<h1 style="font-size: 13px;color:#bbb;margin:0 19px; position:relative;top:2px">70 videa zdarma z city</h1> <div class="related-keywords">...</div> <div class="row-masonry video video-search-results"> flex <div class="row-masonry-cell" style="flex-basis: 355.55555555555554px; flex-grow: 1.7777777777777777; flex-shrink: 1.7777777777777777; max-width: 622.2222222222222px"> <div class="row-masonry-cell-outer" style="padding-top: 56.25%"> <div class="row-masonry-cell-inner"> <div itemscope itemtype="schema.org/videoobject" class="item" data-w="1920" data-h="1980">
<meta itemprop="license" content="https://creativecommons.org/licenses/publicdomain/"> <meta itemprop="contentUrl" content="//player.vimeo.com/external/142621375.mobile.mp4?s=e9a3c9616798b6f3de74d579ea8314acc75fad72&profile_id=116"> <meta itemprop="thumbnailUrl" content="https://i.vimeocdn.com/video/539965294-5d28c2680682aa5173e86fa74acb94671783ba7e2dc2892682e897c8a158af75-d_640x360.jpg"> <meta itemprop="name" content="New York City, Manhattan, Lidé"> <meta itemprop="description" content="New York City, Manhattan, Lidé, Auta, Rozcestí, Amerika"> <meta itemprop="duration" content="TM145"> <meta itemprop="uploadDate" content="2022-02-16"> <a href="/cs/videos/new-york-city-manhattan-lidC3%A9-auta-1944/"> == $ <div class="media" data-mp4="//player.vimeo.com/external/142621375.mobile.mp4?s=e9a3c9616798b6f3de74d579ea8314acc75fad72&profile_id=116">
<img class="video-preview" src="https://i.vimeocdn.com/video/539965294-5d28c26...d 640x360.jpg" alt="New York City, Manhattan, Lidé, Auta">
<i></i> </div> </a> ►<em class="info-corner">-</em>
<em class="info-line">-</em> flex </div> </div> </div>

To extract the values of the href attributes you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get("https://pixabay.com/videos/search/madona/")
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.item>a")))])
Using XPATH:
driver.get("https://pixabay.com/videos/search/madona/")
print([my_elem.get_attribute("href") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#class='item']/a")))])
Console Output:
['https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fthe-blessed-virgin-mary-in-front-of-the-roman-catholic-diocese-public-place-in-gm1297534089-390641862%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=96d4db58e5ed4dfa33719b6789ec2e54c6a9e93c', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fshining-star-landscape-above-the-nativity-scene-in-bethlehem-in-the-middle-of-the-gm1284414826-381542953%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=63c5e9d5424c6ed9234b7558a47c2f9bd34b0b2f', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fvirgin-mary-statue-and-stained-glass-window-cathedral-la-major-marseille-france-gm523679382-92792807%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=eba8300b6dc8d879d5c0c20ad4159667075b13a8', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fmary-and-joseph-kissing-and-touching-baby-jesus-gm1331078248-414315796%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=0d8025f74d6831ec1728a487c6c348fbb0489c0e', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fthe-blessed-virgin-mary-in-front-of-the-roman-catholic-diocese-public-place-in-gm1297536160-390641882%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=1a8d9dc7db4a76fe4fe8b15d050e950a4ef6ceee', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fmary-and-joseph-speaking-and-taking-care-of-baby-jesus-gm1331074139-414312596%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=71968733d6752db862f8f614ec81965c72c045c7', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fvirgin-mary-over-the-village-of-maaloula-in-syria-view-of-the-virgin-mary-in-the-gm1225424859-360681300%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=77ee3cc8968570cce77488fd36324f9a21aa8cdb', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fstatue-of-st-mary-in-the-church-gm1192697685-338971692%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=0bc49cf096c333282f735ce562f59c9eb17309f4', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fnotre-dame-de-paris-exterior-beautiful-statue-of-virgin-and-child-architecture-gm980707058-266394749%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=69ab20290bfcaef4b644d7f888618d569cd0c994', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fmary-and-joseph-with-baby-jesus-in-barn-gm1331079736-414317051%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=dbae2b824a22623d12eaf1127f0fd33f4de2cc41', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fchristmas-nativity-gm113747428-13537033%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=496f4633dd6cd6394f1a06959e4c6dd089d3aeff', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fsculpture-of-the-image-of-nossa-senhora-aparecida-the-patroness-of-brazil-gm1348138435-425434800%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=c745fbc9d7b69eeaa2186c9843f8d566590b5e61', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fchurch-icon-close-up-gm824010680-134889129%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=89a9e0bee0d8d49c4dad52407e63e5ab37a98b55', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fangled-detail-of-icon-image-of-virgin-mary-in-st-nicholas-orthodox-cathedral-in-nice-gm860958856-143133885%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=0fc269e8cf9d4fc2c0a6384537349b33fa98f963', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fvitaleta-chapel-aerial-view-in-the-wonderful-valley-of-orcia-tuscany-la-toscana-drone-gm1309396875-399121012%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=356539420e275a4aae53df6dfc2ec1e22a469f5a', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fburning-candles-in-the-cathedral-of-chartres-gm1358325815-431987333%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=3e6b1cd3f8922c80cb2ea4aacbc5f0ca13b48024', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Flourdes-france-sanctuary-of-our-lady-of-lourdes-a-famous-pilgrimage-place-gm1352878773-428151138%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=e29f6ee34464b11218e1bba7f72d321db571968a', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fsanctuary-of-our-lady-of-lourdes-gm1348213795-425484844%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=00fc1155414568f9c01d4d8e07b3e7d6396e9c76', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Frosaries-on-the-bridge-in-lourdes-gm1348213738-425484842%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=ffc4132d8125f9da46baff336297150fee0617e2', 'https://pixabay.com/link/?ua=t%3Devent%26ec%3Dapi_ad%26ea%3Dnavigate%26el%3Dgetty%26v%3D1%26tid%3DUA-20223345-1&next=https%3A%2F%2Fwww.istockphoto.com%2Fvideo%2Fa-statue-of-the-virgin-mary-in-lourdes-gm1348213561-425484841%3Futm_source%3Dpixabay%26utm_medium%3Daffiliate%26utm_campaign%3DSRP_video_sponsored%26utm_content%3Dhttp%253A%252F%252Fpixabay.com%252Fvideos%252Fsearch%252Fmadona%252F%26utm_term%3Dmadona&hash=198c52a01d8f0d8e2c2079ea5238cfc1f3da8d00']
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Well, first of all, you haven't actually provided what the error is, which would be helpful.
Additionally, your for loop is overwriting the image var which is just bad practice, but shouldn't break anything.
Finally, it looks like you've written an infinite loop with a non-conditional break at the end and no continue. Is this supposed to be a loop?

Scraping nested html with Selenium

I'm looking for some help with scraping with selenium in python.
You need a paid account to view this page so creating a reproducible won't be possible.
The page I'm trying to scrape
I'm attempting to scrape the data from the pitch in the top right corner of the image under 'Spots on Field'.
<div class="player-details-football-map__UEFA player-details-football-map">
<div class="shots">
<div>
<a class="shot episode" style="left: 39.8529%; top: 28.9474%;"></a>
<div class="tooltip" style="left: 39.8529%; top: 28.9474%;">
<div class="tooltip-title">
<div class="tooltip-shoot-type">Shot on target</div>
<div class="tooltip-blow-type">Donyell Malen </div>
<div class="tooltip-shoot-name"></div>
</div>
<div class="tooltip-time">h Viktoria Koln</div>
<div class="tooltip-time">Half 1, 18:22 02/09/20</div>
<div class="tooltip-time">Length: 7.1 m</div>
<div class="tooltip-shoot-xg">Expected goals: 0.17</div>
</div>
</div>
The above is a snippet of just one of the data points I want to scrape.
I've tried using BeautifulSoup
from bs4 import BeautifulSoup
from requests import get
url = 'https://football.instatscout.com/players/294322/shots'
response = get(url)
html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)
shots = html_soup.find_all('div', class_ = 'tooltip')
print(type(shots))
print(len(shots))
and nothing was being returned.
So now I've tried using Selenium.
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Users\James\OneDrive\Desktop\webdriver\chromedriver.exe')
driver.get('https://football.instatscout.com/players/294322/shots')
print("Page Title is : %s" %driver.title)
driver.find_element_by_name('email').send_keys('my username')
driver.find_element_by_name('pass').send_keys('my password')
driver.find_element_by_xpath('//*[contains(concat( " ", #class, " " ), concat( " ", "hRAqIl", " " ))]').click()
goals = driver.find_element_by_class_name('tooltip')
but I'm getting the error of
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".tooltip"}
Can someone please help point me in the right direction? I'm basically trying to scrape everything from the above HTML, that includes 'tooltip' in the class name.
Thanks

Using css selectors with bs4:
from bs4 import BeautifulSoup as soup
import re #for extracting offsets
r = [{**dict(zip(['left', 'top'], re.findall('[\d\.]+', i.div['style']))),
'shoot_type':i.select_one('.tooltip-shoot-type').text,
'name':i.select_one('.tooltip-blow-type').text,
'team':i.select_one('div:nth-of-type(2).tooltip-time').text,
'time':i.select_one('div:nth-of-type(3).tooltip-time').text,
'length':i.select_one('div:nth-of-type(4).tooltip-time').text[8:],
'expected_goals':i.select_one('.tooltip-shoot-xg').text[16:]}
for i in soup(html, 'html.parser').select('div.shots > div')]
Output:
[{'left': '39.8529', 'top': '28.9474', 'shoot_type': 'Shot on target', 'name': 'Donyell Malen ', 'team': 'h Viktoria Koln', 'time': 'Half 1, 18:22 02/09/20', 'length': '7.1 m', 'expected_goals': '0.17'}]

Unable to locate an lists of elements using selenium

I need to scrape some pages. The exact structure of the part that I want is as follows:
<div class="someclasses">
<h3>...</h3> # Not needed
<ul class="ul-class1 ul-class2">
<li id="li1-id" class="li-class1 li-class2">
<div id ="div1-id" class="div-class1 div-class2 ... div-class6">
<div class="div2-class">
<div class="div3-class">...</div> #Not needed
<div class="div4-class1 div4-class2 div4-class3">
<a href="href1" data-control-id="id1" data-control-name="name" id ="a1-id" class="a-class1 a-class2">
<h3 class="h3-class1 h3-class2 h3-class3">Text1</h3>
</a></div>
<div>...</div> # Not needed
</div>
</li>
<li id="li2-id" class="li-class1 li-class2">
<div id ="div2-id" class="div-class1 div-class2 ... div-class6">
<div class="div2-class">
<div class="div3-class">...</div> #Not needed
<div class="div4-class1 div4-class2 div4-class3">
<a href="href2" data-control-id="id2" data-control-name="name" id ="a2-id" class="a-class1 a-class2">
<h3 class="h3-class1 h3-class2 h3-class3">Text2</h3>
</a></div>
<div>...</div> # Not needed
</div>
</li>
# More <li> elements
</ul>
</div>
Now what I want is to get the Texts as well as the hrefs.I have used the naming in above example exactly realistic i.e the same names are also the same in the real webpage. The code that I am currently using is:
elems = driver.find_elements_by_xpath("//div[#class='someclasses']/ul[#class='ul-class1']/li[#class='li-class1']")
print(len(elems))
for elem in elems:
elem1 = driver.find_element_by_xpath("./a[#data-control-name='name']")
names2.append(elem1.text)
print(elem1.text)
hrefs.append(elem.get_attribute("href"))
The result of the print statement above is 0 so basically the elements are not found. Can anyone please tell me what am I doing wrong.

You are using only part of the class name... in XPATH you need the full class name...
FYI: With CSS you can use part of the class name...
If you want to use XPATH try:
elems = driver.find_elements_by_xpath("//div[#class='someclasses']//li//a")
print(len(elems))
for elem in elems:
names2.append(elem.text)
print(elem.text)
new_href = elem.get_attribute("href")
print(new_href)
hrefs.append(new_href)
For CSS use: div.someclasses ul.ul-class1
elems = driver.find_elements_by_css_selector("div.someclasses ul.ul-class1 li a")
for elem in elems:
names2.append(elem.text)
print(elem.text)
new_href = elem.get_attribute("href")
print(new_href)
hrefs.append(new_href)