The problem here is i have a "show more" with infinite scroll, i need to detect when the button opacity becomes 0, or in other words when the button becomes not visible so i can use pagination until the button becomes invisible, which means there's no more info to scrape, i'm talking about over 200 pages to scrape, and no other way to know how many pages are there and use range to iterate the pages, so i need to get the trigger of the opacity attribute. Since the button is always there i can't just use is not None to find the last page.
The button is like this when there are more pages:
<a class="btn btn-primary btn-lg MaisArtigos">SHOW MORE</a>
it turns to this on the last page
<a class="btn btn-primary btn-lg MaisArtigos" style="opacity: 0;">SHOW MORE</a>
I need to trigger when the style attribute is present
I'm using BeautifulSoup and requests library
you can use the Element.getAttribute()
const div1 = document.getElementById('div1');
const exampleAttr= div1.getAttribute('style');
console.log("the attribute is: " + exampleAttr)
<div id="div1" style="opacity:0;">Hi Champ!</div>
well you may want to add an event listener with "scroll" to trigger the get attribute, also if the element has no "style" attribute it will return a NULL, exampleAttr = NULL;
Getting the attribute you can use the css selector:
soup.select_one('a.MaisArtigos[style="opacity: 0;"]')
Checking if the style is set you can use bool() that will give you True if exists and False while not existing:
bool(soup.select_one('a.MaisArtigos[style="opacity: 0;"]'))
Example
To demonstrate, this will only select the styled <a>.
from bs4 import BeautifulSoup
html='''
<a class="btn btn-primary btn-lg MaisArtigos" style="opacity: 0;">SHOW MORE</a>
<a class="btn btn-primary btn-lg MaisArtigos" style="opacity: 100;">SHOW MORE</a>
<a class="btn btn-primary btn-lg MaisArtigos" >SHOW MORE</a>
'''
soup=BeautifulSoup(html,'lxml')
soup.select('a.MaisArtigos[style="opacity: 0;"]')
Related
im trying to get a value from a button that is inside of a form like so:
<form class="search-form" method="post" action="">
<div class="dropdown">
<button class="btn btn-default dropdown-toggle text-primary" type="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="true" name="platform" value="platform">
Choose Platform
<span class="caret"></span>
</button>
<ul class="dropdown-menu" aria-labelledby="dropdownMenu1">
<li>Steam</li>
<li>PSN</li>
</ul>
</div>
</form>
With a JS i change the text inside the Button
$(".dropdown-menu li a").click(function(){
$(this).parents(".dropdown").find('.btn').html($(this).text() + ' <span class="caret"></span>');
$(this).parents(".dropdown").find('.btn').val($(this).data('value'));
Inside of the FlaskApp -> When i try to get the Button's Value from the Form with platform = request.form['platform'] im getting KeyError: 'platform'
I tried to fetch the data from the button element but had no success.
My guess is that Flask/Werkzeug does not capture button's data. I went through the documentation and the debugger but did not find why exactly this is happening.
You can add a hidden field and assign the value to it instead.
Could not click the button English.
It says no such element exists.
Tried the following and Select method.
element = WebDriverWait(driver, 40).until(expected_conditions.element_to_be_clickable((By.XPATH, "/html/body/content/main/div/form")))
select = Select(driver.find_element_by_css_selector("label[for='language-MCQ'] span[class='me-2']"))
Code block:
<input class="form-check-input d-none lang-list" type="checkbox" id="language-MCQ" name="language_ids[]" value="1" wire:model="language_ids.0" wire:change="addLanguageQuestions(1)">
<label class="form-check-label d-flex flex-nowrap align-items-center rounded-pill btn btn-lg btn-lang-select text-white cursor-pointer" for="language-MCQ">
<span class="me-2">
English
</span>
</label>
<div class="form-check ps-0 me-3">
<input class="form-check-input d-none lang-list" type="checkbox" id="language-Kannada" name="language_ids[]" value="2" wire:model="language_ids.1" wire:change="addLanguageQuestions(2)">
<label class="form-check-label d-flex flex-nowrap align-items-center rounded-pill btn btn-lg btn-lang-select text-white cursor-pointer" for="language-Kannada">
<span class="me-2">
Kannada
</span>
</label>
</div>
Did you try this xpath :
//span[text()='English']
if you do not wish to use text()
//label[#for='language-MCQ']/span
also this looks wrong
select = Select(driver.find_element_by_css_selector("label[for='language-MCQ'] span[class='me-2']"))
Note that Select is only applicable for html select tag.
To minimize the possibility of any kind of exception it is recommended that your xpath should be pointed to the interactive element, In this case span or label is not interactive element but the input tag.
and your possible xpath solutions can be -
.//span[contains(text(),'English')]/ancestor::label/preceding-sibling::input
Or
.//input[#id='language-MCQ']
Or
.//label[contains(#class,'btn-lang-select')]/preceding-sibling::input[#value='1']
something like this should work -
element = WebDriverWait(driver, 40).until(expected_conditions.element_to_be_clickable((By.XPATH, ".//span[contains(text(),'English')]/ancestor::label/preceding-sibling::input")))
element.click()
If the above xpaths are not working with the explicit wait and expected conditions then check element should be in iframe.
I have a html page like below
<body>
<div>
<a class="btn btn-info" href="a.php">A</a>
<a class="btn btn-info" href="b.php">B</a>
<a class="btn btn-info" href="c.php">C</a>
<a class="btn btn-info" href="d.php">D</a>
</div>
<a class="btn btn-info" href="f.php">F</a>
</body>
I need to select last link F. I have tried like below
link = driver.find_element_by_xpath("//a[last()]")
It's selecting D
I also tried below way
email = driver.find_element_by_xpath("/body//a[last()]")
In this time unable to locate element. How can I get last link here in an easy way ?
To get the last element. Try following xpath.
link = driver.find_element_by_xpath("(//a)[last()]")
use the driver.findElements() method and put the return value in an array.
The last link will be the last element of the array.
I suggest you to (for various reasons):
do yourArray = driver.findElements(By.tag("a"))
for every element check if the href attribute is not null with element.getAttribute("href") != null and update a myIndex var if it's not
The element at the index "myIndex" of "yourArray" will be the element you're searching for.
I'm learning with Python Selenium and I'm trying to click a button which doesn't seem to have an id. I get an error that the element could not be scrolled into view.
Could anyone help with how I click the element?
It might be useful to know that the button only appears when you mouse over a section of the page
HTML is:
<div class="btn-group" role="group" aria-label="...">
<button type="button" class="grey-bg btn btn-default btn-expand" title="Expand"></button>
<button type="button" class="grey-bg btn btn-default btn-export" title="Export"></button>
<button type="button" class="grey-bg btn btn-default btn-clone" title="Clone"></button>
<button type="button" class="grey-bg btn btn-default btn-stamp" title="Pin"></button>
<button type="button" class="grey-bg btn btn-default btn-delete" title="Delete"></button>
</div>
I've tried to select by xpath:
driver.find_element_by_xpath('//*[#title="Export"]').click()
Thanks!
As you mentioned, it appears when you mouse hover. Kindly refer to this discussion for mouse hover. And try to look for the button after then click using javascript as this doesn't need to be displayed in order to click.
exportBtn = driver.find_element_by_xpath('//*[#title="Export"]') driver.execute_script("arguments[0].click();", exportBtn)
From the error, it looks like the element hasn't been scrolled into view. In order to scroll to the element, just add this line to ur code:
element = driver.find_element_by_xpath('//*[#title="Export"]')
driver.execute_script("arguments[0].scrollIntoView();", element)
element.click()
Hope that this helps!
There are 2 buttons in a page, and the difference between these 2 buttons is "onclick".
<button class="btn btn-primary" style="width: 96px;" type="button" id="YesBtn" onclick="check_security('security_div0')">OK</button>
<button class="btn btn-primary" style="width: 96px;" type="button" id="YesBtn" onclick="check_security('wlan1security_div0')">OK</button>
I was thinking to use xpath:
driver.find_element_by_xpath("//form[#id='update-container-id']/div/div/div/div[2]/div/div[2]/table[1]/tbody/tr[1]/td[8]/div[3]/div/div/div/div[3]/button").click()
But it responses the error as below:
selenium.common.exceptions.ElementNotInteractableException: Message: Element <button id="YesBtn" class="btn btn-primary" type="button"> could not be scrolled into view
Does anyone can help me to click the 2nd button correctly? Thanks a lot.
try with the x-path //button[#onclick="check_security('wlan1security_div0')"]
driver.find_element_by_xpath("//button[#onclick=\"check_security('wlan1security_div0')\"]").click()
Using Action class,
button = driver.find_element_by_xpath("//button[#onclick=\"check_security('wlan1security_div0')\"]")
ActionChains(driver).move_to_element(button).click(button).perform()
using java script executor,
driver.execute_script("javascript:check_security('wlan1security_div0')")
As per the HTML you have provided, to click on the button using the onclick() event you can use the following solution:
First Element(css_selector):
driver.find_element_by_css_selector("button.btn.btn-primary#YesBtn[onclick*='security_div0']").click()
First Element(xpath):
driver.find_element_by_xpath("//button[#class='btn btn-primary' and #id='YesBtn'][#onclick=\"check_security('security_div0')\"]").click()
Second Element(css_selector):
driver.find_element_by_css_selector("button.btn.btn-primary#YesBtn[onclick*='wlan1security_div0']").click()
Second Element(xpath):
driver.find_element_by_xpath("//button[#class='btn btn-primary' and #id='YesBtn'][#onclick=\"check_security('wlan1security_div0')\"]").click()
First and foremost, you are using a really long xpath which will be difficult to maintain. You can narrow it down further.
Now, some xpaths you can try:
1) Get the second button with the id YesBtn (assuming there are only two buttons with that attribute) :
driver.find_element_by_xpath("(//button[#id= 'YesBtn'])[2]");
2) Find by the onclick attribute:
driver.find_element_by_xpath("//button[#onclick= \'check_security(\'wlan1security_div0\')\']");
button1 with HTML :
<button class="btn btn-primary" style="width: 96px;" type="button" id="YesBtn" onclick="check_security('security_div0')">OK</button>
XPATH :
//button[text()='OK' and #onclick="check_security('security_div0')"]
button2 with HTML :
<button class="btn btn-primary" style="width: 96px;" type="button" id="YesBtn" onclick="check_security('wlan1security_div0')">OK</button>
HTML:
//button[text()='OK' and #onclick="check_security('wlan1security_div0')"]
try:
buttonVar = browser.find_element_by_css_selector(cssSel)
ActionChains(browser).move_to_element(buttonVar).click(buttonVar ).perform()
except Exception as ex:
print("button not found)
Firefox gives you several option you can use with selenium find_element or elements;
by pressing f12 to bring up the inspector,
use the "pick an item" icon in the left corner to select the item you want to view.
This will highlight the text for that item in the inspector, which you can then right click on to bring up a menu where you select "copy" where there are at least 2 find_element options, css_selecor & xpath.
I personally don't like xpath.
They are sometimes long & unruly to work with but whatever suits your style. In my case css_selector works fine.