We Keep Coding

Delete dynamic elements from HTML with Selenium and Python

I've used BeautifulSoup to find a specific div class in the page's HTML. I want to check if this div has a span class inside it. If the div has the span class, I want to maintain it on the page's code, but if it doesn't, I want to delete it, maybe using Selenium.
For that I have two lists selecting the elements (div and span). I tried to check if one list is inside the other, and that kind of worked. But how can one delete that found element from the page's source code?
Edit
I've edited the code after a few conversations in the commentaries section. With help, I was able to implement code to remove elements executing javascript.
The code is running with no errors, but nothing is being deleted from the page.
# Import required module
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import time
# Option to launch browser in incognito
options = Options()
options.add_argument("--incognito")
#options.add_argument("--headless")
# Using chrome driver
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
# Web page url request
driver.get('https://www.facebook.com/ads/library/?active_status=all&ad_type=all&country=BR&q=frete%20gr%C3%A1tis%20aproveite&sort_data[direction]=desc&sort_data[mode]=relevancy_monthly_grouped&search_type=keyword_unordered&media_type=all')
driver.maximize_window()
time.sleep(10)
driver.execute_script("""
for(let div of document.querySelectorAll('div._99s5')){
let match = div.innerText.match(/(\d+) ads? use this creative and text/)
let numAds = match ? parseInt(match[1]) : 0
if(numAds < 10){
div.querySelector(".tp-logo")?.remove()
}
}
""")

Since you're deleting them in javascript anyway:
driver.execute_script("""
for(let div of document.querySelectorAll('div._99s5')){
let match = div.innerText.match(/(\d+) ads? use this creative and text/)
let numAds = match ? parseInt(match[1]) : 0
if(numAds < 10){
div.querySelector(".tp-logo")?.remove()
}
}
""")

Note: Question and comments reads a bit confusing so it would be great to improve it a bit. Assuming you like to decompose() some elements, the reason why or what to do after this action is not clear. So this answer will only point out an apporache.
To decompose() the elements that do not contains ads use this creative and text just negate your selection and iterate the ResultSet:
for e in soup.select('div._99s5:has(:not(:-soup-contains("ads use this creative and text")))'):
e.decompose()
Now these elements will no longer be included in your soup and you could process it for your needs.

Webpage formatted in a way that makes selecting text with selenium impossible

This problem is driving me insane: I'm trying to capture the response from a Pandorabot using Selenium but although I can input text and make the bot reply, its webpage is formatted in such a way that makes selecting the output text a nightmare.
This is my code in Python:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
driver = webdriver.Firefox()
driver.get("http://demo.vhost.pandorabots.com/pandora/talk?botid=b0dafd24ee35a477")
elem = driver.find_element_by_name("input")
elem.clear()
elem.send_keys("hello")
elem.send_keys(Keys.RETURN)
line = driver.find_element_by_xpath("(//input)[#name='botcust2']/preceding::font[1]/*")
print(line)
response = line.text
print(response)
driver.close()
which manages to get the first bit of the response ("Chomsky:") but not the rest.
How do I get to properly capture the response text (ideally excluding the bot name)?
Is there a more elegant way to do it (eg jquery script) that wouldn't break so easily if the webpage gets reformatted?
Many thanks!
Edit
So, after playing around a bit more with jQuery I found a workaround to the problem of any URL text not showing.
I set the whole text string into a variable and then I replace any instances of the name and empty lines with ''. So the jQuery code as pointed out by pguardiario becomes:
# get the last child text node
response = self.browser.execute_script("""
var main_str = $('font:has(b:contains("Chomsky:"))').contents().has( "br" ).last().text().trim();
main_str = main_str.replace(/Chomsky:/g,'').replace(/^\\s*[\\r\\n]/gm, '');
return main_str;
""")
I'm sure there may be better/more elegant ways to do the whole thing but for now it works.
Many thanks to pguardiario and everyone else for the suggestions!

Since you asked for jQuery:
from requests import get
body = get("http://code.jquery.com/jquery-1.11.3.min.js").content.decode('utf8')
driver.execute_script(body)
# get the last child text node
response = driver.execute_script("""
return $('font:has(b:contains("Chomsky:"))').contents().last().text().trim()
""")

To capture the response from a Pandorabot using Selenium as the response is within a text node you can execute_script() method as follows:
Code Block:
driver.get('http://demo.vhost.pandorabots.com/pandora/talk?botid=b0dafd24ee35a477')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='input']"))).send_keys("hello")
driver.find_element_by_css_selector("input[value='Ask Chomsky']").click()
print(driver.execute_script("return arguments[0].lastChild.textContent;", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//input[#value='Ask Chomsky']//following-sibling::font[last()]//font")))).strip())
Console Output:
Hi! Can I ask you a question?

Selenium can't find 99% of elements, why?

I'm using Selenium with Python and Firefox on Windows 7.
Running through my script, at some point Selenium can't find any elements beside and after I navigated to a certain part of a website.
The website belongs to an E-Mail provider called freemail (https://web.de/). I have set up a test account to complete a training project from chapter 11 of the book "Automate the Boring Stuff with Python".
Python, Selenium and Firefox are all up to date. And before this certain part of the website everything was working fine (including finding tags, class names, id's etc.).
I tried like 10 different ways to click one of the ways to navigate to the inbox of the E-Mail provider but had no luck in making it work.
Here is the HTMLcode with the highlighted element I want Selenium to click.
Here another image of the Website itself with the button highlighted in the upper left corner.
And this is the code for my script:
## This program navigates through the website of an
## e-mail provider and sends a mail automatically.
# Import necessary modules
from selenium import webdriver
# Mail account info
mailLink = 'https://web.de/'
mailAddress = 'testing2772#web.de' # This is just a test account
mailPw = 'ewaa11ewaa11'
# Recipient % content of the e-mail
recipient = 'pradicradr#matra.top'
mailContent = 'Test this awesome string.'
# Set up browser
browser = webdriver.Firefox()
browser.get(mailLink)
# Log into account
browser.find_element_by_class_name('icon-freemail').click()
browser.find_element_by_id('freemailLoginUsername').send_keys(mailAddress)
pwElem = browser.find_element_by_id('freemailLoginPassword')
pwElem.send_keys(mailPw)
pwElem.submit()
# Close notification overlay if it pops up
try:
browser.find_element_by_class_name('layerCloser').click()
except:
pass
## At this point Selenium can't Find 99% of elements including the one I need
## Testing if Selenium finds basic elements
testElem1 = browser.find_element_by_tag_name('html') # works
testElem2 = browser.find_element_by_tag_name('body') # works
testElem3 = browser.find_element_by_tag_name('div') # works
## Click link to inbox (all of these variants won't work somehow)
linkToInbox = browser.find_element_by_class_name('nx-track-sec-click-communication-newmail') # try by class_name
linkToInbox = browser.find_element_by_xpath("/html/body/div[3]/div/div[3]/div[1]/ul/li[3]/a") # try by XPath
linkToInbox = browser.find_element_by_link_text('E-Mail schreiben') # try by link_text
linkToInbox = browser.find_element_by_css_selector('a.nx-track-sec-click-communication-newmail') # try by CSS selector
linkToInbox = browser.find_element_by_css_selector('a.newmail') # try by CSS selector
linkToInbox = browser.find_element_by_css_selector('a.nx-track nx-track-sec-click-communication-newmail newmail') # another try by CSS selector
linkToInbox = browser.find_element_by_css_selector('a[title="E-Mail schreiben"]') # another try by CSS selector
## TODO: Click link to write a new mail
## TODO: Fill in recipient and mail content
## TODO: Send mail and close program
I guess it has something to do with an iframe? But I'm a beginner and I don't know how to handle this :/
Thanks in advance!
EDIT:
I solved the problem by finding another button that linked to the inbox with
browser.find_element_by_xpath("/html/body/atlas-app/atl-navbar/atl-actions-menu/div[1]/div[1]/atl-menu-item[2]/pos-icon-item/span/span").click(). This button is not within an iframe, so no problem here.
However, it is still strange that I can't enter any iframes. Here is another image to make it more clear. Does anybody know how to switch to that iframe?
browser.switch_to.default_content() and then browser.switch_to.frame('home') doesn't do the trick...
SECOND EDIT:
KunduK's answer did it.
The problem was, that the script needs to wait for the iframe to become available and then for the button to become available after that. See his post further down for the exact approach. Thanks everyone!

To click on the E-Mail schreiben link you need to switched to iframe first and then click on link.Use WebDriverWait and frame_to_be_available_and_switch_to_it and then use element_to_be_clickable
WebDriverWait(browser,20).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,'home')))
WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.XPATH,'//div[#id="navigation"]//ul//li[#class="item"]//a[#title="E-Mail schreiben"]'))).click()
You need to use following imports to execute above code.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

I guess it has something to do with an iframe?
Since the elements you are trying to find are in an iframe, you must first switch to the iframe before finding the elements. Your iframe has a name=home attribute, so you should be able to add:
driver.switch_to.frame('home')
... then find the elements.
Once you are done interacting with the iframe and want to switch back to the top level content, use:
driver.switch_to.default_content()

Option 1
linkToInbox = browser.find_element_by_css_selector('a.nx-track.nx-track-sec-click-communication-newmail.newmail') # try by CSS selector
Option 2
use the title property of the anchor link
linkToInbox = browser.find_element_by_css_selector('a[title="E-Mail schreiben"]')
option 3
re-read the docs: https://selenium-python.readthedocs.io/locating-elements.html

How to extract text from previously inserted input with Selenium?

For some reasons, all I want is to get from the input field what I have just written IN THE INPUT FIELD just to check.
from selenium import webdriver
import os
xpath_user = '//*[#id="login-username"]'
user = 'user#yahoo.com'
dir_path = os.path.dirname(os.path.realpath(__file__))
chromedriver = dir_path + "/chromedriver.exe"
driver = webdriver.Chrome(chromedriver)
driver.implicitly_wait(3)
driver.get('https:\\www.yahoo.com')
driver.find_element_by_xpath(xpath_user).send_keys(user)
element = driver.find_element_by_xpath(xpath_user).text
print(element)
if element == 'user#yahoo.com':
print("Good")
In this example, the output is '', but I want the actual 'user#yahoo.com', but I don't know if it is even possible because 'user#yahoo.com' doesn't appear in the html form of the page. Maybe I am missing something or there is a work around. I'll be glad if someone could help me.
Note that my experience with python is limited.

Try driver.find_element_by_xpath(xpath_user).get_attribute("value")
The text property is for text within the tags of an element.

Select all text in a textbox Selenium RC using Ctrl + A

I am trying to select all the text in a textbox in order to clear the textbox. I am using Ctrl+A to do this using the following Python 2.7 code on Selenium RC standalone 2.20.0.jar Server on Windows 7 firefox:
from selenium import selenium
s = selenium('remote-machine-ip', 4444, '*chrome', 'http://my-website-with-textbox')
locator = 'mylocator-of-textbox'
s.open()
s.type(locator, 'mytext')
s.focus(locator)
s.control_key_down()
s.key_down(locator, "A")
s.key_press(locator, "A")
s.key_up(locator, "A")
s.control_key_up()
# Nothing happens here... I cannot see the text getting selected...
# Nothing gets cleared here except the last char
s.key_down(locator, chr(8)) # Pressing backspace
s.key_press(locator, chr(8))
s.key_up(locator, chr(8))
Any help?
Thanks,
Amit

I'm using clear() in WebDriver without any hassle...
el = self.selenium.find_element_by_name(name)
el.clear()

In Selenium RC, simply use the following to clear a textbox
selenium.type("someLocator", "");

Try using first
element.click()
then use element
element.clear()
It might solve your issue as it really solved mine.

You can do this:
public void selectAll(WebElement element) {
String selectAll = Keys.chord(Keys.CONTROL, "a");
element.sendKeys(selectAll);
}
And when you whant to use it, for example:
selectAll(myDriver.findElement(By.id("testId")));
In this example the WebElement can be a text box, text area and similars.