I've written a script in python to scrape a certain text located within class floorplan which is within right-column which again within modal-body. However, when I run my script, it gives blank output?
link to that site
Elements before click (values are null in floorplan and swing classes):
<div class="right-column">
<div class="field" ng-show="selectedLot.Name !== ''">
<div class="label">Home Design:</div>
<div class="floorplan value ng-binding"></div>
<hr>
</div>
<div class="field" ng-show="selectedLot.ShortDescription !== ''">
<div class="label">Elevation:</div>
<div class="swing value ng-binding"></div>
<hr>
</div>
<div class="field" ng-show="selectedLot.Swing !== ''">
<div class="label">Swing:</div>
<div class="swing value ng-binding"></div>
<hr>
</div>
</div>
After click (values are there now in floorplan and swing classes):
<div class="right-column">
<div class="field" ng-show="selectedLot.Name !== ''">
<div class="label">Home Design:</div>
<div class="floorplan value ng-binding">Delaware</div>
<hr>
</div>
<div class="field" ng-show="selectedLot.ShortDescription !== ''">
<div class="label">Elevation:</div>
<div class="swing value ng-binding">TRA</div>
<hr>
</div>
<div class="field" ng-show="selectedLot.Swing !== ''">
<div class="label">Swing:</div>
<div class="swing value ng-binding">Garage Right</div>
<hr>
</div>
</div>
I've tried so far with (can't make my script click on that image to reveal the data I'm after):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def collect_links(link):
driver.get(link)
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR,"path#ip-loader-circle")))
item = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,".modal-body .right-column .floorplan")))
print(item.get_attribute("innerHTML"))
if __name__ == '__main__':
url = "https://khovsecure.ml3ds-cloud.com/index.html?_ga=2.181197287.1174152084.1550480313-902396065.1550480313#/lotmap/43935"
driver = webdriver.Chrome()
wait = WebDriverWait(driver,20)
collect_links(url)
driver.quit()
Expected output:
Delaware
This is how the information popped up in a box when a click is initiated on that map:
How can I make a click on that map to scrape the desired text from the pop up container?
Code below give you all data in json format:
import requests
if __name__ == '__main__':
headers = {
'fullurl': 'https://khovsecure.ml3ds-cloud.com/index.html?_ga=2.181197287.1174152084.1550480313-902396065.1550480313#/lotmap/43935',
}
response = requests.get('https://khovsecure.ml3ds-cloud.com/resources/data/CommunityData/khovsecure.ml3ds-cloud.com', headers=headers)
print(response.json())
Related
I am trying to navigate to a search box and send_keys with selenium python but completely stuck.
And here is the source code snippet:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<div id="LeftTreeFrame" class="leftNavBackground" >
<div class="ui-widget searchPanelContainer">
<div id="searchPanel" class="search-field-container search-field-container-margin">
<input type="text" doesntDirty id="Search" name="Search" class="search-text-field-left-tree-frame" NoHighlight="nohighlight"/>
<div class="search-field-icon-container">
<a id="searchlbl" href="#"><img src="../images/normal_search_u39.svg" title="Go To Page" /></a>
</div>
</div>
</div>
<div id='pageNavigation'>
<div id='ootbNavigationPage'></div>
<div id='favoriteNavigationPage'></div>
<div id='adminNavigationPage'></div>
<div id='navigationEmptyState' class="treeEmptyState">
<div class="message"></div>
</div>
</div>
<div class="navigation-view-mode-container">
<div class="box" onclick="renderModel(0)">
<button type="button">
<span class="svg-load ootb-icon" data-src="~/images/Reskin/ootb-icon.svg"></span>
</button>
</div>
<div class="star" onclick="renderModel(1)">
<button type="button">
<span class="svg-load star-icon" data-src="~/images/Reskin/star.svg"></span>
</button>
</div>
<div class="person" onclick="renderModel(2)">
<button type="button">
<span class="svg-load person-icon" data-src="~/images/Reskin/person-nav.svg"></span>
</button>
</div>
</div>
</div>
When I try to do
element = driver.find_element(By.XPATH, '//input[#name="Search"]')
element.send_keys('test')
I get error "selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable"
I have tried everything I can imagine, but cannot click the element or send keys.
Also, this page is a new page that opens after the last successful click. I first tried switching to this page by
#printing handles
handles = driver.window_handles
i=0
for handle in handles:
print(f"Handle {i}: {handle}\n")
i +=1
#after confirming new page is second handle via:
driver.switch_to.window(handles[1])
print(f" Title: {driver.title}")
print(f" Current url: {driver.current_url}")
print('\n')
#I can even find the tag I am looking for after switching to new window:
all_div_tags = driver.find_elements(By.TAG_NAME, "input")
for tag in all_div_tags:
print(f"Attribute name: {tag.get_attribute('name')}\n")
#but i cannot get to the search box. Thank you in advance!
Look at the html code, notice that //input[#name="Search"] is contained in an <iframe>. In order to select an element inside an iframe with find_element() you have first to switch to the iframe, as shown in the code
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "frmCode")))
element = driver.find_element(By.XPATH, '//input[#name="Search"]')
...
I only started to study python. I have a problem with my code. I use Python + selenium and I don’t understand how I can click on the first element in this code, then go to products__item, after that return back to catalog and click on the second element, then on the third element using selenium.
This is code from a typical internet shop. I understand I need to use a cycle “for” for it but how to do it I don’t know.
<div class="products__list" style="position: relative">
<div class=" products__item ">
<a href="/catalog/dveri-mezhkomnatnyye/dveri-ekoshpon/bravo/bravo-21-snow-art" class="card">
<div class="card__title"> Product-1 </div>
<div class="card__color"> Snow Art </div>
<div class="card__img"> <div class="card__img-wrapper"> <img src="/storage/products/small/60fa84e2b06096.72907873.jpg"> </div>
<div class="card__img-wrapper"> </div>
<div class=" products__item ">
<a href="/catalog/dveri-mezhkomnatnyye/dveri-ekoshpon/bravo/bravo-21-snow" class="card">
<div class="card__title"> Product-2 </div>
<div class="card__color"> Snow </div>
<div class="card__img"> <div class="card__img-wrapper"> <img src="/storage/products/small/60fa84e2b06096.72907874.jpg"> </div>
<div class="card__img-wrapper"> </div>
<div class=" products__item ">
<a href="/catalog/dveri-mezhkomnatnyye/dveri-ekoshpon/bravo/bravo-21-snow-art-classic" class="card">
<div class="card__title"> Product-3 </div>
<div class="card__color"> Snow Art Classic </div>
<div class="card__img"> <div class="card__img-wrapper"> <img src="/storage/products/small/60fa84e2b06096.72907875.jpg"> </div>
<div class="card__img-wrapper"> </div>
Hope it helps!
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome('path/to/chromedriver')
driver.get("https://dveri.com/catalog/dveri-mezhkomnatnyye?page=1")
wait = WebDriverWait(driver, 10)
elements_xpath = '//div[#class=" products__item "]/a[#class="card"]'
# Wait for emelents to load
wait.until(EC.element_to_be_clickable((By.XPATH, elements_xpath)))
num_elements = len(driver.find_elements(By.XPATH, elements_xpath))
ac = ActionChains(driver)
for i in range(num_elements):
# Wait until elements are clickable
wait.until(EC.element_to_be_clickable((By.XPATH, elements_xpath)))
# Get all elements and select only the i-th one
element = driver.find_elements(By.XPATH, elements_xpath)[i]
# Click the element with the offset from center to actually go to the other page
ac.move_to_element(element).move_by_offset(0, 100).click().perform()
# Here do whatever has to be done on a specific webpage
time.sleep(1)
# Go back to the previous page
driver.execute_script("window.history.go(-1)")
I am looping through a column of excel file to auto-complete the form for me and submit. However, the challenge is that:
I cannot input the value in the excel column into the first field for the form.
The div element for the form gives me element not clickable before sending the value.
See how the form looks here:
Img1:
Img2:
The first field(Artist) automatically opens up with values when you manually start to type the name before the option comes for you to select.
My Python script is below:
from selenium import webdriver
import pandas as pd
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
#Read Data file
df = pd.read_excel('try.xlsx')
#Import Driver components
driver = driver = webdriver.Chrome('/home/dev-mike/.wdm/drivers/chromedriver/linux64/88.0.4324.96/chromedriver')
url = 'myurl'
username = 'myname.com'
password = '****'
driver.get(url)
time.sleep(2)
driver.find_element_by_id("email").send_keys(username)
time.sleep(1)
driver.find_element_by_id("password").send_keys(password)
time.sleep(1)
driver.find_element_by_xpath("//*[#id='app']/main/div/div/div/div/div[2]/form/div[4]/div/button").click()
time.sleep(1)
driver.find_element_by_xpath("/html/body/div/div[1]/div[3]/div[1]/div[2]/div[2]/button").click()
# #Inside Dashboard
for i in df.index:
entry = df.loc[i]
#Name Field
driver.find_element_by_xpath('/html/body/div/div[2]/aside[1]/div/div[2]/div/form/div[1]/div/div/div/div[2]/span').click()
time.sleep(10)
artist_input = driver.find_element_by_xpath('/html/body/div/div[2]/aside[1]/div/div[2]/div/form/div[1]/div/div/div/div[3]/ul/li[1]/span/span')
artist_input.send_keys(entry['Artist'])
time.sleep(1)
title_input = driver.find_element_by_id('input-2')
title_input.send_keys(entry['Title'])
time.sleep(1)
medium_input = driver.find_element_by_id('input-3')
medium_input.send_keys(entry['Medium'])
time.sleep(1)
A cross-section of my form-field elements looks like this:
This is based on the assumption that is typing Paul in that input field.
<div class="fb-select fb-element">
<div class="v-popover display-block">
<div
aria-describedby="popover_qmz2wjbcfu"
class="trigger"
style="display: inline-block"
>
<span class="fb-select__label">Artist</span>
<div tabindex="-1" class="multiselect fb-select__field">
<div class="multiselect__select"></div>
<div class="multiselect__tags">
<div class="multiselect__tags-wrap" style="display: none"></div>
<!---->
<div class="multiselect__spinner" style="display: none"></div>
<input
name=""
type="text"
autocomplete="nope"
placeholder="Start typing..."
tabindex="0"
class="multiselect__input is-focusable"
style="width: 0px; position: absolute; padding: 0px"
/>
<!---->
<span class="multiselect__placeholder"> Start typing... </span>
</div>
<div
tabindex="-1"
class="multiselect__content-wrapper"
style="max-height: 300px; display: none"
>
<ul class="multiselect__content" style="display: inline-block">
<!---->
<li class="multiselect__element">
<span
data-select=""
data-selected=""
data-deselect=""
class="multiselect__option"
><span>Paul Gauguin</span></span
>
<!---->
</li>
<li class="multiselect__element">
<span
data-select=""
data-selected=""
data-deselect=""
class="multiselect__option"
><span>Paul Kaspa</span></span
>
<!---->
</li>
<li class="multiselect__element">
<span
data-select=""
data-selected=""
data-deselect=""
class="multiselect__option multiselect__option--highlight"
><span>Paula Rego</span></span
>
<!---->
</li>
<li style="display: none">
<span class="multiselect__option"
>No elements found. Consider changing the search query.</span
>
</li>
<li style="display: none">
<span class="multiselect__option">Nothing found</span>
</li>
</ul>
</div>
</div>
<!---->
</div>
</div>
</div>
I am open to make you understand further if this explanation is not clear?
use :
driver.find_element_by_xpath(f"//div[#role='option'][normalize-space(.)='{your_input_text}']").click()
if it doesn't work, please say what site is it and I check what for you.
I am trying to iterate through a set of results, similar to the below, so to select that I perform the below:
for a in browser.find_elements_by_css_selector(".inner-row"):
What I then want to do is return:
a. The x in the class next to time-x (e.g. 26940 in the example)
b. filter to bananas only
c. Grab the suffix to "row-x" in the id
For each result. I can then iterate through the results for each of these that meets the parameters.
I have tried the get attribute function but this doesn't return any results, and .text is out of the question due to no real information between the tags.
<div id="bookingResults bookingGroup-111">
<div id="row-1522076067"
class="row row-time group-111 time-26940 amOnly bananas groupOnly rule-1252"
style="display: block;">
<div class="lockOverlay lock-row-124" style="display: none;"><div class="lockInfoCont"><p class="lockedText">Locked <span class="miclub-icon icon-lock"></span></p></div><div class="lockTimer"></div></div>
<div class="col-lg-3 col-md-4 col-sm-4 col-xs-4 row-heading " id="heading-1522076067" >
<div class="row">
<div class="col-lg-4 col-md-4 col-sm-5 col-xs-5 row-heading-inner">
<h3>07:29 am</h3>
<h4>
Choose Me
<br/>
<span id="rule-name-row-1522076067" style="display: none">
</span>
</h4>
</div>
<div class="col-lg-8 col-md-8 col-sm-7 col-xs-7 row-heading-inner">
<button id="btn-book-group-1522076067"
class="btn btn-book-group hide"
title="Book Row" >
<span class="btn-label">BOOK GROUP</span>
</button>
<div class="row-information">
</div>
</div>
</div>
</div>
I guess this html is the each of the a in your code then you can exract the id and time with following code:
for a in browser.find_elements_by_css_selector(".inner-row"):
try:
el = a.find_element_by_css_selector("div.bananas")
print("id: %s", el.get_attribute("id").split("-")[1])
print("time: %s", [s for s in el.get_attribute("class")(" ") if "time-" in s][0].split("time-")[1])
except NoSuchElementException as e:
pass
You can get the ids like so
print([e.get_attribute('id') for e in driver.find_elements(By.CSS_SELECTOR, 'div.bananas')])
Prints ['row-1522076067']
To handle dynamic element Induce WebDriverWait() and wait for visibility_of_all_elements_located() and following css selector.
Then use regular expression to get the value from element attribute.
Code
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import re
driver=webdriver.Chrome()
driver.get("URL here")
elements=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div[id^='bookingResults']>div.bananas")))
for element in elements:
print(re.findall("row-(\d+)",element.get_attribute("id"))[0])
classatr=element.get_attribute("class")
print(re.findall("time-(\d+)",classatr)[0])
I have a problem with finding a WebElement in Selenium. I tried to fill the input field, this is the html:
<div class="container">
<div class="tp-fl" id="btnSearchStockAutoComplete">
<span class="tp-icon tp-search tp-co-3 "></span>
</div>
<angucomplete placeholder="search" searchfields="Not Found" pause="400" set-focus="tpFocus" selectedobject="tpSelected" titlefield="label" inputclass="form-control form-control-small" matchclass="highlight" api="tpApi" disable="disableSearchBox" stockdetails="stockdetails" class="ng-isolate-scope">
<div class="tp-re angucomplete-holder tp-width">
<div id="fulltextContainer" style="display: none" class="symbol-info tp-etc">
</div>
<input id="txt_search" class="search-box tp-co-1 tp-pa-rl-5 tp-re tp-bo-bo" type="text" placeholder="Search" onmouseup="this.select();" autocomplete="off">
<div id="auto-list-container" class="auto-list-container tp-bg-2 tp-co-1 tp-bo-4 tp-bo tp-width angucomplete-dropdown" style="min-width: 250px; display: none">
<div style="display: none" id="loading" class="tp-h-35 tp-pa-rl-10 angucomplete-searching">Searching ...</div>
<div style="display: none" id="norecord" class="tp-35 tp-pa-rl-10 angucomplete-searching">Not Found </div>
<div id="list_dropdown">
</div>
</div>
</div>
</angucomplete>
</div>
this is my code in python:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser=webdriver.Chrome('path')
inputTxt=browser.find_element_by_xpath('//*[#id="txt_search"]')
inputTxt.send_keys('stock')
but when I run this program, it gives me an error and throws an exception, "selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable", I was wondering if there is any possible solution to this problem.
When you write the CSS selector app-content#txt_search, that means the HTML is something like <app-content id='txt_search'>...</app-content>. Joined together with no space between means they are all part of the same element. What you want is either app-content > #txt_search (> indicates child) or app-content #txt_search ( indicates descendant). From the HTML posted in your question, I'm assuming you want space because I don't see the parent <app-content> tag.