I am currently doing web UI automation using Robot Framework. For my current task I have a div tag which shows a table inside. The table only shows 25 elements at load time. See image of elements for reference.
After scrolling the top div tag slowly, it creates new set of tables. See below image:
The point is it doesn't show all tables at one go. Also it keeps on adding tables with _startrow attribute to a random value. It also deletes tables from earlier rows on scrolling.
I am not sure how to deal with such dynamic content. Using Execute Javascript keyword has not helped as well.
Any new help would be appreciated. Thank you.
Your situation is a bit specific (on the js framework, and the html it produces), but I'll take a stab of providing a bit generic answer - with pseudo code you'll need to adapt.
What does a user do to make sure the new data is loaded? They see an entry, scroll a bit, wait for that entry to disappear, and for a new one to be visualized. So let's try to simulate it; "see" the current entry:
${old table}= Get Webelement ${locator for a generic table in the div} # something like //div/table[#summary=“Task Types“]
Now the value of that variable is going to be a specific webelement - an actual object in the DOM, not the "current <table> tag in the html, in any given moment".
Then, you do the operation that will make the system reload the tables - and wait for the one that was present to disappear:
Operation So Data Is Changed
Wait Until Element Is Not Visible ${old table}
Once the "Wait For" returns execution/completes, you are sure the old table is no longer visible. And then - you need to wait for the new one to appear:
Wait Until Element Is Visible ${locator for a generic table in the div} # something like //div/table[#summary=“Task Types“]
This approach is based off the SeleniumLibrary, and YMMV on the exact operations and locators, but that's the general idea.
Related
I have a problem getting text from a table that can be shown by a button click, the data for that table are present even before the button be clicked, but when I try to get it, it returns nothing, if I try to push the button first, it successfully gets the text, is there a solution for this ? By the way, I am using Xpath to get the element.
table = f'(//div[child::table[child::tbody[child::tr[child::td[text()="Student Name:"]]]]])[{self.row}]'
self.studentName = self.dataBrowser.find_element(By.XPATH, table+'/table[1]/tbody/tr[1]/td[2]').text
Probably you are missing a delay here.
I mean probably you are extracting the text from web element before it is fully loaded, containing it text content.
In several cases waiting for the visibility of the element you are extracting the text from will resolve your problem.
To give more precise answer we need to see all your code, including the page you are working on to make some debugging and testing.
I'm working on a project trying to autonomously monitor item prices on an Angular website.
Here's what a link to a particular item would look like:
https://www.<site-name>.com/categories/<sub-category>/products?prodNum=9999999
Using Selenium (in Python) on a page with product listings, I can get some useful information about the items, but what I really want is the prodNum parameter.
The onClick attribute for the items = clickOnItem(item, $index).
I do have some information for items including the presumable item and $index values which are visible within the html, but I'm doubtful there is a way of seeing what is actually happening in clickOnItem.
I've tried looking around using dev-tools to find where clickOnItem is defined, but I haven't been successful.
Considering that I don't see any way of getting prodNum without clicking, I'm wondering, is there's a way I could simulate a click to see where it would redirect to, but without actually loading the link- as this would take way too much time to do for each item?
Note: I want to get the specific prodNumber. I want to be able to hit the item page directly without first going though the main listing page.
I have a table which just shows 25 rows on inspection. It loads new set of rows on scrolling. But at a time there are only 25 rows that are visible. The ELEMENT also changes its ID from 1 to 25 to let's say 8 to 32 after little scroll. Point is it doesn't show entire content at any given time and only loads 25 IDs at any point. My requirement is to fetch all the rows(98) from the table. I have tried using Scroll Element Into View, but that has not helped me. Also using Execute JavaScript window.scrollTo(0, document.body.scrollHeight) has not helped as well.
I have seen so many questions on this one, but no appropriate solution to this.
How to do scrolling inside infinite load table or list
Robot framework, Python - handle lazy load on page with dynamic content
Please help if possible.
You have to write recursive keyword that would do following things:
Collects required data
Triggers loading of new rows (scrolls to the bottom, clicks button)
If there are new rows trigger this keyword once more
Combine results
I am having trouble finding elements on a customer-facing webpage that I am scraping data from, using Robot Framework + Selenium. My trouble, I think, has to do with the desired data existing in a popup/frame. The data I seek is located on a customer's invoice, which pops up when I press a button ("View Current Invoice"). I've been successful with logging into the site and navigating around, and at one point I was successful pressing the View Current Invoice button to cause the invoice to pop up - but forgot to commit that code and lost it. :-(
In any case, eve if I manually enter the popped up invoice by pressing the button when my script is expecting it to be pressed, I can't seem to scrape the subsequent data. I have tried to identify elements on the invoice using locators (from Right-Click-Inspect capability built into Firefox and Chrome; Katalaon Recorder; Selenium IDE; etc.). I get what looks like a valid locator (almost always Xpath); yet when I run my Robot script, it fails to find the element in question. I have spent a lot of time poring over the page's source code, but since I am not as savvy with HTML/JS/CSS as I should be, I haven't been successful.
Here is a screenshot of the invoice button:
And here is what I see when the button is pressed. I want to scrape all the invoice data, like Amount Due, Invoice Number, Due Date, etc.
Does anyone have any idea what I am missing here? What would you do to get the data on the invoice if you were in my shoes? I know my question probably sounds vague and naiive, but I am at the end of my rope, so to speak. I am willing to share page source code, more screenshots, whatever is required.
EDIT I used Rahul Rai's method to inspect the popup while it was popped up; then searched for "iframe". There were 10 matches; #7, when clicked on, resulted in the invoice popup being highlighted in blue:
I assume this means this is the iframe referencing the popup? If so, I should be able to find information about the "handle" to the iframe in the inspection code, but I don't see anything there that matches the locators I am used to (e.g. name, id, xpath). I even tried "Select Frame 1599252503952", but that just resulted in a
"Element with locator '1599252503952' not found" error.
As per above screen you have shared I can see your Invoice details are inside iframe. So after clicking on View Current Invoice button you can use below code to navigate inside frame and then scrape required information.
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'invoice_detail_container']")))
#Code to scrape data
ele = driver.find_element_by_xpath('<xpath>')
print(ele.txt)
......
......
#After your work is done in this frame to navigate back to main window
driver.switch_to.default_content()
Note: I have assumed your main frame for invoice is not in side any other iframe ( Based on screen shared). Also before elements start there is no other nested frame. If there is any other nested frame you need to navigate first into that.
I was finally able to scrape data from the Invoice popup after inspecting the HTML source, and seeing this:
<iframe frameborder="0" src="/cmc/invoice_detail_container.pyt?direction=//my.hughesnet.com/cmc/invoice_detail.pyt%3Finvnumber%1234-567890&portletId=863" name="1599391562960" class="cboxIframe" allowfullscreen="true" webkitallowfullscreen="true" mozallowfullscreen="true"></iframe>
I was then able to use the Select Frame keyword in Robot Framework, specifying the iframe locator for the popup, using the 'class' strategy. I also had to explicitly select the main body frame first. In the end, the code that allowed me to enter the iframe and scrape was:
Select Frame body
Select Frame class:cboxIframe
Big thank you to Rahul Rai for pushing me closer to the solution; and thanks to the others who answered as well.
You need to switch your site to frame/popup, you can use like below example, may be it will help you.
IList<IWebElement> textfields = new List<IWebElement>();
textfields = driver.FindElements(By.TagName("iframe"));
driver.SwitchTo().Frame(textfields[count); // number of textfields list.
please try to implement as per your scenario, let me know if any question.
You can try to do :
driver.switch_to_active_element()
and then scrape the popup to close it. Then I think it will be okay...
Greetings fellow scrapers,
I'm trying to use Selenium in Python 3.4 to open a website's clunky interface and get the information I need from it. Sadly, since I need to submit a form (which directs to a different page without bringing up a new window) each time and there is no way to get all of the information in one go. The whole process goes:
"For" all relevant fields: # as of right now, this cannot be a for loop
# because the elements are lost when I submit
(1) click on the necessary buttons (make it visible)
in order to select a field in the form
(2) submit the form
(3) get the necessary information from the new page
(4) go back to the first page
If possible, I would like to find a way of doing this that does not involve me clicking my way to a button each time- that is, I would like to eliminate step (1) of my loop and change step (4) to simply switching windows.
For clarification on how the form is submitted, there is a javascript function within the code that takes care of submission but the built-int Selenium method Element.submit() still works (so I can either .click() on the submit button or I can .submit() from any element in the form).
To make my question clearer, is there a way that the form output can be opened in a new window/tab, whether through inserting javascript or otherwise, so that I don't have to reload the first page and click its buttons so many times?
Please PM me if you would like more specific details about the code and the website, or tell me in an answer or comment if there is any crucial information missing from the question.
So, I did some digging, and it turns out that it is possible to do this, in some cases.
I found this link about setting attributes, and after having read that forms can have attributes, especially target = _blank, which opens the form in a new (blank) window
This actually depends on how the form is submitted. If in the HTML the code has <form...id = someID, and the form element has some identifier (in this case, someID) you can manually change this attribute.
This opens forces the form to open in a new window the next time it is submitted:
Browser.execute_script('document.getElementById("someID").'setAttribute("target","_blank")')
Browser is the name I gave my WebDriver. This executes internal javaScript, which sets the target attribute of the form to _blank- just what I wanted.