Greetings fellow scrapers,
I'm trying to use Selenium in Python 3.4 to open a website's clunky interface and get the information I need from it. Sadly, since I need to submit a form (which directs to a different page without bringing up a new window) each time and there is no way to get all of the information in one go. The whole process goes:
"For" all relevant fields: # as of right now, this cannot be a for loop
# because the elements are lost when I submit
(1) click on the necessary buttons (make it visible)
in order to select a field in the form
(2) submit the form
(3) get the necessary information from the new page
(4) go back to the first page
If possible, I would like to find a way of doing this that does not involve me clicking my way to a button each time- that is, I would like to eliminate step (1) of my loop and change step (4) to simply switching windows.
For clarification on how the form is submitted, there is a javascript function within the code that takes care of submission but the built-int Selenium method Element.submit() still works (so I can either .click() on the submit button or I can .submit() from any element in the form).
To make my question clearer, is there a way that the form output can be opened in a new window/tab, whether through inserting javascript or otherwise, so that I don't have to reload the first page and click its buttons so many times?
Please PM me if you would like more specific details about the code and the website, or tell me in an answer or comment if there is any crucial information missing from the question.
So, I did some digging, and it turns out that it is possible to do this, in some cases.
I found this link about setting attributes, and after having read that forms can have attributes, especially target = _blank, which opens the form in a new (blank) window
This actually depends on how the form is submitted. If in the HTML the code has <form...id = someID, and the form element has some identifier (in this case, someID) you can manually change this attribute.
This opens forces the form to open in a new window the next time it is submitted:
Browser.execute_script('document.getElementById("someID").'setAttribute("target","_blank")')
Browser is the name I gave my WebDriver. This executes internal javaScript, which sets the target attribute of the form to _blank- just what I wanted.
Related
I'm working on a project trying to autonomously monitor item prices on an Angular website.
Here's what a link to a particular item would look like:
https://www.<site-name>.com/categories/<sub-category>/products?prodNum=9999999
Using Selenium (in Python) on a page with product listings, I can get some useful information about the items, but what I really want is the prodNum parameter.
The onClick attribute for the items = clickOnItem(item, $index).
I do have some information for items including the presumable item and $index values which are visible within the html, but I'm doubtful there is a way of seeing what is actually happening in clickOnItem.
I've tried looking around using dev-tools to find where clickOnItem is defined, but I haven't been successful.
Considering that I don't see any way of getting prodNum without clicking, I'm wondering, is there's a way I could simulate a click to see where it would redirect to, but without actually loading the link- as this would take way too much time to do for each item?
Note: I want to get the specific prodNumber. I want to be able to hit the item page directly without first going though the main listing page.
I am currently doing web UI automation using Robot Framework. For my current task I have a div tag which shows a table inside. The table only shows 25 elements at load time. See image of elements for reference.
After scrolling the top div tag slowly, it creates new set of tables. See below image:
The point is it doesn't show all tables at one go. Also it keeps on adding tables with _startrow attribute to a random value. It also deletes tables from earlier rows on scrolling.
I am not sure how to deal with such dynamic content. Using Execute Javascript keyword has not helped as well.
Any new help would be appreciated. Thank you.
Your situation is a bit specific (on the js framework, and the html it produces), but I'll take a stab of providing a bit generic answer - with pseudo code you'll need to adapt.
What does a user do to make sure the new data is loaded? They see an entry, scroll a bit, wait for that entry to disappear, and for a new one to be visualized. So let's try to simulate it; "see" the current entry:
${old table}= Get Webelement ${locator for a generic table in the div} # something like //div/table[#summary=“Task Types“]
Now the value of that variable is going to be a specific webelement - an actual object in the DOM, not the "current <table> tag in the html, in any given moment".
Then, you do the operation that will make the system reload the tables - and wait for the one that was present to disappear:
Operation So Data Is Changed
Wait Until Element Is Not Visible ${old table}
Once the "Wait For" returns execution/completes, you are sure the old table is no longer visible. And then - you need to wait for the new one to appear:
Wait Until Element Is Visible ${locator for a generic table in the div} # something like //div/table[#summary=“Task Types“]
This approach is based off the SeleniumLibrary, and YMMV on the exact operations and locators, but that's the general idea.
I have a URL with different items. Every item has an associated button_item and when this button_item is clicked on, a pop-up window appears with a marked default option and a submit button. This pop-up window is apparently the same for all items in the source window but actually it isn't because the server knows which button_item you clicked on (we know this).
How can I implement in python/selenium the action of this submit button without having to click on it, not even opening the corresponding popup window?
I am guessing I would have to find in the HTLM code the form or information corresponding to that popup window and then apply a submit() method. My idea was to find the form through the ID of the particular item which is a string specific for every item and I guess it's the element used by the server to know which item you cliked on exactly.
Something like:
driver.find_element_by_???("itemID").submit()
I don't know which method to use after "driver." for finding the right form and submitting it.
Thanks
I am having trouble finding elements on a customer-facing webpage that I am scraping data from, using Robot Framework + Selenium. My trouble, I think, has to do with the desired data existing in a popup/frame. The data I seek is located on a customer's invoice, which pops up when I press a button ("View Current Invoice"). I've been successful with logging into the site and navigating around, and at one point I was successful pressing the View Current Invoice button to cause the invoice to pop up - but forgot to commit that code and lost it. :-(
In any case, eve if I manually enter the popped up invoice by pressing the button when my script is expecting it to be pressed, I can't seem to scrape the subsequent data. I have tried to identify elements on the invoice using locators (from Right-Click-Inspect capability built into Firefox and Chrome; Katalaon Recorder; Selenium IDE; etc.). I get what looks like a valid locator (almost always Xpath); yet when I run my Robot script, it fails to find the element in question. I have spent a lot of time poring over the page's source code, but since I am not as savvy with HTML/JS/CSS as I should be, I haven't been successful.
Here is a screenshot of the invoice button:
And here is what I see when the button is pressed. I want to scrape all the invoice data, like Amount Due, Invoice Number, Due Date, etc.
Does anyone have any idea what I am missing here? What would you do to get the data on the invoice if you were in my shoes? I know my question probably sounds vague and naiive, but I am at the end of my rope, so to speak. I am willing to share page source code, more screenshots, whatever is required.
EDIT I used Rahul Rai's method to inspect the popup while it was popped up; then searched for "iframe". There were 10 matches; #7, when clicked on, resulted in the invoice popup being highlighted in blue:
I assume this means this is the iframe referencing the popup? If so, I should be able to find information about the "handle" to the iframe in the inspection code, but I don't see anything there that matches the locators I am used to (e.g. name, id, xpath). I even tried "Select Frame 1599252503952", but that just resulted in a
"Element with locator '1599252503952' not found" error.
As per above screen you have shared I can see your Invoice details are inside iframe. So after clicking on View Current Invoice button you can use below code to navigate inside frame and then scrape required information.
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'invoice_detail_container']")))
#Code to scrape data
ele = driver.find_element_by_xpath('<xpath>')
print(ele.txt)
......
......
#After your work is done in this frame to navigate back to main window
driver.switch_to.default_content()
Note: I have assumed your main frame for invoice is not in side any other iframe ( Based on screen shared). Also before elements start there is no other nested frame. If there is any other nested frame you need to navigate first into that.
I was finally able to scrape data from the Invoice popup after inspecting the HTML source, and seeing this:
<iframe frameborder="0" src="/cmc/invoice_detail_container.pyt?direction=//my.hughesnet.com/cmc/invoice_detail.pyt%3Finvnumber%1234-567890&portletId=863" name="1599391562960" class="cboxIframe" allowfullscreen="true" webkitallowfullscreen="true" mozallowfullscreen="true"></iframe>
I was then able to use the Select Frame keyword in Robot Framework, specifying the iframe locator for the popup, using the 'class' strategy. I also had to explicitly select the main body frame first. In the end, the code that allowed me to enter the iframe and scrape was:
Select Frame body
Select Frame class:cboxIframe
Big thank you to Rahul Rai for pushing me closer to the solution; and thanks to the others who answered as well.
You need to switch your site to frame/popup, you can use like below example, may be it will help you.
IList<IWebElement> textfields = new List<IWebElement>();
textfields = driver.FindElements(By.TagName("iframe"));
driver.SwitchTo().Frame(textfields[count); // number of textfields list.
please try to implement as per your scenario, let me know if any question.
You can try to do :
driver.switch_to_active_element()
and then scrape the popup to close it. Then I think it will be okay...
I'm trying to swtich to an iframe in IE9 so I can send_keys() to the text area. When I do switch, I can tell that the webdriver thinks it switched (when I print page_source, it's right) but the cursor is still blinking on another textfield (not TinyMCE), at this point, if I send keys, the keys get appended to the other textfield and not to TinyMCE.
So I've been trying workarounds, If I select the the tinyMCE iframe and click(), the cursor is in the right place and I can send keys but the I can't return (switch back to the original frame/window) to submit the input.
Has anyone else run into this in IE9, are there workarounds?
This works, in Firefox and Chrome, just not IE9.
I had a problem like this once, and its quite complicated to work around it since TinyMCE generates some dynamic content. What I ended up doing to manipulate the contents of the TinyMCE editor was calling the API directly via page.execute_script and just doing it all on JavaScript.
A sample of my JS code is:
jQuery('textarea.tinymce').tinymce().setContent('test text in editor');
jQuery('textarea.tinymce').tinymce().selection.select(jQuery('textarea.tinymce').tinymce().dom.select('p')[0]);
jQuery('textarea.tinymce').tinymce().execCommand('Italic','true');
jQuery('textarea.tinymce').tinymce().execCommand('Underline','true');
jQuery('textarea.tinymce').tinymce().execCommand('Bold','true');
The first line adds text in TinyMCE's textarea, the second selects it (simulating a user cursor select), the third, fourth and fifth just manipulate the controls.
.execCommand() was particularly useful for activating the different extensions. After that I just validated that the form fields I was using were set with the expected HTML tags and called it a day.
I hope it helps!