How to scrape a dropdown list with python - python

I have to scrape the data from this web page: http://www.mlsa.am/?page_id=368. This is a dropdown list where the options are: Regions, Areas, Communities, Type of Subsidy, Month and Year. Once these options are selected a table shows up with information on the citizens of these places who get the different kinds of subsidies. The difficulties I am facing at the time to scrape it is that the "Areas" field depends on which "Region" I select, and "Communities" depends on both of them.
This is how the web page looks like when I inspect it, this information
belongs to the "Regions" (first option) cell:
`<!--Մարզեր-->
<div class="td-pb-row">
<div class="td-pb-span2"></div>
<div class="td-pb-span5">
Մարզեր <span class="ben-required">*</span>
<select id="ref_regions_id" name="ref_regions" style="border:1px solid #0790A2;" >
<option value="0" > Ընտրել </option>
<option value="1"> ԱՐԱԳԱԾՈՏՆ</option>`
<option value="2"> ԱՐԱՐԱՏ</option>
<option value="3"> ԱՐՄԱՎԻՐ</option>
<option value="4"> ԳԵՂԱՐՔՈՒՆԻՔ</option>
<option value="5"> ԼՈՌԻ</option>
<option value="6"> ԿՈՏԱՅՔ</option>
<option value="7"> ՇԻՐԱԿ</option>
<option value="8"> ՍՅՈՒՆԻՔ</option>
<option value="9"> ՎԱՅՈՑ ՁՈՐ</option>
<option value="10"> ՏԱՎՈՒՇ</option>
<option value="11"> ԵՐԵՎԱՆ</option>`
</select>
</div>
I am using selenium with python and so far this is my code:
`import time
import requests
from selenium import webdriver
from selenium.common.exceptions import WebDriverException
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
chrome_path = r"C:\Users\ivrav\selenium-2.25.0\Driver\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
print("loading url into browser...")
def get_all_pages():
payload={'value':'1'}
driver.get("http://www.mlsa.am/?page_id=368")
print(url.text)
time.sleep(2)`

sorry I can't understand the web page
you should create a selenium action chain to select every option that you need then click the button.
here is the reference

Related

How can I select item from dropdown, when option's element not interactable via selenium/python?

I want to select the item from the dropdown, but the options which are to be selected are not interactable, and show (Alert: This element is not interactable through selenium(automation) as it is not visible in UI. Try any near by element. Learn more...) alert.
Here the Html
<div class="col-md-6">
<div class="floating-label input_design p-0 mt-3">
<select class="floating-select" id="coursedd" name="title">
<option disabled="disabled" selected="selected">Select course</option>
<option value="1">Class A CDL</option>
<option value="2">Test Course Title B</option>
<option value="5">NA</option>
<option value="6">NA</option>
<option value="7">NA</option>
</select>
</div>
I tried with this method but it's not working
select_course = Select(driver.find_element(By.XPATH, "//input[#id='cpassword']"))
select_course.select_by_value('2')
This works per the HTML you have provided:
sel = Select(WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.ID, "coursedd"))))
by_val = sel.select_by_value('2')
print(sel.first_selected_option.text)
Additional imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Output:
Test Course Title B
Process finished with exit code 0

How to select a option value, click in another menu and then hit submit

I have this page with two menus and a submit button.
I would like to select an option in the first menu (companies), then select an item in the second menu (type) and finally hit the submit button (Send)
Here is the simplified HTML page:
<select name="companies" multiple="multiple" id="IDcompanies" style="width:200px;">
<option value="01">Facebook</option>
<option value="02">Oracle </option>
<option value="03">AWS</option>
<option value="04">Tesla</option>
</select>
<select name="type" id="IDtype" style="width:200px;">
<option value="T1">Type1 </option>
<option value="T2">Type2 </option>
<option value="T3">Type3</option>
</select>
<input type="submit" name="Button1" value="Send" id="ID_Button1" />
In python, I'm trying this to the first part: click on the first company (Facebook):
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get(url)
box = driver.find_element(By.ID, "IDcompanies")
box.select_by_index(0)
But gives me an error: AttributeError: 'WebElement' object has no attribute 'select_by_index'
I appreciate if someone can help with this error and guide me on how to proceed to make this sequence of clicking on the first menu, then on the second and then on the submit button.
In order to use special Selenium methods like select_by_index, select_by_value and select_by_visible_text you should define and initialize the special Selenium Select object, as following:
companies_select = Select(driver.find_element(By.ID, "IDcompanies"))
companies_select.select_by_index(0)
For more details see here

click on element in dropdown option menu - Selenium Python

I want to click on one element from a dropdown menu which has the following html code:
<select style="font-size:10px" onchange="dg_send('contractNonParticipationsDatagrid1-form', 'contractNonParticipationsDatagrid1', 'search', '/masterdata/datagridContractNonParticipations/dg_page/1/tabId/non-participations/id/1/licenseHolderId/1/dg_sort/dvec.full_name/dg_order/asc/dg_rowlimit/' + this.options[this.selectedIndex].value,false); "><option selected="selected" value="10">10</option><option value="25">25</option><option value="50">50</option><option value="100">100</option><option value="999999999">alle</option></select>
<option selected="selected" value="10">10</option>
<option value="25">25</option>
<option value="50">50</option>
<option value="100">100</option>
<option value="999999999">all</option>
I would like to select the last one, so that all are displayed.
Thank you!
To get the last option of that tag try and get it's xpath value and do the following.
from selenium.webdriver.support.select import Select
sel = Select(driver.find_element_by_xpath("//select[#style='font-size:10px']"))
sel.select_by_visible_text("all")
You could also try this
from selenium import webdriver
from shutil import which
path = which("chromedriver")
driver = webdriver.Chrome(executable_path=path)
all_btn = driver.find_element_by_xpath("//select/option[contains(text(),'all'))
all_btn.click()

element not interactable: Element is not currently visible and may not be manipulated

Newbie to selenium! I'm trying to list a house for sale on Craigslist using selenium.
And I am having trouble selecting an option from a drop-down select box.
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'D:/apps/chromedriver/chromedriver.exe')
driver.get('https://post.craigslist.org/k/tKNKfCkr6hG7ghq71YXqTA/oj7w8?s=edit')
driver.find_element_by_css_selector("select.housing_type > option[value='6']").click()
I get the following error:
ElementNotInteractableException: Message: element not interactable:
Element is not currently visible and may not be manipulated
Using "Select" also gives the same error:
from selenium.webdriver.support.ui import Select
housing_type = Select(driver.find_element_by_css_selector("select.housing_type"))
housing_type.select_by_visible_text('house')
The element is present:
housing_type = driver.find_element_by_css_selector("select.housing_type")
housing_type
But it is not intractable:
housing_type.click()
ElementNotInteractableException: Message: element not interactable
I noticed the select element is hidden and selection is controlled by next element overplayed on the select box.
<label class="json-form-item select housing_type std variant-select">
<div class="label-wrapper"><span class="label">housing type</span></div>
<select tabindex="1" name="housing_type" class="json-form-input no-js housing_type" id="ui-id-1" style="display: none;">
<option value="1" selected="">apartment</option>
<option value="2">condo</option>
<option value="3">cottage/cabin</option>
<option value="4">duplex</option>
<option value="5">flat</option>
<option value="6">house</option>
<option value="7">in-law</option>
<option value="8">loft</option>
<option value="9">townhouse</option>
<option value="10">manufactured</option>
<option value="11">assisted living</option>
<option value="12">land</option>
</select>
<span class="ui-selectmenu-button ui-widget ui-state-default ui-corner-all" tabindex="0" id="ui-id-1-button" role="combobox" aria-expanded="false" aria-autocomplete="list" aria-owns="ui-id-1-menu" aria-haspopup="true" style="width: 88%;">
<span class="ui-icon ui-icon-triangle-1-s"></span>
<span class="ui-selectmenu-text">apartment</span>
</span>
</label>
I am able to make the selection by activating the element and using down/return keys as follows, but this is not an elegant solution.
from selenium.webdriver.common.keys import Keys
housing_type = driver.find_element_by_id("ui-id-1-button")
housing_type.click()
for i in range(0,5):
housing_type.send_keys(Keys.DOWN)
housing_type.send_keys(Keys.RETURN)
Is there a better way to make the selection?
To select an item from the Housing type dropdown on the page you provided, I would first invoke WebDriverWait on the dropdown menu to ensure that it exists before you try to interact with it. Then, you can use Javascript to click the dropdown trigger and expand the options.
After that, we invoke WebDriverWait once more on the option we wish to click. The following code sample will click the option 'flat':
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# start driver
driver = webdriver.Chrome(executable_path=r'D:/apps/chromedriver/chromedriver.exe')
driver.get('https://post.craigslist.org/k/tKNKfCkr6hG7ghq71YXqTA/oj7w8?s=edit')
# wait for dropdown to exist
WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, "//label[contains(#class, 'housing_type')]")))
# expand housing type dropdown using javascript
dropdown_trigger = driver.find_element_by_xpath("//label[contains(#class, 'housing_type')]/span/span[contains(#class, 'ui-icon')]")
driver.execute_script("arguments[0].click();", dropdown_trigger)
# select an option -- this selects 'flat'
dropdown_option = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, "//li[text()='flat']")))
dropdown_option.click()
Unfortunately, this seems to be a problem with how the overlay/dropdown is being used.
I noticed the select element is hidden and selection is controlled by next element overplayed on the select box.
Since the visibility of the dropdown is controlled by some overlay, you'll be forced to click on the dropdown button in order to activate the overlay and make the dropdown button visible. After the dropdown button becomes visible, you should be able to use the Selenium code without further issue. Namely, this snippet of code should be good to use:
housing_type = Select(driver.find_element_by_css_selector("select.housing_type"))
housing_type.select_by_visible_text('house')
I suggest using this approach, rather than your alternative of using the Keys.DOWN actions, as this is a better use of Selenium's built-in functions

I am trying to get the XPath for a drop down

I am trying to get the XPath for a drop down field which I can use in my Webdriver, Python code.
There are two drop down fields in the html. I need to identify the drop down which has the options, Person Name, Organisation Name, Address etc.
I would like to select Address value from the drop down.
I have tried the following XPath:
//select[#class="gwt-ListBox marginbelow"]
It identifies both the drop down fields.
<div class="clear">
<span class="gwt-InlineLabel marginbelow myinlineblock" style="width: 8em;">Type</span>
<select class="gwt-ListBox marginbelow" style="display: inline;">
<option value="Person name">Person name</option>
<option value="Organisation name">Organisation name</option>
<option value="Address">Address</option>
<option value="Date">Date</option>
<option value="Email">Email</option>
<option value="Phone">Phone</option>
<option value="Integer">Integer</option>
<option value="Numeric">Numeric</option>
<option value="String">String</option>
<option value="User-defined">User-defined</option>
</select>
</div>
<div class="clear">
<div class="clear">
<div class="clear">
<div style="display: none;" aria-hidden="true">
<div class="clear">
<span class="gwt-InlineLabel marginbelow myinlineblock" style="width: 8em;">TestDB</span>
<select class="gwt-ListBox marginbelow" style="display: inline;">
<option value="paf_uk15051">paf_uk15051</option>
</select>
</div>
</div>
I will then use the XPath like this, e.g.
from selenium.webdriver.support import By
from selenium.webdriver.support.ui import select
data_objects_element_Type.Select(self.driver.find.element(By.XPATH, '//select[#class="gwt-ListBox marginbelow"]'))
data_objects_element_Type.select_by_visible_text(("Address"))
Thanks for suggestions.
Riaz
If you can, I suggest editing the HTML, to contain ID's, so that you can specifically identify your two select boxes separately. If you cannot, here is a solution. I have made a jsfiddle site with your html-code, so that you can execute the script and see for yourself. The only thing you need to change, is the path, and choice of webdriver:
from selenium.webdriver.chrome.webdriver import WebDriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.common.by import By
driver = WebDriver("path_to_webdriver")
driver.get("http://jsfiddle.net/ksftLk9j/")
wait = WebDriverWait(driver, 20, 2)
wait.until(expected_conditions.frame_to_be_available_and_switch_to_it((By.NAME, 'result')))
element = driver.find_element_by_xpath('//select[#class="gwt-ListBox marginbelow"]/option[#value="Address"]')
element.click()
the WebDriverWait, and switch_to_frame-statements are specific for this jsfiddle-case. The two bottom lines are not. You do not say anything about what you want to do with the element, but I'm gonna go out on a limb, and assume that you want to select it. Presto.
You can try by checking the contains function.
Like this:
//option[contains(text(),'Address')]
Try this to find the element and perform click on it directly. This allows you to bypass the use of Select class
//select/option[#value='Address']
you can try it this way :
//select[#class="gwt-ListBox marginbelow"]/option[#value = 'Address']/text()
Try this:
//select[contains(#class, 'ListBox')]/option[#value = 'Address']
So you code will be:
from selenium.webdriver.support import By
from selenium.webdriver.support.ui import select
data_objects_element_Type.Select(self.driver.find.element(By.XPATH, '//select[contains(#class, 'ListBox')]'))
data_objects_element_Type.select_by_visible_text(("Address"))
^ I don't know python, but you need method smth. like selectByValue

Categories