I'm trying to store address values generated by a postcode lookup and then create a list that I can use the python Random module to select a random value using random.choice
Scenario:
Enter postcode, click 'Search' - dropdown list is dynamically populated with available options.
I'm using a dictionary to store my form values as xpaths and then using the webdriver to find_elements_by_xpath or find_element_by_xpath.
Code looks something like this (not properly formatted, just reference):
__author__ = 'scott'
from selenium import webdriver
import random
driver = webdriver.Firefox()
driver.maximize_window()
driver.get('https://www.somerandomsite')
formFields = {'postcode' : "//INPUT[#id='postcode']",
'county' : "//SELECT[#id='address']/option"}
pcList = ['BD23 1DN', 'BD20 0JZ']
#picks a random postcode from pcList#
driver.find_element_by_xpath(formFields['postcode']).send_keys(random.choice(pcList))
driver.find_elements_by_xpath(formFields['county'])
#####now need to store the values from county and select a random option from the list######
driver.close()
Using the random module on my postcodes isn't a problem.
If anyone can give some guidance or point me in a direction to take for reference it would be much appreciated - I'm only a noob to Selenium and Python - made steady progress but seem to be going round in circles on this problem - my first issue was using find_element_by_xpath a simple 's' missing off 'element' threw me for a while.
Use the Select class provided by the python selenium bindings out-of-the-box - it is a nice abstraction layer over select->option HTML structures:
from selenium.webdriver.support.ui import Select
# initialize the select instance
select = Select(driver.find_element_by_id('address'))
# get the list of options and choose a random one
options = [o.text for o in select.options]
option = random.choice(options)
# select it
select.select_by_visible_text(option)
Related
I am writing a python script which will call a webpage and will select an option from the drop down to download that file. To do this task, I am using chropath. It is a browser extension which can give you the relative xpath or id for any button or field on the webpage and using that we can call it from python selenium script.
Above image shows the drop down menu in which I have to select 2019 as year and the download the file. In the lower part of the image, you can see that I have used chropath to get the relative xpath of the drop down menu which is //select[#id='rain']
Below is the code I am using:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("<URL>")
driver.maximize_window()
grbf = driver.find_element_by_xpath("//select[#id='rain']")
grbf.send_keys('2019')
grbf_btn = (By.XPATH, "//form[1]//input[1]")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(grbf_btn)).click()
from the above code, you can see that I am using xpath to select the drop down grbf = driver.find_element_by_xpath("//select[#id='rain']") and then sending keys as 2019 i.e. grbf.send_keys('2019') and after that I am calling download button to download it. But for some reason, its always selecting year 1999 from the drop down. I am not able to understand what is wrong in this. Is this correct approach to solve this. Please help. Thanks
I had the same problem time ago. Try this:
from selenium.webdriver.support.ui import Select
grbf = Select(driver.find_element_by_xpath("//select[#id='rain']"))
grbf.select_by_value('2019')
In the select_by_value() you have to use the value of the element in the dropdown.
By the way, if an element has id, use it.
grbf = Select(driver.find_element_by_id('rain'))
Try below code:
select = Select(driver.find_element_by_xpath("//select[#id='rain']"))
select.select_by_visible_text('2019')
Another approches to deal with dropdown:
Using Index of Dropdown :
select.select_by_index(Paass index)
Using valueof Dropdown :
select.select_by_value('value of element')
Using visible text of Dropdown :
select.select_by_visible_text('element_text')
In my opinion, I don't think this is the correct approach. You try to select the option which is dropdown (not a text box like ), so send key command does not work.
What you need to do is try to inspect HTML changing when clicking the dropdown and try to XPath for an option that you want to select.
If you still stuck at this problem, I recommend using katalon recorder which is a chrome extension to allow you to record and do UI testing
So I am quite new to using Selenium and thus and quite unsure how to do this, or even word it for this matter.
But what I am trying to do is to use selenium to grab the following values and then store them into a list.
Image provided of the inspector window of Firefox, to show what I am trying to grab (Highlighted)
https://i.stack.imgur.com/rHk9R.png
In Selenium, you access elements using functions find_element(s)_by_xxx(), where xxx is for example the tag name, element name or class name (and more). The functions find_element_... return the first element that matches the argument, while find_elements_... return all matching elements.
Selenium has a [good documentation][1], in section "Getting started" you can find several examples of basic usage.
As to your question, the following code should collect the values you want:
from selenium import webdriver
driver = webdriver.Firefox() # driver for the browser you use
select_elem = driver.find_element_by_name('ctl00_Content...') # full name of the element
options = select_elem.find_elements_by_tag_name('option')
values = []
for option in options:
val = option.get_attribute('value')
values.append(val)
I am new in python and scraping.
I am trying to extract information about Tripadvisor. First of all, I need Selenium for crawling but when I run the program in diferents times the paths change.
I show you a example:
import urllib.request
import urllib.parse
from selenium import webdriver
import csv
from selenium.webdriver.common.action_chains import ActionChains
import time
from _datetime import datetime
from selenium.webdriver.common.keys import Keys
options=webdriver.ChromeOptions()
options.headless=False
prefs={"profile.default_content_setting_values.notofications" :2}
options.add_experimental_option("prefs",prefs)
chromedriver = "C:/Users/rober/OneDrive/Escritorio/tfm/chromedriver.exe"
driver=webdriver.Chrome(chromedriver)
driver.maximize_window()
time.sleep(5)
driver.get("https://www.tripadvisor.es/")
//*[#id="component_5"]/div/div/div/span[3]/div/div/div/a/span[2]
#Click Restaurants
driver.find_element_by_xpath('//*[#id="component_5"]/div/div/div/span[3]/div/div/div/a').click()
#Introduce localization
driver.find_element_by_xpath('//*[#id="BODY_BLOCK_JQUERY_REFLOW"]/div[14]/div/div/div[1]/div[1]/div/input').send_keys("madrid")
In the last part of code, sometimes div[14] is div[13] or div[15]. is it possible absolute xpath or use other form?
Thank you
You should not use Xpath with a longer path. That makes the test brittle
Please use shorter xpaths. An Xpath like this "//input[#class="Smftgery"]" should help you click on the same input field.
Also to click on Restaurantes, you can use //*[text()='Restaurantes']
Your Xpath is too specific, find some uniqueness in the deeper levels of the DOM. This uniqueness can be also a combination of multiple levels.
e.g. if there is only one input field inside BODY_BLOCK_JQUERY_REFLOW you can ignore all the levels in between:
'//*[#id="BODY_BLOCK_JQUERY_REFLOW"]//input'
or use some other attribute of input e.g. if it has a data attribute://input[#data="the-data-of-the-input-field"]
I am trying to simulated clicking through multiple options on an online data tool that ends with downloading an excel sheet given your filters.
I am currently using selenium and identifying xpaths.
I am able to get through a single iteration and get a single excel sheet, but I need to do it for every possible permutation of drop down choices. To do by hand is unrealistic, as there are thousands of options.
The website for context: https://data.cms.gov/mapping-medicare-disparities
Does anyone know of a function that can be done in selenium that will work?
My current strategy is to create lists with the xpaths and then try to do a permutation function to get all the combinations. However, this has not worked because the function: b.find_element_by_xpath only allows one xpath at a time.
examples of lists:
geography county state/territory
G1 = '//select[#id="geography"]//option[#value="c"]'
G2 = '//select[#id="geography"]//option[#value="s"]'
Geo = [G1, G2]
creating pool of combinations
import itertools
from itertools import product
for perm in product(Geo, Adjust, Analysis, Domain):
print(perm)
actual code to use selenium
**from** selenium **import** webdriver
**from** selenium.webdriver.common.keys **import** Keys
b = webdriver.Firefox()
code to click through a popup
pop_up = b.find_element_by_xpath('/html/body/div[1]/button')
pop_up.click()
code trying to use xpath to select all options at once
b.find_element_by_xpath(('//select[#id="geography"]//option[#value="c"],
'//select[#id="adjust"]//option[#value="1"],'//select[#id="analysis"]
//option[#value="base"],'//select[#id="domain"]//option[#value="d1"]'))
error message: InvalidArgumentException: Message: invalid type: sequence, expected a string at line 1 column 28
This is because the find_element_by_xpath (I am assuming) will only look at 1 xpath at a time.
your syntax in code trying to use xpath... is wrong anyway, but you could just put all the xpaths in a list and loop through it.
xpathlist=['//select[#id="geography"]//option[#value="c"]', '//select[#id="adjust"]//option[#value="1"]',.....]
for xp in xpathlist:
b.find_element_by_xpath(xp)
#then add code to click or download or whatever
I am trying to use Selenium, python, and firefox to enter data into an input box on a website and select from a dropdown menu in the text box, but have been unable to do so. Formerly it was "just" a text box, but the website was changed.
The website is located at http://ecos.fws.gov/ecp/
the input textbox ID is "searchTerm". The site allows one to enter a scientific name (or part of a scientific name) and then select from options. For example, if you enter "Acipenser brevirostrum" into the textbox, it will provide you with a single option to click on.
Unfortunately, not sure how to code this up. Any help would be appreciated. So far I have this.
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.support.ui import Select
binary = FirefoxBinary(r'C:/Program Files (x86)/Mozilla Firefox/firefox.exe')
driver = webdriver.Firefox(firefox_binary=binary)
driver.get("http://ecos.fws.gov/ecp")
SciName = driver.find_element_by_id('searchTerm')
SciName.send_keys(names)
SciName.send_keys(Keys.RETURN)
The last three lines used to work, but now that it is both an input box with a drop down menu, it fails.
Any help would be appreciated.
After you enter the name in the search box (id -- searchTerm), check for the visibility of the div (class -- autocomplete-suggestions) with an explicit wait. This contains the suggestions that the site throws up. Then you have to find the option which works for you. Try this xpath and click on the element. You should use findelements option as your text might not match anything and you will get an error.
"//div[#class='autocomplete-suggestion'][contains(.,'Acipenser brevirostrum')]"
If you have problems getting the autocomplete dropdown when you type the search text, send the text one character at a time with a small interval in between to trigger the call.
you can simply use driver.find_element_by_name,as html is:
<input type="text" class="form-control" name="query" placeholder="Search ECOS">
This code will work:
from selenium import webdriver
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("http://ecos.fws.gov/ecp/")
driver.find_element_by_name("query").click()
driver.find_element_by_name("query").send_keys("test")
driver.find_element_by_css_selector("button.btn.btn-default").click()