I am trying to simulated clicking through multiple options on an online data tool that ends with downloading an excel sheet given your filters.
I am currently using selenium and identifying xpaths.
I am able to get through a single iteration and get a single excel sheet, but I need to do it for every possible permutation of drop down choices. To do by hand is unrealistic, as there are thousands of options.
The website for context: https://data.cms.gov/mapping-medicare-disparities
Does anyone know of a function that can be done in selenium that will work?
My current strategy is to create lists with the xpaths and then try to do a permutation function to get all the combinations. However, this has not worked because the function: b.find_element_by_xpath only allows one xpath at a time.
examples of lists:
geography county state/territory
G1 = '//select[#id="geography"]//option[#value="c"]'
G2 = '//select[#id="geography"]//option[#value="s"]'
Geo = [G1, G2]
creating pool of combinations
import itertools
from itertools import product
for perm in product(Geo, Adjust, Analysis, Domain):
print(perm)
actual code to use selenium
**from** selenium **import** webdriver
**from** selenium.webdriver.common.keys **import** Keys
b = webdriver.Firefox()
code to click through a popup
pop_up = b.find_element_by_xpath('/html/body/div[1]/button')
pop_up.click()
code trying to use xpath to select all options at once
b.find_element_by_xpath(('//select[#id="geography"]//option[#value="c"],
'//select[#id="adjust"]//option[#value="1"],'//select[#id="analysis"]
//option[#value="base"],'//select[#id="domain"]//option[#value="d1"]'))
error message: InvalidArgumentException: Message: invalid type: sequence, expected a string at line 1 column 28
This is because the find_element_by_xpath (I am assuming) will only look at 1 xpath at a time.
your syntax in code trying to use xpath... is wrong anyway, but you could just put all the xpaths in a list and loop through it.
xpathlist=['//select[#id="geography"]//option[#value="c"]', '//select[#id="adjust"]//option[#value="1"]',.....]
for xp in xpathlist:
b.find_element_by_xpath(xp)
#then add code to click or download or whatever
Related
I'm having some trouble finding elements with Selenium in Python, it works fine for every element on all other websites I have tested yet on a game website it can only find certain elements.
Here is the code I'm using:
from selenium import webdriver
import time
driver = webdriver.Chrome("./chromedriver")
driver.get("https://www.jklm.fun")
passSelf = input("Press enter when in game...")
time.sleep(1)
syllable = driver.find_element_by_xpath("/html/body/div[2]/div[2]/div[2]/div[2]/div").text
print(syllable)
Upon running the code, the element /html/body/div[2]/div[2]/div[2]/div[2]/div isn't found. In the image you can see the element it is trying to find:
Element the code is trying to find
However running the same code but replacing the XPath with something outside of the main game (for example the room code in the top right) it successfully finds the element:
Output of the code being run on a different element
I've tried using the class name, name, selector and XPath to find the original element but no prevail the only things I can think that are affecting it is that:
The elements are changing periodically (not sure if this affects it)
The elements are in the "Canvas area" and it is somehow blocking it.
I'm not certain whether these things matter as I'm new to using selenium any help is appreciated. The website the game is on is https://www.jklm.fun/ if you want to have a look through the elements
Element you are trying to access is inside an iframe. Switch to the frame first like this
driver.switch_to_frame(driver.find_element_by_xpath("//div[#class='game']/iframe[contains(#src,'jklm.fun')]"))
driver.get("https://jklm.fun/JXUS")
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.XPATH, "//button[#class='styled']"))).click()
time.sleep(10)
driver.switch_to.frame(0)
while True:
Get_Text = driver.find_element_by_xpath("//div[#class='round']").text
print(Get_Text)
I am attempting to get a list of games on
https://www.xbox.com/en-US/live/gold#gameswithgold
According to Firefox's dev console, it seems that I found the correct class: https://i.imgur.com/M6EpVDg.png
In fact, since there are 3 games, I am supposed to get a list of 3 objects with this code: https://pastebin.com/raw/PEDifvdX (the wait is so Seleium can load the page)
But in fact, Selenium says it does not exist: https://i.imgur.com/DqsIdk9.png
I do not get what I am doing wrong. I even tried css selectors like this
listOfGames = driver.find_element_by_css_selector("section.m-product-placement-item f-size-medium context-game gameDiv")
Still nothing. What am I doing wrong?
You are trying to get three different games so you need to give different element path or you can use some sort of loop like this one
i = 1
while i < 4:
link = f"//*[#id='ContentBlockList_11']/div[2]/section[{i}]/a/div/h3"
listGames = str(driver.find_element_by_xpath(link).text)
print(listGames)
i += 1
you can use this kind of loop in some places where there is slight different in xpath,css or class
in this way it will loop over web element one by one and get the list of game
as you are trying to get name I think so you need to put .text which will only get you the name nothing else
Another option with a selector that isn't looped over and changed-- also one that's less dependent on the page structure and a little easier to read:
//a[starts-with(#data-loc-link,'keyLinknowgame')]//h3
Here's sample code:
from selenium import webdriver
from selenium.common.exceptions import StaleElementReferenceException
driver = webdriver.Chrome()
url = f"https://www.xbox.com/en-US/live/gold#gameswithgold"
driver.get(url)
driver.implicitly_wait(10)
listOfGames = driver.find_elements_by_xpath("//a[starts-with(#data-loc-link,'keyLinknowgame')]//h3")
for game in listOfGames:
try:
print(game.text)
except StaleElementReferenceException:
pass
If you're after more than just the title, remove the //h3 selection:
//a[starts-with(#data-loc-link,'keyLinknowgame')]
And add whatever additional Xpath you want to narrow things down to the content/elements that you're after.
So I am quite new to using Selenium and thus and quite unsure how to do this, or even word it for this matter.
But what I am trying to do is to use selenium to grab the following values and then store them into a list.
Image provided of the inspector window of Firefox, to show what I am trying to grab (Highlighted)
https://i.stack.imgur.com/rHk9R.png
In Selenium, you access elements using functions find_element(s)_by_xxx(), where xxx is for example the tag name, element name or class name (and more). The functions find_element_... return the first element that matches the argument, while find_elements_... return all matching elements.
Selenium has a [good documentation][1], in section "Getting started" you can find several examples of basic usage.
As to your question, the following code should collect the values you want:
from selenium import webdriver
driver = webdriver.Firefox() # driver for the browser you use
select_elem = driver.find_element_by_name('ctl00_Content...') # full name of the element
options = select_elem.find_elements_by_tag_name('option')
values = []
for option in options:
val = option.get_attribute('value')
values.append(val)
I'm new in Selenium with Python. I'm trying to scrape some data but I can't figure out how to parse outputs from commands like this:
driver.find_elements_by_css_selector("div.flightbox")
I was trying to google some tutorial but I've found nothing for Python.
Could you give me a hint?
find_elements_by_css_selector() would return you a list of WebElement instances. Each web element has a number of methods and attributes available. For example, to get an inner text of the element, use .text:
for element in driver.find_elements_by_css_selector("div.flightbox"):
print(element.text)
You can also make a context-specific search to find other elements inside the current element. Taking into account, that I know what site you are working with, here is an example code to get the departure and arrival times for the first-way flight in a result box:
for result in driver.find_elements_by_css_selector("div.flightbox"):
departure_time = result.find_element_by_css_selector("div.departure p.p05 strong").text
arrival_time = result.find_element_by_css_selector("div.arrival p.p05 strong").text
print [departure_time, arrival_time]
Make sure you study Getting Started, Navigating and Locating Elements documentation pages.
I'm trying to store address values generated by a postcode lookup and then create a list that I can use the python Random module to select a random value using random.choice
Scenario:
Enter postcode, click 'Search' - dropdown list is dynamically populated with available options.
I'm using a dictionary to store my form values as xpaths and then using the webdriver to find_elements_by_xpath or find_element_by_xpath.
Code looks something like this (not properly formatted, just reference):
__author__ = 'scott'
from selenium import webdriver
import random
driver = webdriver.Firefox()
driver.maximize_window()
driver.get('https://www.somerandomsite')
formFields = {'postcode' : "//INPUT[#id='postcode']",
'county' : "//SELECT[#id='address']/option"}
pcList = ['BD23 1DN', 'BD20 0JZ']
#picks a random postcode from pcList#
driver.find_element_by_xpath(formFields['postcode']).send_keys(random.choice(pcList))
driver.find_elements_by_xpath(formFields['county'])
#####now need to store the values from county and select a random option from the list######
driver.close()
Using the random module on my postcodes isn't a problem.
If anyone can give some guidance or point me in a direction to take for reference it would be much appreciated - I'm only a noob to Selenium and Python - made steady progress but seem to be going round in circles on this problem - my first issue was using find_element_by_xpath a simple 's' missing off 'element' threw me for a while.
Use the Select class provided by the python selenium bindings out-of-the-box - it is a nice abstraction layer over select->option HTML structures:
from selenium.webdriver.support.ui import Select
# initialize the select instance
select = Select(driver.find_element_by_id('address'))
# get the list of options and choose a random one
options = [o.text for o in select.options]
option = random.choice(options)
# select it
select.select_by_visible_text(option)