We Keep Coding

Selenium Webdriver returns None when trying to access text

I'm trying to scrape some data from a website and I can't seem to get the text in between two tags as it keeps returning None. This is my code, could somebody tell me what is wrong?
uq = "https://my.uq.edu.au/programs-courses/requirements/program/2451/2021"
driver = webdriver.Firefox(executable_path="/Users/Connor/Downloads/geckodriver")
driver.get(uq)
groups = driver.find_element_by_xpath("/html/body/div/div[2]/div/div/div[3]/div[1]/div[2]/div[4]/div[1]/div[2]/a[1]/span[1]")
print(groups)
print(groups.text) #This won't get the text between the span tags e.g. <span>hello</span>

To get that text you'll need to click the parent element first.
However, you can avoid it with .get_attribute("innerHTML") and waiting for the presence of the element. Also, make sure you are using stable locators.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
options = FirefoxOptions()
options.add_argument('window-size=1920x1080')
driver = webdriver.Firefox(options=options)
uq = "https://my.uq.edu.au/programs-courses/requirements/program/2451/2021"
driver.get(uq)
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[title='Theory of Computing']>span:nth-of-type(1)")))
group = driver.find_element_by_css_selector("a[title='Theory of Computing']>span:nth-of-type(1)").get_attribute("innerHTML")
print(group)
Prints: COMP2048

You have to set an action to get the value of the WebElement.
So you have to use this code instead:
print(groups.getText());

Yes, same for me. I have tried every way possible. Trying this:
print(speed.get_attribute('outerHTML'))
I get
<span data-download-status-value="NaN" class="result-data-large number result-data-value download-speed"> </span>

How can I get this SPECIFIC element in selenium in python

UPDATE:
So thanks to the voted answer it displayed some information not the right information, it shows 0kb out of 100 and when in the inspect element console if doing console.log($0) then the item would be displayed in console how do I fetch this
I want to create a python 3.x programme that gets my stats off of netlify and easybase using selenium. The issue I have come across already is that the element does not have a specific class name and the text widget isn't just a tag nor a tag. Here is a screenshot of the html of netlify the screenshot, and this is the code that I used
element = driver.find_element_by_name("github")
element.click()
login = driver.find_element_by_name("login")
login.send_keys(email)
password = driver.find_element_by_name("password")
password.send_keys(passwordstr)
loginbtn = driver.find_element_by_name("commit")
loginbtn.click()
getbandwidth = driver.find_element_by_xpath('//*[#id="main"]/div/div[1]/div/section/div/div/div/dl/div/dd')
print(getbandwidth.text)

getbandwidth = driver.find_element_by_xpath("//dd[#class='tw-text-xl tw-mt-4px tw-leading-none']")
You can use this to grab the first xpath with that class. Below does the same but if you want to index other elements with similar classes.
(//dd[#class='tw-text-xl tw-mt-4px tw-leading-none'])[1]
Normally we use webdriver waits to allow for the element to become visible.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
getbandwidth = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,"//dd[#class='tw-text-xl tw-mt-4px tw-leading-none']")))

Finding dynamically named elements on a page using Selenium

I'm trying to locate and click on a section of a web page using Selenium so I can add a comment. I'm having a bit of trouble figuring out how to do this, though.
It seems like the class of the element changes from page-to-page. It also doesn't help that there are multiple similar elements in the document. Here's what I've come up so far:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=/Users/me/Library/Application/Support/Google/Chrome/Default")
chrome_options.add_argument('--profile-directory=Profile 1')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("https://link-i-want-to-visit")
comment_section = driver.find_elements_by_xpath(("//input"))
print comment_section
comment_section.click()
Here's some of the markup from the relevant page:
<input class="sc-iKpIOp igoGaM" placeholder="Add a comment…">
On each different URL on this site, the class name appears to change. How can I circumvent that limitation, click into the input field, and send my comment?
Any guidance would be much appreciated. If it helps, this input field appears to be the last on the page, but I don't know if that's relevant (semantically).

Can't you use absolute xpath not including class?
e.g. /html/body/div/div/div/div/div/div/div/div/div/p[1] is your "I'm trying to..." paragraph on this page.
Also, try //input[placeholder="Add a comment…"]

comment_section = driver.find_elements_by_xpath("//input")
This will return list.So you can't click on list. You should use driver.find_element_by_xpath("//input") to click on the element.
However for a best practice Use WebDriverWait and Wait for the element element_to_be_clickable
and then click.
comment_section=WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,"//input[#placeholder='Add a comment…']")))
comment_section.click()
OR
comment_section=WebDriverWait(driver,30).until(EC.element_to_be_clickable((By.XPATH,"//input[contains(#placeholder,'Add a comment')]")))
comment_section.click()
You need to use following imports to execute above code.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Python with selenium webscraping unable to find elements

I am trying to write a webscraping in python that will activate the "onclick" functionality of certain buttons on a webpage because the tables with the data I want are converted to csv, which makes it much easier to access. But the problem is that I am unable to locate elements by xpath at all when using PhantomJs. How can I click the element and access the csv content that I want?
This is my code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import *
url = "http://www.pro-football-reference.com/boxscores/201609180nwe.htm"
xpath = "//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button"
path_to_phantomjs = 'browser/phantomjs'
browser = webdriver.PhantomJS(executable_path = path_to_phantomjs)
browser.get(url)
delay=3
element_present = EC.presence_of_element_located((By.ID, 'all_player_offense'))
WebDriverWait(browser, delay).until(element_present)
browser.find_element_by_xpath(xpath).click()
And I get this error:
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"153","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50989","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"sessionId\": \"93ff24f0-9cbe-11e6-8711-bdfa3ff9cfb1\", \"value\": \"//*[#id='all_player_offense']/div[1]/div/ul/li[1]/div/ul/li[3]/button\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/93ff24f0-9cbe-11e6-8711-bdfa3ff9cfb1/element"}}
Screenshot: available via screen

IMPORTANT THING I FORGOT TO MENTION: As described in this this issue on GitHub, try putting set_window_size(width, height) or maximize_window()after setting the webdriver. You should also consider telling the webdriver to implicitly_wait(10) for the element to appear.
So there's a special maneuver you have to perform in order for the Selenium Webdriver to properly emulate what you're doing. In essence, to get the desired data, you'd have to:
A: Hover over the "Share & More" dropdown menu. Then
B: Click "Get table as CSV (Excel)".
For A, this involves having to place the emulated cursor on the element without clicking it. This idea of "mouse over" can be done with the move_to_element() function provided in the ActionChains class. So at the top you'd insert this:
from selenium.webdriver.common.action_chains import ActionChains
You want Selenium to find the specific element and move to it. You can achieve this with 2 lines of code:
dropdown = browser.find_element_by_xpath('//*[#id="all_player_offense"]/div[1]/div/ul/li[1]')
ActionChains(browser).move_to_element(dropdown).perform()
If you omit the above, you'll get an ElementNotVisibleException.
Now for B, you should be able to do browser.find_element_by_xpath(xpath).click().

Python: Selenium send_key not working

I am trying to use Selenium in Python, as as I am a beginner in doing so I cannot get send_key to work, but most probably it is straight forward and I am missting something.
Here is an example of what I have done so far:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://semantria.com/demo")
item = driver.find_element_by_id("analyze_url_form")
item.send_keys("http://finance.yahoo.com/news/skystar-bio-pharmaceutical-company-provides-133000048.html")
go_button = driver.find_element_by_id("analyze_url_button")
go_button.click()
The idea is that in the https://semantria.com/demo website, there is an empty space that one can enter a website link, and then click on the Go button.
However, it looks like my code does not do this.
Am I doing something wrong? Does this website do something that I should be aware of and change my code accordingly?
Any help on this is really appreciated.

The problem is that you are sending keys to the form element, not the input element inside.
Plus, you can just send the URL with a new line at the end which is the same as you've entered the URL and pressed ENTER key which results in the form being submitted. Works for me:
item = driver.find_element_by_css_selector("form#analyze_url_form input[name=link]")
item.send_keys("http://finance.yahoo.com/news/skystar-bio-pharmaceutical-company-provides-133000048.html" + "\n")
As a bonus, here is how you can grab the sentiment value (you have to let selenium know what to wait for via WebDriverWait and Expected Conditions):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("https://semantria.com/demo")
item = driver.find_element_by_css_selector("form#analyze_url_form input[name=link]")
item.send_keys("http://finance.yahoo.com/news/skystar-bio-pharmaceutical-company-provides-133000048.html" + "\n")
wait = WebDriverWait(driver, 30)
sentiment_value = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "strong.sentiment_score_value")))
print(sentiment_value.text)
Prints positive (+0.230).