We Keep Coding

Dynamic content not rendered when scraping using selenium

I am trying to use selenium for scraping (the script used to work in python 3.7).
Last week I had to reset my PC and I installed the latest versions of python and all the packages used in the script.
What I observed was that none of the dynamic values are getting rendered and are displayed with header tags. Please see below some of the outputs:
<tr>
<td class="textsr">Close</td>
<td class="textvalue">{{ScripHeaderData.Header.Close}}</td>
</tr>
<tr>
<td class="textsr">WAP</td>
<td class="textvalue">{{StkTrd.WAP}}</td>
</tr>
<tr>
<td class="textsr">Big Value</td>
<td class="textvalue">{{checknullheader(CompData.BigVal)?'-':(CompData.BigVal)}}</td>
</tr>
I have been using the script for my research purpose and need it back in shape, hence appreciate any guidance.
Here's the snippet for reference:
target_url = q.get(timeout=1)
time.sleep(1)
driver = webdriver.Chrome('./chromedriver',options=opts)
driver.get(target_url)
# this is just to ensure that the page is loaded
time.sleep(5)
html_content = driver.page_source
soup = BeautifulSoup(html_content, features="html.parser")
table_rows = soup.find_all('tr')
for row in table_rows:
table_cols = row.find_all('td')
for col in table_cols:
label_value = col.text

I had referred a lot of forums and tried many suggestions (waits, driver options, changing web drivers, switching content etc.) however my issue seems to be more specific and did not get resolved.
Eventually fell back to my old setup (runs python 3.9.6) and then it went back to working state.
Thanks to you Joe Carboni for your time and inputs on this.
It is a bit frustrating that I could not find the root cause of the issue and a workaround to resolve it. But just posting what I did here in case if it helps someone, cheers.

While it may be tempting to use time.sleep to wait for the page to load, it's better to use Selenium Waits with conditions to wait for, likely related to the elements you want.
https://www.selenium.dev/documentation/webdriver/waits/
Here's another thread with a good answer about Waits and conditions vs. time.sleep: How to sleep Selenium WebDriver in Python for milliseconds

Python Selenium - Cannot find element on page

I've perused SO for quite a while and cannot find the exact or similar solution to my current problem. This is my first post on SO, so I apologize if my formatting is off.
The Problem -
I'm trying to find a button on a webpage to punch me into a timeclock automatically. I am able to sign in and navigate to the correct page (it seems the page is dynamically loaded, as switching from different tabs like "Time Management" or "Pay Period" do not change the URL).
Attempts to solve -
I've tried using direct and indirect XPaths, CSS Selectors, IDs, Classes, Names, and all have failed. Included below are the different code attempts to find the button, and also a snippet of code including the button.
Button - HTML
Full Page HTML Source Code
<td>
<a onclick="return OnEmpPunchClick2(this);" id="btnEMPPUNCH_PUNCH" class="timesheet button icon " href="javascript:__doPostBack('btnEMPPUNCH_PUNCH','')">
<span> Punch</span></a>
<input type="hidden" name="hdfEMPPUNCH_PUNCH" id="hdfEMPPUNCH_PUNCH" value="0">
</td>
Attempts - PYTHON - ALL FAIL TO FIND
#All these return: "Unable to locate element"
self.browser.find_element_by_id("btnEMPPUNCH_PUNCH")
self.browser.find_element_by_xpath("//a[#id='btnEMPPUNCH_PUNCH']")
self.browser.find_element_by_css_selector('#btnEMPPUNCH_PUNCH')
#I attempted a manual wait:
wait=WebDriverWait(self.browser,30)
button = wait.until(expected_conditions.element_to_be_clickable((By.CSS_SELECTOR,'#btnEMPPUNCH_PUNCH')))
#And even manually triggering the script:
self.browser.execute_script("javascript:__doPostBack('btnEMPPUNCH_PUNCH','')")
self.browser.execute_script("__doPostBack('btnEMPPUNCH_PUNCH','')")
#Returns Message: ReferenceError: __doPostBack is not defined
None of these work, and I cannot seem to figure out why that is. Any help will be greatly appreciated!

Website submit button and python, no action attribute

I am building a program in Python that interacts with an online store. So far I am able to find the desired item and navigate to the page using BeautifulSoup, but I am having issues clicking the "Add to cart" button. Most of the solutions I've found online using robobrowser and similar would work except that they are dealing with the tag which has a method attribute. The for on the site I am dealing with looks like this:
<input class="button" name="commit" type="submit" value="add to cart">
How would I go about "clicking" this button? What libraries would I need. I'm using python 3 by the way so I can't use mechanize. Thanks in advance for the help.

You can consider using Selenium in Python.
Please use the code snippet below as a reference:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("url")
button = driver.find_element_by_css_selector("input[class='button']")
button.click()
In case you get multiple matches, you can narrow it down by involving more attributes:
button = driver.find_element_by_css_selector("input[class='button'][name='commit']")
Please refer to this link for more examples on Python Selenium.
http://selenium-python.readthedocs.io/locating-elements.html

Cannot switch to a frame

I am having super difficulty understanding a problem I have while working on automating a page using Chromedriver. I am in the login page and here is how the HTML for the page looks:
<frame name="mainFrame" src>
<body>
<table ..>
<tr>
<td ..>
<input type="password" name="ui_pws">
</td>
..
..
..
</frame>
This is gist, the page of course has multiple tables, divs, etc ...
I am trying to enter the password in the input element using xpath //input[#name="ui_pws"].
But the element was not found.
So I thought it might be because of wrong frame and I tried:
driver.switch_to_frame('mainFrame')
and it failed with NoSuchFrameException.
So I switched to:
main_frame = driver.find_element_by_xpath('//frame[#name="mainFrame"]')
driver.switch_to_frame(main_frame)
Then to cross verify I got the current frame element using:
current_frame = driver.execute_script("return window.frameElement")
And to my surprise I got two different elements when printed it out.
Now I am really confused as to what I should be doing to switch frames or access the password field in the webpage. I have had 4 cups of coffee since morning and still have a brain freeze.
Can anyone please guide me with this?

You can try, this is in Java should be almost similar in python too
driver.switchTo().defaultContent();
WebElement frameElement = driver.findElement(By.xpath("//frame[#name='mainFrame']"));
drive.switchTo().frame(frameElement);
SwitchTo defaultContent helps bring in focus properly, and later we can switch to the desired frame in the window.

driver.switchTo().frame(driver.findElement(By.xpath("//frame[# name='mainFrame']")));
//perform operation which you want to perform on web elements present inside the frame(mainFrame), once you finish your operation come back to default
driver.switchTo().defaultContent();

Choose element by partial ID using Selenium with python?

I am trying to reach an web element in Selenium in Python 2.7. The element ID is like this:
cell.line.order(240686080).item(250444868).unitCost
The first number string '240686080' is known, but the second number '250444868' is unknown beforehand.
I tried to reach it by something like this.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("somewebsite.com")
driver.find_elements_by_id('input.line.order(240417939).item('+ '\d{9}'+').unitCost')
So my questions is, is there anyway we can search and reach this element only with part of the ID known?
I found similar question answered below, but it was in C#. And unfortunately, I don't know C#.
Finding an element by partial id with Selenium in C#
Thank you in advance!
Edited 4/8
The element is coded like this:
<td id="cell.line.order(240417939).item(250159165).unitCost" class="or_monetarydata">
<input id="input.line.order(240417939).item(250159165).unitCost" type="hidden" value="135.00"
name="order(240417939).item(250159165).unitcost"></input>
135.00
</td>
I could get the element list by
value=driver.find_elements_by_xpath("//*[contains(#id, 'input.line.order(240417939)')]")
Thank you!

Although the answer is in C#, the underlying solution is still the same, by using CSS selectors:
driver.find_elements_by_css_selector('input[id*='cell.line.order(240686080)']')
or XPath will also be able to do this:
driver.find_elements_by_xpath('//*[contains(#id, 'cell.line.order(240686080)')]')