We Keep Coding

Python Selenium - Cannot find element on page

I've perused SO for quite a while and cannot find the exact or similar solution to my current problem. This is my first post on SO, so I apologize if my formatting is off.
The Problem -
I'm trying to find a button on a webpage to punch me into a timeclock automatically. I am able to sign in and navigate to the correct page (it seems the page is dynamically loaded, as switching from different tabs like "Time Management" or "Pay Period" do not change the URL).
Attempts to solve -
I've tried using direct and indirect XPaths, CSS Selectors, IDs, Classes, Names, and all have failed. Included below are the different code attempts to find the button, and also a snippet of code including the button.
Button - HTML
Full Page HTML Source Code
<td>
<a onclick="return OnEmpPunchClick2(this);" id="btnEMPPUNCH_PUNCH" class="timesheet button icon " href="javascript:__doPostBack('btnEMPPUNCH_PUNCH','')">
<span> Punch</span></a>
<input type="hidden" name="hdfEMPPUNCH_PUNCH" id="hdfEMPPUNCH_PUNCH" value="0">
</td>
Attempts - PYTHON - ALL FAIL TO FIND
#All these return: "Unable to locate element"
self.browser.find_element_by_id("btnEMPPUNCH_PUNCH")
self.browser.find_element_by_xpath("//a[#id='btnEMPPUNCH_PUNCH']")
self.browser.find_element_by_css_selector('#btnEMPPUNCH_PUNCH')
#I attempted a manual wait:
wait=WebDriverWait(self.browser,30)
button = wait.until(expected_conditions.element_to_be_clickable((By.CSS_SELECTOR,'#btnEMPPUNCH_PUNCH')))
#And even manually triggering the script:
self.browser.execute_script("javascript:__doPostBack('btnEMPPUNCH_PUNCH','')")
self.browser.execute_script("__doPostBack('btnEMPPUNCH_PUNCH','')")
#Returns Message: ReferenceError: __doPostBack is not defined
None of these work, and I cannot seem to figure out why that is. Any help will be greatly appreciated!

Can't scrape data from webpage with popup/frame

I am having trouble finding elements on a customer-facing webpage that I am scraping data from, using Robot Framework + Selenium. My trouble, I think, has to do with the desired data existing in a popup/frame. The data I seek is located on a customer's invoice, which pops up when I press a button ("View Current Invoice"). I've been successful with logging into the site and navigating around, and at one point I was successful pressing the View Current Invoice button to cause the invoice to pop up - but forgot to commit that code and lost it. :-(
In any case, eve if I manually enter the popped up invoice by pressing the button when my script is expecting it to be pressed, I can't seem to scrape the subsequent data. I have tried to identify elements on the invoice using locators (from Right-Click-Inspect capability built into Firefox and Chrome; Katalaon Recorder; Selenium IDE; etc.). I get what looks like a valid locator (almost always Xpath); yet when I run my Robot script, it fails to find the element in question. I have spent a lot of time poring over the page's source code, but since I am not as savvy with HTML/JS/CSS as I should be, I haven't been successful.
Here is a screenshot of the invoice button:
And here is what I see when the button is pressed. I want to scrape all the invoice data, like Amount Due, Invoice Number, Due Date, etc.
Does anyone have any idea what I am missing here? What would you do to get the data on the invoice if you were in my shoes? I know my question probably sounds vague and naiive, but I am at the end of my rope, so to speak. I am willing to share page source code, more screenshots, whatever is required.
EDIT I used Rahul Rai's method to inspect the popup while it was popped up; then searched for "iframe". There were 10 matches; #7, when clicked on, resulted in the invoice popup being highlighted in blue:
I assume this means this is the iframe referencing the popup? If so, I should be able to find information about the "handle" to the iframe in the inspection code, but I don't see anything there that matches the locators I am used to (e.g. name, id, xpath). I even tried "Select Frame 1599252503952", but that just resulted in a
"Element with locator '1599252503952' not found" error.

As per above screen you have shared I can see your Invoice details are inside iframe. So after clicking on View Current Invoice button you can use below code to navigate inside frame and then scrape required information.
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'invoice_detail_container']")))
#Code to scrape data
ele = driver.find_element_by_xpath('<xpath>')
print(ele.txt)
......
......
#After your work is done in this frame to navigate back to main window
driver.switch_to.default_content()
Note: I have assumed your main frame for invoice is not in side any other iframe ( Based on screen shared). Also before elements start there is no other nested frame. If there is any other nested frame you need to navigate first into that.

I was finally able to scrape data from the Invoice popup after inspecting the HTML source, and seeing this:
<iframe frameborder="0" src="/cmc/invoice_detail_container.pyt?direction=//my.hughesnet.com/cmc/invoice_detail.pyt%3Finvnumber%1234-567890&portletId=863" name="1599391562960" class="cboxIframe" allowfullscreen="true" webkitallowfullscreen="true" mozallowfullscreen="true"></iframe>
I was then able to use the Select Frame keyword in Robot Framework, specifying the iframe locator for the popup, using the 'class' strategy. I also had to explicitly select the main body frame first. In the end, the code that allowed me to enter the iframe and scrape was:
Select Frame body
Select Frame class:cboxIframe
Big thank you to Rahul Rai for pushing me closer to the solution; and thanks to the others who answered as well.

You need to switch your site to frame/popup, you can use like below example, may be it will help you.
IList<IWebElement> textfields = new List<IWebElement>();
textfields = driver.FindElements(By.TagName("iframe"));
driver.SwitchTo().Frame(textfields[count); // number of textfields list.
please try to implement as per your scenario, let me know if any question.

You can try to do :
driver.switch_to_active_element()
and then scrape the popup to close it. Then I think it will be okay...

How to press multiple buttons on a webpage without knowing the id or xpath with Selenium (Python)

I'm trying to automate the logging in and unfollowing of people on the website Depop. I am having a bit of trouble clicking all of the 'unfollow' buttons. Each of the buttons have a different ID and Class and the only similarities between them is this HTML code:
<span> Following <span>
Is there any way to click every button with this HTMl code?
I have already tried finding the XPath of all the buttons but because there are so many of them, it would be difficult to find the XPath of every single button. I have also tried to find the class of the follow buttons but they are all different. There are no similarities in the css selectors.
I have tried doing something like this but to no success.
driver.get_attribute('<span> Following </span>').click()
For optimum results, the program would go through the page, unfollowing everyone that is currently being followed.

Here is the logic.
unfollows = driver.find_elements_by_xpath("//span[normalize-space(.)='Following']")
for btn in unfollows:
btn.click()
If you get staleElement exception with the above approach then you have to follow the following approach.
while len(driver.find_elements_by_xpath("//span[normalize-space(.)='Following']"))>0:
btn = driver.find_element_by_xpath("(//span[normalize-space(.)='Following'])[1]")
btn.location_once_scrolled_into_view
driver.execute_script("arguments[0].click();",btn)

How to Bypass confirm age model through scrapy

If you go to the site, you'd notice that there is an age confirmation window which I want to bypass through scrapy but I messed up with that and I had to move on to selenium webdriver and now I'm using
driver.find_element_by_xpath('xpath').click()
to bypass that age confirmation window. Honestly I don't want to go with selenium webdriver because of its time consumption. Is there any way to bypass that window?
I searched a lot in stackoverflow and google
but didn't get any answer which may resolves my problem. If you've any link or idea of resolving it by Scrapy, that'd be appreciated. A single helpful comment will be up-voted!

To expand on Chillie's answer.
The age verification is irrelavant here. The data you are looking for is loaded via AJAX request:
See related question: Can scrapy be used to scrape dynamic content from websites that are using AJAX? to understand how they work.
You need to figure out how https://ns5bwtai8m-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%203.19.1&x-algolia-application-id=NS5BWTAI8M&x-algolia-api-key=e676b05f3844d3adf54a29732af6e43c url works and how can you retrieve in it scrapy.

But the age verification "window" is just a div that gets hidden when you press the button, not a real separate window:
<div class="age-check-modal" id="age-check-modal">
You can use the browser's Network tab in developer tools to see that no new info is uploaded or sent when you press the button. So everything is already loaded when you request a page. The "popup" is not even a popup, just an element whose display is changed to none when you click the button.
So Scrapy doesn't really care what's meant to be displayed as long as all html is loaded. If the elements are loaded, they are accessible. Or have you seen some information being unavailable without pressing the button?
You should inspect the html code more to see what each website does, this might make your scraping tasks easier.
Edit: After inspecting the original html you can see the following:
<div class="products-list">
<div class="products-container-block">
<div class="products-container">
<div id="hits" class='row'>
</div>
</div>
</div>
</div>
You can also see a lot of JS script tags.
The browser element inspector shows us the following:
The ::before part gives away that this was manipulated by JS, as you cannot do this with simple CSS. See Granitosaurus' answer for details on this.
What this means is that you need to somehow execute the arbitrary JS code on those pages. So you either need a solution with Scrapy, or just use Selenium, as many do, and as you already have.

Django - turn field into tags with dropdown

In my form on django, I have a field called package_includes and a field called price. Right now, the content in package_includes it is simply text. So if I input "paper, glue, glitter" it will display exactly as i typed it, unable to change anything. However, I want the text to work like individual tags ->How I want tags to look
So when you click on one of the individual items (EX: "paper", "glue", or "glitter") i want it to display a drop down that allows you the option to put extra glitter for an additional $2.00 to the total.
Also, at the end of the text I want a tag that displays "Add" and this function would allow you to add items not included in this specific package. For example, you should be able to add crayons for $5, highlighter for $2.50, etc.
I am new to programming and don't fully understand how to add a function to text like this in a field in django. Thank you in advance for your help!

The answer to your question is unfortunately not a step-by-step instruction of how to go from 0 to a complete working example - only a pointer in the right direction.
What you need is achieved through CSS (producing the visual effect you want) and JavaScript (handling the drop-down functionality; it most probably has to be your own script based on jQuery). All of what you want needs to live (execute) within the Internet browser, and Django can only help there by serving the static CSS and script files, and referencing them in the actual page, when you put them in the correct template.
Here is an example of Wagtail Admin doing something similar:
If you open the page source in your browser, all you will see as HTML code for this is the following:
<div class="field-content">
<div class="input ">
<input id="id_tags" name="tags" type="text" value=""hello world", blog" /><script>initTagField("id_tags", "/admin/tag-autocomplete/");</script>
<span></span>
</div>
</div>
But if you open the page with FireBug, your browser's Development Tools, or the Web Developer extension, with visual styles applied it will decompose to something like this:
If you notice, the applied style to the form input element is display: none;. What you see is actually the styled unordered list elements right after. In order to be able to use them in your script, you need to be able to get to them/their contents from within your JS function. One strategy would be to assign an id attribute to all of the visible <li> elements. It can be something counter-based (e.g. id="shopping-cart-item-0", id="shopping-cart-item-1", etc.). Another way would be to assign an id to the <div class="input"> element, and within your function get all its DOM descendants of type <li>. Whatever works better for you. Then you could parse the label (inner text) of the list element in your script to get the type of item you are dealing with, and from then on find the discount price and apply it...
I would suggest that you start with a working example - find some app that uses taggit or similar package for Django and install it in a test application. Then you can use your browser's or preferred web development extension to play around with the CSS on them and see how different options affect the visual aspect of what you need. Once you are comfortable with that, see how you can build drop-down menus with jQuery. I believe the jQuery site has enough tutorials on the topic.