We Keep Coding

Wait for an element to appear, instead of keeping on refreshing the page using selenium

I am using selenium webdriver, python to write a test case wherein I have to perform an action when a button appears on the screen. The tricky part over here is that if the button is not already present, I have to keep on refreshing the page till it comes and then perform the necessary actions.
I wanted to know if there's any other way around this so that I won't have to keep on refreshing the page?
Thank you for the help.

How can I click "invisible" reCAPTCHA buttons using Selenium web automation?

I am using Python and Selenium to automate this website: https://prenotami.esteri.it
The script I made fills out a form and then clicks a button to advance to the next page. These actions are carried out using Selenium's find_element_by_xpath() function. Recently, the website added a reCAPTCHA that pops up after the button is clicked, and must be completed before advancing.
I have already written a Python script that is capable of surpassing this type of captchas by using the audio option. However, in this particular website, I am not able to find the xpath to the audio button of the reCAPTCHA. Although there is an iframe that contains the reCAPTCHA, there seems not to be anything inside it.
In the first attached image you can see how this website's reCAPTCHA looks like in HTML, compared to other website that is visible in the second image, where a #document can be seen inside the iframe.
My intention is to run this program using headless Chrome, so I can't relay in any mouse control functions offered by pyautogui for example.
I've been scratching my head around this problem for a while, so any advice is useful. Thanks!
Edit: after some research I have found that this type of reCAPTCHA that doesn't need to check a "I am not a robot" checkbox is called "invisible reCAPTCHA". The captcha only pops up if the detected activity is suspicious (for example clicking too fast). I have tried adding random waits and movements to mimic human behaviour, but the captcha still appears after some tries. Since I don't think there is a way to avoid the captcha from appearing 100% of the times, the question of how to click the buttons using Selenium's find_element_by_xpath() function remains the same. Leaving this as a note just in case someone finds it useful.

Ever tried to use the following function:
add_argument("-auto-open-devtools-for-tabs")
I managed to interact with captcha

If the position is always fixed, you can use PyAutoGUI to move the mouse and click on it
import pyautogui
pyautogui.click(100, 100) # button coordinates

Since, it is in iframe, we need to move our selenium pointing to iframe and then use your xpath.
driver.switch_to.frame("c-la7g7xqfbit4")
capchaBtn = driver.find_element_by_xpath("(//button[#id='recaptcha-audio-button'])[2]")

Python - requests_html screen scraping

I’m trying to log in to a pretty complex (to my beginner’s eye) website and make a reservation. Did not know a single python statement before starting the project. After many starts and stops have successfully logged in using requests_html/HTMLSession. Have overcome the security/authorization issues and arrived at target page. The page displays the server time on it and I cannot hit the proper key until the time reaches 7:00 AM.
I am unable to access the field. I have tried the .search and .find commands, but nothing. I am hoping someone can tell me how to download the time into my program so I can test the time and wait until it reaches, or almost reaches 7:00. (I say almost because the reservation is for tee times and there is a real crunch at 7 – the whole point of this application is to automate the process and be the fastest!)
So I need to be able to load the time into my python and click a date file when the clock reaches 7:00.

No idea what scraping tool you are using, but generaly you would access this elemen via xpath or css selector:
response.css(".jquery_server_clock::text").extract()
This example is if you are using scrapy

Maybe you would be better off using selenium.
Selenium allows you to automate a browser window, so it's possible that it is not possible to interact with the site using requests, but using selenium the site you visit thinks you are using a normal browser but you can automate everything.
So what I would do if I were you:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("your_url.com")
input("Navigate to the desired page, then press enter")
while not driver.find_element_by_class_name("jquery_server_clock").text[0] == "7":
pass
driver.find_element_by_class_name("other_button").click()
This would wait until it is 7 AM and then click the other button immediately.

Selenium with Python 3.5 and Chrome - failed to click element located via xpath

I have a script which gets me info from Polish Avon's website. So essentially every month they change prices, and to make my girl's life easier I just download the prices to have a look up table in excel.
Anyways, so I have the script which navigates to this website:
https://www.avon.pl/szukaj/po-kodzie-produktu/
Once the page is loaded it enters a number between 00000 and 99999 into the search box, which I find using xpath:
find_box_path = '//*[#id="ShopByProductNumber"]/div[2]/div[3]/div[1]/div/input'
Only some of the codes are valid, so if the search is successful, the script will click on the item, which opens in a new window and processes the information, if nothing is found it moves on to the next number. The script checks for the xpath to figure out if the code is valid or not. The following exctract would click on the element and open a new tab:
# ------------ click the product ------------
find_item_text_element = WebDriverWait(driver, 10).until(lambda driver: driver.find_element_by_xpath(find_text_path))
find_item_text_element.click()
time.sleep(0.4)
The find_text_path variable is declared earlier as:
find_text_path = '//*[#id="ShopByProductNumber"]/div[2]/div[3]/p/a'
The interesting bit, is that for more than a year, my script worked like a charm. Only 2 days ago the script was running, I got through maybe 25000 combinations until it stooped. From that point, when the script gets to the bit above, it shuts down and resets. I understand why it resets - that is intended - but I have no idea why it won't click on the element. The IDE doesn't show any error.
I use XPather to find the xpaths, and the one above is a valid xpath. And as I said it work fine until now. I understand that website itself could have done something to prevent automation, but I don't see the problem. Can anyone see/point out the problem? Maybe some workarounds?
Location of both elements in question
EDIT:
The issue was resolved. The zoom in my chrome profile I was using, was set to 105% instead of 100%. This cause the webdriver to click wrong spot on the page.

Can you show us the specific html code which is causing the issue? I'm not able to find it on the website. Maybe the element you're trying to reach is now included in a frame, but I can only suppose without the html.

Weird thing with selenium test (python) on angular js based UI

We are testing a web application based on angular js. I have encountered it twice now. this time I need to click a dropdown embedded in a link tag . I can manually manipulate it with ipython, but once it run in script, the dropdown popup will not appear as what I can do it with the terminal.
Do you have any idea about this?

Judging from your comment "there is no error actually, the element is clicked", then I would suspect that the script is running fast enough to click the element before the JavaScript actions have been bound to the event. You can verify this by adding a
import time
time.sleep(4)
If the action works when there is a deliberate pause then you can be quite sure that it is a race condition between the JavaScript being bound and Selenium clicking the element.
How you deal with this is up to you. You could mark the DOM in some way when the events had been bound. That's a technique I've used in the past.
You could execute a bit of JavaScript in a loop that returned some information about the global state of the page, and use that to decide when the page was ready to interact with.