I am using selenium to scrape some data.
This is my code, simplified:
def get_usrs():
#DO SOMETHING
def scroll_down():
#SCROLL UNTIL ARRIVES TO THE BOTTOM OF THE PAGE
#CONTINUE WITH GET_USRS()
The problem is that when the code gets to scroll_down() it doesn't wait until it finishes but continues with get_usrs() and obviously encounters an error.
How can I solve this? Thanks in advance!
So, my code was running with:
try:
get_usrs()
except Exception as e:
print(e)
Now it's running with:
if __name__ == '__main__':
get_usrs()
Works fine.
Related
This script loops and even if it crashes it restarts.
Now I want it to restart the script even if it has NOT CRASHED yet.
while True:
try:
do_main_logic()
except:
pass
I have the loop that restart on crash, but I want it to restart on 60 seconds.
It is pretty hard to understand what you are asking for but i can still show how if works:
while True:
try:
#Try to do something
except:
#if it failed
else:
#if it succeded
You can do this :
from time import sleep
while True:
try:
do_main_logic()
except:
sleep(60)
pass
So I've got a selenium python script that occasionally runs into the following error at differing sections in my code:
Exception has occurred: WebDriverException
Message: unknown error: cannot determine loading status
from target frame detached
But when I encounter this error, if I re-run the program without changing anything, everything works again (so I know that it's either a problem with the website or the webdriver). In the meantime, I was wondering if there was a way to tell python to restart the program if a WebDriverException is encountered. Any help would be greatly appreciated, thank you.
You could try os.execv(), according to here, it enables a python script to be restarted, but you need to clean the buffers etc using this C-like function sys.stdout.flush()
try:
<your block of code>
except WebDriverException:
print(<Your Error>)
import os
import sys
sys.stdout.flush()
os.execv(sys.argv[0], sys.argv)
You could simply use the os module to do this: os.execv(sys.argv[0], sys.argv)
Use a main() function as a starting point for your program, and trigger that function again whenever you need to restart. Something like
def foo1():
return
def foo2():
try:
...
except:
main()
def main():
foo1()
foo2()
if __name__ == "main":
main()
If the error occurs because it does not find a certain tag, you could put a wait
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
or worst case
browser.implicitly_wait(5)
or
time.sleep(5)
which waits 5 seconds for the element to appear
When running the below code, if an error occurs or a connection times out, it triggers the loop so that the 'retrying code' is printed. But the code doesn't restart. It continuously prints 'retrying code' but that is all.
I wonder if it is anything to with the fact that the 'testingfile123.py' is working with http.requests?
import time
import os
os.system('python testingfile123.py')
while True:
try:
testingfile123.py()
print('running code')
except:
time.sleep(5)
print('retrying code')
Isn't this caused by the while True loop? Even if the code goes to the except section, it will just wait 5 seconds and print "retrying code". You would need some break statement.
I am having a main program which is defined like this:
main.py
def main():
try:
registry.start_server()
except:
print("Shutting down the program")
pass
if __name__ == '__main__':
main()
registry.start_server() is the method in another module which looks like this:
def start_server():
t_server = threading.Thread(target=server.start)
t_server.start()
try:
t_server.join()
except KeyboardInterrupt:
print("Error")
raise ValueError
finally:
fp.close()
server.start is the method in another module which does some listening work in a while(True) manner. I am not sure how to stop the whole program when clicking Stop in PyCharm which is Ctrl + C (Signal). I tried with Event but without success. I get to the main.py by raising an exception when the signal gets caught but that does not terminate the whole program. It shows Waiting for program to detach. The only way is to use SIGKILL. I don't understand where does the program keeps hanging? I have also tried calling sys.exit(0) when the signal gets caught and creating the thread as Deamon but that didnt help either.
EDIT
While True method in another module
def start(self, event):
try:
while True:
if event.is_set():
if self.pubsub.channels:
print("It enters here")
message = self.pubsub.get_message(True)
if message:
.
.
.
else:
return
To solve the problem, all you need to do is:
let the child-thread exit, and
let main thread join the child-thread.
i am making an Flask-API for my project and i want to achieve something when the server restarts or runs, meaning whenever the main block is executed i want to do a check.
the code:
if __name__ == '__main__':
try:
with open('x.p','rb') as pkl_PR:
ps=pickle.load(pkl_PR)
with open('y.p','rb') as pkl_df:
df=pickle.load(pkl_df)
with open('z.p','rb') as pkl_spl:
spl_df = pickle.load(pkl_spl)
except Exception as e:
logger.debug(e)
app.run(debug=True)
so if any one of the pickle file doesn't exist, i dont want to start the server and save a log file with error.
so how do i go about it?
You can call sys.exit() from inside the except block, that will cause your program to exit before starting the flask server.