I have a Python program which sends several (about 5-6) long poll requests in parallel using different threads for each poll via requests package. And I realized that some of my threads sometimes just freeze. When this happens, the server I am sending the request does not receive the request. Also I set a timeout on the request and it does not work.
try:
print("This line prints")
response = requests.head(poll_request_url, timeout=180)
print("This line does not print when freeze occurs")
except ReadTimeout:
print("Request exception.")
except RequestException as e:
print("Request exception.")
except Exception:
print("Unknown exception.")
print("This line does not print either when freeze occurs.")
I am doing this on Raspberry Pi 2 hardware with Raspbian OS.
I used this same program without a problem when I was using Python 2.7. Recently I switched to Python 3.5. I tested using both requests versions with 2.8.1 and 2.9.1.
This problem does not occur very frequently but happens 2-3 times per day on different threads.
What might be the problem? How can I debug this?
Edit: The problem is solved by updating the Linux kernel.
According to the docs:
http://docs.python-requests.org/en/master/user/quickstart/#timeouts
It should be throwing a Timeout exception, when the timeout happens. That would mean that the line:
print("This line does not print when freeze occurs")
Would never be called it a timeout actually happens.
Are you catching the exception? Or any other exception? It might be that it's timing out fine, but you just aren't seeing this. Maybe try something like this:
try:
response = requests.head(poll_request_url, timeout=180)
except requests.exceptions.Timeout:
print("Timeout occurred")
So you can see if that's what is going on.
EDIT: possibly it's the "connect" step that's not timing out correctly. It may be the large timeout value for the "connect" step is messing it up somehow. Perhaps trying having a shorter timeout for that (as mentioned here):
http://docs.python-requests.org/en/master/user/advanced/#timeouts
e.g.
response = requests.head(poll_request_url, timeout=(3, 180))
Failing that it might be some sort of DNS lookup issue? Maybe see if hardcoding the IPs presents the same problem?
Solved my problem using timers (from threading import Timer). If no result next 10 seconds - repeat, if no result next 10 seconds - print 'Error' and go on. You can't monitor timer status with if statement if request freezes, but you can do it through while loop, adding time if result is ok (Python: Run code every n seconds and restart timer on condition).
Related
I have a script in Playwright 1.23.1 with Python 3.10.2 running in Ubuntu LTS 20.4 to extract data from a page, the scripts run for about an hour because it is a lot of data. The problem is that sometimes the execution freezes and I can't catch it and I have to give the SIGINT sign (CTRL + C) to stop the execution. I don't know if it is a problem of Playwright or it is of the page that it is browsing. This page is really not that good, it's slow and have some problems in general.
My code is something like this:
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError
from playwright.sync_api import Error as PlaywrightError
with sync_playwright() as spw:
# connect to the page
browser, page = connect(spw)
# sign in with credentials
sign_in(username, page)
data_code = 1
while True:
try:
# Extract the data from the page using playwright methods
# including select_option, frame_locator, evaluate, click, and reload
end_signal = extract_data(page, data_code)
if end_signal:
break
except (TimeoutError, PlaywrightTimeoutError, PlaywrightError) as e:
print(f"Error is: {e}")
page.reload(timeout=90000, wait_until='domcontentloaded')
data_code += 1
print("Exit cycle")
That's the basic structure of my code, the problem is that sometimes when trying to use click, select_option, or reloading the execution freezes. There are some prints inside extract_data but they don't appear, and sometimes print(f"Error is: {e}") does not print, so it does reach it, therefore, my conclusion is that the execution is freezed.
I have researched a little about Playwright debug, but in general I can't find the problem, and this error happens sometimes not always, so I can't repliacte it for a proper debug with playwights. The logs at least let me identify where it freezes, but nothing else so far.
Hope someone can help me, thanks.
I have tried the schedule library however my code stalls sometimes which also means the schedule code stalls.
import schedule
import time
import app
schedule.every(3).seconds.do(app.run)
while 1:
try:
schedule.run_pending()
except Exception as e:
print e
pass
app.run is the script, it using Queues and Threads to request exchange data. One of the exchanges randomly throws an error and because the of threading (I think) the code goes into a state of limbo. I can't seem to fix the bug, but a dirty fix to the problem would be force run the script every x time (in my case I want 10 seconds) Any ideas ?
Besides the obvious way (have a thread that with while True: containing action() and sleep(10)), you can use threading.Timer or the sched module.
I have a python program that opens threads in a loop, each thread runs a method that sends an http packet.
The program is supposed to emulate heavy traffic on a certain server we're working on.
The thread creation code looks something like this:
while True:
try:
num_of_connections += 1
thread_obj = HTTP_Tester.Threaded_Test(ip)
thread_obj.start()
except RuntimeError as e:
print("Runtime Error!")
So again, the thread_obj is running a method which sends HTTP requests to the ip it is given, nothing fancy.
When running this code under python 3.5, I am able to open around 880 threads until the RuntimeError "can't open a new thread" is thrown.
When running this code under python 3.4 however, the number of threads keeps growing and growing - I got up to 2000+ threads until the machine it was running on became unresponsive.
I check the amount of threads that are open by looking at the num_of_connections counter and also using TCPView to verify in fact that the number of Sockets is actually growing. Under python 3.4 TCPView actually shows 2000+ sockets open for the program so I deduce that there are in fact 2000+ threads open
I googled around and saw that people suggested the threading.stack_size is the culprit - not here, I changed the size but the number of possible threads doesn't change either way
The question is, how come with 3.5 the limit is so low, whereas in 3.4 its (presumably) high? Also can I change the limit? I would prefer to use 3.5 but want to open as many threads as I can
Thank you!
I am using exscripts module which has a call conn.connect('IP address').
It tries to open a telnet session to that IP.
It will generate an error after connection times out.
The timeout exception is set somewhere in the code of the module or it would be what the default for telnet is. (not sure)
This timeout time is too long and slowing down the script if 1 device is not reachable. Is there something we can do with the try except here ? Like
Try for 3 secs:
then process the code
except:
print " timed out"
We changed the API. Mike Pennington only recently introduced the new connect_timeout parameter for that specific use case.
New solution (current master, latest release on pypi 2.1.451):
conn = Telnet(connect_timeout=3)
We changed the API because you usually don't want to wait for unreachable devices, but want to wait for commands to finish (some take a little longer).
I think you can use
conn = Telnet(timeout=3)
I dont know whether timeout in seconds. If microseconds, try 3000
I'm still relatively new to Python, so if this is an obvious question, I apologize.
My question is in regard to the urllib2 library, and it's urlopen function. Currently I'm using this to load a large amount of pages from another server (they are all on the same remote host) but the script is killed every now and then by a timeout error (I assume this is from the large requests).
Is there a way to keep the script running after a timeout? I'd like to be able to fetch all of the pages, so I want a script that will keep trying until it gets a page, and then moves on.
On a side note, would keeping the connection open to the server help?
Next time the error occurs, take note of the error message. The last line will tell you the type of exception. For example, it might be a urllib2.HTTPError. Once you know the type of exception raised, you can catch it in a try...except block. For example:
import urllib2
import time
for url in urls:
while True:
try:
sock=urllib2.urlopen(url)
except (urllib2.HTTPError, urllib2.URLError) as err:
# You may want to count how many times you reach here and
# do something smarter if you fail too many times.
# If a site is down, pestering it every 10 seconds may not
# be very fruitful or polite.
time.sleep(10)
else:
# Success
contents=sock.read()
# process contents
break # break out of the while loop
The missing manual of urllib2 might help you