I tried to read WWW-site on my Python project. However, the code will crash if I can't connect to the Internet. How can catch the exception if there is no connection during some point of reading the site?
import sys
import time
import urllib3
# Gets the weather from foreca.fi.
def get_weather(url):
http = urllib3.PoolManager()
r = http.request('GET', url)
return (r.data)
time = time.strftime("%Y%m%d")
url = "http://www.foreca.fi/Finland/Kuopio/Saaristokaupunki/details/" + time
weather_all = get_weather(url)
print(weather_all)
I tested your code with no connection, if there's no connection it will raise and MaxRetryError ("Raised when the maximum number of retries is exceeded.") so you can handle the exception something like:
try:
# Your code here
except urllib3.exceptions.MaxRetryError:
# handle the exception here
Another thing you can do is to use a timeout and do something special when it times out, so that you have additional control. which in a sense what the exception raised it telling you, that it hit the max amount
Also, consider working with requests library.
I presume urllib3 would throw a URLError exception if there is no route to the specified server (i.e. the internet connection is lost), so perhaps you could use a simply try catch? I'm not particularly well versed in urllib3, but for urllib it would be something like:
E.g.
try:
weather_all = get_weather(url)
except urllib.error.URLError as e:
print "No connection to host"
Related
I am trying to create a simple HTTP server that uses the Python HTTPServer which inherits BaseHTTPServer. [https://github.com/python/cpython/blob/main/Lib/http/server.py][1]
There are numerous examples of this approach online and I don't believe I am doing anything unusual.
I am simply importing the class via:
"from http.server import HTTPServer, BaseHTTPRequestHandler"
in my code.
My code overrides the do_GET() method to parse the path variable to determine what page to show.
However, if I start this server and connect to it locally (ex: http://127.0.0.1:50000) the first page loads fine. If I navigate to another page (via my first page links) that too works fine, however, on occasion (and this is somewhat sporadic), there is a delay and the server log shows a Request timed out: timeout('timed out') error. I have tracked this down to the handle_one_request method in the BaseHTTPServer class:
def handle_one_request(self):
"""Handle a single HTTP request.
You normally don't need to override this method; see the class
__doc__ string for information on how to handle specific HTTP
commands such as GET and POST.
"""
try:
self.raw_requestline = self.rfile.readline(65537)
if len(self.raw_requestline) > 65536:
self.requestline = ''
self.request_version = ''
self.command = ''
self.send_error(HTTPStatus.REQUEST_URI_TOO_LONG)
return
if not self.raw_requestline:
self.close_connection = True
return
if not self.parse_request():
# An error code has been sent, just exit
return
mname = 'do_' + self.command ## the name of the method is created
if not hasattr(self, mname): ## checking that we have that method defined
self.send_error(
HTTPStatus.NOT_IMPLEMENTED,
"Unsupported method (%r)" % self.command)
return
method = getattr(self, mname) ## getting that method
method() ## finally calling it
self.wfile.flush() #actually send the response if not already done.
except socket.timeout as e:
# a read or a write timed out. Discard this connection
self.log_error("Request timed out: %r", e)
self.close_connection = True
return
You can see where the exception is thrown in the "except socket.timeout as e:" clause.
I have tried overriding this method by including it in my code but it is not clear what is causing the error so I run into dead ends. I've tried creating very basic HTML pages to see if there was something in the page itself, but even "blank" pages cause the same sporadic issue.
What's odd is that sometimes a page loads instantly, and almost randomly, it will then timeout. Sometimes the same page, sometimes a different page.
I've played with the http.timeout setting, but it makes no difference. I suspect it's some underlying socket issue, but am unable to diagnose it further.
This is on a Mac running Big Sur 11.3.1, with Python version 3.9.4.
Any ideas on what might be causing this timeout, and in particular any suggestions on a resolution. Any pointers would be appreciated.
After further investigation, this particular appears to be an issue with Safari. Running the exact same code and using Firefox does not show the same issue.
Im not able to free the main function from this code so that tasks are completed in parallel and i can receive another get.
in this code when i open in chrome http://0.0.0.0:8082/envioavisos?test1=AAAAAA&test2=test the get_avisos_grupo() function is excecuted in secuence and not in parallel and untill the function ends and not able to send another request to http://0.0.0.0:8082/envioavisos?test1=AAAAAA&test2=test
#!/usr/bin/env python3
import asyncio
import time
from sanic import Sanic
from sanic.response import text
from datetime import datetime
import requests
avisos_ips = ['1.1.1.1','2.2.2.2']
app = Sanic(name='server')
async def get_avisos_grupo(ip_destino,test1,test2):
try:
try:
print(datetime.now().strftime("%d/%m/%Y %H:%M:%S,%f"),'STEP 2',ip_destino)
r = requests.post('http://{}:8081/avisosgrupo?test1={}&test2={}'.format(ip_destino,test1,test2), timeout=10)
await asyncio.sleep(5)
except Exception as e:
print('TIME OUT',str(e))
pass
except Exception as e:
print(str(e))
pass
#app.route("/envioavisos", methods=['GET','POST'])
async def avisos_telegram_send(request): ## enviar avisos
try:
query_components = request.get_args(keep_blank_values=True)
print(datetime.now().strftime("%d/%m/%Y %H:%M:%S,%f"),'>--------STEP 1',query_components['test1'][0])
for ip_destino in avisos_ips:
asyncio.ensure_future(get_avisos_grupo(ip_destino,query_components['test1'][0],query_components['test2'][0]))
except Exception as e:
print(str(e))
pass
print(datetime.now().strftime("%d/%m/%Y %H:%M:%S,%f"),'STEP 4')
return text('ok')
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8082, workers=4)
Expected result is to post everything in parallel.
I'm getting this result
06/04/2021 16:25:18,669074 STEP 2 1.1.1.1
TIME OUT HTTPConnectionPool(host='1.1.1.1', port=8081): Max retries exceeded with url: '))
06/04/2021 16:25:28,684200 STEP 2 2.2.2.2
TIME OUT HTTPConnectionPool(host='2.2.2.2', port=8081): Max retries exceeded with url: '))
i expect to have something like this
06/04/2021 16:25:18,669074 STEP 2 1.1.1.1
06/04/2021 16:25:28,684200 STEP 2 2.2.2.2
TIME OUT HTTPConnectionPool(host='1.1.1.1', port=8081): Max retries exceeded with url: '))
TIME OUT HTTPConnectionPool(host='2.2.2.2', port=8081): Max retries exceeded with url: '))
Asyncio is not a magic bullet that parallelizes operations. Indeed Sanic doesn't either. What it does is make efficient use of the processor to allow for multiple functions to "push the ball forward" a little at a time.
Everything runs in a single thread and a single process.
You are experiencing this because you are using a blocking HTTP call. You should replace requests with an async compatible utility so that Sanic can put the request aside to handle new requests while the outgoing operation takes place.
take a look at this:
https://sanicframework.org/en/guide/basics/handlers.html#a-word-about-async
A common mistake!
Don't do this! You need to ping a website. What do you use? pip install your-fav-request-library 🙈
Instead, try using a client that is async/await capable. Your server will thank you. Avoid using blocking tools, and favor those that play well in the asynchronous ecosystem. If you need recommendations, check out Awesome Sanic
Sanic uses httpx inside of its testing package (sanic-testing) 😉.
I'm testing the behaviour of my application when it tries to reach an unknown URL. It takes 5 seconds to raise a ConnectionError. How can I modify to fail, let's say, at 100ms instead?
I have tried setting the timeout parameter with requests.get(url, timeout=0.1), but it changes nothing.
try:
requests.get(url, timeout=0.1)
except Exception as error:
return False
I found that the problem was Docker and not the http library. Outside Docker it works properly.
I'm using requests to get a URL, such as:
while True:
try:
rv = requests.get(url, timeout=1)
doSth(rv)
except socket.timeout as e:
print e
except Exception as e:
print e
After it runs for a while, it quits working. No exception or any error, just like it suspended. I then stop the process by typing Ctrl+C from the console. It shows that the process is waiting for data:
.............
httplib_response = conn.getresponse(buffering=True) #httplib.py
response.begin() #httplib.py
version, status, reason = self._read_status() #httplib.py
line = self.fp.readline(_MAXLINE + 1) #httplib.py
data = self._sock.recv(self._rbufsize) #socket.py
KeyboardInterrupt
Why is this happening? Is there a solution?
It appears that the server you're sending your request to is throttling you - that is, it's sending bytes with less than 1 second between each package (thus not triggering your timeout parameter), but slow enough for it to appear to be stuck.
The only fix for this I can think of is to reduce the timeout parameter, unless you can fix this throttling issue with the Server provider.
Do keep in mind that you'll need to consider latency when setting the timeout parameter, otherwise your connection will be dropped too quickly and might not work at all.
The default requests doesn't not set a timeout for connection or read.
If for some reason, the server cannot get back to the client within the time, the client will stuck at connecting or read, mostly the read for the response.
The quick resolution is to set a timeout value in the requests object, the approach is well described here: http://docs.python-requests.org/en/master/user/advanced/#timeouts
(Thanks to the guys.)
If this resolves the issue, please kindly mark this a resolution. Thanks.
I'm developing a Python Service(Class) for accessing Redis Server. I want to know how to check if Redis Server is running or not. And also if somehow I'm not able to connect to it.
Here is a part of my code
import redis
rs = redis.Redis("localhost")
print rs
It prints the following
<redis.client.Redis object at 0x120ba50>
even if my Redis Server is not running.
As I found that my Python Code connects to the Server only when I do a set() or get() with my redis instance.
So I dont want other services using my class to get an Exception saying
redis.exceptions.ConnectionError: Error 111 connecting localhost:6379. Connection refused.
I want to return proper message/Error code. How can I do that??
If you want to test redis connection once at startup, use the ping() command.
from redis import Redis
redis_host = '127.0.0.1'
r = Redis(redis_host, socket_connect_timeout=1) # short timeout for the test
r.ping()
print('connected to redis "{}"'.format(redis_host))
The command ping() checks the connection and if invalid will raise an exception.
Note - the connection may still fail after you perform the test so this is not going to cover up later timeout exceptions.
The official way to check if redis server availability is ping ( http://redis.io/topics/quickstart ).
One solution is to subclass redis and do 2 things:
check for a connection at instantiation
write an exception handler in the case of no connectivity when making requests
As you said, the connection to the Redis Server is only established when you try to execute a command on the server. If you do not want to go head forward without checking that the server is available, you can just send a random query to the server and check the response. Something like :
try:
response = rs.client_list()
except redis.ConnectionError:
#your error handlig code here
There are already good solutions here, but here's my quick and dirty for django_redis which doesn't seem to include a ping function (though I'm using an older version of django and can't use the newest django_redis).
# assuming rs is your redis connection
def is_redis_available():
# ... get redis connection here, or pass it in. up to you.
try:
rs.get(None) # getting None returns None or throws an exception
except (redis.exceptions.ConnectionError,
redis.exceptions.BusyLoadingError):
return False
return True
This seems to work just fine. Note that if redis is restarting and still loading the .rdb file that holds the cache entries on disk, then it will throw the BusyLoadingError, though it's base class is ConnectionError so it's fine to just catch that.
You can also simply except on redis.exceptions.RedisError which is the base class of all redis exceptions.
Another option, depending on your needs, is to create get and set functions that catch the ConnectionError exceptions when setting/getting values. Then you can continue or wait or whatever you need to do (raise a new exception or just throw out a more useful error message).
This might not work well if you absolutely depend on setting/getting the cache values (for my purposes, if cache is offline for whatever we generally have to "keep going") in which case it might make sense to have the exceptions and let the program/script die and get the redis server/service back to a reachable state.
I have also come across a ConnectionRefusedError from the sockets library, when redis was not running, therefore I had to add that to the availability check.
r = redis.Redis(host='localhost',port=6379,db=0)
def is_redis_available(r):
try:
r.ping()
print("Successfully connected to redis")
except (redis.exceptions.ConnectionError, ConnectionRefusedError):
print("Redis connection error!")
return False
return True
if is_redis_available(r):
print("Yay!")
Redis server connection can be checked by executing ping command to the server.
>>> import redis
>>> r = redis.Redis(host="127.0.0.1", port="6379")
>>> r.ping()
True
using the ping method, we can handle reconnection etc. For knowing the reason for error in connecting, exception handling can be used as suggested in other answers.
try:
is_connected = r.ping()
except redis.ConnectionError:
# handle error
Use ping()
from redis import Redis
conn_pool = Redis(redis_host)
# Connection=Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>
try:
conn_pool.ping()
print('Successfully connected to redis')
except redis.exceptions.ConnectionError as r_con_error:
print('Redis connection error')
# handle exception