We Keep Coding

History of retries using Request Library

I'm building a new retry feature in my Orchestrate script and I want to know how many times, and, if possible, what error my request method got when trying to connect to a specific URL.
For now, I need this for logging purposes, because I'm working on a messaging system and I may need this 'retry' information to understand when and why I'm facing any kind of problem in HTTP requests, once I work in a micro-service environment.
So far, I debugged and certify that retries are working as expected (I have a mocked flask server for all micro services that we use), but I couldn't find a way to got the 'retries history' data.
In other words, for example, I want to see if a specific micro-service may respond only after the third request, and those kind of thing.
Below is the code that I'm using now:
from requests import exceptions, Session
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
def open_request_session():
# Default retries configs
toggle = True #loaded from config file
session = Session()
if toggle:
# loaded from config file as
parameters = {'total': 5,
'backoff_factor': 0.0,
'http_error_to_retry': [400]}well
retries = Retry(total=parameters['total'],
backoff_factor=parameters['backoff_factor'],
status_forcelist=parameters['http_error_to_retry'],
# Do not force an exception when False
raise_on_status=False)
session.mount('http://', HTTPAdapter(max_retries=retries))
session.mount('https://', HTTPAdapter(max_retries=retries))
return session
# Request
with open_request_session() as request:
my_response = request.get(url, timeout=10)
I see in the urllib3 documentation that Retry has a history attribute, but when I try to consult the attribute it is empty.
I don't know if I'm doing wrong or forgetting something, once Software Development is not my best skill.
So, I have two questions:
Does anyone know a way to got this history information?
How can I do create tests to verify if the retries behavior is working as expected? (So far I only test in debug mode)
I'm using Python 3.6.8.
I know that I can create a while statement to 'control' this, but I'm trying to avoid complexity. And this is why I'm here, I'm looking for an alternative based on Python and community best practices.

A bit late, but I just figured this out myself so thought I'd share what I found.
Short answer:
response.raw.retries.history
will get you what you are looking for.
Long answer:
You cannot get the history off the original Retry instance created. Under the covers, urllib3 creates a new Retry instance for every attempt.
urllib3 does store the last Retry instance on the response when one is returned. However, the response from the requests library is a wrapper around the urllib3 response. Luckily, requests stores the original urllib3 response on the raw field.

SOUNDCLOUD API: get body of the response for error 429 with soundcloud-python

I'm using soundcloud-python https://github.com/soundcloud/soundcloud-python for Soundcloud API on Ubuntu Server 16.04.1 (installed with pip install soundcloud).
Soundcloud API Rate Limits official page https://developers.soundcloud.com/docs/api/rate-limits#play-requests says that, in case an app exceeds the API rate limits, the body of the 429 Client Error response would be a JSON object, containing some additional info.
I'm interested in getting reset_time field, to inform the user when the block will be over.
The problem is that when, for example, like rate limits is exceeded, doing response = client.put('/me/favorites/%d' % song_id) the app crashes and response is null.
How can I get the JSON response body?

Why don't you read the package's source code and find out by yourself ?
Let's see... You don't explain you got that client object but browsing the source code we can see there's a "client.py" module that defines a Client class. This class does'nt not define a put method explicitely but it defines the __getattr__ hook :
def __getattr__(self, name, **kwargs):
"""Translate an HTTP verb into a request method."""
if name not in ('get', 'post', 'put', 'head', 'delete'):
raise AttributeError
return partial(self._request, name, **kwargs)
Ok, so Client.put(...) returns a partial object wrapping Client._request, which is a quite uselessly convoluted way to define Client.put(**kwargs) as return self._request("put", **kwargs).
Now let's look at Client._request: it basically make a couple sanity checks, updates **kwargs and returns wrapped_resource(make_request(method, url, kwargs)).
Looking up the imports at the beginning of the module, we can see that make_request comes from "request.py" and wrapped_resource from "resources.py".
You mention that doing an api call while over the rate limit "crashes the application" - I assume you mean "raises an exception" (BTW please post exceptions and tracebacks when saking about such problems) - so assuming this is handled at the lower level, let's start with request.make_request. A lot of data formatting / massaging obviously and finally the interesting part: a call to response.raise_for_status(). This is a hint that we are actually delegating to the famous python-requests package, which is actually confirmed a few lines above and in the requirements file
If we read python-requests fine manual, we find out what raise_for_status does - it raises a requests.exceptions.HTTPError for client (4XX) and server (5XX) status codes.
Ok now we know what exception we have. Note that you had all those informations already in your exception and traceback, which would have saved us a lot of pain here had you posted it.
But anyway... It looks like we won't get the response content, does it ? Well, wait, we're not done yet - python-requests is a fairly well designed package, so chances are we can still rescue our response. And indeed, if we look at requests.exceptions source code, we find out that HttpError is a subclass of RequestException, and that RequestException is "Initialize(d)" with "request and response objects."
Hurray, we do have our response - in the exception. So all we have to do is catch the exception and check it's response attribute - which should contains the "additional informations".
Now please understand that this took me more than half an hour to write, but about 7 minutes to sort out without the traceback - with the traceback it would have boiled down to a mere 2 minutes, the time to go to the requests.exceptions source code and make sure it keeped the request and response. Ok I'm cheating, I'm used to read source code and I use python-requests a lot, but still: you could have solved this by yourself in less than an hour, specially with python's interactive shell which let's you explore and test live objects in real time.

Custom abort mapping/exceptions in Flask

The default message for Flask 400 exception (abort()) is:
{
"message": "The browser (or proxy) sent a request that this server could not understand."
}
For 404:
{
"message": "The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. You have requested this URI [/obj/] but did you mean /obj/ or /obj/<int:id>/ or /obj/<int:id>/kill/ ?"
}
I have trouble comprehending these messages when I'm getting them as replies in my API (especially the first one, I thought there's something wrong with encryption or headers) and I thing it's kinda tiresome to try to override text manually for every abort() exception. So I change the mapping:
from flask import abort
from werkzeug.exceptions import HTTPException
class BadRequest(HTTPException):
code = 400
description = 'Bad request.'
class NotFound(HTTPException):
code = 404
description = 'Resource not found.'
abort.mapping.update({
400: BadRequest,
404: NotFound
})
For the case of 400 it works beautifully. But when it comes to 404 it is still the same message. I tested it in the same place in my code - it works for abort(400), abort(403) and some of the others, but it gets mysteriously overridden by default message on abort(404). Debugging didn't help much. What may be the culprit here?
Update. Yes, I'm using abort imported from flask not flask_restful as the latter doesn't have the mapping and it's a function not an Aborter object. Besides, it does work for most exceptions, so it's probably not the real issue here.
Update 2. The abort.mapping seems to be perfectly fine on execution. The exceptions in question are overridden, including 404.
Update 3: I've put together a little sandbox, that I use for debugging. (removed the repo since the mystery is long solved).

It took me some time, but now I actually found the place, where it all derails on 404 error. It's actually an undocumented feature in flask-restful. Look at the code here. Whatever message you chose persists until that very place and then it becomes the default. What we need now is just put ERROR_404_HELP = False to our config and everything works as intended.
Why is this code even there in the first place? OK, perhaps, I can live with that, but it should be all over the documentation. Even when I googled the name of the constant, I only got a couple of GitHub issues (1, 2).
Anyways, the mystery is officially solved.

By the way... I can't point to documentation for how I discovered this, I just tried it (that's how I learn most development!) but, you can simply abort with the desired response code but instead return a custom string with it. I think this makes sense because you're using the framework the way it's intended, you're not writing tons of code, you're returning the correct response code and in the fashion the framework expects, and you're informing any human who reads it as to the application's context for the error.
from flask import abort
abort(404, "And here's why.")

Disable the "400 Bad Request" error for missing keys in form

The documentation for Flask explains that it will return a 400 Bad Request error if code attempts to access a form key that does not exist. This makes debugging difficult because the error doesn't have information about what key is missing or if something else caused the error.
Is there a way to disable the 400 error and return a "500 Internal Server Error" with a traceback in the logs instead?

If you are not sure if the form contains a key, you should use .get(key), which returns None if it is not found, rather than directly indexing it.
Typically, form validation should not raise an error at all, but should return messages about why validation failed. Consider using a form library such as WTForms to handle validation for you, so you get nice messages instead of errors.
If you still want to mess with the errors rather than doing validation, you can add a custom error handler to log the exception and return a generic 500 error.
from werkzeug.exceptions import abort, BadRequestKeyError
#app.errorhandler(BadRequestKeyError)
def handle_key_error(e):
app.logger.exception('Missing key {}'.format(e.args[0]))
# raise a new 500 exception
# abort doesn't work from inside an error handler
# so simulate how Flask would handle it
try:
abort(500)
except Exception as new_e:
app.handle_user_exception(new_e)
Your logging setup may be different. The app logger may require extra configuration when not in debug mode.

So I figured out a great solution. At least with Flask-restful, but I suspect that this is the case with the Flask-request in general. If you get a 400 Bad Request back, examining the JSON response from the server will give you more data. For example, in the Firefox development console, go to the network take, click on the request that generated the bad request, click the "Response" tab, and BAM, there's your useful debug error. Works at least in development mode on my own server.

image download problem (python)

I was trying to download images with multi-thread, which has a limited max_count in python.
Each time a download_thread is started, I leave it alone and activate another one. I hope the download process could be ended in 5s, which means downloading is failed if opening the url costs more than 5s.
But how can I know it and stop the failed thread???

Can you tell which version of python you are using?
Maybe you could have posted a snippet too.
From Python 2.6, you have a timeout added in urllib2.urlopen.
Hope this will help you. It's from the python docs.
urllib2.urlopen(url[, data][,
timeout]) Open the URL url, which can
be either a string or a Request
object.
Warning HTTPS requests do not do any
verification of the server’s
certificate. data may be a string
specifying additional data to send to
the server, or None if no such data is
needed. Currently HTTP requests are
the only ones that use data; the HTTP
request will be a POST instead of a
GET when the data parameter is
provided. data should be a buffer in
the standard
application/x-www-form-urlencoded
format. The urllib.urlencode()
function takes a mapping or sequence
of 2-tuples and returns a string in
this format. urllib2 module sends
HTTP/1.1 requests with
Connection:close header included.
The optional timeout parameter
specifies a timeout in seconds for
blocking operations like the
connection attempt (if not specified,
the global default timeout setting
will be used). This actually only
works for HTTP, HTTPS and FTP
connections.
This function returns a file-like
object with two additional methods:
geturl() — return the URL of the
resource retrieved, commonly used to
determine if a redirect was followed
info() — return the meta-information
of the page, such as headers, in the
form of an mimetools.Message instance
(see Quick Reference to HTTP Headers)
Raises URLError on errors.
Note that None may be returned if no
handler handles the request (though
the default installed global
OpenerDirector uses UnknownHandler to
ensure this never happens).
In addition, default installed
ProxyHandler makes sure the requests
are handled through the proxy when
they are set.
Changed in version 2.6: timeout was
added.