I have a python code which recovers information from an HTTP API using the requests module. This code is run over and over again with an interval of few milliseconds between each call.
The HTTP API which I'm calling can send me 3 different responses, which can be:
text 'EMPTYFRAME' with HTTP status 200
text 'CAMERAUNAVAILABLE' with HTTP status 200
JPEG image with HTTP status 200
This is part of the code which handles this situation:
try:
r = requests.get(url,
auth=(username, pwd),
params={
'camera': camera_id,
'ds': int((datetime.now() - datetime.utcfromtimestamp(0)).total_seconds())
}
)
if r.text == 'CAMERAUNAVAILABLE':
raise CameraManager.CameraUnavailableException()
elif r.text == 'EMPTYFRAME':
raise CameraManager.EmptyFrameException()
else:
return r.content
except ConnectionError:
# handles the error - not important here
The critical part is the if/elif/else section, this comparison is taking way too long to complete and if I completely remove and simply replace it by return r.content, I have the performance I wish to, but checking for these other two responses other than the image is important for the application flow.
I also tried like:
if len(r.text) == len('CAMERAUNAVAILABLE'):
raise CameraManager.CameraUnavailableException()
elif len(r.text) == len('EMPTYFRAME'):
raise CameraManager.EmptyFrameException()
else:
return r.content
And:
if r.text[:17] == 'CAMERAUNAVAILABLE':
raise CameraManager.CameraUnavailableException()
elif r.text[:10] == 'EMPTYFRAME':
raise CameraManager.EmptyFrameException()
else:
return r.content
Which made it faster but still not as fast as I think this can get.
So is there a way to optimize this comparison?
EDIT
With the accepted answer, the final code is like this:
if r.headers['content-type'] == 'image/jpeg':
return r.content
elif len(r.text) == len('CAMERAUNAVAILABLE'):
raise CameraManager.CameraUnavailableException()
elif len(r.text) == len('EMPTYFRAME'):
raise CameraManager.EmptyFrameException()
Checking the response's Content-Type provided a much faster way to assure an image was received.
Comparing the whole r.text (which may contain the JPEG bytes) is probably slow.
You could compare the Content-Type header the server should set:
ct = r.headers['content-type']
if ct == "text/plain":
# check for CAMERAUNAVAILABLE or EMPTYFRAME
else:
# this is a JPEG
Related
I have a problem which I can't sort out for a day, and kindly asking for help of the community.
I have prepared Pandas DF which looks like this:
What I need to do:
Create URL links using 'name' and 'namespace' columns of df
Ask each URL
Save parameters from page if code = 200 and I have data, otherwise - find error code and save to the new column if df, called 's2s_status'
What I have for now:
start_slice = 1
total_end = 20
while start_slice <= total_end:
end_slice = start_slice + 1
if end_slice>total_end:
end_slice=total_end
var_name=df.loc[df.index == start_slice,'name'].values[0]
var_namespace=df.loc[df.index == start_slice, 'namespace'].values[0]
url=f"http://{var_name}.{var_namespace}.prod.s.o3.ru:84/config"
r=requests.get(url, timeout=(12600,1))
data=r.json()['Values']['s2s_auth_requests_sign_grpc']
if r.status_code == 404:
df['s2s_status']="404 error"
elif r.status_code == 500:
df['s2s_status']="500 Error"
elif r.status_code == 502:
df['s2s_status']="502 Error"
elif r.status_code == 503:
df['s2s_status']="503 Error"
else:
data=r.json()['Values']['s2s_auth_requests_sign_grpc']
df['s2s_status']="sign"
if end_slice == total_end:
break
else:
start_slice = end_slice
print(r)
print(url)
print(df)
This code iterates over first 20 records, but:
Brings wrong errors, e.g. page like 'http://exteca-auth.eea.prod.s.o3.ru:84/config' not found at all, but it gives me 404 error.
Less important, but still - didn't imagine how to handle cases when page won't return anything (no data with 200 code, not 404/500/502/503/etc. error)
Thank you in advance.
Request Method: PATCH
There is a Query String Parameter section
Your code cannot be run to reproduce your error. Check what you get as a response here:
r = requests.patch(url, headers=self._construct_header(),data=body)
response = getattr(r,'_content').decode("utf-8")
response_json = json.loads(response)
If you pass invalid json to json.loads(), then an error occurs with a similar message.
import json
response = b'test data'.decode("utf-8")
print(response)
response_json = json.loads(response)
print(response_json)
Output:
test data
Traceback (most recent call last):
...
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
EDIT:
In your case, to avoid an error, you need to add an if-else block. After receiving the response, you need to check what exactly you received.
r = requests.patch(url, headers=self._construct_header(),data=body)
# if necessary, check content type
print(r.headers['Content-Type'])
response = getattr(r,'_content').decode("utf-8")
if r.status_code == requests.codes.ok:
# make sure you get the string "success"
# if necessary, do something with the string
return response
else:
# if necessary, check what error you have: client or server errors, etc.
# or throw an exception to indicate that something went wrong
# if necessary, make sure you get the error in json format
# you may also get an error if the json is not valid
# since your api returns json formatted error message:
response_dict = json.loads(response)
return response_dict
In these cases, your function returns the string "success" or dict with a description of the error.
Usage:
data = {
'correct_prediction': 'funny',
'is_accurate': 'False',
'newLabel': 'funny',
}
response = aiservice.update_prediction(data)
if isinstance(response, str):
print('New Prediction Status: ', response)
else:
# provide error information
# you can extract error description from dict
I have an ELIF function to determine whether or not a website exists. The elif works, but is incredibly slow. I'd like to create a dictionary to apply the ELIF function to the list of URLs I have. Ideally I'm looking to get the outputs into a new table listing the URL and the result from the function.
I'm creating a dictionary for the potential outputs outlined in the elif statement posted below
check = {401:'web site exists, permission needed', 404:'web site does not exist'}
for row in df['sp_online']:
r = requests.head(row)
if r.status_code == 401:
print ('web site exists, permission needed')
elif r.status_code == 404:
print('web site does not exist')
else:
print('other')
How can I get the results of the confirmation function to show each url's result as a new column in the dataframe?
I think your should try a Thread or Multiprocessing approach. Instead of requesting one site at a time, you can pool n websites and wait for their responses. With ThreadPool you can achieve this with a few extra lines. Hope this is of use to you!
import requests
from multiprocessing.pool import ThreadPool
list_sites = ['https://www.wikipedia.org/', 'https://youtube.com', 'https://my-site-that-does-not-exist.com.does.not']
def get_site_status(site):
try:
response = requests.get(site)
except requests.exceptions.ConnectionError:
print("Connection refused")
return 1
if response.status_code == 401:
print('web site exists, permission needed')
elif response.status_code == 404:
print('web site does not exist')
else:
print('other')
return 0
pool = ThreadPool(processes=1)
results = pool.map_async(get_site_status, list_sites)
print('Results: {}'.format(results.get()))
I think you are looking for Series.map
df = pd.DataFrame({'status': [401, 404, 500]})
check = {401:'web site exists, permission needed', 404:'web site does not exist'}
print(df['status'].map(check))
prints
0 web site exists, permission needed
1 web site does not exist
2 NaN
Name: status, dtype: object
Assign to a new column in the normal way
df['new_col'] = df['status'].map(check)
I'm testing a flask application with py.test with the following code:
response = flask_app_test_client.post('/users', data=json.dumps(user))
assert response.status_code == 201
assert response.content_type == 'application/json'
assert isinstance(response.json, dict)
assert set(response.json.keys()) >= {'id', 'status', 'created_at', 'updated_at'}
assert response.json['name'] == user['name']
assert response.json['status'] == 'pending'
When some assertion fails I'm getting something like this:
response = test_client.post('/users', data=json.dumps(user))
> assert response.status_code == 201
E assert 400 == 201
E + where 400 = <JSONResponse streamed [400 BAD REQUEST]>.status_code
============== 1 failed, 3 passed in 0.10 seconds ===================
I do a lot of TDD so I expect my test fails frequently while developing. My problem is the assertion error message is kind of useless without the rest of the response data (body, headers, etc).
I only get in the output that the response.status_code is 400 but I don't get the error description that is in the response body: {"errors": ["username is already taken", "email is required"]}. Ideally I would like a full dump of the request and response (headers + body) when an assertion fails.
How I can print a summary of the response on each failed assertion?
Assert statement graamar
assert response.status_code == 201, "Anything you want"
You can be as verbose as you want. You can also use UnitTest's suite of helper methods -
without test case classes through this bit of abuse - https://github.com/nose-devs/nose2/blob/master/nose2/tools/such.py#L34
I'm came up with two different solutions.
Solution #1: try/catch
try:
assert response.status_code == 201
assert response.content_type == 'application/json'
assert isinstance(response.json, dict)
assert set(response.json.keys()) >= {'id', 'status', 'created_at', 'updated_at'}
assert response.json['name'] == user['name']
assert response.json['status'] == 'pending'
except AssertionError as e:
except AssertionError as e:
raise ResponseAssertionError(e, response)
class ResponseAssertionError(AssertionError):
def __init__(self, e, response):
response_dump = "\n + where full response was:\n" \
"HTTP/1.1 {}\n" \
"{}{}\n".format(response.status, response.headers, response.json)
self.args = (e.args[0] + response_dump,)
Solution #2: no try/catch needed (if repr is too long sometimes is cut off...)
Extend and override Flask response object
import json
class JSONResponse(Response):
def __repr__(self):
headers = {}
while len(self.headers) > 0:
tuple_ = self.headers.popitem()
headers[tuple_[0]] = tuple_[1]
data = {
'status': self.status,
'headers': headers,
'body': self.json
}
return json.dumps(data)
and
#pytest.fixture(scope='session')
def test_client(flask_app):
flask_app.response_class = JSONResponse
return flask_app.test_client()
I know this is an older question, but Pytest has an option --pdb that will pop you into a PDB shell should your test fail. Very handy way to "just look around" rather than having to pass tons of stuff to an exception message.
i have an python API call, and the server response is coming to me as JSON output.
How can I assert that the "status" from this output is 0 for example:
def test_case_connection():
req = requests.get_simple(url=Server.MY_SERVER, params=my_vars)
assert req["status"]="0"
is what i've tried.
The response is looking like:
{"status" : 0, ......}
error i got is:
TypeError: 'Response' object has no attribute '__getitem__'
If you simply need to check that the request was successful, using request.status_code will do the trick:
def test_case_connection():
req = requests.get_simple(url=Server.MY_SERVER, params=my_vars)
assert req.status_code == 200
If you want instead to check for the presence of a specific key-value pair in the response, you need to convert your response payload from json to a dict:
import json
def test_case_connection():
req = requests.get_simple(url=Server.MY_SERVER, params=my_vars)
data = json.loads(req.content)
assert data["status"] == "0"
If you are using the Requests library you can avoid converting json manually by using its builtin json decoder.
It should be assert req['status'] == 0, i. e. comparison (==) instead of assignment (=) and 0 as integer not "0" as string (not entirely sure about the latter).
status code in assertion:
response.ok #it is True if response status code is 200.
In context with pytest it would be like that:
#pytest.mark.parametrize("fixture_name", [(path, json)], indirect=True)
def test_the_response_status_code_first(fixture_name):
assert fixture_name.ok, "The message for the case if the status code != 200."
# the same with checking status code directly:
assert fixture_name.status_code == 200, "Some text for failure. Optional."