I'm reading some source code which raises errors for bad requests in the following way:
import requests
response = requests.get("www.google.com") # This won't work because it's missing the http://
if response.ok is False or response.json()['status'] != 'success':
raise Exception("API error: {}".format(response.json()['message']))
I was thinking that the last two lines could be replaced with
response.raise_for_status()
I actually don't see any difference in the error returned. In both cases it is
Traceback (most recent call last):
File "/home/kurt/Documents/Scratch/requests_test.py", line 3, in <module>
response = requests.get("www.google.com") # This won't work because it's missing the http://
File "/home/kurt/.local/lib/python2.7/site-packages/requests/api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "/home/kurt/.local/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/home/kurt/.local/lib/python2.7/site-packages/requests/sessions.py", line 451, in request
prep = self.prepare_request(req)
File "/home/kurt/.local/lib/python2.7/site-packages/requests/sessions.py", line 382, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/home/kurt/.local/lib/python2.7/site-packages/requests/models.py", line 304, in prepare
self.prepare_url(url, params)
File "/home/kurt/.local/lib/python2.7/site-packages/requests/models.py", line 362, in prepare_url
to_native_string(url, 'utf8')))
requests.exceptions.MissingSchema: Invalid URL 'www.google.com': No schema supplied. Perhaps you meant http://www.google.com?
It seems to me that raise_for_status() is more succinct and perhaps also doesn't lose information on the original exception (cf. Use of "except Exception" vs. "except ... raise" in Python). Would this indeed be a better approach?
response.raise_for_status() only raises an exception if the response status code is not a 200 response. The second case, where response.json()['status'] != 'success' is true, is not covered.
You have a different error, however. You never get to the if test, because the exception is raised by the requests.get() call. You failed to pass in a schema (no http:// or https:// at the start). Because the exception is raised in the requests.get() expression, the next line is simply never executed. No request is even sent, so you can't make any assertions about the response either.
The test has more problems:
requests.ok is False is not idiomatic Python; you'd use not requests.ok instead
If requests.ok is False is true, then the requests.json() call will most likely fail altogether, so the next line using response.json()['message']) will produce a different exception.
Related
(I used a translator when writing this article. Please understand that some words may be incorrect.)
I tested it using the requests module. If the site cannot be found, a 404 code should be returned, but with an error. I don't know what the reason is. Any help would be appreciated. How to properly return a 404 code?
---Below is the code.
import requests as re
a = re.get(input())
print(a.status_code)
error :
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\82104_dvfqr9f\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\82104_dvfqr9f\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\82104_dvfqr9f\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 515, in request
prep = self.prepare_request(req)
File "C:\Users\82104_dvfqr9f\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 443, in prepare_request
p.prepare(
File "C:\Users\82104_dvfqr9f\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 318, in prepare
self.prepare_url(url, params)
File "C:\Users\82104_dvfqr9f\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 392, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'eeee.com': No scheme supplied. Perhaps you meant http://eeee.com?
You can use this link for how to work with requests module.
import requests
try:
r = requests.get('https://www.google.com/search?q=ggg')
print(r.status_code)
if r.status_code==404:
print("this url dosn't exist")
except Exception as error:
print(error)
Can anyone please tell me How do I remove this Traceback (most recent call last): Error in python. I am using python 2.7.9
Take a look over the code.
import requests
import optparse
parser = optparse.OptionParser()
parser.add_option("-f", '--filename', action="store" ,dest="filee")
options, args = parser.parse_args()
file = options.filee
fopen = open(file, 'r')
for x in fopen.readlines():
print "Checking for Clickjacking vulnerability\n"
url = x.strip('\n')
req = requests.get(url)
try:
print "[-]Target:" + url + " Not vulnerable\n The targeted victim has %s header\n" % (req.headers['X-Frame-Options'])
except Exception as e:
print "[+] Target:" + url +" Vulnerable to clickjacking"
After running the code successfully I go this error at the end
Traceback (most recent call last):
File "C:\Python27\utkarsh3.py", line 17, in <module>
req = requests.get(url)
File "C:\Python27\lib\site-packages\requests\api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 494, in request
prep = self.prepare_request(req)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 437, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "C:\Python27\lib\site-packages\requests\models.py", line 305, in prepare
self.prepare_url(url, params)
File "C:\Python27\lib\site-packages\requests\models.py", line 379, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?
Which really irritate me. I know there are so many peoples who are already asking this before. But I can't understand it so I ask.
And please tell me How we beginners handle these errors?
in a eli5 fashion, a Traceback is a log of what the program was trying to do before actual error happened. Your actual error is requests.exceptions.MissingSchema
The line that follows Invalid URL '': No schema supplied. Perhaps you meant http://? describes the exact problem.
File "C:\Python27\utkarsh3.py", line 17, in <module>
req = requests.get(url)
These above lines describe where the error started..
So, if you go to line 17 of your program you must see this exact same line.
Making a context out of these two things, i get that url is a string that is just example.com and not http://example.com or something on those lines.
I can only speculate so much on what your code might be. But, feel free to provide your code snippets to explain more.
But, hope this helps you to read future tracebacks.
Edit1 : Now that you added snippet. Try printing url just before requests.get(url) and see what exactly you are trying to reach. And, if you have the right schema prepended.
I am unable to PUT Nest data like ambient_temperature_f to https://developer-api.nest.com or the redirected Firebase URL. I suspect there is something specific to the Nest that will need to be tweaked in the Firebase module I am using (https://ozgur.github.io/python-firebase/).
From firebase.py:
#http_connection(60)
def make_put_request(url, data, params, headers, connection):
"""
Helper function that makes an HTTP PUT request to the given firebase
endpoint. Timeout is 60 seconds.
`url`: The full URL of the firebase endpoint (DSN appended.)
`data`: JSON serializable dict that will be stored in the remote storage.
`params`: Python dict that is appended to the URL like a querystring.
`headers`: Python dict. HTTP request headers.
`connection`: Predefined HTTP connection instance. If not given, it
is supplied by the `decorators.http_connection` function.
The returning value is a Python dict deserialized by the JSON decoder. However,
if the status code is not 2x or 403, an requests.HTTPError is raised.
connection = connection_pool.get_available_connection()
response = make_put_request('http://firebase.localhost/users',
'{"1": "Ozgur Vatansever"}',
{'X_FIREBASE_SOMETHING': 'Hi'}, connection)
response => {'1': 'Ozgur Vatansever'} or {'error': 'Permission denied.'}
"""
timeout = getattr(connection, 'timeout')
response = connection.put(url, data=data, params=params, headers=headers, timeout=timeout)
print('[make_put_request]: [%s][%s][%s][%s]\n' %(url, data, params, headers))
if response.ok or response.status_code == 403:
return response.json() if response.content else None
else:
response.raise_for_status()
Prints out:
[make_put_request]: [https://developer-api.nest.com/devices/thermostats/DEVICE_ID/ambient_temperature_f.json?auth=VALID_AUTH_TOKEN][71][{}][{}]
Returns error:
Traceback (most recent call last):
File "C:\py\nest_auth.py", line 90, in <module>
put_result = firebase.put(data_url, field_name, 71)
File "C:\Python34\lib\site-packages\firebase\decorators.py", line 19, in wrapped
return f(*args, **kwargs)
File "C:\Python34\lib\site-packages\firebase\firebase.py", line 312, in put
connection=connection)
File "C:\Python34\lib\site-packages\firebase\decorators.py", line 19, in wrapped
return f(*args, **kwargs)
File "C:\Python34\lib\site-packages\firebase\firebase.py", line 77, in make_put_request
response.raise_for_status()
File "C:\Python34\lib\site-packages\requests\models.py", line 808, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request
This worked when using a firebaseio.com target but isn't working for Nest:
put_result = firebase.put('/devices/thermostats/DEVICE_ID/', 'ambient_temperature', 71)
According to the documentation ambient_temperature_f is a read only field that represent's the reported ambient temperature from the thermostat. It wouldn't make sense to override that since it is a sensor reading.
I think you want to write to target_temperature_f, which is the temperature the thermostat should heat or cool to.
I wrote a feature to run a task asynchronously via celery, tested it locally and it's all good. Shipped to my staging environment and when the celery tries to consume the tasks it fails with the following traceback.
I'm not even sure how I can go about debugging this error, as it's being called by celery, and happening deep in the python standard lib. Any ideas?
Traceback (most recent call last):
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
return self.run(*args, **kwargs)
File "/home/ubuntu/Hypnos/hypnos/recs_jobber/tasks.py", line 5, in send_sms_action
msg = twilio_client.sms.messages.create(body = sms_action.body, to=sms_action.to_number, from_=TW_NUMBER)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/twilio/rest/resources/sms_messages.py", line 167, in create
return self.create_instance(kwargs)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/twilio/rest/resources/base.py", line 352, in create_instance
data=transform_params(body))
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/twilio/rest/resources/base.py", line 204, in request
resp = make_twilio_request(method, uri, auth=self.auth, **kwargs)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/twilio/rest/resources/base.py", line 129, in make_twilio_request
resp = make_request(method, uri, **kwargs)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/twilio/rest/resources/base.py", line 101, in make_request
resp, content = http.request(url, method, headers=headers, body=data)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1252, in _conn_request
conn.connect()
File "/home/ubuntu/hypnos-venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1017, in connect
sock.settimeout(self.timeout)
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
TypeError: a float is required
Used Celery's rdb to get into the frame and seems that the socket timeout is not set.
(Pdb) self.timeout
<Unset Timeout Value>
Any ideas how I might solve this? The error is flowing from Twilio -> httplib2 -> socket.py which is a wrapper to _socket. This is over my head and not sure even how to approach the problem.
By default, Twilio's lib sets timeout to:
class _UnsetTimeoutKls(object):
""" A sentinel for an unset timeout. Defaults to the system timeout. """
def __repr__(self):
return '<Unset Timeout Value>'
# None has special meaning for timeouts, so we use this sigil to indicate
# that we don't care
UNSET_TIMEOUT = _UnsetTimeoutKls()
So something must be happening in between instantiating a TwilioRestClient and the actual socket call that is evaluating that _UnsetTimeoutKls to something different than None.
Setting timeout to None when initialising TwilioRestClient seems to fix the error:
self.client = TwilioRestClient(TWILIO_SID,
TWILIO_AUTH_TOKEN,
timeout=None)
Here's what's going on:
You're telling the Twilio library to create an SMS message.
Twilio goes over to httplib to connect to the Twilio servers.
httplib, while connecting to the Twilio servers, sets the timeout of a socket.
The only problem is that for some reason, self.timeout in the penultimate stack frame is not a float as required. You may want to try running your application under the Python debugger, e.g.:
python -m pdb myapp.py
You'll get to a prompt from which you can type run to run your application. Once the error occurs, it should drop you back to the prompt. You should then be able to type up until you get to the offending frame and see what self.timeout is. You may then want to look around to see where self.timeout is getting set to that, and why. You should then be able to resolve the issue by fixing that.
I am seeing the python-requests library crash with the following traceback:
Traceback (most recent call last):
File "/usr/lib/python3.2/http/client.py", line 529, in _read_chunked
chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./app.py", line 507, in getUrlContents
response = requests.get(url, headers=headers, auth=authCredentials, timeout=http_timeout_seconds)
File "/home/dotancohen/code/lib/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/home/dotancohen/code/lib/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/home/dotancohen/code/lib/requests/sessions.py", line 338, in request
resp = self.send(prep, **send_kwargs)
File "/home/dotancohen/code/lib/requests/sessions.py", line 441, in send
r = adapter.send(request, **kwargs)
File "/home/dotancohen/code/lib/requests/adapters.py", line 340, in send
r.content
File "/home/dotancohen/code/lib/requests/models.py", line 601, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/home/dotancohen/code/lib/requests/models.py", line 542, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/home/dotancohen/code/lib/requests/packages/urllib3/response.py", line 222, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/home/dotancohen/code/lib/requests/packages/urllib3/response.py", line 173, in read
data = self._fp.read(amt)
File "/usr/lib/python3.2/http/client.py", line 489, in read
return self._read_chunked(amt)
File "/usr/lib/python3.2/http/client.py", line 534, in _read_chunked
raise IncompleteRead(b''.join(value))
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
self.run()
File "./app.py", line 298, in run
self.target(*self.args)
File "./app.py", line 400, in provider_query
url_contents = getUrlContents(str(providerUrl), '', authCredentials)
File "./app.py", line 523, in getUrlContents
except http.client.IncompleteRead as error:
NameError: global name 'http' is not defined
As can be seen, I've tried to catch the http.client.IncompleteRead: IncompleteRead(0 bytes read) error that requests is throwing with the line except http.client.IncompleteRead as error:. However, that is throwing a NameError due to http not being defined. So how can I catch that exception?
This is the code throwing the exception:
import requests
from requests_oauthlib import OAuth1
authCredentials = OAuth1('x', 'x', 'x', 'x')
response = requests.get(url, auth=authCredentials, timeout=20)
Note that I am not including the http library, though requests is including it. The error is very intermittent (happens perhaps once every few hours, even if I run the requests.get() command every ten seconds) so I'm not sure if added the http library to the imports has helped or not.
In any case, in the general sense, if included library A in turn includes library B, is it impossible to catch exceptions from B without including B myself?
To answer your question
In any case, in the general sense, if included library A in turn includes library B, is it impossible to catch exceptions from B without including B myself?
Yes. For example:
a.py:
import b
# do some stuff with b
c.py:
import a
# but you want to use b
a.b # gives you full access to module b which was imported by a
Although this does the job, it doesn't look so pretty, especially with long package/module/class/function names in real world.
So in your case to handle http exception, either try to figure out which package/module within requests imports http and so that you'd do raise requests.XX.http.WhateverError or rather just import it as http is a standard library.
It's hard to analyze the problem if you don't give source and just the stout,
but check this link out : http://docs.python-requests.org/en/latest/user/quickstart/#errors-and-exceptions
Basically,
try and catch the exception whereever the error is rising in your code.
Exceptions:
In the event of a network problem (e.g. DNS failure, refused connection, etc),
Requests will raise a **ConnectionError** exception.
In the event of the rare invalid HTTP response,
Requests will raise an **HTTPError** exception.
If a request times out, a **Timeout** exception is raised.
If a request exceeds the configured number of maximum redirections,
a **TooManyRedirects** exception is raised.
All exceptions that Requests explicitly raises inherit
from **requests.exceptions.RequestException.**
Hope that helped.