I'm trying to create a post request in python, but I get an internal server error when issuing the request.
I'm trying to intercept it with a try-statement, but that doesn't seem to work.
import logging
import requests
logging.basicConfig(filename='python.log', filemode='w', level=logging.DEBUG)
url = "https://redacted-url.com/my-api/check_email"
json = {"email":request.params["email"].strip(), "list":request.params["list"]}
headers = {"Content-Type":"application/json", "Accept": "text/plain"}
try:
r = requests.post(url, headers=headers, json=json)
except requests.exceptions.RequestException as e:
logging.error(e, exc_info=True)
A: I have no idea where that logging-file would be stored. Do I have to add the full server-part? What If I just use «python.log»? Where would it be stored?
B: the try/except doesn't seem to work, I still get an internal server error
C: the error definitely occurs on the line r = requests.post(url, headers=headers, json=json). If I comment that out, the error doesn't occur.
D: Since I don't get an error that's meaningful: What am I doing wrong with that request? This is actually my main problem, but it would be nice to figure out how to log that error and how to intercept it.
Last but not least: If I run the same command from the terminal, the request is processed fine. WTH???
Related
i am trying to trigger jenkins job via python requests.POST()
reply = requests.post(url,
data=data,
cookies=self.__cookies,
auth=(self.__user, self.__pass),
files=files,
)
Received error:
No valid crumb was included in request for //job/Deploy/job/JOBNAME/buildWithParameters by . Returning 403.
Then Tried to pass crumb in header using the following code:
session = requests.Session()
crum = session.get('https://xxxxxx.xxx.net:7xxx/crumbIssuer/api/json',
auth=(self.__user, self.__pass)).json()
reply = requests.post(url,
data=data,
cookies=self.__cookies,
params={crum['crumbRequestField']: crum['crumb']},
auth=(self.__user, self.__pass),
headers={crum['crumbRequestField']: crum['crumb']},
files=files,
)
Received following error:
Found invalid crumb 09ee0x69284910425bd94378a1735b3471826e8d5ec077c526562. If you are calling this URL with a script, please use the API Token instead. More information: https://jenkins.io/redirect/crumb-cannot-be-used-for-script
we are looking for a solution with via requests.post()
To get it to work you need a valid session, auth and crumb. We also use tokens, so I've added that as well
with requests.Session() as s:
s.auth = auth
crumb = s.get(jenkins_url + "crumbIssuer/api/json").json()['crumb']
s.headers.update({"token": token, "Jenkins-Crumb": crumb})
then we can
s.post(url)
and we get back a 201 Created. It looks like you got everything except you're not doing it inside the session, but instead with a bare request. It's probably sufficient to just replace the requests.post with session.post
I'm using Python 3.7 with urllib.
All work fine but it seems not to athomatically redirect when it gets an http redirect request (307).
This is the error i get:
ERROR 2020-06-15 10:25:06,968 HTTP Error 307: Temporary Redirect
I've to handle it with a try-except and manually send another request to the new Location: it works fine but i don't like it.
These is the piece of code i use to perform the request:
req = urllib.request.Request(url)
req.add_header('Authorization', auth)
req.add_header('Content-Type','application/json; charset=utf-8')
req.data=jdati
self.logger.debug(req.headers)
self.logger.info(req.data)
resp = urllib.request.urlopen(req)
url is an https resource and i set an header with some Authhorization info and content-type.
req.data is a JSON
From urllib documentation i've understood that the redirects are authomatically performed by the the library itself, but it doesn't work for me. It always raises an http 307 error and doesn't follow the redirect URL.
I've also tried to use an opener specifiyng the default redirect handler, but with the same result
opener = urllib.request.build_opener(urllib.request.HTTPRedirectHandler)
req = urllib.request.Request(url)
req.add_header('Authorization', auth)
req.add_header('Content-Type','application/json; charset=utf-8')
req.data=jdati
resp = opener.open(req)
What could be the problem?
The reason why the redirect isn't done automatically has been correctly identified by yours truly in the discussion in the comments section. Specifically, RFC 2616, Section 10.3.8 states that:
If the 307 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
Back to the question - given that data has been assigned, this automatically results in get_method returning POST (as per how this method was implemented), and since that the request method is POST, and the response code is 307, an HTTPError is raised instead as per the above specification. In the context of Python's urllib, this specific section of the urllib.request module raises the exception.
For an experiment, try the following code:
import urllib.request
import urllib.parse
url = 'http://httpbin.org/status/307'
req = urllib.request.Request(url)
req.data = b'hello' # comment out to not trigger manual redirect handling
try:
resp = urllib.request.urlopen(req)
except urllib.error.HTTPError as e:
if e.status != 307:
raise # not a status code that can be handled here
redirected_url = urllib.parse.urljoin(url, e.headers['Location'])
resp = urllib.request.urlopen(redirected_url)
print('Redirected -> %s' % redirected_url) # the original redirected url
print('Response URL -> %s ' % resp.url) # the final url
Running the code as is may produce the following
Redirected -> http://httpbin.org/redirect/1
Response URL -> http://httpbin.org/get
Note the subsequent redirect to get was done automatically, as the subsequent request was a GET request. Commenting out req.data assignment line will result in the lack of the "Redirected" output line.
Other notable things to note in the exception handling block, e.read() may be done to retrieve the response body produced by the server as part of the HTTP 307 response (since data was posted, there might be a short entity in the response that may be processed?), and that urljoin is needed as the Location header may be a relative URL (or simply has the host missing) to the subsequent resource.
Also, as a matter of interest (and for linkage purposes), this specific question has been asked multiple times before and I am rather surprised that they never got any answers, which follows:
How to handle 307 redirection using urllib2 from http to https
HTTP Error 307: Temporary Redirect in Python3 - INTRANET
HTTP Error 307 - Temporary redirect in python script
API http://admin.wechat.com/wiki/index.php?title=Transferring_Multimedia_Files
import requests
r='https://api.wechat.com/cgi-bin/token?grant_type=client_credential&appid=wx82c6ebdb6e33ad33&secret=c2861ec348b3c94087c4b64cbe166fbb' #credentials sharing no problem
a=(requests.get(r).json()['access_token'])
print(a)
params = (
('access_token', a),
('type', 'image'),
)
import os
files = {
'media': ('1.jpg', open('static/1.jpg', 'rb'),'image/jpg',),
}
print()
response = requests.post('http://file.api.wechat.com/cgi-bin/media/upload', params=params, files=files)
def uprint(x,file=None):
try:
pass
if x:
print(x.encode('cp65001',errors='backslashreplace').decode('cp1252'),file=file)
except Exception as e:
return f'{x}\n{e}'
def prin(*a):print(ascii(*a))
print(response.text,file=open('z.html','a',encoding="utf-8"))
print(response.headers)
Looks like you are using http for the upload call. I have seen this error returned from sites for this reason before.
The HTTP error 503 means "Service Unavailable". It is usually returned by a server when it is unable to handle the request due to a temporary overloading or maintenance of the server.
After checking the API documentation for wechat, I notided this:
This API must be used via HTTPS.
And then I noticed this in the Q&A:
Q: Which server should I send API requests to?
A: If you have an International Official Account, use api.wechat.com.
If you have a China Official Account, use api.weixin.qq.com.
So, in your case I think you need to use https, and the domain api.wechat.com, like this:
response = requests.post('https://api.wechat.com/cgi-bin/media/upload', params=params, files=files)
When I run code locally and try to fetch data from URL and then parse it to text everything work properly.
When I run exactly the same code on the remote server and try to fetch data from URL error HTTP Error 403: Forbidden occur
Answers from questions:
HTTP error 403 in Python 3 Web Scraping,
urllib2.HTTPError: HTTP Error 403: Forbidden helped me when I tried to run it locally and everything work fine.
Do you know what can be different in fetching data from remote server while code is the same(locally and on the server) and way of running code is the same but result is absolutely different?
URL that I want to fetch:
url=https://bithumb.cafe/notice
Code that I was trying to use to fetch data(once it work, second not)
try:
request = urllib.request.Request(url)
request.add_header('User-Agent', 'cheese')
logger.info("request: {}".format(request))
content = urllib.request.urlopen(request).read()
logger.info('content: {}'.format(content))
decoded = content.decode('utf-8')
logger.info('content_decoded: {}'.format(decoded))
return decoded
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''`
second way of fetching data(also work locally but on the remote server not):
class AppURLopener(urllib.request.FancyURLopener):
version = "Mozilla/5.0"
method:
try:
opener = AppURLopener()
response = opener.open(url)
logger.info("request response: {}. response type: {}. response_dict: {}"
.format(response, type(response), response.__dict__))
html_response = response.read()
logger.info("html_Response".format(html_response))
encoding = response.headers.get_content_charset('utf-8')
decoded_html = html_response.decode(encoding)
logger.info('content_decoded: {}'.format(decoded_html))
return decoded_html
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''
How do I access the error stream using the Python requests library? For example, using HttpURLConnection in Java, I would do something like:
InputStream errorStream = conn.getErrorStream();
Is there a function like this with requests?
I'm looking for the error message in the response that was supplied by the source, NOT the error status e.g. Internal Server Error
requests won't raise an exception if HTTP error occurs, but you can get the error message in the response content. Example:
url = 'https://stackoverflow.com/does_not_exist'
r = requests.get(url)
print(r.status_code, r.reason)
print(r.text)