We Keep Coding

Getting HTTPError on API request

I recently started learning Python 3 and am trying to write my first program. The essence of the program is the auto-display of items on the trading floor. I use the API https://market.csgo.com/docs-v2. Everything would be fine, if not the errors that appear while the script is running. I know to use "TRY and EXECPT", but how to do it right? My code:
while True:
try:
ip = {'18992549780':'10000', '18992548863':'20000','18992547710':'30000','18992546824':'40000', '18992545927':'50000', '18992544515':'60000', '18992543504':'70000', '18992542365':'80000', '18992541028':'90000', '18992540218':'100000'}
for key,value in ip.items():
url3 = ('https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id={id}&price={price}&cur=RUB')
addtosale = url3.format(id = key, price = value)
onsale = requests.get(addtosale)
onsale.raise_for_status()
r = onsale.json()
print(addtosale)
print(onsale.raise_for_status)
print(r)
time.sleep(5)
except requests.HTTPError as exception:
print(exception)
My task is to run this piece of code between TRY and EXCEPT again on any error (5xx for example)
Traceback (most recent call last):
File "D:\Python\tmsolve1.py", line 30, in <module>
onsale.raise_for_status()
File "C:\Users\���������\AppData\Local\Programs\Python\Python38-32\lib\site-packages\requests\models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id=18992545927&price=50000&cur=RUB
502 Server Error: Bad Gateway for url: https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id=18992549780&price=10000&cur=RUB

Error handling can be done in multiple ways. You have 10 API calls. You can either stop the code on first error, retry the request or continue with additional call.
The example below will continue running through all requests.
Also except requests.HTTPError as exception may not be needed. This error is thrown by response.raise_for_status(). You can preform logging before calling .raise_for_status(). The try/catch only allows the code to continue in the loop.
import requests
import time
import json
# while True: # This will make the code loop continuously
try:
ip = {'18992549780':'10000', '18992548863':'20000','18992547710':'30000','18992546824':'40000', '18992545927':'50000', '18992544515':'60000', '18992543504':'70000', '18992542365':'80000', '18992541028':'90000', '18992540218':'100000'}
for key,value in ip.items():
url= 'https://market.csgo.com/api/v2/add-to-sale'
payload = {'key': 'MYAPIKEY', 'id': id, 'price': value, 'cur': 'RUB'}
response = requests.get(url, params=payload)
print(f'Status code: { response.status_code}')
print(f'Response text: { response.text}') # This will contain an error message or json results.
response.raise_for_status() # This will only error if status code is 4xx or 5xx
results = response.json()
if results.get('error'): # "results" can contains {"error":"Bad KEY","success":false}
raise Exception('Error in response json')
print(json.dumps(results))
time.sleep(5)
except requests.HTTPError as exception: # Captures response.raise_for_status() - 4xx or 5xx status code. If you remove this, then code will use generic handle
print(exception)
except Exception as exception: # Generic error handler for raise Exception('Error in response json') and "Max retries exceeded."
print(exception)

HTTP Error 403: Forbidden while fetching html source on the server

When I run code locally and try to fetch data from URL and then parse it to text everything work properly.
When I run exactly the same code on the remote server and try to fetch data from URL error HTTP Error 403: Forbidden occur
Answers from questions:
HTTP error 403 in Python 3 Web Scraping,
urllib2.HTTPError: HTTP Error 403: Forbidden helped me when I tried to run it locally and everything work fine.
Do you know what can be different in fetching data from remote server while code is the same(locally and on the server) and way of running code is the same but result is absolutely different?
URL that I want to fetch:
url=https://bithumb.cafe/notice
Code that I was trying to use to fetch data(once it work, second not)
try:
request = urllib.request.Request(url)
request.add_header('User-Agent', 'cheese')
logger.info("request: {}".format(request))
content = urllib.request.urlopen(request).read()
logger.info('content: {}'.format(content))
decoded = content.decode('utf-8')
logger.info('content_decoded: {}'.format(decoded))
return decoded
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''`
second way of fetching data(also work locally but on the remote server not):
class AppURLopener(urllib.request.FancyURLopener):
version = "Mozilla/5.0"
method:
try:
opener = AppURLopener()
response = opener.open(url)
logger.info("request response: {}. response type: {}. response_dict: {}"
.format(response, type(response), response.__dict__))
html_response = response.read()
logger.info("html_Response".format(html_response))
encoding = response.headers.get_content_charset('utf-8')
decoded_html = html_response.decode(encoding)
logger.info('content_decoded: {}'.format(decoded_html))
return decoded_html
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''

python, when http response is None, how to get a response code

In python, when a http request is invalid, response is None, in this case, how to get the response code from the response? The invalid request in my code are caused by two reasons, one is a invalid token, I expect to get 401 in this case, another reason is invalid parameter, I expect to get 400 in this case, but under both cases, response is always None and I'm not able to get the response code by calling response.getcode(), how to solve this?
req = urllib2.Request(url)
response = None
try: response = urllib2.urlopen(req)
except urllib2.URLError as e:
res_code = response.getcode() #AttributeError: 'NoneType' object has no attribute 'getcode'

You can't get the status code when URLError is raised. Because when it is raised (ex: DNS couldn't resolve domain name), it means request hasn't been sent to server yet so there is no HTTP response generated.
In your scenario, (for 4xx HTTP status code), urllib2 throws HTTPError so you can derive the status code from it.
The documentation says:
code
An HTTP status code as defined in RFC 2616. This numeric value corresponds to a value found in the dictionary of codes as found in BaseHTTPServer.BaseHTTPRequestHandler.responses.
import urllib2
request = urllib2.Request(url)
try:
response = urllib2.urlopen(request)
res_code = response.code
except urllib2.HTTPError as e:
res_code = e.code
Hope this helps.

Get URL when handling urllib2.URLError

This pertains to urllib2 specifically, but custom exception handling more generally. How do I pass additional information to a calling function in another module via a raised exception? I'm assuming I would re-raise using a custom exception class, but I'm not sure of the technical details.
Rather than pollute the sample code with what I've tried and failed, I'll simply present it as a mostly blank slate. My end goal is for the last line in the sample to work.
#mymod.py
import urllib2
def openurl():
req = urllib2.Request("http://duznotexist.com/")
response = urllib2.urlopen(req)
#main.py
import urllib2
import mymod
try:
mymod.openurl()
except urllib2.URLError as e:
#how do I do this?
print "Website (%s) could not be reached due to %s" % (e.url, e.reason)

You can add information to and then re-raise the exception.
#mymod.py
import urllib2
def openurl():
req = urllib2.Request("http://duznotexist.com/")
try:
response = urllib2.urlopen(req)
except urllib2.URLError as e:
# add URL and reason to the exception object
e.url = "http://duznotexist.com/"
e.reason = "URL does not exist"
raise e # re-raise the exception, so the calling function can catch it
#main.py
import urllib2
import mymod
try:
mymod.openurl()
except urllib2.URLError as e:
print "Website (%s) could not be reached due to %s" % (e.url, e.reason)

I don't think re-raising the exception is an appropriate way to solve this problem.
As #Jonathan Vanasco said,
if you're opening a.com , and it 301 redirects to b.com , urlopen will automatically follow that because an HTTPError with a redirect was raised. if b.com causes the URLError , the code above marks a.com as not existing
My solution is to overwrite redirect_request of urllib2.HTTPRedirectHandler
import urllib2
class NewHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
m = req.get_method()
if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
or code in (301, 302, 303) and m == "POST"):
newurl = newurl.replace(' ', '%20')
newheaders = dict((k,v) for k,v in req.headers.items()
if k.lower() not in ("content-length", "content-type")
)
# reuse the req object
# mind that req will be changed if redirection happends
req.__init__(newurl,
headers=newheaders,
origin_req_host=req.get_origin_req_host(),
unverifiable=True)
return req
else:
raise HTTPError(req.get_full_url(), code, msg, headers, fp)
opener = urllib2.build_opener(NewHTTPRedirectHandler)
urllib2.install_opener(opener)
# mind that req will be changed if redirection happends
#req = urllib2.Request('http://127.0.0.1:5000')
req = urllib2.Request('http://www.google.com/')
try:
response = urllib2.urlopen(req)
except urllib2.URLError as e:
print 'error'
print req.get_full_url()
else:
print 'normal'
print response.geturl()
let's try to redirect the url to an unknown url:
import os
from flask import Flask,redirect
app = Flask(__name__)
#app.route('/')
def hello():
# return 'hello world'
return redirect("http://a.com", code=302)
if __name__ == '__main__':
port = int(os.environ.get('PORT', 5000))
app.run(host='0.0.0.0', port=port)
And the result is:
error
http://a.com/
normal
http://www.google.com/

Response Code 202, Not a Qualified Error Code

I'm working with an API which I post files to. However, when I receive the response, the HTTP status code is a 202. This is to be expected, but in addition the API will also respond with XML content.
So in my try/except block urllib2.urlopen will result in a raised urllib2.HTTPError and destroying the XML content.
try:
response = urllib2.urlopen(req)
except urllib2.HTTPError, http_e:
if http_e.code == 202:
print 'accepted!'
pass
print response.read() # UnboundLocalError: local variable 'response' referenced before assignment
How can I expect the 202 and keep the response content, but not raise an error?

Edit
Being silly, I forgot to inspect the exception that is returned by urllib2. It features all of the properties I've been waxing on about for httplib. This should do the trick for you:
try:
urllib2.urlopen(req)
except urllib2.HTTPError, e:
print "Response code",e.code # prints 404
print "Response body",e.read() # prints the body of the response...
# ie: your XML
print "Headers",e.headers.headers
Original
In this case, given that you're using HTTP as your transport protocol, you'll probably have more luck with the httplib library:
>>> import httplib
>>> conn = httplib.HTTPConnection("www.stackoverflow.com")
>>> conn.request("GET", "/dlkfjadslkfjdslkfjd.html")
>>> r = conn.getresponse()
>>> r.status
301
>>> r.reason
'Moved Permanently'
>>> r.read()
'<head><title>Document Moved</title></head>\n<body><h1>Object Moved</h1>
This document may be found
here</body>'
You can further use r.getheaders() and so forth to inspect other aspects of the response.