I wrote this function to call weather api
def weather_data(df):
import json
query_data = urllib.request.urlopen(f"https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/timeline/{df['Latitude']},{df['Longitude']}/{df['date_time_epoch']}?key= any key ID")
json_decode = query_data.read().decode('utf-8')
extracted_info = json.loads(json_decode)
temp = extracted_info['days'][0]['temp']
feelslike = extracted_info['days'][0]['feelslike']
return pd.Series([temp, feelslike])
When I test to call 10K-35K records, There is no problem. However, when I run it to call 500K ->400K -> 200K record, it shows HTTPError: HTTP Error 502: Bad Gateway. I pay the subscription fee to use this service.
Does anyone know how to fix this?
Related
I'm not quite sure why this piece of code fails, please I'd be happy to hear your thoughts. I've used examples from boto3 and its works in general, but I'd get an AttributeError all of a sudden in some cases. Please explain what I am doing wrong because if a payload is None, I will get a JSON decoding error but not a None object.
Here is a simplified version of a code that causes the exception.
import boto3
import json
client = boto3.client('lambda', region_name='...')
res = client.invoke(
FunctionName='func',
InvocationType='RequestResponse',
Payload=json.dumps({'param': 123})
)
payload = json.loads(res["Payload"].read().decode('utf-8'))
for k in payload.keys():
print(f'{k} = {payload[k]}')
The error
----
[ERROR] AttributeError: 'NoneType' object has no attribute 'keys'
Traceback (most recent call last):
.....
Just replicated your issue in my environment by creating a lambda that doesn't return anything and calling it using boto3. The "null" object passes through the json loads without error but doesn't have any keys since it's not a dictionary.
def lambda_handler(event, context):
pass
I created my code just like yours and got the same error. Weirdly, I was able to get the json error by attempting to print out the streaming body with this line
print(res["Payload"].read().decode('utf-8'))
before loading it. I have no idea why this happens.
Edit: Looks like once you read from the StreamingBody object it's empty from then on. https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html#botocore.response.StreamingBody. My recommendation is to read the data from the streaming body and check for "null" and process as normal.
I 'think' I have some caching issue.
my webservice is receiving data through POST request and saving it to disk.
Following is my code:
#app.route('/ws_part' , methods=['POST'])
def ws_part():
request_data = request.get_json()
#placeholder for workstation number and part number
received_data ={'ws_no':request_data['workstation'],
'part_no':request_data['part']}
#Checking if workstation number is already available
global updated
updated = 'no'
for i in repository:
if i['ws'] == received_data['ws']:
i['part'] = received_data['part']
updated = 'yes'
if(updated!='yes'):
new_input = received_data
repository.append(new_input)
return jsonify({'repository': repository})
Issue I am facing:
1.The very first request goes through successfully and gives 200 OK response and workstation number and part coming with the request get saved in 'repository' placeholder.
2.But the very next request throws 500 server error
Error:
File "API_ws_par.py", line 23, in ws_part
if i['ws'] == received_data['ws']:
KeyError: 'ws'
3.Funny thing happening is:
If at this point I restart my web service and trigger the POST request again, the data coming with this new request is getting overwritten and I am losing my previously saved data.
4.Also, the consequent second POST request throws the same 500 server error
Please, guide if this is something to do with cache?
If yes, then please let me know how to do it?
Thanks in advance.
this is not a caching problem, it's a problem in the logic of the function,
on the first call, you probably don't get to the line of code generating the exception,
The problem is on the second call, the key name is not 'ws' it's 'ws_no',
if you change these lines:
for i in repository:
if i['ws'] == received_data['ws_no']:
i['part'] = received_data['part_no']
updated = 'yes'
to use 'ws_no', and 'part_no' you won't have the key error.
I'm not yet very familiar with error handling in Python. I'd like to know how to retrieve the error code when I get an error using the python-twitter package :
import twitter
#[...]
try:
twitter_connexion.friendships.create(screen_name = "someone_who_blocked_me", follow = True)
except twitter.TwitterHTTPError as twittererror:
print(twittererror)
Twitter sent status 403 for URL: 1.1/friendships/create.json using parameters: (follow=True&oauth_consumer_key=XXX&oauth_nonce=XXX&oauth_signature_method=HMAC-SHA1&oauth_timestamp=XXX&oauth_token=XXX&oauth_version=1.0&screen_name=someone_who_blocked_me&oauth_signature=XXX)
details: {'errors': [{'message': 'You have been blocked from following this account at the request of the user.', 'code': 162}]}
In this case, I'd like to retrieve the 'code': 162 part, or just the 162.
twittererror.args is a tuple with one element in it, which is a string, print(twittererror.args[0][0:10]) outputs Twitter se
How can I get the 162 ?
(Obviously, twittererror.args[0][582:585] is not the answer I'm looking for, as any other error message will be of a different length and the error code won't be at [582:585])
Looking at how the TwitterHTTPError is defined in this GitHub repo, you should obtain the dict with twittererror.response_data.
Therefore you can do something like that:
import twitter
#[...]
try:
twitter_connexion.friendships.create(screen_name = "someone_who_blocked_me", follow = True)
except twitter.TwitterHTTPError as twittererror:
for error in twittererror.response_data.get("errors", []):
print(error.get("code", None))
I have a Google App Engine API using Python and NDB working except for HTTP response code/error checking. I put in some code to handle 406 (to only accept json requests) and 400 errors (to prevent a user from leaving a required field blank) to the post function for one of my entities but now it seems to have broken my code. This is the code with the error checking included:
class Task_action(webapp2.RequestHandler):
def post(self):
#Only allows a JSON, if not, then error
if 'application/json' not in self.request.accept:
self.response.status = 406
self.response.status_message = "Not Acceptable, API only supports application/json MIME type"
return
new_task = Task(parent=PARENT_KEY,
name = self.request.get("task_name"),
hours = int(self.request.get("task_hours")),
id = self.request.get("task_name"))
#has error code, since name and hours is required
if name:
new_task.name = name
else:
self.response.status = 400
self.response.status_message = "Invalid request, task name is Required."
if hours:
new_task.hours = hours
else:
self.response.status = 400
self.response.status_message = "Invalid request, task hours is Required."
key = new_task.put()
out = new_task.to_dict()
self.response.write(json.dumps(out))
I am using curl to test it:
curl --data-urlencode "name=clean" -H "Accept: application/json" http://localhost:15080/task
I know the problem is in the error checking code (all the if else statements) because when I take it out the curl test works fine and the object is added to the ndb database correctly. However, with the error checking code included my curl test does not add the object as it should. Does anyone have an idea why the error checking code is breaking my post statement? Is there a better way to return HTTP error response codes?
You had some uninitialized variables in the code (name, hours, maybe PARENT_KEY) and also you didn't return after preparing the error response, thus flowing in areas where the code wouldn't work.
I'd suggest re-organizing the error checking code for minimal impact on the functional code (checks should be done as early as possible, to simplify the remaining functional code. Also, I prefer to use the more concise webapp2.abort() function (which doesn't need a return statement).
Something along these lines:
class Task_action(webapp2.RequestHandler):
def post(self):
# Only allows a JSON, if not, then error
if 'application/json' not in self.request.accept:
webapp2.abort(406, details="Not Acceptable, API only supports application/json MIME type")
# request must contain a valid task name
name = self.request.get("task_name")
if not name:
webapp2.abort(400, details="Invalid request, task name is Required.")
# request must contain a valid task hours
try:
hours = int(self.request.get("task_hours"))
except Exception:
hours = 0
if not hours:
webapp2.abort(400, details="Invalid request, task hours is Required.")
new_task = Task(parent=PARENT_KEY, name=name, hours=hours, id=hours)
new_task.name = name # isn't this done by Task() above?
new_task.hours = hours # isn't this done by Task() above?
key = new_task.put()
out = new_task.to_dict()
self.response.write(json.dumps(out))
Another note: you're specifying the id parameter in the Task() call, which doesn't work unless you know each Task() entity has a unique hours value. You may want to let the datastore assign IDs automatically.
I'm trying to make a custom error page in Flask, and I'd like to give the error handler access to the request that generated the API call that caused the error so that the error page it returns can change depend on the circumstances. For instance, say there are two endpoints:
(1) #app.route('/get_item')
(2) #app.route('/submit_item')
If an error occurs during a call to get_item, I want to display a certain error page ("Sorry, an error occurred...") however, if an error occurs during a call to submit_item, I want it to say something more informative, like:
"An error occured! Please contact us.
Your user id: request.json['userid']
Your submission id: request.json['submission']"
Is it possible to allow the error handler to have access to this, or do I just have to wrap the whole of submit_item in try/except statements?
You can use the request context in the error handler function,
something along those lines:
from flask import request
def exception_handler(*args):
if request.url.endswith('submit_item'):
return "MyMoreDescriptiveErrorMessage", 500
else:
return "Something wrong happened", 500
I would probably create a custom exception and specify an error handler for it similar to this example.
class CustomException(Exception):
def __init__(self, message=None, status_code=None, payload=None):
Exception.__init__(self)
if message is None:
message = "Sorry, an error occurred..."
self.message = message
if status_code is not None:
self.status_code = status_code
self.payload = payload
#app.errorhandler(CustomException)
def handle_custom(error):
response = render_template('error.html', message=error.message)
response.status_code = error.status_code
return response
#app.route('/submit_item')
def submit_item():
message = "An error occured! Userid: %(userid)d, submission: %(submission_id)d"
message = message % (request.json)
raise CustomException(message)