We Keep Coding

Python requests no Exception on status 400

I'm setting up a small Python script so my colleagues can collect data from a certain internal API based on a few inputs using the following code:
url = "https://....."
params = dict(...)
client = BackendApplicationClient(client_id=client_id)
client.prepare_request_body(scope=[])
session = OAuth2Session(client=client)
response = session.get(url=url, params=params, verify=session.verify)
where the params are based on the manual inputs. I can guarantee some of the inputs will not conform to the API's requirements fully (like lower case letters where upper case is needed, etc.). In this case, the API will return a response with status 400:
>> response
<Response [400]>
>> response.text
{"statusCode":400,"errorMessage":"Bad Request","errors": ...}
>> response.status_code
400
I thought I could capture this with response.raise_for_status(), but no Exception is raised, and the returned value is None:
>> response.raise_for_status()
None
Why is this? I thought the raise_for_status function was supposed to raise an Exception based on the response's status_code

raise_for_status() on a response from the requests module will raise an HTTPError exception if the HTTP status code is 400. This is a peculiarity of OAuth2Session which you can read about here

Twitter V2 API 429 HTTP error not being caught

Background and Code
I have the below function to handle rate limiting in Twitter's V2 API based on the HTTP status codes.
from datetime import datetime
from osometweet.utils import pause_until
def manage_rate_limits(response):
"""Manage Twitter V2 Rate Limits
This method takes in a `requests` response object after querying
Twitter and uses the headers["x-rate-limit-remaining"] and
headers["x-rate-limit-reset"] headers objects to manage Twitter's
most common, time-dependent HTTP errors.
Wiki Reference: https://github.com/osome-iu/osometweet/wiki/Info:-HTTP-Status-Codes-and-Errors
Twitter Reference: https://developer.twitter.com/en/support/twitter-api/error-troubleshooting
"""
while True:
# The x-rate-limit-remaining parameter is not always present.
# If it is, we want to use it.
try:
# Get number of requests left with our tokens
remaining_requests = int(response.headers["x-rate-limit-remaining"])
# If that number is one, we get the reset-time
# and wait until then, plus 15 seconds (your welcome Twitter).
# The regular 429 exception is caught below as well,
# however, we want to program defensively, where possible.
if remaining_requests == 1:
buffer_wait_time = 15
resume_time = datetime.fromtimestamp( int(response.headers["x-rate-limit-reset"]) + buffer_wait_time )
print(f"One request from being rate limited. Waiting on Twitter.\n\tResume Time: {resume_time}")
pause_until(resume_time)
except Exception as e:
print("An x-rate-limit-* parameter is likely missing...")
print(e)
# Explicitly checking for time dependent errors.
# Most of these errors can be solved simply by waiting
# a little while and pinging Twitter again - so that's what we do.
if response.status_code != 200:
# Too many requests error
if response.status_code == 429:
buffer_wait_time = 15
resume_time = datetime.fromtimestamp( int(response.headers["x-rate-limit-reset"]) + buffer_wait_time )
print(f"Too many requests. Waiting on Twitter.\n\tResume Time: {resume_time}")
pause_until(resume_time)
# Twitter internal server error
elif response.status_code == 500:
# Twitter needs a break, so we wait 30 seconds
resume_time = datetime.now().timestamp() + 30
print(f"Internal server error # Twitter. Giving Twitter a break...\n\tResume Time: {resume_time}")
pause_until(resume_time)
# Twitter service unavailable error
elif response.status_code == 503:
# Twitter needs a break, so we wait 30 seconds
resume_time = datetime.now().timestamp() + 30
print(f"Twitter service unavailable. Giving Twitter a break...\n\tResume Time: {resume_time}")
pause_until(resume_time)
# If we get this far, we've done something wrong and should exit
raise Exception(
"Request returned an error: {} {}".format(
response.status_code, response.text
)
)
# Each time we get a 200 response, exit the function and return the response object
if response.ok:
return response
This function is fed a response object from a requests call like the below
response = requests.get(
url,
headers=self._header,
params=payload
)
response = manage_rate_limits(response)
In the above response call the parameters are the following:
where
url = Twitter's base endpoint URL (in this case it is the full archive academic search)
params/payload = a combination of endpoint search operators (these should be irrelevant but I can include if necessary)
headers/self._bearer_token is a user bearer_token in the below proper header form
self._header = {"Authorization": f"Bearer {MY_BEARER_TOKEN}"}
Question & Error:
Using the above code, I get a long-running script that returns the below error from the rate_limit_manager function.
Traceback (most recent call last):
File "/scratch/mdeverna/Superspreaders/src/get_rts_of_user.py", line 218, in get_rts_of_user
full_archive_search = True
File "/nfs/nfs5/home/scratch/mdeverna/osometweet/osometweet/api.py", line 248, in search
response = self._oauth.make_request(url, payload)
File "/nfs/nfs5/home/scratch/mdeverna/osometweet/osometweet/oauth.py", line 181, in make_request
response = manage_rate_limits(response)
File "/nfs/nfs5/home/scratch/mdeverna/osometweet/osometweet/rate_limit_manager.py", line 67, in manage_rate_limits
response.status_code, response.text
Exception: Request returned an error: 429 {"title":"Too Many Requests","type":"about:blank","status":429,"detail":"Too Many Requests"}
What I don't understand is that the line that prints this exception is...
# If we get this far, we've done something wrong and should exit
raise Exception(
"Request returned an error: {} {}".format(
response.status_code, response.text
)
... and this illustrates the response.status_code prints (equals) 429, however, the conditional earlier in this function checks for exactly this status code but seems to miss it. It seems like the condition which checks if the status code = 429 is being skipped, only to print down below that the status code is 429?
What is going on here?

Even if the status code is 429 or 500 or 503, you're going to flow off the bottom of the if/elif/elif sequence and right into the raise. Did you intend to return at the end of each? Or did you mean for the raise to be in an else: clause?

404 when closing pull request on github api

I have the following code for interacting with pull requests on the github api.
def merge(pull):
url = "https://api.github.com/repos/{}/{}/pulls/{}/merge".format(os.environ.get("GITHUB_USERNAME"), os.environ.get("GITHUB_REPO"), pull['number'])
response = requests.put(url, auth=get_auth(), data={})
if response.status_code == 200:
#Merge was successful
return True
else:
#Something went wrong. Oh well.
return response.status_code
def close(pull):
url = "https://api.github.com/repos/{}/{}/pulls/{}".format(os.environ.get("GITHUB_USERNAME"), os.environ.get("GITHUB_REPO"), pull['number'])
payload = {"state" : "closed"}
response = requests.put(url, auth=get_auth(), data=payload)
if response.status_code == 200:
#Close was successful
return True
else:
#Something went wrong. Oh well.
return response.status_code
Now merge works just fine, when I run it with a pull request, the pull request is merged and it feels good.
But close gives me a 404. This is strange since merge can clearly find the pull request, and also shows that I clearly have permissions set up properly so I can close the request.
I have also confirmed that I can close the request manually by logging in on github and pressing the 'close pull request' button.
Why does github give me a 404 for the close function but not the merge function? What is different between these two functions?

The answer is that the 'update a pull request' api call should be a POST request, not a put request.
Changing
response = requests.put(url, auth=get_auth(), data=payload)
to
response = requests.post(url, auth=get_auth(), data=payload)
Fixed the issue.

Returning response REST API error messages with python

I'm sending a simple REST API call in python:
import json
import requests
import time
url = 'https://server:9899/123456789/open.scan'
data = {'alpha':'one','beta':'two'}
print 'Sending open.scan command: ' + str(data)
try:
response = requests.post(url, data=data, verify=false)
print response.status_code
except:
print "failed"
This all works correctly and triggers the API. However, if the apikey I feed is bad I get back a 405 error (as expected?).
The issue I'm having is this:
When I issue this API call over a web browser as simply:
https://server:9899/api/wrongapikeyhere/open.sesame
The page returned to me actually states "Wrong API key used"
I'm trying to figure out how to get that returned string into my python code so that I can actually print out what the specific error is and not simply a 405.

Facebook webhook making multiple calls for the same message?

I made and echo bot on AWS with Python and serverless.
I keep getting the same request again and again.
I read the faq where it says u have to deliver a status code 200 else it will keep retrying the webhook.
I'm not sure How I do this.
I have noticed that the sequence number is always the same for the calls so I assume the reply I sent was not acknowledged.
my code is here
import os
import json
import requests
import random
from datetime import datetime
######################
# helper functions
######################
##recursively look/return for an item in dict given key
def find_item(obj, key):
item = None
if key in obj: return obj[key]
for k, v in obj.items():
if isinstance(v,dict):
item = find_item(v, key)
if item is not None:
return item
##recursivley check for items in a dict given key
def keys_exist(obj, keys):
for key in keys:
if find_item(obj, key) is None:
return(False)
return(True)
##send txt via messenger to id
def send_message(send_id, msg_txt):
print("Send message called")
print (datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')[:-3])
params = {"statusCode": 200,"access_token": os.environment['accesstoken']}
headers = {"statusCode": "200","Content-Type": "application/json"}
data = json.dumps({"statusCode": "200","recipient": {"id": send_id},
"message": {"text": msg_txt}})
r = requests.post("https://graph.facebook.com/v2.9/me/messages", params=params, headers=headers, data=data)
print (r.text)
if r.status_code != 200:
print(r.status_code)
print(r.text)
#-----------------------------------------------------------
def hello(event, context):
#debug
event=json.loads(json.dumps(event))
print("event:" )
print(event)
# print("context")
# print(context)
#handle webhook challenge
try:
if keys_exist(event, ["queryStringParameters","hub.verify_token","hub.challenge"]):
print("subscribe to webhook invoked")
v_token = str(find_item(event, 'hub.verify_token'))
challenge = find_item(event, 'hub.challenge')
if ("strongtoken" == v_token):
response = {
"statusCode": 200,
"body": str(challenge)
}
print(challenge)
return response
#handle messaging events
if keys_exist(event, ['body']):
event_entry=json.loads(event['body'])
if ((len(event_entry['entry'])>0) & (keys_exist(event_entry['entry'][0],['messaging'])) ):
messaging_event = event_entry['entry'][0]['messaging'][0]
if (keys_exist(messaging_event,['message'])):
msg_txt = messaging_event['message']['text']
sender_id = messaging_event['sender']['id']
print(sender_id)
first_word = msg_txt.split(" ")[0]
send_message(sender_id, msg_txt)
else:
print("Did not send message")
pass
else:
print("Did not send message")
pass
else:
pass
except:
pass
I have given the status code 200 in soo many places and I'm not sure y I still keep getting the same problem.

If you are getting multiple messages your server did not return 200 status code to webhook request from Facebook server. This means an error occurred on your part otherwise 200 should be returned. It looks to me that problems are in following lines:
params = {"statusCode": 200,"access_token": os.environment['accesstoken']}
headers = {"statusCode": "200","Content-Type": "application/json"}
data = json.dumps({"statusCode": "200","recipient": {"id": send_id},
"message": {"text": msg_txt}})
Firstly you are passing statusCode in a data body of message and according to this documentation message response should not contain it.
Another problem could be sending status code inside params. I would remove status code completely from send_message method. I doubt its needed there. You are basically trying to return status 200 on wrong end. You are trying to return it on output instead of input (from Facebooks point of view).
So its quite possible you are correctly getting message from Facebook but you are still returning wrong status code because you are calling send_message method from inside messaging event and send_message method should return status "400 bad request" because you are sending faulty request. Thus your server also returns wrong response code.
Just make sure your code works correctly and 200 should be returned.
EDIT:
So i would try with following code:
params = {"access_token": os.environment['accesstoken']}
headers = {"Content-Type": "application/json"}
data = json.dumps({"recipient": {"id": send_id},
"message": {"text": msg_txt}})

I'm working on a Facebook Messenger chatbot with a Node/Express server and was having the same exact issue. The server was appropriately sending 200 responses and yet the issue persisted. I resolved the problem by caching message ID's and checking for duplicates before processing:
var NodeCache = require('node-cache');
var myCache = new NodeCache();
app.post('/webhook/', function(req, res) {
var messageID = req.body.entry[0].messaging[0].message.mid;
var checkForDupe = myCache.get(messageID);
if (checkForDupe == undefined) {
var setID = myCache.set(req.body.entry[0].messaging[0].message.mid, req.body.entry[0].messaging[0].message.mid);
//Handle message .....
I hope this helps someone. It was driving me nuts.

Your webhook callback should always return a 200 OK HTTP response when invoked by Facebook. Failing to do so may cause your webhook to be unsubscribed by the Messenger Platform. (copied from messenger docs.)
You have to search problem from your returning response. It is not with the sending data.
response = requests.post(POST_MESSAGE_URL, headers={
'Content-Type': 'application/json'
}, params={
'access_token': ACCESS_TOKEN
}, data=json.dumps(DATA_JSON)
this code working fine for me

I am from Java background but still I will try to help you.
In handle webhook challenge I can see you returning 200 status code
But in handle messaging block 200 is not returned You have set 200 status code in send message. But it will not work you have to return 200 from handle messaging block.
Also 200 should be returned even if any exception occurs in this block otherwise on exception it will get blocked. So may be in finally block you can return 200.

As an alternative, you can use chatbotproxy.com which always return 200 in 1ms and pass the exact same request to your endpoint. Since immediate response is very important for Facebook Messenger platform, 1 millisecond time is a very promising feature. If you need help to try, let me know.