We're using Rauth to connect to various OAuth 1 APIs. It works fine for a single request, but trying to do 2 or more requests against the given session results in 401 not authorized errors from the APIs.
Twitter API example:
import requests
from rauth import OAuth1Service
from rauth import OAuth1Session
consumer_key = {the consumer key}
consumer_secret = {the consumer secret}
access_token = {the access token}
access_token_secret = {the access token secret}
oauth_service = OAuth1Service(consumer_key = consumer_key,
consumer_secret = consumer_secret)
oauth_session = oauth_service.get_session(token = (access_token, access_secret))
url = 'https://api.twitter.com/1.1/statuses/home_timeline.json'
params = {'include_rts': 'true'}
r = oauth_session.get(url, params=params) # THIS WORKS
r = oauth_session.get(url, params=params) # THIS RETURNS 401 ERROR
This happens on both Twitter and LinkedIn APIs. How do we execute multiple requests against a single OAuth1Session object?
VERSIONS:
rauth==0.5.4
requests==1.1.0
UPDATE:
Strangely, if the params argument is not included then multiple requests can be made- but once params are included, even if it is an empty dict, we get 401s.
Example 1:
r = oauth_session.get(url) # THIS WORKS
r = oauth_session.get(url) # THIS WORKS
Example 2:
r = oauth_session.get(url, params={}) # THIS WORKS
r = oauth_session.get(url, params={}) # THIS RETURNS 401 ERROR
Carrying over from the comments, using session.get(..., header_auth=True) should do the trick. It's hard to say exactly why it doesn't work without this, but for the record, header-based authentication is preferred by the spec and given Twitter's position, I wouldn't be surprised if they also prefer it as a provider.
A quick search reveals dozens upon dozens of reports of their API failing where it ostensibly should work and one remedy is to prefer header authentication. From what I can tell, rauth is signing appropriately, so perhaps this is something to do with the way the provider is showing preference and handling non-header authenticated requests.
Update
It looks like either rauth or Requests was not properly handling params. It's odd because the signature base string and oauth_signature seemed to be correct, in that they were appropriately different on each respective request and the data they operated on seemed to checkout. So it seems like it should have validated the request.
At any rate, to correct this, we need to deepcopy elements of the request parameters that are mutable types, e.g. dictionaries. I've got a patch that should correct this, so you should be able to use this without header_auth. However, header authentication is the preferred method so I would still recommend it.
Related
Context:
I'm working on a side project to pull data from the Spotify API into a
Microsoft SQL Server database as part of a refreshing ETL job. I need
to use the "Authorization Code Flow" so I can authorize/authenticate
programmatically, so my table will populate each day.
I'm using the Python requests library for this, and I don't want to
make an Object Oriented Solution for this if possible (not my
preference).
Problem:
I'm having trouble getting the Access Token after authenticating.
Looking at similar issues, it's very similar to this one:
Spotify API Authorization Code Flow with Python
.
I'm not sure why I'm getting a Response 400 (Bad Request) from this.
Can someone please advise here?
Code:
# used to to encode byte string from CLIENT_ID : CLIENT_SECRET, then decode for Authentication Header
import base64
# used to make HTTP requests from Spotify API
import requests
# used to access the environment variables
import os
def request_user_authorization():
'''
HTTP GET request to gain access to data (Authorization Code Flow)
HTTP POST request to send the code and receive an Authorization Token (current issue)
https://developer.spotify.com/documentation/general/guides/authorization/code-flow/
'''
# URLs
AUTH_URL = 'https://accounts.spotify.com/authorize'
TOKEN_URL = 'https://accounts.spotify.com/api/token'
BASE_URL = 'https://api.spotify.com/v1'
SPOTIFY_URI = 'https://api.spotify.com/v1/me/player/recently-played'
# sensitive items
CLIENT_ID = os.environ.get('SPOTIFY_CLIENT_ID_ENV')
CLIENT_SECRET = os.environ.get('SPOTIFY_CLIENT_SECRET_ENV')
# make a request to the /authorize endpoint to get an authorization code
user_authorization_code = requests.get(
AUTH_URL, {
'client_id': CLIENT_ID,
'response_type': 'code',
'redirect_uri': SPOTIFY_URI,
'scope': 'user-read-recently-played',
}
)
# Code 200 = "OK"
print(user_authorization_code)
#----------------------------------------------------------#
api_header_string = base64.urlsafe_b64encode((CLIENT_ID + ':' + CLIENT_SECRET).encode('ascii'))
api_headers={
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': 'Basic %s' % api_header_string.decode('ascii')
}
api_payload = {
'grant_type': 'authorization_code',
'code': user_authorization_code,
'redirect_uri': SPOTIFY_URI,
'client_id': CLIENT_ID,
'client_secret': CLIENT_SECRET
}
#issue here:
# Make a request to the /token endpoint to get an access token
access_token_request = requests.post(url=TOKEN_URL, data=api_payload, headers=api_headers)
# returns <Response [400]>
# https://datatracker.ietf.org/doc/draft-ietf-httpbis-semantics/
# 15.5.1. 400 Bad Request
# The _400 (Bad Request)_ status code indicates that the server cannot
# or will not process the request due to something that is perceived to
# be a client error (e.g., malformed request syntax, invalid request
# message framing, or deceptive request routing).
# print(access_token_request)
#----------------------------------------------------------#
request_user_authorization()
You seem to have misunderstood how the Authorizatuon Code Flow works.
The redirect_uri in this kind of flow is used by the provider api (here spotify) as a callback to give you the authorization code.
The spotify API will call this url with a code parameter that you can use to ask for a token.
Meaning that for this flow to work you need a web server ready to receive requests on the uri that you have given in your code request (and specified when creating your app on the spotify developer portal). You might be better off using the Client Credentials Flow for your use case.
Also you should always use the name of the keywords arguments when using requests.get, requests.post ... It makes the code clearer and the order of the arguments differ for each method so it can get confusing if you don't.
#Speedlulu you're correct, that was the problem.
For anyone in the future reading this question: this is what I learned since posting the question:
What I misunderstood was the flow of data, and that Client Credentials Flow (Application to Spotify only) was the better choice because I don't need to have a "User" portion to this program.
Spotify's Client Credentials Flow Documentation: https://developer.spotify.com/documentation/general/guides/authorization/client-credentials/
# used to access environment variables securely (sensitive data)
import os
# used to encode strings into bytes and back
import base64
# used to convert JSON data into strings
import json
# endpoint that I'm connecting to on Spotify's servers
token_request_url = "https://accounts.spotify.com/api/token"
CLIENT_ID = os.environ.get('SPOTIFY_CLIENT_ID_ENV')
CLIENT_SECRET = os.environ.get('SPOTIFY_CLIENT_SECRET_ENV')
# encode credentials into bytes, then decode into a string for the HTTP POST request to Spotify to authenticate
BASE64_ENCODED_HEADER_STRING = base64.b64encode(bytes(f"{CLIENT_ID}:{CLIENT_SECRET}", "ISO-8859-1")).decode("ascii")
#initializing dictionaries for HTTP POST request
headers = {}
data = {}
headers['Authorization'] = f"Basic {BASE64_ENCODED_HEADER_STRING}"
data['grant_type'] = "client_credentials"
data['json'] = True
data['scope'] = 'user-read-recently-played'
r = requests.post(url=token_request_url, headers=headers, data=data)
# prints the response from the server regarding the access token data (formatted to be easier to read)
print(json.dumps(r.json(), indent=2))
# store the token value in a variable for HTTP GET request
token = r.json()['access_token']
What was unclear is that I first need to POST my request with the credentials to get the token (using the specific URL to do), store the r.json()['access_token'] value in a variable, then use that as part of the following GET request to access my specific data.
I would like to make a HTTP call to this resource :
https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/jobs
As I read to the documentation I use an API key generated from my GCP project to be authenticated. So with requests I make a simple call like this:
import requests
params = {'key': 'MY_API_KEY'}
base_url = 'https://bigquery.googleapis.com'
project_id = 'MY_PROJECT_ID'
r = requests.get(f'{base_url}/bigquery/v2/projects/{project_id}/jobs', params=params)
Unfortunately it returns a response 401 and I can't figure out why.
Thanks a lot and have a nice day !
Update code after guillaume blaquiere reply :
from google.auth.transport.requests import AuthorizedSession
from google.oauth2 import service_account
base_url = 'https://bigquery.googleapis.com'
project_id = 'project_id'
credentials = service_account.Credentials.from_service_account_file(
'service_account.json',
scopes=['https://www.googleapis.com/auth/bigquery',
'https://www.googleapis.com/auth/cloud-platform'],
)
authed_session = AuthorizedSession(credentials)
response = authed_session.request('GET', f'{base_url}/bigquery/v2/projects/{project_id}/jobs')
print(response.json())
# this returns : {'etag': 'tAZvk1k2f2GY8yHaQF7how==', 'kind': 'bigquery#jobList'}
The API Key no longer works for a large number of Google API. Only some legacy continue to accept an API key.
Now, you need an authenticated request. You can find exemple in the google-auth python library documentation. Look at Refresh and Authorized_session.
Don't hesitate to comment if you need help about the credential obtention, I can also help you on this.
EDIT
When you perform the request, it's, by default, only on the current user. In your case, it's the service account when you use the Python code, and your User account when you use the API Explorer (the swagger like in the Google Documentation).
In your case, I guess that your service account has never performed a job (query or load job) and thus, there is no entry for it.
According with the documentation, is you want to see all the user jobs, you have to add the param ?allUsers=true at the end of your URL
response = authed_session.request('GET', f'{base_url}/bigquery/v2/projects/{project_id}/jobs?allUsers=true')
This is what I have so far. No real success. Trying to retrieve a token, but nothing seems to work. Just returns a giant mess of characters.
import requests
import json
auth_url = "http://learn.ZZZZZZZ.com/oauth2/authorize"
#credential
auth_client_id = "BBBBBBBBBBBBBBBBBBBBBBBB"
auth_client_secret = "YYYYYYYYYYYYYYYYYYYYY"
payload={'grant_type':'client_credentials', 'client_id':auth_client_id,'client_secret':auth_client_secret}
headers={'Accept':'application/json', 'Content-Type':'application/x-www-form-urlencoded'}
response = requests.post(auth_url,headers=headers,data=payload)
response.text
Use the token endpoint to obtain an access token.
auth_url = "http://learn.ZZZZZZZ.com/oauth2/token"
Docebo API documentation
After looking at several articles online, StackOverflow, and the Yelp Google Group, I've been unable to figure out the problem to an Invalid Signature error being produced from my Yelp API request.
Here is the exact error:
{'error': {'text': 'Signature was invalid', 'description': 'Invalid signature. Expected signature base string: [some text here with keys]}}
And the code I've written to go along with it:
import rauth
import time
def get_results():
#Obtain these from Yelp's manage access page
consumer_key = ''
consumer_secret = ''
token = ''
token_secret = ''
session = rauth.OAuth1Session(
consumer_key = consumer_key
,consumer_secret = consumer_secret
,access_token = token
,access_token_secret = token_secret)
request = session.get("http://api.yelp.com/v2/search?location=Boston&term=food")
#Transforms the JSON API response into a Python dictionary
data = request.json()
print(data)
session.close()
return data
if __name__=="__main__":
print(get_results())
So what exactly is causing this error? I've done some modifications prior to this attempt, and the previous attempts I made I got similar errors; except one time I only got a "Invalid Signature" error, with no "Expect signature base string" message
There are more steps to authentication as per the docs
Making a Request
Each request must contain the following OAuth protocol parameters:
OAuth Parameter Value
oauth_consumer_key Your OAuth consumer key (from Manage API Access).
oauth_token The access token obtained (from Manage API Access).
oauth_signature_method hmac-sha1
oauth_signature The generated request signature, signed with the oauth_token_secret obtained (from Manage API Access).
oauth_timestamp Timestamp for the request in seconds since the Unix epoch.
oauth_nonce A unique string randomly generated per request.
These parameters may be passed in the HTTP (Authorization) header as URL query keys or in the POST data. Generating the OAuth signature is done by applying the HMAC-SHA1 with the oauth_token_secret. You may view your OAuth consumer key at Manage API Access. OAuth libraries are available to generate these requests.
You are not passing oauth_timestamp which is required or applying the HMAC-SHA1 so you get an Invalid Signature error, it is clearly outlined in the docs above what you need to send.
There is also an actual python yelp api you could use but to make a request you can use the example below based on the request function from the example code: to make a request using oauth2 and requests:
import requests
import oauth2
def request(url, url_params=None):
consumer_key = ""
consumer_secret = ""
token = ""
token_secret =""
url_params = url_params or {}
consumer = oauth2.Consumer(consumer_key, consumer_secret)
oauth_request = oauth2.Request(method="GET", url=url, parameters=url_params)
oauth_request.update(
{
'oauth_nonce': oauth2.generate_nonce(),
'oauth_timestamp': oauth2.generate_timestamp(),
'oauth_token': token,
'oauth_consumer_key': consumer_key
}
)
token = oauth2.Token(token, token_secret)
oauth_request.sign_request(oauth2.SignatureMethod_HMAC_SHA1(), consumer, token)
signed_url = oauth_request.to_url()
print(u'Querying {0} ...'.format(url))
return requests.get(signed_url).json()
Which using your url outputs a whole load of json, the start of which is:
Querying http://api.yelp.com/v2/search?location=Boston&term=food ...
{'region': {'center': {'longitude': -71.05460875, 'latitude': 42.35028894954365}, 'span': {'latitude_delta': 0.0325510910039668, 'longitude_delta': 0.04668455000000904}}, 'total': 8351, 'businesses': [{'name': "Giacomo's Ristorante", 'url': 'http://www.yelp.com/biz/giacomos-ristorante-boston', 'mobile_url': 'http://m.yelp.com/biz/giacomos-ristorante-boston', 'rating_img_url_large': 'http://s3-media2.fl.yelpcdn.com/assets/2/www/img/ccf2b76faa2c/ico/stars/v1/stars_large_4.png', 'phone':
...............................................................
...............................................................
I am not sure if the api supports python 3 but the code above was tested with python3 and python2 and it works fine, to install oauth2 you can simple pip install oauth2 and the same with requests if you don't have it installed.
Another common issue is that the servers time is out of sync. On linux, one can run
sudo ntpdate -s time.nist.gov
So, there is a code that uses xAuth authentication to call tumblr API methods:
import urllib
import urlparse
import oauth2 as oauth
consumer_key = "..."
consumer_secret = "..."
consumer = oauth.Consumer(consumer_key, consumer_secret)
client = oauth.Client(consumer)
resp, content = client.request('https://www.tumblr.com/oauth/access_token', "POST", urllib.urlencode({
'x_auth_mode': 'client_auth',
'x_auth_username': '...#yandex.ru',
'x_auth_password': '...'
}))
token = dict(urlparse.parse_qsl(content))
print token
token = oauth.Token(token['oauth_token'], token['oauth_token_secret'])
client = oauth.Client(consumer, token)
response, data = client.request('http://api.tumblr.com/v2/blog/good.tumblr.com/followers', method='GET')
print data
It works perfect with User methods from tumblr API that require OAuth authentication.
But it fails when i try to call any Blog method with OAuth authentication (/followers for example):
{"meta":{"status":401,"msg":"Not Authorized"},"response":[]}
Except one thing. If i use my blog name as {base-hostname} parameter it works without any errors.
Weird. How is that possible? Is something wrong with the code?
Well that is because your OAuth access token grants you access to your blogs. OAuth can't give you permission to access Blog methods that you do not own because then you could post to them.
When you make POST request the enctype must be "multipart/form-data".
I had the same problem with Zend_Oauth (php), but is resolved now.