I have set up API access to a website and can use their test GUI interface to connect (HTTP 200) using the username, passsword and API key.
If I try code below I get error Response 403:
from requests.auth import HTTPDigestAuth
url = 'https://website.com'
result = requests.get(url, auth=HTTPDigestAuth('username', 'password'))
I think I need to include the API key with the original requests.get but I'm not sure how to do that.
It's a REST API. On the website I generated an API key by providing a Name and then set the API login details (username and password). Under info on the HTTP requests it lists Request Header components:
X-API-KEY: k The API key k (obtained from My Account) is how we identify and authorise the calling application
CST / Authorization A valid access token identifying the client.
X-SECURITY-TOKEN / ACCOUNT-ID A valid account token or account id identifying the client's current account.
Content-Type: application/json Request format type. This should always be set as indicated to json only
Accept: application/json; charset=UTF-8 Response format type. This should always be set as indicated to json only Version:
Version: v API version v (defaults to 1, if not specified)
Is it possible that the API key, username and password are all contained with the header? The username and password are not used to set the API key.
You appear to be using the IG Labs REST Trading API; the documentation states there are two modes of authentication, depending on the API version you specify. The header documentation you quote is a general overview of the different headers that are involved. That section is not very useful to actually understanding how the authentication / authorization works for that API.
On the same page is a Authentication and authorisation section, and it is that section you need to study. This is where the two modes of authentication are described. There is also a separate examples section that explains how the two different modes work with concrete requests and responses shown.
To authenticate, you must first send a POST request to the /session route, with your API key present in the X-IG-API-KEY header. This is where you'll need the username and password. A valid username / password then gives you security tokens to be used for subsequent requests. For API versions 1 and 2 an additional field encryptedPassword can be included to state whether or not the password is plain text or encrypted; you only need this to satisfy IG Singapore login restrictions, and the field can safely be omitted.
For v1 and v2 requests, you get those tokens in the response headers, API version v3 gives you the required information in the body. The tokens have a limited lifetime. The v1/v2 tokens are valid for 6 hours, but are automatically extended up to 72 hours by making requests with those tokens, before you need to log in again. The v3 token is only valid for 60 seconds, but a separate refresh_token lets you get a new token by sending the refresh token to the /session/refresh-token endpoint.
V1 / v2 requests then use the tokens in the exact same headers the /session response uses them, so copy across the CST ("client session token") and the X-SECURITY-TOKEN headers and you are on your way.
V3 requests use the standard OAuth Authorization header, with method set to Bearer. Just construct that header with the access_token value from the oauthToken structure, and save the refresh_token for when the access_token has expired. The documentation also recommends you set the IG-ACCOUNT-ID header, to identify the account (the token identifies the client only).
I strongly recommend you use a requests.Session() object to make header handling simpler.
For versions v1 and v2 of the API, use:
import requests
url = 'https://website.com'
API_KEY = '.... your API key ....'
username = 'username'
password = 'password'
session = requests.Session()
# these are sent along for all requests
session.headers['X-IG-API-KEY'] = API_KEY
# not strictly needed, but the documentation recommends it.
session.headers['Accept'] = "application/json; charset=UTF-8"
# log in first, to get the tokens
response = session.post(
url + '/session',
json={'identifier': username, 'password': password},
headers={'VERSION': '2'},
)
response.raise_for_status() # if not a 2xx response, raise an exception
# copy across the v1/v2 tokens
session.headers['CST'] = response.headers['CST']
session.headers['X-SECURITY-TOKEN'] = response.headers['X-SECURITY-TOKEN']
Now you are set up to use session to continue to access the API.
For v3, the tokens live in the JSON body:
session = requests.Session()
# these are sent along for all requests
session.headers['X-IG-API-KEY'] = API_KEY
# log in first, to get the tokens
response = session.post(
url + '/session',
json={'identifier': username, 'password': password},
headers={'VERSION': '3'},
)
response.raise_for_status() # if not a 2xx response, raise an exception
response_data = response.json()
oauth_tokens = response_data['oauthToken']
session.headers['Authorization'] = 'Bearer ' + oauth_tokens['access_token']
session.headers['IG-ACCOUNT-ID'] = response_data['accountId']
Now you are set up to use session to continue to access the API until oauth._tokens['expires_in'] seconds have passed or a 401 response is given:
if response.status == 401 and response.json()['errorCode'] == 'error.security.oauth-token-invalid':
# refresh required, old token is done.
You need to then use a session.post(url + '/session/refresh-token', json={'refresh_token': oauth_tokens['refresh_token']}) to get a freshoauthToken` structure:
# refresh the access token
del session.headers['Authorization']
response = session.post(
url + '/session/refresh-token',
json={'refresh_token': oauth_tokens['refresh_token']},
headers={'VERSION': '1'},
)
response.raise_for_status() # if not a 2xx response, raise an exception
oauth_tokens = response.json()
session.headers['Authorization'] = 'Bearer ' + oauth_tokens['access_token']
Note that I keep sending the VERSION header with each individual request; their API uses version numbers per end point, so /session has 3 versions, but /session/refresh-token only has version 1, and you can't set VERSION to anything else or it'll break.
The V3 /session format may appear to be more cumbersome, but that version lets you refresh your access indefinitely, provided you do so within 10 minutes from your last token use; the V1 / V2 access tokens expire after 72 hours tops, whatever you do.
Related
Context:
I'm working on a side project to pull data from the Spotify API into a
Microsoft SQL Server database as part of a refreshing ETL job. I need
to use the "Authorization Code Flow" so I can authorize/authenticate
programmatically, so my table will populate each day.
I'm using the Python requests library for this, and I don't want to
make an Object Oriented Solution for this if possible (not my
preference).
Problem:
I'm having trouble getting the Access Token after authenticating.
Looking at similar issues, it's very similar to this one:
Spotify API Authorization Code Flow with Python
.
I'm not sure why I'm getting a Response 400 (Bad Request) from this.
Can someone please advise here?
Code:
# used to to encode byte string from CLIENT_ID : CLIENT_SECRET, then decode for Authentication Header
import base64
# used to make HTTP requests from Spotify API
import requests
# used to access the environment variables
import os
def request_user_authorization():
'''
HTTP GET request to gain access to data (Authorization Code Flow)
HTTP POST request to send the code and receive an Authorization Token (current issue)
https://developer.spotify.com/documentation/general/guides/authorization/code-flow/
'''
# URLs
AUTH_URL = 'https://accounts.spotify.com/authorize'
TOKEN_URL = 'https://accounts.spotify.com/api/token'
BASE_URL = 'https://api.spotify.com/v1'
SPOTIFY_URI = 'https://api.spotify.com/v1/me/player/recently-played'
# sensitive items
CLIENT_ID = os.environ.get('SPOTIFY_CLIENT_ID_ENV')
CLIENT_SECRET = os.environ.get('SPOTIFY_CLIENT_SECRET_ENV')
# make a request to the /authorize endpoint to get an authorization code
user_authorization_code = requests.get(
AUTH_URL, {
'client_id': CLIENT_ID,
'response_type': 'code',
'redirect_uri': SPOTIFY_URI,
'scope': 'user-read-recently-played',
}
)
# Code 200 = "OK"
print(user_authorization_code)
#----------------------------------------------------------#
api_header_string = base64.urlsafe_b64encode((CLIENT_ID + ':' + CLIENT_SECRET).encode('ascii'))
api_headers={
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': 'Basic %s' % api_header_string.decode('ascii')
}
api_payload = {
'grant_type': 'authorization_code',
'code': user_authorization_code,
'redirect_uri': SPOTIFY_URI,
'client_id': CLIENT_ID,
'client_secret': CLIENT_SECRET
}
#issue here:
# Make a request to the /token endpoint to get an access token
access_token_request = requests.post(url=TOKEN_URL, data=api_payload, headers=api_headers)
# returns <Response [400]>
# https://datatracker.ietf.org/doc/draft-ietf-httpbis-semantics/
# 15.5.1. 400 Bad Request
# The _400 (Bad Request)_ status code indicates that the server cannot
# or will not process the request due to something that is perceived to
# be a client error (e.g., malformed request syntax, invalid request
# message framing, or deceptive request routing).
# print(access_token_request)
#----------------------------------------------------------#
request_user_authorization()
You seem to have misunderstood how the Authorizatuon Code Flow works.
The redirect_uri in this kind of flow is used by the provider api (here spotify) as a callback to give you the authorization code.
The spotify API will call this url with a code parameter that you can use to ask for a token.
Meaning that for this flow to work you need a web server ready to receive requests on the uri that you have given in your code request (and specified when creating your app on the spotify developer portal). You might be better off using the Client Credentials Flow for your use case.
Also you should always use the name of the keywords arguments when using requests.get, requests.post ... It makes the code clearer and the order of the arguments differ for each method so it can get confusing if you don't.
#Speedlulu you're correct, that was the problem.
For anyone in the future reading this question: this is what I learned since posting the question:
What I misunderstood was the flow of data, and that Client Credentials Flow (Application to Spotify only) was the better choice because I don't need to have a "User" portion to this program.
Spotify's Client Credentials Flow Documentation: https://developer.spotify.com/documentation/general/guides/authorization/client-credentials/
# used to access environment variables securely (sensitive data)
import os
# used to encode strings into bytes and back
import base64
# used to convert JSON data into strings
import json
# endpoint that I'm connecting to on Spotify's servers
token_request_url = "https://accounts.spotify.com/api/token"
CLIENT_ID = os.environ.get('SPOTIFY_CLIENT_ID_ENV')
CLIENT_SECRET = os.environ.get('SPOTIFY_CLIENT_SECRET_ENV')
# encode credentials into bytes, then decode into a string for the HTTP POST request to Spotify to authenticate
BASE64_ENCODED_HEADER_STRING = base64.b64encode(bytes(f"{CLIENT_ID}:{CLIENT_SECRET}", "ISO-8859-1")).decode("ascii")
#initializing dictionaries for HTTP POST request
headers = {}
data = {}
headers['Authorization'] = f"Basic {BASE64_ENCODED_HEADER_STRING}"
data['grant_type'] = "client_credentials"
data['json'] = True
data['scope'] = 'user-read-recently-played'
r = requests.post(url=token_request_url, headers=headers, data=data)
# prints the response from the server regarding the access token data (formatted to be easier to read)
print(json.dumps(r.json(), indent=2))
# store the token value in a variable for HTTP GET request
token = r.json()['access_token']
What was unclear is that I first need to POST my request with the credentials to get the token (using the specific URL to do), store the r.json()['access_token'] value in a variable, then use that as part of the following GET request to access my specific data.
that is my first try with an API, said API being called OPS.
I would like to get information using the API (OAuth 2) within my python code.
The ressource URL is :
http://ops.epo.org/3.2/rest-services/register/{publication}/{EPODOC}/{EP2814089}/biblio
I also received :
Consumer Key: O220VlTQqAmodifiedsf0YeqgM6c
Consumer Secret Key: swWmodified3edjORU
The documentation states that:
OPS uses the OAuth framework for Authentication and Authorization. At this point in
time, only the “Client Credentials” flow is supported using a Consumer key and
Consumer secret.
The actual steps to follow are:
Step 1: Client converts Consumer key and Consumer secret to
Base64Encode(Consumer key:Consumer secret).
This should be done programmatically using the language you are developing the client
application in. For the purposes of this example, a public website was used to perform
this conversion.
By entering the colon separated Client credentials, an encoded response is generated.
This response is then be used for basic Authentication.
Step 2: Client requests an access token using Basic Authentication, supplying its
Consumer key and Consumer secret with base64Encoding over encrypted HTTPS
connection:
OPS authenticates the client credentials passed in the Authorization header using basic
authentication method.
If credentials are valid, OPS responds with a valid access token.
Step 3: Client accesses OPS resources with access token in authorization header
(bearer tokens) over encrypted HTTPS connection
I tried a few samples of code with requests but, until now, nothing worked.
The client credentials flow is described in the OAuth2 RFC-6749. The client id and secret are base64 encoded in a Basic authentication scheme as described in RFC-7617
You should be able to get a token using Python code like:
import requests
import base64
url = 'https://ops.epo.org/3.2/auth/accesstoken'
data = {"grant_type": "client_credentials"}
creds = base64.b64encode("O220VlTQqAmodifiedsf0YeqgM6c:swWmodified3edjORU".encode())
headers = {'Authorization': 'Basic ' + creds.decode('UTF-8'), 'Content-Type': 'application/x-www-form-urlencoded'}
response = requests.post(url, headers=headers, data=data)
access_token = response.json()["access_token"]
When using the previous response I can obtain a token. (Thanks a lot for your answer)
So I tried :
myUrl = 'http://ops.epo.org/3.2/rest-services/register/publication/EPODOC/EP2814089/biblio'
header = {'PRIVATE-TOKEN': myToken}
response = requests.get(myUrl, headers=header)
print(response.text)
but I obtained a 403 error.
I finally got a specific library to do the job :
EPO OPS Library
But I still don't know how to do it on my own...
When trying to authorize spotify using python 3, I get a "server_error" with the description "Unexpected status: 400".
I am using the correct authorization code and the spotify documentation (https://developer.spotify.com/web-api/authorization-guide/) instructed me to use a post command with those parameters.
I'm quite a noob in this and I do not know what I am doing wrong.
Here is the code:
import requests
params = {'grant_type': 'authorization_code', 'code': authcode, 'redirect_uri': 'https://example.com/callback','client_id':'example', 'client_secret':'example'}
req=requests.post('https://accounts.spotify.com/api/token', params=params)
print(req.content)
According to spotify's own guide (see step #4):
https://developer.spotify.com/web-api/authorization-guide/
The authorization info for requesting a new token must go in the header via an "Authorization" variable:
Authorization: Required. Base 64 encoded string that contains the
client ID and client secret key. The field must have the format:
Authorization: Basic base64 encoded client_id:client_secret
You have it instead in the request body itself.
So you should do something like:
import requests
import base64
authcode = 'valid_authcode_from_prev_authorization_step'
params = {'grant_type': 'authorization_code', 'code': authcode, 'redirect_uri': 'https://example.com/callback'}
client_id = 'example_id'
client_secret = 'example_secret'
b64_val = base64.b64encode("%s:%s" % (client_id, client_secret))
req = requests.post(
'https://accounts.spotify.com/api/token', params=params,
headers={'Authorization': b64_val})
However, for this to work you need a valid auth code which you can only get by having the user go through the auth step which precedes the token acquisition step.
This code gets sent to the callback you have registered in your app settings, which won't work if you have a fake callback set (ie: http://example.com/callback).
I've been following the guide for Twitter's 3-legged oauth setup:
https://dev.twitter.com/docs/auth/implementing-sign-twitter
Step 1: Obtaining a request token
For their authentication, step 1 requires making a post request containing the base64 encoded public and secret key.
key = "CONSUMER_KEY"
secret = "CONSUMER_SECRET"
auth = base64.encodestring("%s:%s" % (key, secret)).replace("\n", "")
data = {}
data["grant_type"] = "client_credentials"
headers = {}
headers["Authorization"] = "Basic " + auth
headers["Content-Type"] = "application/x-www-form-urlencoded;charset=UTF-8"
headers["Accept-Encoding"] = "gzip"
response = requests.post("https://api.twitter.com/oauth2/token",
headers=headers, data=data)
This first request returns a valid response code 200 along with an access token. The response looks like this:
{u'access_token': u'AAAAAAAAAAAAAAAAAAAAAHHHHH... ...vncbi', u'token_type': u'bearer'}
Step 2: Redirecting the user
This is where the problem is occurring. According to the docs, the user then just needs to be redirected to the authorization url formatted like this:
https://api.twitter.com/oauth/authenticate?oauth_token=AAAAAAAAAAAAAAAAAAAAAHHHHH... ...vncbi
However when I get to this page I get an error message:
Is there something I missed? The access_token is being generated without an issue. I'm not sure if this message is showing up because I set something up incorrectly earlier in the process. I'm also not sure how to check if the oauth token has expired.
Actually, you have been following https://dev.twitter.com/docs/api/1.1/post/oauth2/token which is quite different, e.g. only used for public resources and not private like status updates. For the three step one checkout https://gist.github.com/ib-lundgren/4487236 or better yet http://twython.readthedocs.org/en/latest/
If you only want to access public resources like user timelines you can do so via the code below.
# OBS: If you want to look at per user details and make status updates
# you want the OAuth1 version. This is only for publicly available
# resources such as user timelines.
from requests_oauthlib import OAuth2Session
from oauthlib.oauth2 import BackendApplicationClient
# Credentials you get from registering a new application
client_id = '<the id you get from github>'
client_secret = '<the secret you get from github>'
# TODO remove
client_id = 'VVq5UniipB5nXFAqtTA'
client_secret = 'PlaHnaSDbeY4eYkv8XiqxS1nzGWyKoq5WYSNjdeaw'
client_id = 'I1Xi7fOeYnA9jabyvGUaZxY20'
client_secret = 'k5PZpINooRpjAfQccGwLUr2ZMEtRJtoX8cKaooHjKewWupxRBG'
token_url = 'https://api.twitter.com/oauth2/token'
client = BackendApplicationClient(client_id)
twitter = OAuth2Session(client_id, client=client)
headers = {
'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
}
twitter.fetch_token(token_url, headers=headers, auth=(client_id, client_secret))
# Only public resources available to this application-only clients.
r = twitter.get('https://api.twitter.com/1.1/statuses/user_timeline.json?count=100&screen_name=twitterapi')
print r.content
Make sure you use the github version of the libraries
pip install git+https://github.com/idan/oauthlib.git
pip install git+https://github.com/requests/requests-oauthlib.git
I'm new to Oauth. In the past for twitter applications written in Python i used python-oauth2 library to initialize client like this:
consumer = oauth.Consumer(key = CONSUMER_KEY, secret = CONSUMER_SECRET)
token = oauth.Token(key = ACCESS_KEY, secret = ACCESS_SECRET)
client = oauth.Client(consumer, token)
That was easy because twitter provides both CONSUMER and ACCESS keys and secrets. But now i need to do the same for tumblr. The problem is that tumblr provides only CONSUMER_KEY, CONSUMER_SECRET and these urls:
Request-token URL http://www.tumblr.com/oauth/request_token
Authorize URL http://www.tumblr.com/oauth/authorize
Access-token URL http://www.tumblr.com/oauth/access_token
Using this data how can i initialize client to access tumblr API?
UPD
jterrace suggested a code i tried to use before. The problem with it is oauth_callback. If i don't specify any, api returns error "No oauth_callback specified", but if i do specify some url like "http://example.com/oauthcb/" and follow the link http://www.tumblr.com/oauth/authorize?oauth_token=9ygTF..., then press Allow button, tumblr doesn't show any PIN code page, it immediately redirects to that callback url, which is useless since it's desktop application. Why PIN code isn't shown?
UPD 2
Tumblr API doesn't support PIN code authorization. Use xAuth instead - https://groups.google.com/group/tumblr-api/browse_thread/thread/857285e6a2b4268/15060607dc306c1d?lnk=gst&q=pin#15060607dc306c1d
First, import the oauth2 module and set up the service's URL and consumer information:
import oauth2
REQUEST_TOKEN_URL = 'http://www.tumblr.com/oauth/request_token'
AUTHORIZATION_URL = 'http://www.tumblr.com/oauth/authorize'
ACCESS_TOKEN_URL = 'http://www.tumblr.com/oauth/access_token'
CONSUMER_KEY = 'your_consumer_key'
CONSUMER_SECRET = 'your_consumer_secret'
consumer = oauth2.Consumer(CONSUMER_KEY, CONSUMER_SECRET)
client = oauth2.Client(consumer)
Step 1: Get a request token. This is a temporary token that is used for
having the user authorize an access token and to sign the request to obtain
said access token.
resp, content = client.request(REQUEST_TOKEN_URL, "GET")
request_token = dict(urlparse.parse_qsl(content))
print "Request Token:"
print " - oauth_token = %s" % request_token['oauth_token']
print " - oauth_token_secret = %s" % request_token['oauth_token_secret']
Step 2: Redirect to the provider. Since this is a CLI script we do not
redirect. In a web application you would redirect the user to the URL
below.
print "Go to the following link in your browser:"
print "%s?oauth_token=%s" % (AUTHORIZATION_URL, request_token['oauth_token'])
# After the user has granted access to you, the consumer, the provider will
# redirect you to whatever URL you have told them to redirect to. You can
# usually define this in the oauth_callback argument as well.
oauth_verifier = raw_input('What is the PIN? ')
Step 3: Once the consumer has redirected the user back to the oauth_callback
URL you can request the access token the user has approved. You use the
request token to sign this request. After this is done you throw away the
request token and use the access token returned. You should store this
access token somewhere safe, like a database, for future use.
token = oauth2.Token(request_token['oauth_token'], request_token['oauth_token_secret'])
token.set_verifier(oauth_verifier)
client = oauth2.Client(consumer, token)
resp, content = client.request(ACCESS_TOKEN_URL, "POST")
access_token = dict(urlparse.parse_qsl(content))
print "Access Token:"
print " - oauth_token = %s" % access_token['oauth_token']
print " - oauth_token_secret = %s" % access_token['oauth_token_secret']
print
Now that you have an access token, you can call protected methods with it.
EDIT: Turns out that tumblr does not support the PIN authorization method. Relevant post here.
If you just want to gain an access-token/secret to sign, you could just setup your callback URL as: http://localhost/blah
Fireup the CLI-app (after modifying the callback-url, secret and token ofcourse)
Follow the link in your browser
Allow app
View addressbar of the page you've been redirected to in the browser after allowing your app. It should look something like:
http://localhost/blah?oauth_token=xxxxxxxxxxxxxxxxxxxxxxxxxx0123456789ABCDEFGHIJKLMN&oauth_verifier=XXXXXXXXXXXXXXXXXXXXXXXXX0123456789abcdefghijklmn
Use the value of the query-parameter 'oauth_verifier' as your PIN:
XXXXXXXXXXXXXXXXXXXXXXXXX0123456789abcdefghijklmn
The CLI should print out your oauth-token and oauth-token-secret.
HTH! Got this working for tumblr in this way :)
Have a look at https://github.com/ToQoz/Pyblr
It uses oauth2 and urllib to provide a nice wrapper for exactly what you're trying to do.
It seems that what you're trying to do is access an OAuth 1 API with an OAuth 2 client.
See https://github.com/simplegeo/python-oauth2 and look for “three-legged OAuth example”.
had this problem with oauth2 and facebook.
#deepvanbinnen's answer lead me into the right direction.
facebook actually redirected to a page similar to this
'http://localhost/blah?code=AQAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX#_=_'
using then the ' AQAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX#_=_ as the PIN actually got me the access to the requested facebook account.
#jterrance's answer is good. However, realize it is a one _time_ manual procedure to get the access token. The access token is the key that you use for all subsequent API calls. (That's why he recommends saving the access token in a database.) The string referred to as 'PIN' (aka the verification key) is not necessarily a number. It can be a printable string in any form. That verification key is displayed on the authorization page at the URL printed in step 2 then pasted into the prompt for a the 'PIN'.