We Keep Coding

CloudKit Server-to-Server auth: Keep getting 401 Authentication failed

I have been recently exploring the CloudKit and related frameworks. I got the communication with my app working, as well as with my website using CloudKitJS. Where I am struggling is the Server-to-Server communication (which I would need for exporting data from public database in csv.
I have tried Python package requests-cloudkit, which others were suggesting. I have created a Server-to-Server token, and have copied only the key between START and END line once creating the eckey.pem file. I then got this code:
from requests_cloudkit import CloudKitAuth
from restmapper import restmapper
import json
KEY_ID = '[my key ID from CK Dashboard]'
SECRET_FILE_KEY = 'eckey.pem'
AUTH = CloudKitAuth(KEY_ID, SECRET_FILE_KEY)
PARAMS = {
'query':{
'recordType': '[my record type]'
},
}
CloudKit = restmapper.RestMapper("https://api.apple-cloudkit.com/database/1/[my container]/development/")
cloudkit = CloudKit(auth=AUTH)
response = cloudkit.POST.public.records.query(json.dumps(PARAMS))
I am then getting the 401 Authentication failed response. I am stuck on this for days, so I would be grateful for any help or advice. 😊

Creating the server-to-server key is an important first step, but in order to make HTTP requests after that, you have to sign each request.
Look for the Authenticate Web Service Requests section near the bottom of this documentation page.
It's a little bit convoluted, but you have to carefully construct signed headers to include with each request you make. I'm not familiar with how to do it in Python, but here's how I do it in NodeJS which may help:
//Get the timestamp in a very specific format
let date = moment().utc().format('YYYY-MM-DD[T]HH:mm:ss[Z]')
//Construct the subpath
let endpoint = '/records/lookup'
let path = '/database/1/iCloud.*****/development/public'
let subpath = path+endpoint
//Get the key file
let privateKeyFile = fs.readFileSync('../../'+SECRET_FILE_KEY, 'utf8')
//Make a string out of your JSON query
let query = {
recordType: '[my record type]'
}
let requestBody = JSON.stringify(query)
//Hash the query
let bodyHash = crypto.createHash('sha256').update(requestBody, 'utf8').digest('base64')
//Assemble the components you just generated in a special format
//[Current date]:[Request body]:[Web service URL subpath]
let message = date+':'+bodyHash+':'+subpath
//Sign it
let signature = crypto.createSign('RSA-SHA256').update(message).sign(privateKeyFile, 'base64')
//Assemble your headers and include them in your HTTP request
let headers = {
'X-Apple-CloudKit-Request-KeyID': KEY_ID,
'X-Apple-CloudKit-Request-ISO8601Date': date,
'X-Apple-CloudKit-Request-SignatureV1': signature
}
This is a bit hairy at first, but I just put all this stuff in a function that I reuse whenever I need to make a request.
Apple's documentation has pretty much been abandoned and it's hard to find good help with CloudKit Web Services these days.

Updating records with Python 3 through API on Rails 4 application

I'm working on a Rails 4 / mongoid application which needs to expose APIs for other applications and scripts. I need to be able to update documents in one of the models through an API with Python 3 script. I'm a bit fresh with Python hence asking for help here.
I already found out how to query Rails APIs with Python 3 and urllib but struggling with updates. I was trying to go through Python 3.5 docs for urllib2 but struggling to apply this to my script.
What goes to data and how to add authentication token to headers, which in curl would look something like this
-H 'Authorization:Token token="xxxxxxxxxxxxxxxx"'
-X POST -d "name=A60E001&status=changed"
I would greatly appreciate if somebody explained how to, for example, update status based on name (name is not unique yet but will be). My biggest challenge is the Python side. Once I have the data in params on Rails side I think I can handle it. I think.
I included my model and update action from the controller below.
app/models/experiment.rb
class Experiment
include Mongoid::Document
include Mongoid::Timestamps
field :name, type: String
field :status, type:String
end
app/controllers/api/v1/experiments_controller.rb
module Api
module V1
class ExperimentsController < ActionController::Base
before_filter :restrict_access
...
def update
respond_to do |format|
if #expt_proc.update(expt_proc_params)
format.json { render :show, status: :ok, location: #expt_proc }
else
format.json { render json: #expt_proc.errors, status: :unprocessable_entity }
end
end
end
...
private
def restrict_access
authenticate_or_request_with_http_token do |token, options|
ApiKey.where(access_token: token).exists?
end
end
...

I figured out who to send a PATCH request with Python 3 and update the the record's status by name.
Thanks to this post found out about requests module. I used requests.patch to update the record and it works great.
python code
import requests
import json
url = 'http://0.0.0.0:3000/api/v1/update_expt_queue.json'
payload = {'expt_name' : 'myExperiment1', 'status' : 'finished' }
r = requests.patch(url, payload)
There are two problems remaining:
How to add headers to this request which would allow me to go through token based authentication. At the moment it's disabled. request.patch only takes 2 parameters, it doesn't take headers parameter.
How to access the JSON which is sent in response by Rails API.

How to validate a webhook signature using python and openssl

I am trying to validate an incoming webhook and so far the resulting hash is not matching the test hash generated by the api.
The docs list the following example for Ruby however I am using Python/Django so any help to 'convert' this function would be appreciated!
Ruby Function
# request_signature - the signature sent in Webhook-Signature
# request_body - the JSON body of the webhook request
# secret - the secret for the webhook endpoint
require "openssl"
digest = OpenSSL::Digest.new("sha256")
calculated_signature = OpenSSL::HMAC.hexdigest(digest, secret, request_body)
if calculated_signature == request_signature
# Signature ok!
else
# Invalid signature. Ignore the webhook and return 498 Token Invalid
end
This is roughly what I have put together myself so far using https://docs.python.org/3/library/hashlib.html.
Python Attempt
import hashlib
secret = "xxxxxxxxxxxxxxxxxx"
json_data = {json data}
h = hashlib.new('sha256')
h.update(secret)
h.update(str(json_data))
calculated_signature = h.hexdigest()
if calculated_signature == webhook_signature:
do_something()
else:
return 498
When I run the above the hashes never match obviously due to my incorrect Python implementation.
Any help/pointers would be greatly appreciated!

I believe it should be something like this:
import hmac
import hashlib
digester = hmac.new(secret, request_body, hashlib.sha256)
calculated_signature = digester.hexdigest()
A few notes:
Use the actual request body. Don't rely on str(json_data) equalling the request body. This will almost certainly fail as python will print out inner strings using repr which will likely leave a bunch of spurious u"..." that aren't actually in the response. json.dumps won't necessarily do better because there could be whitespace differences that are isignificant to JSON, but are very significant to the hmac signature.
hmac is your friend :-)

Insufficient permissions when trying to create a Quizlet set

I am trying to create a set on Quizlet.com, using its API found here: https://quizlet.com/api/2.0/docs/sets#add
Here is my code of a set I am trying to create:
import requests
quizkey = my_client_id
authcode = my_secret_code # I'm not sure if I need this or not
data = {"client_id":quizkey, "whitespace":1, "title":"my-api-set",
"lang_terms":"it", "lang_definitions":"en",
"terms":['uno','due'], "definitions":["one","two"]}
apiPrefix = "https://api.quizlet.com/2.0/sets"
r = requests.post(url=apiPrefix, params=data)
print r.text
The response is:
{
"http_code": 401,
"error": "invalid_scope",
"error_title": "Not Allowed",
"error_description": "You do not have sufficient permissions to perform the requested action."
}
I also tried "access_token":authcode instead of "client_id":quizkey, but this resulted in the error: "You do not have sufficient permissions to perform the requested action."
How can I fix this and not get a 401 error?

Alright so 3 and a half years later (!!) I've looked into this again and here's what I've discovered.
To add a set you need an access token - this is different to the client_id (what I call quizkey in my code), and to be quite honest I don't remember what authcode in my code is.
This token is obtained by going through the user authentication flow. To summarise it:
Send a POST request to https://quizlet.com/authorize like so:
https://quizlet.com/authorize?response_type=code&client_id=MY_CLIENT_ID&scope=read&state=RANDOM_STRING
Keep the response_type as code, replace client_id with your client_id, keep the scope as read, and state can be anything
I believe this requires human intervention because you're literally authorising your own account? Not sure of another way...
You'll receive a response back with a code
Let's call this RESPONSE_CODE for now
Send a POST request to https://api.quizlet.com/oauth/token, specifying 4 mandatory parameters:
grant_type="authorization_code" (this never changes)
code=RESPONSE_CODE
redirect_uri=https://yourredirecturi.com (this can be found at your personal API dashboard)
client ID and secret token separated by a colon and then base64-encoded (the user authentication flow link above tells you what this is if you don't want to do any of the encoding)
You'll receive the access_token from this API call
Now you can use that access_token in your call to create a set like I've done above (just replace "client_id":quizkey with "access_token":access_token)

You will need to authenticate in order to make sets. This link gives an overview:
https://quizlet.com/api/2.0/docs/making_api_calls
And this one provides details about the authentication process:
https://quizlet.com/api/2.0/docs/authorization_code_flow

How do you use cookies and HTTP Basic Authentication in CherryPy?

I have a CherryPy web application that requires authentication. I have working HTTP Basic Authentication with a configuration that looks like this:
app_config = {
'/' : {
'tools.sessions.on': True,
'tools.sessions.name': 'zknsrv',
'tools.auth_basic.on': True,
'tools.auth_basic.realm': 'zknsrv',
'tools.auth_basic.checkpassword': checkpassword,
}
}
HTTP auth works great at this point. For example, this will give me the successful login message that I defined inside AuthTest:
curl http://realuser:realpass#localhost/AuthTest/
Since sessions are on, I can save cookies and examine the one that CherryPy sets:
curl --cookie-jar cookie.jar http://realuser:realpass#localhost/AuthTest/
The cookie.jar file will end up looking like this:
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
localhost FALSE / FALSE 1348640978 zknsrv 821aaad0ba34fd51f77b2452c7ae3c182237deb3
However, I'll get an HTTP 401 Not Authorized failure if I provide this session ID without the username and password, like this:
curl --cookie 'zknsrv=821aaad0ba34fd51f77b2452c7ae3c182237deb3' http://localhost/AuthTest
What am I missing?
Thanks very much for any help.

So, the short answer is you can do this, but you have to write your own CherryPy tool (a before_handler), and you must not enable Basic Authentication in the CherryPy config (that is, you shouldn't do anything like tools.auth.on or tools.auth.basic... etc) - you have to handle HTTP Basic Authentication yourself. The reason for this is that the built-in Basic Authentication stuff is apparently pretty primitive. If you protect something by enabling Basic Authentication like I did above, it will do that authentication check before it checks the session, and your cookies will do nothing.
My solution, in prose
Fortunately, even though CherryPy doesn't have a way to do both built-in, you can still use its built-in session code. You still have to write your own code for handling the Basic Authentication part, but in total this is not so bad, and using the session code is a big win because writing a custom session manager is a good way to introduce security bugs into your webapp.
I ended up being able to take a lot of things from a page on the CherryPy wiki called Simple authentication and access restrictions helpers. That code uses CP sessions, but rather than Basic Auth it uses a special page with a login form that submits ?username=USERNAME&password=PASSWORD. What I did is basically nothing more than changing the provided check_auth function from using the special login page to using the HTTP auth headers.
In general, you need a function you can add as a CherryPy tool - specifically a before_handler. (In the original code, this function was called check_auth(), but I renamed it to protect().) This function first tries to see if the cookies contain a (valid) session ID, and if that fails, it tries to see if there is HTTP auth information in the headers.
You then need a way to require authentication for a given page; I do this with require(), plus some conditions, which are just callables that return True. In my case, those conditions are zkn_admin(), and user_is() functions; if you have more complex needs, you might want to also look at member_of(), any_of(), and all_of() from the original code.
If you do it like that, you already have a way to log in - you just submit a valid session cookie or HTTPBA credentials to any URL you protect with the #require() decorator. All you need now is a way to log out.
(The original code instead has an AuthController class which contains login() and logout(), and you can use the whole AuthController object in your HTTP document tree by just putting auth = AuthController() inside your CherryPy root class, and get to it with a URL of e.g. http://example.com/auth/login and http://example.com/auth/logout. My code doesn't use an authcontroller object, just a few functions.)
Some notes about my code
Caveat: Because I wrote my own parser for HTTP auth headers, it only parses what I told it about, which means just HTTP Basic Auth - not, for example, Digest Auth or anything else. For my application that's fine; for yours, it may not be.
It assumes a few functions defined elsewhere in my code: user_verify() and user_is_admin()
I also use a debugprint() function which only prints output when a DEBUG variable is set, and I've left these calls in for clarity.
You can call it cherrypy.tools.WHATEVER (see the last line); I called it zkauth based on the name of my app. Take care NOT to call it auth, or the name of any other built-in tool, though .
You then have to enable cherrypy.tools.WHATEVER in your CherryPy configuration.
As you can see by all the TODO: messages, this code is still in a state of flux and not 100% tested against edge cases - sorry about that! It will still give you enough of an idea to go on, though, I hope.
My solution, in code
import base64
import re
import cherrypy
SESSION_KEY = '_zkn_username'
def protect(*args, **kwargs):
debugprint("Inside protect()...")
authenticated = False
conditions = cherrypy.request.config.get('auth.require', None)
debugprint("conditions: {}".format(conditions))
if conditions is not None:
# A condition is just a callable that returns true or false
try:
# TODO: I'm not sure if this is actually checking for a valid session?
# or if just any data here would work?
this_session = cherrypy.session[SESSION_KEY]
# check if there is an active session
# sessions are turned on so we just have to know if there is
# something inside of cherrypy.session[SESSION_KEY]:
cherrypy.session.regenerate()
# I can't actually tell if I need to do this myself or what
email = cherrypy.request.login = cherrypy.session[SESSION_KEY]
authenticated = True
debugprint("Authenticated with session: {}, for user: {}".format(
this_session, email))
except KeyError:
# If the session isn't set, it either wasn't present or wasn't valid.
# Now check if the request includes HTTPBA?
# FFR The auth header looks like: "AUTHORIZATION: Basic <base64shit>"
# TODO: cherrypy has got to handle this for me, right?
authheader = cherrypy.request.headers.get('AUTHORIZATION')
debugprint("Authheader: {}".format(authheader))
if authheader:
#b64data = re.sub("Basic ", "", cherrypy.request.headers.get('AUTHORIZATION'))
# TODO: what happens if you get an auth header that doesn't use basic auth?
b64data = re.sub("Basic ", "", authheader)
decodeddata = base64.b64decode(b64data.encode("ASCII"))
# TODO: test how this handles ':' characters in username/passphrase.
email,passphrase = decodeddata.decode().split(":", 1)
if user_verify(email, passphrase):
cherrypy.session.regenerate()
# This line of code is discussed in doc/sessions-and-auth.markdown
cherrypy.session[SESSION_KEY] = cherrypy.request.login = email
authenticated = True
else:
debugprint ("Attempted to log in with HTTBA username {} but failed.".format(
email))
else:
debugprint ("Auth header was not present.")
except:
debugprint ("Client has no valid session and did not provide HTTPBA credentials.")
debugprint ("TODO: ensure that if I have a failure inside the 'except KeyError'"
+ " section above, it doesn't get to this section... I'd want to"
+ " show a different error message if that happened.")
if authenticated:
for condition in conditions:
if not condition():
debugprint ("Authentication succeeded but authorization failed.")
raise cherrypy.HTTPError("403 Forbidden")
else:
raise cherrypy.HTTPError("401 Unauthorized")
cherrypy.tools.zkauth = cherrypy.Tool('before_handler', protect)
def require(*conditions):
"""A decorator that appends conditions to the auth.require config
variable."""
def decorate(f):
if not hasattr(f, '_cp_config'):
f._cp_config = dict()
if 'auth.require' not in f._cp_config:
f._cp_config['auth.require'] = []
f._cp_config['auth.require'].extend(conditions)
return f
return decorate
#### CONDITIONS
#
# Conditions are callables that return True
# if the user fulfills the conditions they define, False otherwise
#
# They can access the current user as cherrypy.request.login
# TODO: test this function with cookies, I want to make sure that cherrypy.request.login is
# set properly so that this function can use it.
def zkn_admin():
return lambda: user_is_admin(cherrypy.request.login)
def user_is(reqd_email):
return lambda: reqd_email == cherrypy.request.login
#### END CONDITIONS
def logout():
email = cherrypy.session.get(SESSION_KEY, None)
cherrypy.session[SESSION_KEY] = cherrypy.request.login = None
return "Logout successful"
Now all you have to do is enable both builtin sessions and your own cherrypy.tools.WHATEVER in your CherryPy configuration. Again, take care not to enable cherrypy.tools.auth. My configuration ended up looking like this:
config_root = {
'/' : {
'tools.zkauth.on': True,
'tools.sessions.on': True,
'tools.sessions.name': 'zknsrv',
}
}