We Keep Coding

CloudKit Server-to-Server auth: Keep getting 401 Authentication failed

I have been recently exploring the CloudKit and related frameworks. I got the communication with my app working, as well as with my website using CloudKitJS. Where I am struggling is the Server-to-Server communication (which I would need for exporting data from public database in csv.
I have tried Python package requests-cloudkit, which others were suggesting. I have created a Server-to-Server token, and have copied only the key between START and END line once creating the eckey.pem file. I then got this code:
from requests_cloudkit import CloudKitAuth
from restmapper import restmapper
import json
KEY_ID = '[my key ID from CK Dashboard]'
SECRET_FILE_KEY = 'eckey.pem'
AUTH = CloudKitAuth(KEY_ID, SECRET_FILE_KEY)
PARAMS = {
'query':{
'recordType': '[my record type]'
},
}
CloudKit = restmapper.RestMapper("https://api.apple-cloudkit.com/database/1/[my container]/development/")
cloudkit = CloudKit(auth=AUTH)
response = cloudkit.POST.public.records.query(json.dumps(PARAMS))
I am then getting the 401 Authentication failed response. I am stuck on this for days, so I would be grateful for any help or advice. 😊

Creating the server-to-server key is an important first step, but in order to make HTTP requests after that, you have to sign each request.
Look for the Authenticate Web Service Requests section near the bottom of this documentation page.
It's a little bit convoluted, but you have to carefully construct signed headers to include with each request you make. I'm not familiar with how to do it in Python, but here's how I do it in NodeJS which may help:
//Get the timestamp in a very specific format
let date = moment().utc().format('YYYY-MM-DD[T]HH:mm:ss[Z]')
//Construct the subpath
let endpoint = '/records/lookup'
let path = '/database/1/iCloud.*****/development/public'
let subpath = path+endpoint
//Get the key file
let privateKeyFile = fs.readFileSync('../../'+SECRET_FILE_KEY, 'utf8')
//Make a string out of your JSON query
let query = {
recordType: '[my record type]'
}
let requestBody = JSON.stringify(query)
//Hash the query
let bodyHash = crypto.createHash('sha256').update(requestBody, 'utf8').digest('base64')
//Assemble the components you just generated in a special format
//[Current date]:[Request body]:[Web service URL subpath]
let message = date+':'+bodyHash+':'+subpath
//Sign it
let signature = crypto.createSign('RSA-SHA256').update(message).sign(privateKeyFile, 'base64')
//Assemble your headers and include them in your HTTP request
let headers = {
'X-Apple-CloudKit-Request-KeyID': KEY_ID,
'X-Apple-CloudKit-Request-ISO8601Date': date,
'X-Apple-CloudKit-Request-SignatureV1': signature
}
This is a bit hairy at first, but I just put all this stuff in a function that I reuse whenever I need to make a request.
Apple's documentation has pretty much been abandoned and it's hard to find good help with CloudKit Web Services these days.

Create Shared Access Signature in Python - Azure Api Rest Management

sorry for my english.
I'm interacting with the Azure admin rest API and I want to programmatically create a SAS token. In Azure documentation is explained for C # (I attach the code below) but I need to implement it in Python (I'm new) and I can't get the data encoding and signing process correctly, even though I've searched a lot of information and tested in some different ways. Could someone help me to "translate" this code?. Thank you very much.
c#
using System;
using System.Text;
using System.Globalization;
using System.Security.Cryptography;
public class Program
{
public static void Main()
{
var id = "account-name";
var key = "account-key";
var expiry = DateTime.UtcNow.AddDays(10);
using (var encoder = new HMACSHA512(Encoding.UTF8.GetBytes(key)))
{
var dataToSign = id + "\n" + expiry.ToString("O", CultureInfo.InvariantCulture);
var hash = encoder.ComputeHash(Encoding.UTF8.GetBytes(dataToSign));
var signature = Convert.ToBase64String(hash);
var encodedToken = string.Format("SharedAccessSignature uid={0}&ex={1:o}&sn={2}", id, expiry, signature);
Console.WriteLine(encodedToken);
}
}
}

Based on #GaJsu 's solution
Following is my solution:
import base64
import hmac
import hashlib
from datetime import datetime
from dateutil.relativedelta import relativedelta
identifier = 'the "Identifier" value in "management API" tab'
end_date = datetime.now() + relativedelta(months=+6)
expiry = f'{end_date.isoformat()}0Z'
key_azure = 'the key in "management API" tab, primary/secondary key'
string_to_sign = f'{identifier}\n{expiry}'
signature = (
base64.b64encode(
hmac.new(
bytearray(key_azure, "utf-8"),
bytearray(string_to_sign,"utf-8") ,
hashlib.sha512).digest()
)
).decode("utf-8")
auth_sas = f"SharedAccessSignature uid={identifier}&ex={expiry}&sn={signature}"

Finally I can create the SAS token. Here is the code for the signature:
string_to_sign = '{}{}{}'.format(id_azure,'\n',expiry)
signature = (base64.b64encode(hmac.new(bytearray(key_azure, "utf-8") , bytearray(string_to_sign,"utf-8") , hashlib.sha512).digest())).decode("utf-8").replace("\n", "")

POST request remake from Java to python gets null

I am totally new to python flask and encountered a problem when writing some code using the requests and flask modules.
I am working on a project using the web API offered by the Panther platform. The project provided an example using Apache Java.
The source code is as below (see more for details).
public class TestProject {
public static void main(String args[]) throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpPost httppost = new HttpPost("http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?");
StringBody organism = new StringBody("Homo sapiens", ContentType.TEXT_PLAIN);
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
StringBody enrichmentType = new StringBody("process", ContentType.TEXT_PLAIN);
StringBody testType = new StringBody("FISHER", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("FDR", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("BONFERRONI", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("NONE", ContentType.TEXT_PLAIN);
StringBody type = new StringBody("enrichment", ContentType.TEXT_PLAIN);
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("organism", organism)
.addPart("geneList", fileData)
.addPart("enrichmentType", enrichmentType)
.addPart("test_type", testType)
.addPart("type", type)
//.addPart("correction", cor)
.build();
httppost.setEntity(reqEntity);
CloseableHttpResponse response = httpclient.execute(httppost);
try {
//System.out.println("----------------------------------------");
//System.out.println(response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
System.out.println(IOUtils.toString(resEntity.getContent(), StandardCharsets.UTF_8));
}
EntityUtils.consume(resEntity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
The part I am mostly interested in is .addPart("organism", organism) and all the other code with similar structures. They will help pass the parameters from a third-party website to the web API offered by Panther.
I remade the JAVA code into python3 using requests. The code is as follows:
uploadTemp = {'file':open('./app/static/data_temp/temp.txt','rb')}
url="http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
params = {"organism":organism,"geneList":pantherName,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
# or params = {"organism":organism,"geneList":uploadTemp,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
Pantherpost= requests.post(url, params = params)
print(Pantherpost.text)
I am expecting an XML object from the web API including some basic biological information. However, the result I got was null (or \n\n\rnull\n when I print Pantherpost.content)
It seems that the parameters I have got from my own web were not correctly sent to the web API.
In addition to this getting null problem, as a beginner, I am also not quite sure about whether the "geneList" part should be receiving a plain-text object or a file. The manual says it is expecting a file, however, it may have been reformatted into plain-text by this command
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
Anyway, I did try both interpretations: pantherName is a list with name correctly formatted in plain-text and uploadTemp is a .txt file generated for the project. There must be some extra bugs in my code since it returned null in both cases.
Can someone please help out? Thank you very much.

I've found the following issues with your python code:
One. If you want to POST a file using requests, you should use keyword files=.
Two. Keys in files object should match respective parameters of the request (you're using file instead).
Three. You put your parameters in the wrong place of the request by writing params=params.
Function annotation from requests source code:
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:Request.
In example Java code StringBody is used to create parameters, which implies that parameters should be placed inside the body of HTTP request, not query string. So you should use data= keyword instead. If you use params=, output will be null.
SO article on difference between data and params keywords in requests.
So I've spent some time reading thier manual and made a test script:
import requests
url = "http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
filepath = "C:\\data\\YOUR_DATA.txt" # change to your file location
# all required parameters according to manual, except geneList which is a file (see below)
params = { # using defaults from manual
"type": "enrichment",
"organism": "Homo sapiens",
"enrichmentType": "process",
"test_type": "FISHER",
"correction": "FDR",
}
# note that the key here is the name of paramter: geneList
files = {'geneList': open(filepath, 'rb')}
# it outputs null, when 'params=params' is used
r = requests.post(url, data=params, files=files)
print(r.status_code)
print(r.text)
Output:
200
Id Name GeneId raw P-value FDR

Using a Zapier Custom Request Webhook with JSON Web Tokens

I need to access an API that uses JSON Web Tokens as their authentication method. Is there a good way to use a python code step to create this token then add that token as a header to a custom request webhook step?

My experience authenticating with APIs has been using the simple API key method. As such I first read your question and didn't fully understand. I decided to do some research and hopefully learn something along the way, and I certainly did. I share my findings and answer below:
For starters I began reading into JSON Web Tokens(JWT) which lead me to the JWT website, which was an excellent resource. It very clearly spells out the components that make up a JWT and how they need to be formatted, I would highly recommend having a look.
From the JWT website I found that a JWT is made up of three components:
A base64 URL safe encoded header.
A base64 URL safe encoded payload.
A base64 URL safe encoded signature.
All three of the above combined form the correctly formatted JWT. Fortunately the JWT website has a list of libraries made for Python. Unfortunately none of these third-party libraries are available in the vanilla Python offered by the Zapier code module. To get this done required reading some source code and leveraging what libraries we do have available. So after a few hours and lots of trial and error I was able to come up with the following solution for generating a correctly formatted JWT:
import hashlib
import hmac
import requests
from base64 import urlsafe_b64encode
def base64url_encode(payload):
if not isinstance(payload, bytes):
payload = payload.encode('utf-8')
encode = urlsafe_b64encode(payload)
return encode.decode('utf-8').rstrip('=')
def generate_JWT(header, payload, secret):
encoded_header = base64url_encode(header)
encoded_payload = base64url_encode(payload)
signature = hmac.new(secret,
encoded_header + "." + encoded_payload,
hashlib.sha256)
encoded_signature = base64url_encode(signature.digest())
return encoded_header + "." + encoded_payload + "." + encoded_signature
def get_request(url, jwt):
headers = {
"Authorization" : "Bearer " + jwt
}
result = requests.get(url, headers=headers)
return result
secret = "yoursecrettoken"
header = '{"alg":"HS256","typ":"JWT"}'
payload = '{"sub":"1234567890","name":"John Doe","iat":1516239022}'
jwt = generate_JWT(header, payload, secret)
response = get_request("https://SomeApiEndpoint.com/api/", jwt)
You can test the output of this against the JWT's debugger here.
Note: For the encoding to work properly for the header and payload objects you have to convert them to a string object. I tried doing this by calling the JSON.dumps() function and passing the dictionary objects, but when I encoded the return values they did not match what was shown on the JWT debugger. The only solution I could find was by wrapping the dictionary objects in quotations and ensuring there were no spaces within it.
And so with the JWT in hand you can use it in your Zapier Webhooks custom get request step, or you could save the zap and send the request in the same code module using Python's request library as I have in my code example.
Thanks for the learning opportunity, and I hope this helps.

I had a similar issue trying to generate and use a JWT in a custom integration. Unfortunately, this code above did not work for my situation. I'm currently using the below javascript and it seems to be functioning perfectly.
const toBase64 = obj => {
const str = JSON.stringify (obj);
return Buffer.from(str).toString ('base64');
};
const replaceSpecialChars = b64string => {
// this will match the special characters and replace them with url-safe substitutes
return b64string.replace (/[=+/]/g, charToBeReplaced => {
switch (charToBeReplaced) {
case '=':
return '';
case '+':
return '-';
case '/':
return '_';
}
});
};
const crypto = require('crypto');
const signatureFunction = crypto.createSign('RSA-SHA256');
const headerObj = {
alg: 'RS256',
typ: 'JWT',
};
const payloadObj = {
iat: Math.round(Date.now() / 1000), // lists the current Epoch time
exp: Math.round(Date.now() / 1000) + 3600, // adds one hour
sub: '1234567890'
name: 'John Doe'
};
const base64Header = toBase64(headerObj);
const base64Payload = toBase64(payloadObj);
const base64UrlHeader = replaceSpecialChars(base64Header);
const base64UrlPayload = replaceSpecialChars(base64Payload);
signatureFunction.write(base64UrlHeader + '.' + base64UrlPayload);
signatureFunction.end();
// The private key without line breaks
const privateKey = `-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQC5Q+0Je6sZ6BuX
cTsN7pEzAaj4819UE7gM+Tf7U5AKHSKk3hN5UILtp5EuEO7h7H+lyknn/5txltA4
-----END PRIVATE KEY-----`;
const signatureBase64 = signatureFunction.sign(privateKey, 'base64');
const signatureBase64Url = replaceSpecialChars(signatureBase64);
console.log("Your JWT is: " + base64UrlHeader + "." + base64UrlPayload + "." + signatureBase64Url);
I have this code in a Zapier code-step prior to calling the custom integration and pass in the generated token object to authenticate the call.
Hopefully, this helps someone!

Twilio Programmable Voice call immediately completes

I am using the new Twilio Programmable Voice SDK with Swift and Python. I have begun with the respective quick starter projects, and for the most part things work. I'm able to get a valid access token, I'm able to successfully create a call, and I am even able to receive the call. The problem is on the caller side of the house.
When I try to make a call via the Swift SDK the call is disconnected before it ever starts to ring on the other end.
I have read in the Twilio docs that the client.calls.create function will immediately return a status of complete if you do not handle the status_callback event. I have tried to add this but whenever I do I get an error saying the the key status_callback is not a valid parameter for the client.calls.create function. Also, I cannot find any examples anywhere of actually how to handle the call status.
My question is what in the wold am I doing wrong here? Any help would be greatly appreciated.
Here is my Python code
#app.route('/outgoing', methods=['GET', 'POST'])
def outgoing():
account_sid = os.environ.get("ACCOUNT_SID", ACCOUNT_SID)
api_key = os.environ.get("API_KEY", API_KEY)
api_key_secret = os.environ.get("API_KEY_SECRET", API_KEY_SECRET)
from_number = [HARD_CODED_PHONE_NUMBER_FOR_CALLER]
to_number = [HARD_CODED_PHONE_NUMBER_FOR_RECIPIENT]
client = Client(api_key, api_key_secret, account_sid)
call = client.calls.create(url='http://twimlets.com/holdmusic?Bucket=com.twilio.music.ambient', to=to_number, from_=from_number)
# return str(call.sid)
resp = twilio.twiml.Response()
resp.say("Thank you for calling");
return str(resp)
Here is my relevant iOS code. Please bear in mind that this is NOT my full source. I have only provided what should be necessary in this event. My full source does include handling the registry and invite delegates. I also did not include my source that shows/hides my active call UI as there are no problems with that. This is simply to show how I am placing a call and receiving the call complete delegate.
class VoiceManager: NSObject, PKPushRegistryDelegate, TVONotificationDelegate, TVOCallDelegate, AVAudioPlayerDelegate {
//MARK: - Singleton
static let sharedManager = VoiceManager()
//MARK: - Private Constants
private let baseURLString = [MY_WEBAPP_ENDPOINT]
private let accessTokenEndpoint = "/accessToken"
//MARK: - Private Variables
private var deviceTokenString:String?
private var callInvite: TVOCallInvite?
private var call: TVOCall?
private var status: VoiceStatus = .idle
//MARK: - Getters
private func fetchAccessToken() -> String? {
guard let accessTokenURL = URL(string: baseURLString + accessTokenEndpoint) else {
return nil
}
return try? String.init(contentsOf: accessTokenURL, encoding: .utf8)
}
func placeCall(withParameters params: VoiceParameters, completion: #escaping (_ success: Bool, _ error: VAError?) -> Void) {
if (call != nil) {
call?.disconnect()
completion(false, .phoneCallInProgress)
status = .callEnded
hideActiveCallUI()
} else {
guard let accessToken = fetchAccessToken() else {
completion(false, .phoneAccessTokenFetchFailed)
return
}
guard let paramsDict = params.toDictionary() else {
completion(false, .phoneAccessTokenFetchFailed)
return
}
playOutgoingRingtone(completion: { [weak self] in
if let strongSelf = self {
strongSelf.call = VoiceClient.sharedInstance().call(accessToken, params: [:], delegate: strongSelf) //NOTE: The params here are not necessary as the phone numbers for now are hard coded on the server
if (strongSelf.call == nil) {
strongSelf.status = .callEnded
completion(false, .phoneCallFailed)
return
} else {
strongSelf.status = .callConnecting
self?.showActiveCallUI(withParameters: params)
completion(true, nil)
}
}
})
}
}
// MARK: TVOCallDelegate
func callDidConnect(_ call: TVOCall) {
NSLog("callDidConnect:")
self.call = call
status = .inCall
routeAudioToSpeaker()
}
func callDidDisconnect(_ call: TVOCall) {
NSLog("callDidDisconnect:")
playDisconnectSound()
self.call = nil
status = .callEnded
hideActiveCallUI()
}
func call(_ call: TVOCall, didFailWithError error: Error) {
NSLog("call:didFailWithError: \(error)");
self.call = nil
status = .callEnded
hideActiveCallUI()
}
}

Twilio developer evangelist here.
Your Swift code says that your phone numbers are hard coded on the server right now. The issue, as Robert has said too, is that you are using the REST API to generate a call when you get a callback from Twilio to your /outbound endpoint.
What is actually happening is that when you generate the call in Swift on the device that starts the call for the application. Twilio then makes an HTTP request to your /outbound endpoint to see what to do with that call. So, instead of generating a new call with the REST API, you need to respond with TwiML to tell Twilio what to do with the call next.
In this case, it sounds like you are trying to dial straight onto another number. For that, you should try the following response:
#app.route('/outgoing', methods=['GET', 'POST'])
def outgoing():
from_number = [HARD_CODED_PHONE_NUMBER_FOR_CALLER]
to_number = [HARD_CODED_PHONE_NUMBER_FOR_RECIPIENT]
resp = twilio.twiml.Response()
with resp.dial(callerId=from_number) as r:
r.number(to_number)
return str(resp)
Let me know if that helps.

Note: I have also responded in the ticket you created with Twilio Support.
Please check your account debugger for the whole pile of error notifications you are getting. Here's an example:
An attempt to retrieve content from
https://voiceapp-twilio.herokuapp.com/outgoing
https://voiceapp-twilio.herokuapp.com/outgoing returned the HTTP
status code 500.
The Python web server returned an error message, which includes this:
TypeError: create() got an unexpected keyword argument
'status_events' // Werkzeug Debugger
It looks like a code error in your Python outgoing() function. Most notably, you are attempting to use the REST API to create a new call, when you should actually be returning TwiML. You should be returning TwiML that includes the Dial verb to create the outgoing call leg.