We Keep Coding

Unable to create URI with whitespace in MarkLogic

I have created a Marklogic transform which tries to convert some URL encoded characters: [ ] and whitespace when ingesting data into database. This is the xquery code:
xquery version "1.0-ml";
module namespace space = "http://marklogic.com/rest-api/transform/space-to-space";
declare function space:transform(
$context as map:map,
$params as map:map,
$content as document-node()
) as document-node()
{
let $puts := (
xdmp:log($params),
xdmp:log($context),
map:put($context, "uri", fn:replace(map:get($context, "uri"), "%5B+", "[")),
map:put($context, "uri", fn:replace(map:get($context, "uri"), "%5D+", "]")),
map:put($context, "uri", fn:replace(map:get($context, "uri"), "%20+", " ")),
xdmp:log($context)
)
return $content
};
When I tried this with my python code below
def upload_document(self, inputContent, uri, fileType, database, collection):
if fileType == 'XML':
headers = {'Content-type': 'application/xml'}
fileBytes = str.encode(inputContent)
elif fileType == 'TXT':
headers = {'Content-type': 'text/*'}
fileBytes = str.encode(inputContent)
else:
headers = {'Content-type': 'application/octet-stream'}
fileBytes = inputContent
endpoint = ML_DOCUMENTS_ENDPOINT
params = {}
if uri is not None:
encodedUri = urllib.parse.quote(uri)
endpoint = endpoint + "?uri=" + encodedUri
if database is not None:
params['database'] = database
if collection is not None:
params['collection'] = collection
params['transform'] = 'space-to-space'
req = PreparedRequest()
req.prepare_url(endpoint, params)
response = requests.put(req.url, data=fileBytes, headers=headers, auth=HTTPDigestAuth(ML_USER_NAME, ML_PASSWORD))
print('upload_document result: ' + str(response.status_code))
if response.status_code == 400:
print(response.text)
The following lines are from the xquery logging:
2023-02-13 16:59:00.067 Info: {}
2023-02-13 16:59:00.067 Info:
{"input-type":"application/octet-stream",
"uri":"/Judgment/26856/supportingfiles/[TEST] 57_image1.PNG", "output-type":"application/octet-stream"}
2023-02-13 16:59:00.067 Info:
{"input-type":"application/octet-stream",
"uri":"/Judgment/26856/supportingfiles/[TEST] 57_image1.PNG", "output type":"application/octet-stream"}
2023-02-13 16:59:00.653 Info: Status 500: REST-INVALIDPARAM: (err:FOER0000)
Invalid parameter: invalid uri:
/Judgment/26856/supportingfiles/[TEST] 57_image1.PNG

The MarkLogic REST API is very opinionated about what a valid URI is, and it doesn't allow you to insert documents that have spaces in the URI. If you have an existing URI with a space in it, the REST API will retrieve or update it for you. However, it won't allow you to create a new document with such a URI.
If you need to create documents with spaces in the URI, then you will need to use lower-level APIs. xdmp:document-insert() will let you.

Issue in uploading image for prediction using a MultiPart POST Request

The call is currently happening via a Flutter application which makes a multi-part POST request.
Flutter Code
var request = http.MultipartRequest(
'POST',
Uri.parse('https://techfarmtest.herokuapp.com/upload'),
);
Map<String, String> headers = {"Content-type": "multipart/form-data"};
request.files.add(
http.MultipartFile(
widget.selectedImage.toString(),
widget.selectedImage.readAsBytes().asStream(),
widget.selectedImage.lengthSync(),
filename: widget.selectedImage.path.split('/').last,
),
);
request.headers.addAll(headers);
var res = await request.send();
http.Response response = await http.Response.fromStream(res);
var data = jsonDecode(response.body);
return data;
I intend to upload the image to the backend and then perform the prediction and retrieve the result in JSON format and the backend is scripted using Flask.
Flask Code
#app.route('/upload',methods=["POST"])
def upload_image():
if request.method == "POST":
imageFile = request.files['image']
fileName = werkzeug.utils.secure_filename(imageFile.filename)
print('\nRecieved File name : ' + imageFile.filename)
imageFile.save('./uploadedImages/' + fileName)
pred('./uploadedImages/fileName')
def pred(sampleFile):
model = load_model('./model.h5')
# model.summary()
sample_file = sampleFile
sample_img = image.load_img(sample_file,target_size = (256,256,3))
sample_img = image.img_to_array(sample_img)
sample_img = np.expand_dims(sample_img,axis=0)
prediction_arr = model.predict(sample_img)
result = {
'Sample' : str(sampleFile),
'Label' : str(class_names[prediction_arr.argmax()]),
'Confidence' : str(prediction_arr.max())
}
return jsonify(result)
The current issue I am facing is that I am making a bad request (400).This is a rough code (pseudocode) that I have figured out from various resources. Is there any way to go about it.

So, I figured it out by myself.
I will be attaching the code below for future references.
Flutter Code :
var request = http.MultipartRequest(
'POST',
Uri.parse('https://techfarmtest.herokuapp.com/upload'),
);
request.files.add(
await http.MultipartFile.fromPath('image', img.path),
);
var res = await request.send();
You can verify the POST request by using the below logs :
log('${res.statusCode}', name: 'POST-request-statusCode');
log('${res.reasonPhrase}', name: 'POST-request-status');
With respect to Flask :
#app.route('/upload',methods=["POST","PUT"])
def upload_image():
if request.method == "POST":
imageFile = request.files['image']
***you can perform any operation on the file you have recieved from the request now***
Thank you!

Python Dynamics 365 package to read write data

We managed to connect to our dynamics 365 cloud instance with token auth. However, we were hoping to find package like "pydynamics".
The package "https://libraries.io/pypi/dynamics365crm-python" seems newer, but it seems only to be able to handle standard objects, not custom objects.
Our current solution works with REST only.
import requests
import json
#set these values to retrieve the oauth token
crmorg = 'https://org.crm4.dynamics.com' #base url for crm org
clientid = '<id>' #application client id
client_secret = '<secret>'
username = 'dynamics-api#org.com' #username
userpassword = 'pw' #password
tokenendpoint = 'https://login.microsoftonline.com/bb23defa-be1d-4137-969b-324f8468f15a/oauth2/token' #oauth token endpoint
authorizationendpoint = 'https://login.microsoftonline.com/bb23defa-be1d-4137-969b-324f8468f15a/oauth2/authorize'
#build the authorization token request
tokenpost = {
'client_id':clientid,
'resource':crmorg,
'client_secret':client_secret,
'username':username,
'password':userpassword,
'grant_type':'password',
'oauthUrl':authorizationendpoint
}
#make the token request
tokenres = requests.post(tokenendpoint, data=tokenpost)
#set accesstoken variable to empty string
accesstoken = ''
#extract the access token
try:
accesstoken = tokenres.json()['access_token']
except(KeyError):
#handle any missing key errors
print('Could not get access token')
#set these values to query your crm data
crmwebapi = 'https://<org>.crm4.dynamics.com/api/data/v9.1' #full path to web api endpoint
crmwebapiquery = '/new_households' #web api query (include leading /)
#prepare the crm request headers
crmrequestheaders = {
'Authorization': 'Bearer ' + accesstoken,
'OData-MaxVersion': '4.0',
'OData-Version': '4.0',
'Accept': 'application/json',
'Content-Type': 'application/json; charset=utf-8',
'Prefer': 'odata.maxpagesize=500',
'Prefer': 'odata.include-annotations=OData.Community.Display.V1.FormattedValue'
}
#make the crm request
crmres = requests.get(crmwebapi+crmwebapiquery, headers=crmrequestheaders)
try:
#get the response json
crmresults = crmres.json()['value'][0]
#loop through it
for key,value in crmresults.items():
print (key, value)
except KeyError:
#handle any missing key errors
print('Could not parse CRM results')
Anyone know a package?

REST API is the way to go.
We use them every time with JavaScript and parase the response as Json.
Below is sample call which I perform within crm, you will see the response been transformed into Json
Don't worry much about details, for you important is this line var results = JSON.parse(this.response);
var req = new XMLHttpRequest();
req.open("GET", Xrm.Page.context.getClientUrl() + "/api/data/v9.1/accounts?$select=accountcategorycode,accountclassificationcode,accountid,accountnumber,accountratingcode", false);
req.setRequestHeader("OData-MaxVersion", "4.0");
req.setRequestHeader("OData-Version", "4.0");
req.setRequestHeader("Accept", "application/json");
req.setRequestHeader("Content-Type", "application/json; charset=utf-8");
req.setRequestHeader("Prefer", "odata.include-annotations=\"*\",odata.maxpagesize=10");
req.onreadystatechange = function() {
if (this.readyState === 4) {
req.onreadystatechange = null;
if (this.status === 200) {
var results = JSON.parse(this.response);
for (var i = 0; i < results.value.length; i++) {
var accountcategorycode = results.value[i]["accountcategorycode"];
var accountcategorycode_formatted = results.value[i]["accountcategorycode#OData.Community.Display.V1.FormattedValue"];
var accountclassificationcode = results.value[i]["accountclassificationcode"];
var accountclassificationcode_formatted = results.value[i]["accountclassificationcode#OData.Community.Display.V1.FormattedValue"];
var accountid = results.value[i]["accountid"];
var accountnumber = results.value[i]["accountnumber"];
var accountratingcode = results.value[i]["accountratingcode"];
var accountratingcode_formatted = results.value[i]["accountratingcode#OData.Community.Display.V1.FormattedValue"];
}
} else {
Xrm.Utility.alertDialog(this.statusText);
}
}
};
req.send();

Upload PDF from Python as attachment to Salesforce Object

I am trying to upload a pdf generated in Python as an attachment to a salesforce object using the simple_salesforce Python package. I have tried several different ways to accomplish this, but have had no luck so far. Here is the code
import base64
import json
from simple_salesforce import Salesforce
instance = ''
sessionId = sf.session_id
def pdf_encode(pdf_filename):
body = open(pdf_filename, 'rb') #open binary file in read mode
body = body.read()
body = base64.encodebytes(body)
body = pdf_encode('PDF_Report.pdf')
response = requests.post('https://%s.salesforce.com/services/data/v29.0/sobjects/Attachment/' % instance,
headers = { 'Content-Type': 'application/json', 'Authorization': 'Bearer %s' % sessionId },
data = json.dumps({
'ParentId': parent_id,
'Name': 'test.txt',
'body': body
})
)
I get this error.
TypeError: Object of type bytes is not JSON serializable
I have also tried to use
body = base64.encodebytes(body).decode('ascii')
in my code, but I can't get that to work either. I get the error
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Any suggestions on how to upload a PDF in Python 3 into Salesforce as an attachment using simple_salesforce?

I was working on this and found a few resources to upload files. I created one for myself using that.
Below is the code that you can use for Python and have the file uploaded on Salesforce.
import requests
import base64
import json
params = {
"grant_type": "password",
"client_id": "Your_Client_Id",
"client_secret": "Your_Client_Secret",
"username": "YOUR_EMAIL#procureanalytics.com.pcsandbox", # The email you use to login
"password": "YOUR_PASSWORD+YOUR_SECURITY_TOKEN" # Concat your password and your security token
}
r = requests.post("https://test.salesforce.com/services/oauth2/token", params=params)
# if you connect to a Sandbox, use test.salesforce.com instead
access_token = r.json().get("access_token")
instance_url = r.json().get("instance_url")
print("Access Token:", access_token)
print("Instance URL", instance_url)
#######################################################################################
# Helper function
#######################################################################################
def sf_api_call(action, parameters = {}, method = 'get', data = {}):
"""
Helper function to make calls to Salesforce REST API.
Parameters: action (the URL), URL params, method (get, post or patch), data for POST/PATCH.
"""
headers = {
'Content-type': 'application/json',
'Accept-Encoding': 'gzip',
'Authorization': 'Bearer %s' % access_token
}
if method == 'get':
r = requests.request(method, instance_url+action, headers=headers, params=parameters, timeout=30)
elif method in ['post', 'patch']:
r = requests.request(method, instance_url+action, headers=headers, json=data, params=parameters, timeout=10)
else:
# other methods not implemented in this example
raise ValueError('Method should be get or post or patch.')
print('Debug: API %s call: %s' % (method, r.url) )
if r.status_code < 300:
if method=='patch':
return None
else:
return r.json()
else:
raise Exception('API error when calling %s : %s' % (r.url, r.content))
# Test connection
print(json.dumps(sf_api_call('/services/data/v40.0/query/', {
'q': 'SELECT Account.Name, Name, CloseDate from Opportunity where IsClosed = False order by CloseDate ASC LIMIT 1'
}), indent=2))
#######################################################################################
# File Upload from directory
#######################################################################################
# 1) Create a ContentVersion
path = "Folder_name\Sample_pdf.pdf"
with open(path, "rb") as f:
encoded_string = base64.b64encode(f.read()).decode("utf-8")
ContentVersion = sf_api_call('/services/data/v40.0/sobjects/ContentVersion', method="post", data={
'Title': 'Sample_pdf file',
'PathOnClient': path,
'VersionData': encoded_string,
})
ContentVersion_id = ContentVersion.get('id')
# 2) Get the ContentDocument id
ContentVersion = sf_api_call('/services/data/v40.0/sobjects/ContentVersion/%s' % ContentVersion_id)
ContentDocument_id = ContentVersion.get('ContentDocumentId')
# 3) Create a ContentDocumentLink
Id = "Abcd123" # This Id can be anything: Account_Id or Lead_Id or Opportunity_Id
ContentDocumentLink = sf_api_call('/services/data/v40.0/sobjects/ContentDocumentLink', method = 'post', data={
'ContentDocumentId': ContentDocument_id,
'LinkedEntityId': Id,
'ShareType': 'V'
})
How to use
Step 1:
Key in your email address and password here. Please note that the password here is a string of 'your password' and your 'security token'.
# Import libraries
import requests
import base64
import json
params = {
"grant_type": "password",
"client_id": "Your_Client_Id",
"client_secret": "Your_Client_Secret",
"username": "YOUR_EMAIL#procureanalytics.com.pcsandbox", # The email you use to login
"password": "YOUR_PASSWORD+YOUR_SECURITY_TOKEN" # Concat your password and your security token
}
r = requests.post("https://test.salesforce.com/services/oauth2/token", params=params)
# if you connect to a Sandbox, use test.salesforce.com instead
access_token = r.json().get("access_token")
instance_url = r.json().get("instance_url")
print("Access Token:", access_token)
print("Instance URL", instance_url)
You can get your security token on Salesforce through Account >> Settings >> Reset My Security Token.
You will receive an email from salesforce with your security token.
Step 2:
Choose appropriate link for request.post
For Sandbox environment:
r = requests.post("https://test.salesforce.com/services/oauth2/token", params=params)
For Production enviroment:
r = requests.post("https://login.salesforce.com/services/oauth2/token", params=params)
After your initial connection is ready, the output on the second cell should look something like this:
Access Token: !2864b793dbce2ad32c1ba7d71009ec84.b793dbce2ad32c1ba7d71009ec84
Instance URL https://your_company_name--pcsandbox.my.salesforce.com
Step 3:
Under the 'File upload from a directory' cell (Cell #5), specify your file path.
In my case, this is
# 1) Create a ContentVersion
path = "Folder_name\Sample_pdf.pdf"
with open(path, "rb") as f:
encoded_string = base64.b64encode(f.read()).decode("utf-8")
Step 4:
Under the same cell, mention the Id in which you would like to upload your file.
The sample code below is uploading a file on Accounts object for an account with Id: Abcd123
# 3) Create a ContentDocumentLink
Id = "Abcd123" # This Id can be anything: Account_Id or Lead_Id or Opportunity_Id
ContentDocumentLink = sf_api_call('/services/data/v40.0/sobjects/ContentDocumentLink', method = 'post', data={
'ContentDocumentId': ContentDocument_id,
'LinkedEntityId': Id,
'ShareType': 'V'
})

Yahoo BOSS V2 authorization troubles

I'm having an awfully hard time with Yahoo's authentication/authorization. I've enabled BOSS in my account, set up a payment method, and now I'm trying to run a search using some python code:
import urllib2
import oauth2 as oauth
import time
OAUTH_CONSUMER_KEY = "blahblahblah"
OAUTH_CONSUMER_SECRET = "blah"
def oauth_request(url, params, method="GET"):
params['oauth_version'] = "1.0",
params['oauth_nonce'] = oauth.generate_nonce(),
params['oauth_timestamp'] = int(time.time())
consumer = oauth.Consumer(key=OAUTH_CONSUMER_KEY,
secret=OAUTH_CONSUMER_SECRET)
params['oauth_consumer_key'] = consumer.key
req = oauth.Request(method=method, url=url, parameters=params)
req.sign_request(oauth.SignatureMethod_HMAC_SHA1(), consumer, None)
return req
if __name__ == "__main__":
url = "http://yboss.yahooapis.com/ysearch/web"
req = oauth_request(url, params={"q": "cats dogs"})
req_url = req.to_url()
print req_url
result = urllib2.urlopen(req_url)
I keep getting a urllib2.HTTPError: HTTP Error 401: Unauthorized exception. I can't figure out whether there's something wrong with my key, or the method of signing, or if I'm somehow tampering with my data after signing, or what the deal is. Anyone have suggestions?

I made some small changes to make your example work. See code for comments.
import urllib2
import oauth2 as oauth
import time
OAUTH_CONSUMER_KEY = "blahblahblah"
OAUTH_CONSUMER_SECRET = "blah"
def oauth_request(url, params, method="GET"):
# Removed trailing commas here - they make a difference.
params['oauth_version'] = "1.0" #,
params['oauth_nonce'] = oauth.generate_nonce() #,
params['oauth_timestamp'] = int(time.time())
consumer = oauth.Consumer(key=OAUTH_CONSUMER_KEY,
secret=OAUTH_CONSUMER_SECRET)
params['oauth_consumer_key'] = consumer.key
req = oauth.Request(method=method, url=url, parameters=params)
req.sign_request(oauth.SignatureMethod_HMAC_SHA1(), consumer, None)
return req
if __name__ == "__main__":
url = "http://yboss.yahooapis.com/ysearch/web"
req = oauth_request(url, params={"q": "cats dogs"})
# This one is a bit nasty. Apparently the BOSS API does not like
# "+" in its URLs so you have to replace "%20" manually.
# Not sure if the API should be expected to accept either.
# Not sure why to_url does not just return %20 instead...
# Also, oauth2.Request seems to store parameters as unicode and forget
# to encode to utf8 prior to percentage encoding them in its to_url
# method. However, it's handled correctly for generating signatures.
# to_url fails when query parameters contain non-ASCII characters. To
# work around, manually utf8 encode the request parameters.
req['q'] = req['q'].encode('utf8')
req_url = req.to_url().replace('+', '%20')
print req_url
result = urllib2.urlopen(req_url)

Here is a Python code snippet that works for me against Yahoo! BOSS:
import httplib2
import oauth2
import time
OAUTH_CONSUMER_KEY = "Blah"
OAUTH_CONSUMER_SECRET = "Blah"
if __name__ == "__main__":
url = "http://yboss.yahooapis.com/ysearch/web?q=cats%20dogs"
consumer = oauth2.Consumer(key=OAUTH_CONSUMER_KEY,secret=OAUTH_CONSUMER_SECRET)
params = {
'oauth_version': '1.0',
'oauth_nonce': oauth2.generate_nonce(),
'oauth_timestamp': int(time.time()),
}
oauth_request = oauth2.Request(method='GET', url=url, parameters=params)
oauth_request.sign_request(oauth2.SignatureMethod_HMAC_SHA1(), consumer, None)
oauth_header=oauth_request.to_header(realm='yahooapis.com')
# Get search results
http = httplib2.Http()
resp, content = http.request(url, 'GET', headers=oauth_header)
print resp
print content
Im using an Authenticate Header to submit the OAuth signature.

So I decided to ditch Python and try Perl, and it Just Worked. Here's a minimal code sample:
use strict;
use Net::OAuth;
use LWP::UserAgent;
my $CC_KEY = "blahblahblah";
my $CC_SECRET = "blah";
my $url = 'http://yboss.yahooapis.com/ysearch/web';
print make_request($url, {q => "cat dog", format => "xml", count => 5});
sub make_request {
my ($url, $args) = #_;
my $request = Net::OAuth->request("request token")
->new(
consumer_key => $CC_KEY,
consumer_secret => $CC_SECRET,
request_url => $url,
request_method => 'GET',
signature_method => 'HMAC-SHA1',
timestamp => time,
nonce => int(rand 10**6),
callback => 'oob',
extra_params => $args,
protocol_version => Net::OAuth::PROTOCOL_VERSION_1_0A,
);
$request->sign;
my $res = LWP::UserAgent->new(env_proxy=>1)->get($request->to_url);
return $res->content if $res->is_success;
die $res->status_line;
}

Here's another solution, this time back in python-land. This was put together by Tom De Smedt, author of the Pattern web-mining kit.
I'll communicate with the author of python-oauth2 to see if it can be fixed.
OAUTH_CONSUMER_KEY = "blahblahblah"
OAUTH_CONSUMER_SECRET = "blah"
import urllib
import hmac
import time
import random
import base64
try:
from hashlib import sha1
from hashlib import md5
except:
import sha as sha1
import md5; md5=md5.new
def hmac_sha1(key, text):
return hmac.new(key, text, sha1).digest()
def oauth_nonce(length=40):
h = "".join([str(random.randint(0, 9)) for i in range(length)])
h = md5(str(time.time()) + h).hexdigest()
return h
def oauth_timestamp():
return str(int(time.time()))
def oauth_encode(s):
return urllib.quote(s, "~")
def oauth_signature(url, data={}, method="get", secret="", token=""):
# Signature base string: http://tools.ietf.org/html/rfc5849#section-3.4.1
base = oauth_encode(method.upper()) + "&"
base += oauth_encode(url.rstrip("?")) + "&"
base += oauth_encode("&".join(["%s=%s" % (k, v) for k, v in sorted(data.items())]))
# HMAC-SHA1 signature algorithm: http://tools.ietf.org/html/rfc5849#section-3.4.2
signature = hmac_sha1(oauth_encode(secret) + "&" + token, base)
signature = base64.b64encode(signature)
return signature
q = "cat"
url = "http://yboss.yahooapis.com/ysearch/" + "web" # web | images | news
data = {
"q": q,
"start": 0,
"count": 50, # 35 for images
"format": "xml",
"oauth_version": "1.0",
"oauth_nonce" : oauth_nonce(),
"oauth_timestamp" : oauth_timestamp(),
"oauth_consumer_key" : OAUTH_CONSUMER_KEY,
"oauth_signature_method" : "HMAC-SHA1",
}
data["oauth_signature"] = oauth_signature(url, data, secret=OAUTH_CONSUMER_SECRET)
complete_url = url + "?" + urllib.urlencode(data)
response = urllib.urlopen(complete_url)
print response.read()

Here is sample code to access Yahoo! BOSS API v2 using with python-oauth as oauth liberary.
OAUTH_CONSUMER_KEY = "<oauth consumer key>"
OAUTH_CONSUMER_SECRET = "<oauth consumer secret>"
URL = "http://yboss.yahooapis.com/ysearch/web"
import urllib
import oauth.oauth as oauth
data = {
"q": "yahoo boss search",
"start":0,
"count":2,
"format":"json"
}
consumer = oauth.OAuthConsumer(OAUTH_CONSUMER_KEY, OAUTH_CONSUMER_SECRET)
signature_method_plaintext = oauth.OAuthSignatureMethod_PLAINTEXT()
signature_method_hmac_sha1 = oauth.OAuthSignatureMethod_HMAC_SHA1()
oauth_request = oauth.OAuthRequest.from_consumer_and_token(consumer, token=None, http_method='GET', http_url=URL, parameters=data)
oauth_request.sign_request(signature_method_hmac_sha1, consumer, "")
complete_url = oauth_request.to_url()
response = urllib.urlopen(complete_url)
print "REQUEST URL => %s" % complete_url
print ""
print "RESPONSE =>"
print response.read()

I stepped into the urllib2.open code using the debugger, and found that the response has this header:
WWW-Authenticate: OAuth oauth_problem="version_rejected", realm="yahooapis.com"
So I guess I'm having some kind of version mismatch of OAuth.