I am totally new to python flask and encountered a problem when writing some code using the requests and flask modules.
I am working on a project using the web API offered by the Panther platform. The project provided an example using Apache Java.
The source code is as below (see more for details).
public class TestProject {
public static void main(String args[]) throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpPost httppost = new HttpPost("http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?");
StringBody organism = new StringBody("Homo sapiens", ContentType.TEXT_PLAIN);
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
StringBody enrichmentType = new StringBody("process", ContentType.TEXT_PLAIN);
StringBody testType = new StringBody("FISHER", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("FDR", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("BONFERRONI", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("NONE", ContentType.TEXT_PLAIN);
StringBody type = new StringBody("enrichment", ContentType.TEXT_PLAIN);
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("organism", organism)
.addPart("geneList", fileData)
.addPart("enrichmentType", enrichmentType)
.addPart("test_type", testType)
.addPart("type", type)
//.addPart("correction", cor)
.build();
httppost.setEntity(reqEntity);
CloseableHttpResponse response = httpclient.execute(httppost);
try {
//System.out.println("----------------------------------------");
//System.out.println(response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
System.out.println(IOUtils.toString(resEntity.getContent(), StandardCharsets.UTF_8));
}
EntityUtils.consume(resEntity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
The part I am mostly interested in is .addPart("organism", organism) and all the other code with similar structures. They will help pass the parameters from a third-party website to the web API offered by Panther.
I remade the JAVA code into python3 using requests. The code is as follows:
uploadTemp = {'file':open('./app/static/data_temp/temp.txt','rb')}
url="http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
params = {"organism":organism,"geneList":pantherName,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
# or params = {"organism":organism,"geneList":uploadTemp,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
Pantherpost= requests.post(url, params = params)
print(Pantherpost.text)
I am expecting an XML object from the web API including some basic biological information. However, the result I got was null (or \n\n\rnull\n when I print Pantherpost.content)
It seems that the parameters I have got from my own web were not correctly sent to the web API.
In addition to this getting null problem, as a beginner, I am also not quite sure about whether the "geneList" part should be receiving a plain-text object or a file. The manual says it is expecting a file, however, it may have been reformatted into plain-text by this command
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
Anyway, I did try both interpretations: pantherName is a list with name correctly formatted in plain-text and uploadTemp is a .txt file generated for the project. There must be some extra bugs in my code since it returned null in both cases.
Can someone please help out? Thank you very much.
I've found the following issues with your python code:
One. If you want to POST a file using requests, you should use keyword files=.
Two. Keys in files object should match respective parameters of the request (you're using file instead).
Three. You put your parameters in the wrong place of the request by writing params=params.
Function annotation from requests source code:
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:Request.
In example Java code StringBody is used to create parameters, which implies that parameters should be placed inside the body of HTTP request, not query string. So you should use data= keyword instead. If you use params=, output will be null.
SO article on difference between data and params keywords in requests.
So I've spent some time reading thier manual and made a test script:
import requests
url = "http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
filepath = "C:\\data\\YOUR_DATA.txt" # change to your file location
# all required parameters according to manual, except geneList which is a file (see below)
params = { # using defaults from manual
"type": "enrichment",
"organism": "Homo sapiens",
"enrichmentType": "process",
"test_type": "FISHER",
"correction": "FDR",
}
# note that the key here is the name of paramter: geneList
files = {'geneList': open(filepath, 'rb')}
# it outputs null, when 'params=params' is used
r = requests.post(url, data=params, files=files)
print(r.status_code)
print(r.text)
Output:
200
Id Name GeneId raw P-value FDR
Related
The are any solutions to the problem but m not able to find the root cause and also to mention no of the solution is working in my case.
What I am trying to do is to upload a file to Django rest API from a angular Http Client Service.
I am taking up the input form the user passing to the service at the end I have declared a type specific for the type of model I am creating an object for but I am not getting the same error again and again.
I read some where that Django file uploader doesn't understand some char-set format but I still can't understand whats wrong with it.
var report = <CompleteReport> {
title:'heelo',
description:'sdfsdfs',
author: 'report',
article_upload_images: [uuid4(),],
presentation_upload_images: [uuid4(),],
report_article: article_object,
report_image: image_object,
report_podcast: podcast_object,
report_presentation: presentation_object,
report_video: video_object
};
let headers = new HttpHeaders({
'Accept': 'application/json'
});
let options = {headers: headers};
return this.http.post<any>(url, report, options)
I have a python function hosted on aws that takes an event and then does a bunch of things, but essentially the problem lies at parsing the incoming POST request.
My code looks like this:
import json
def lambda_handler(event, context):
# TODO implement
merchantid = json.dumps(event['body'])
return {
'statusCode': 200,
'body': json.dumps(merchantid)
}
To send the post request I am just using the aws API Gateway test that comes with the website, so it shouldn't be any weird encoding.
As I am passing in a JSON object, I expect the merchantid field to be storing a json object.
Instead, it returns this string:
"\"{\\n \\\"merchantEmail\\\": \\\"timcook#gmail.com\\\",\\n \\\"customerEmail\\\": \\\"billgates#gmail.com\\\",\\n \\\"merchantWallet\\\": \\\"mWallet\\\",\\n \\\"transactionTotal\\\": 1.00,\\n \\\"transactionDenomination\\\": \\\"AUD\\\",\\n \\\"customerCurrency\\\": \\\"EUR\\\",\\n \\\"merchantAccess\\\" : {\\n \\\"merchantKey\\\": \\\"key\\\",\\n \\\"merchantSecret\\\": \\\"secret\\\"\\n },\\n \\\"customerAccess\\\" : {\\n \\\"customerKey\\\": \\\"ckey\\\",\\n \\\"customerSecret\\\": \\\"csecret\\\"\\n }\\n}\""
I have never seen a string like this, and I can not seem to get this to return to JSON format.
Does anybody know how I can return this to the orignal JSON format that was submitted in the body?
I should mention that the lambda_handler works perfectly fine with the JSON using the test event, it was only once I started to try and trigger it with the API Gateway that I started having this problem.
edit:
This is the JSON object I am passing as the body of the PUT request:
{
"merchantEmail": "timcook#gmail.com",
"customerEmail": "billgates#gmail.com",
"merchantWallet": "mWallet",
"transactionTotal": 1.00,
"transactionDenomination": "AUD",
"customerCurrency": "EUR",
"merchantAccess" : {
"merchantKey": "key",
"merchantSecret": "secret"
},
"customerAccess" : {
"customerKey": "ckey",
"customerSecret": "csecret"
}
}
edit:
Before I attached the API Gateway I was handling it with
merchantid = event['merchantEmail']
but once I passed it in as the body of a PUT is would return a 502 internal server error
First of all check this what is the datatype of the event params, Is it a string, or Is it dict.
if the datatype of body key value is JSON (dict)
merchantid = event.get('body').get('merchantEmail')
if the datatype of body key value is String (str)
merchantid = json.loads(event.get('body')).get('merchantEmail')
The event argument that you get into the Lambda function is a python dict.
Assuming you know the dict structure - all you need to do is the read the value you are looking for.
Example:
data = event['my_key']
I have been recently exploring the CloudKit and related frameworks. I got the communication with my app working, as well as with my website using CloudKitJS. Where I am struggling is the Server-to-Server communication (which I would need for exporting data from public database in csv.
I have tried Python package requests-cloudkit, which others were suggesting. I have created a Server-to-Server token, and have copied only the key between START and END line once creating the eckey.pem file. I then got this code:
from requests_cloudkit import CloudKitAuth
from restmapper import restmapper
import json
KEY_ID = '[my key ID from CK Dashboard]'
SECRET_FILE_KEY = 'eckey.pem'
AUTH = CloudKitAuth(KEY_ID, SECRET_FILE_KEY)
PARAMS = {
'query':{
'recordType': '[my record type]'
},
}
CloudKit = restmapper.RestMapper("https://api.apple-cloudkit.com/database/1/[my container]/development/")
cloudkit = CloudKit(auth=AUTH)
response = cloudkit.POST.public.records.query(json.dumps(PARAMS))
I am then getting the 401 Authentication failed response. I am stuck on this for days, so I would be grateful for any help or advice. 😊
Creating the server-to-server key is an important first step, but in order to make HTTP requests after that, you have to sign each request.
Look for the Authenticate Web Service Requests section near the bottom of this documentation page.
It's a little bit convoluted, but you have to carefully construct signed headers to include with each request you make. I'm not familiar with how to do it in Python, but here's how I do it in NodeJS which may help:
//Get the timestamp in a very specific format
let date = moment().utc().format('YYYY-MM-DD[T]HH:mm:ss[Z]')
//Construct the subpath
let endpoint = '/records/lookup'
let path = '/database/1/iCloud.*****/development/public'
let subpath = path+endpoint
//Get the key file
let privateKeyFile = fs.readFileSync('../../'+SECRET_FILE_KEY, 'utf8')
//Make a string out of your JSON query
let query = {
recordType: '[my record type]'
}
let requestBody = JSON.stringify(query)
//Hash the query
let bodyHash = crypto.createHash('sha256').update(requestBody, 'utf8').digest('base64')
//Assemble the components you just generated in a special format
//[Current date]:[Request body]:[Web service URL subpath]
let message = date+':'+bodyHash+':'+subpath
//Sign it
let signature = crypto.createSign('RSA-SHA256').update(message).sign(privateKeyFile, 'base64')
//Assemble your headers and include them in your HTTP request
let headers = {
'X-Apple-CloudKit-Request-KeyID': KEY_ID,
'X-Apple-CloudKit-Request-ISO8601Date': date,
'X-Apple-CloudKit-Request-SignatureV1': signature
}
This is a bit hairy at first, but I just put all this stuff in a function that I reuse whenever I need to make a request.
Apple's documentation has pretty much been abandoned and it's hard to find good help with CloudKit Web Services these days.
i have a function written in Python which calls Robinhood(a stock trading broker) API to get quote data("get_quote(self, symbol)" function in following code snipshot). And it works fine. Correct market data was returned.
import requests
import urllib
class Robinhood(object):
# All known endpoints as of September 5th, 2015
endpoints = {
"quotes": "https://api.robinhood.com/quotes/",
"user": "https://api.robinhood.com/user/",
"user/additional_info": "https://api.robinhood.com/user/additional_info/",
"user/basic_info": "https://api.robinhood.com/user/basic_info/",
"user/employment": "https://api.robinhood.com/user/employment/",
"user/investment_profile": "https://api.robinhood.com/user/investment_profile/",
"watchlists": "https://api.robinhood.com/watchlists/"
}
def get_quote(self, symbol):
''' Returns a qoute object for a given symbol including all data returned by Robinhood's API'''
data = { 'symbols' : symbol }
res = self.session.get(self.endpoints['quotes'], params=data)
if res.status_code == 200:
return res.json()['results']
else:
raise Exception("Could not retrieve quote: " + res.text)
I tried to implement this logic in C++ using Curl library. But it doesn't work. There was no compile or run time error but the program returned a single unreadable character instead of the market price of stock. It looks to me like my URL is not correctly set up but I couldn't figure out how to fix it. Does someone have an idea? Thank you!
std::string RobinhoodAPI::GetQuote(std::string ticker)
{
struct response resStr;
init_string(&resStr);
std::string url = "https://api.robinhood.com/quotes/symbols=AVP/";
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &resStr);
resCode = curl_easy_perform(curl);
std::cout << std::string(resStr.ptr);
return std::string(resStr.ptr);
}
I have created an open api specification for the unofficial documentation of the robinhood api. With this you can generate http client for most languages.
Please visit here https://github.com/sabareeshkkanan/robinhood for the specification. And please visit this repo for how to generate client using this specification https://github.com/swagger-api/swagger-codegen
I'm writing an application that utilizes Paypal's permissions API. I'm currently working on the sandbox. I get the verification code correctly but when I try to GetAccessToken, I get the error:
{"responseEnvelope":{"timestamp":"2013-09-03T08:32:16.580-07:00","ack":"Failure","correlationId":"3527b7033f20f","build":"2210301"},"error":[{"errorId":"560022","domain":"PLATFORM","subdomain":"Application","severity":"Error","category":"Application","message":"The X-PAYPAL-APPLICATION-ID header contains an invalid value","parameter":["X-PAYPAL-APPLICATION-ID"]}]}
I'm using the sandbox APP_ID and all the Verification code is also gotten dynamically. Here is my code fragment.
token = "AAAAAAAYaraTSVjvkUBT"
verification = "mgnnWDVfFmgAES0q371Hug"
headers2 = {
"X-PAYPAL-SECURITY-USERID": settings.USERNAME,
"X-PAYPAL-SECURITY-PASSWORD": settings.PASSWORD,
"X-PAYPAL-SECURITY-SIGNATURE": settings.SIGNATURE,
"X-PAYPAL-REQUEST-DATA-FORMAT": "JSON",
"X-PAYPAL-RESPONSE-DATA-FORMAT": "JSON",
"X-PAYPAL-APPLICATION-ID": "APP-80W284485P519543T",
}
url = "https://svcs.paypal.com/Permissions/GetAccessToken/?token=%s&verifier=%s" %(token, verification)
dat2 = {"requestEnvelope": {"errorLanguage":"en_US"}}
req2 = urllib2.Request(url, simplejson.dumps(dat2), headers2)
res2 = urllib2.urlopen(req2).read()
What I'm I doing wrong??
You cannot use the sandbox application id on the live environment. See https://developer.paypal.com/webapps/developer/docs/classic/lifecycle/goingLive/#register to learn how to obtain a live application id.
The endpoint should be https://svcs.sandbox.paypal.com as Siddick said above. The paypal API documentation is so inconsistent, the endpoint i had used previously had been used in a sandbox situation in the documentation.