We Keep Coding

AioHttp: How can I create a data form for an aiohttp post

This is what the data looks like when I'm using requests and it works fine.
data ={
"srt": srt,
"firstname" : firstname,
"lastname" : lastname,
"Email" : email,
"password" : password,
"promotion" : "true",
"action" : {"name":"EMAIL_REG_FORM_SUBMIT"},
"ri" : "NORU",
"ets" : ets
}
I'm trying to convert it so that it works with aiohttp and this is what I have. I think I'm getting an error because of line: "action" : {"name":"EMAIL_REG_FORM_SUBMIT"},
data = aiohttp.FormData()
data.add_field("srt", srt)
data.add_field("firstname", firstname)
data.add_field("lastname", lastname)
data.add_field("Email", email)
data.add_field("password", password)
data.add_field("promotion", 'true')
data.add_field("action", {"name":"EMAIL_REG_FORM_SUBMIT"})
data.add_field("ri", 'NORU')
data.add_field("ets", ets)
If anyone has any ideas on how to make this work pls leave a comment. Essentially I need an async requests with a session, if you know how to do that pls let me know.

I was able to get a submission of a full dictionary in a field by simply converting it to a JSON string:
data.add_field("action", json.dumps({"name":"EMAIL_REG_FORM_SUBMIT"}))
Depending on the data in the dictionary, you might need to add a serialization class to json.dumps to handle "special" data types that json.dumps default serialization class can't handle or for which you need to serialize to JSON following some special format (for example converting a DateTime with time zone to some special text format the server is expecting)
You can also add files with additional calls to add_field, specifying the name of the file you have in the form field(s) as the name field of your data.add_field() call.
Under the hood, FormData tries to transform all your fields and files in a properly formatted multipart/form-data payload.
(I found out most of this after fighting a whole day with both the aiohttp client docs and the server I was submitting data to that was doing "unsmart" things)

JSON Post body not parsing as expected

I have a python function hosted on aws that takes an event and then does a bunch of things, but essentially the problem lies at parsing the incoming POST request.
My code looks like this:
import json
def lambda_handler(event, context):
# TODO implement
merchantid = json.dumps(event['body'])
return {
'statusCode': 200,
'body': json.dumps(merchantid)
}
To send the post request I am just using the aws API Gateway test that comes with the website, so it shouldn't be any weird encoding.
As I am passing in a JSON object, I expect the merchantid field to be storing a json object.
Instead, it returns this string:
"\"{\\n \\\"merchantEmail\\\": \\\"timcook#gmail.com\\\",\\n \\\"customerEmail\\\": \\\"billgates#gmail.com\\\",\\n \\\"merchantWallet\\\": \\\"mWallet\\\",\\n \\\"transactionTotal\\\": 1.00,\\n \\\"transactionDenomination\\\": \\\"AUD\\\",\\n \\\"customerCurrency\\\": \\\"EUR\\\",\\n \\\"merchantAccess\\\" : {\\n \\\"merchantKey\\\": \\\"key\\\",\\n \\\"merchantSecret\\\": \\\"secret\\\"\\n },\\n \\\"customerAccess\\\" : {\\n \\\"customerKey\\\": \\\"ckey\\\",\\n \\\"customerSecret\\\": \\\"csecret\\\"\\n }\\n}\""
I have never seen a string like this, and I can not seem to get this to return to JSON format.
Does anybody know how I can return this to the orignal JSON format that was submitted in the body?
I should mention that the lambda_handler works perfectly fine with the JSON using the test event, it was only once I started to try and trigger it with the API Gateway that I started having this problem.
edit:
This is the JSON object I am passing as the body of the PUT request:
{
"merchantEmail": "timcook#gmail.com",
"customerEmail": "billgates#gmail.com",
"merchantWallet": "mWallet",
"transactionTotal": 1.00,
"transactionDenomination": "AUD",
"customerCurrency": "EUR",
"merchantAccess" : {
"merchantKey": "key",
"merchantSecret": "secret"
},
"customerAccess" : {
"customerKey": "ckey",
"customerSecret": "csecret"
}
}
edit:
Before I attached the API Gateway I was handling it with
merchantid = event['merchantEmail']
but once I passed it in as the body of a PUT is would return a 502 internal server error

First of all check this what is the datatype of the event params, Is it a string, or Is it dict.
if the datatype of body key value is JSON (dict)
merchantid = event.get('body').get('merchantEmail')
if the datatype of body key value is String (str)
merchantid = json.loads(event.get('body')).get('merchantEmail')

The event argument that you get into the Lambda function is a python dict.
Assuming you know the dict structure - all you need to do is the read the value you are looking for.
Example:
data = event['my_key']

POST request remake from Java to python gets null

I am totally new to python flask and encountered a problem when writing some code using the requests and flask modules.
I am working on a project using the web API offered by the Panther platform. The project provided an example using Apache Java.
The source code is as below (see more for details).
public class TestProject {
public static void main(String args[]) throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpPost httppost = new HttpPost("http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?");
StringBody organism = new StringBody("Homo sapiens", ContentType.TEXT_PLAIN);
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
StringBody enrichmentType = new StringBody("process", ContentType.TEXT_PLAIN);
StringBody testType = new StringBody("FISHER", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("FDR", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("BONFERRONI", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("NONE", ContentType.TEXT_PLAIN);
StringBody type = new StringBody("enrichment", ContentType.TEXT_PLAIN);
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("organism", organism)
.addPart("geneList", fileData)
.addPart("enrichmentType", enrichmentType)
.addPart("test_type", testType)
.addPart("type", type)
//.addPart("correction", cor)
.build();
httppost.setEntity(reqEntity);
CloseableHttpResponse response = httpclient.execute(httppost);
try {
//System.out.println("----------------------------------------");
//System.out.println(response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
System.out.println(IOUtils.toString(resEntity.getContent(), StandardCharsets.UTF_8));
}
EntityUtils.consume(resEntity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
The part I am mostly interested in is .addPart("organism", organism) and all the other code with similar structures. They will help pass the parameters from a third-party website to the web API offered by Panther.
I remade the JAVA code into python3 using requests. The code is as follows:
uploadTemp = {'file':open('./app/static/data_temp/temp.txt','rb')}
url="http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
params = {"organism":organism,"geneList":pantherName,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
# or params = {"organism":organism,"geneList":uploadTemp,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
Pantherpost= requests.post(url, params = params)
print(Pantherpost.text)
I am expecting an XML object from the web API including some basic biological information. However, the result I got was null (or \n\n\rnull\n when I print Pantherpost.content)
It seems that the parameters I have got from my own web were not correctly sent to the web API.
In addition to this getting null problem, as a beginner, I am also not quite sure about whether the "geneList" part should be receiving a plain-text object or a file. The manual says it is expecting a file, however, it may have been reformatted into plain-text by this command
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
Anyway, I did try both interpretations: pantherName is a list with name correctly formatted in plain-text and uploadTemp is a .txt file generated for the project. There must be some extra bugs in my code since it returned null in both cases.
Can someone please help out? Thank you very much.

I've found the following issues with your python code:
One. If you want to POST a file using requests, you should use keyword files=.
Two. Keys in files object should match respective parameters of the request (you're using file instead).
Three. You put your parameters in the wrong place of the request by writing params=params.
Function annotation from requests source code:
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:Request.
In example Java code StringBody is used to create parameters, which implies that parameters should be placed inside the body of HTTP request, not query string. So you should use data= keyword instead. If you use params=, output will be null.
SO article on difference between data and params keywords in requests.
So I've spent some time reading thier manual and made a test script:
import requests
url = "http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
filepath = "C:\\data\\YOUR_DATA.txt" # change to your file location
# all required parameters according to manual, except geneList which is a file (see below)
params = { # using defaults from manual
"type": "enrichment",
"organism": "Homo sapiens",
"enrichmentType": "process",
"test_type": "FISHER",
"correction": "FDR",
}
# note that the key here is the name of paramter: geneList
files = {'geneList': open(filepath, 'rb')}
# it outputs null, when 'params=params' is used
r = requests.post(url, data=params, files=files)
print(r.status_code)
print(r.text)
Output:
200
Id Name GeneId raw P-value FDR

API call fails because of additional escape character in json

I am trying to make an API call to service running locally. The service expects post to have json data like this:
{"tool" : "name", "version" : "1", "payload" : "{"branch" : "main"}"
All these are member of a class where variables are all string (tool, version and payload) and I initialize them with respective strings: "name", "version" and "{\"branch\" : \"main\"}"
Now when I dump the class variable using json.dumps(self.__dict__), I get following json, which I send to my api call:
json_upload = {"tool" : "name", "version" : "1", "payload" : "{\\"branch\\" : \\"main\\"}"
I added a line json_upload.replace("\\\\", "\\") but every time API call fails with 415 and I see the data sent to the api endpoint has double escape character. This is really frustrating, as I don't understand how to strip that extra "\\". How do I fix this? I am using standard library json for encoding.

Seems like the data got escaped twice.
Now when I dump the class variable using json.dumps(self.dict), I get following json, which I send to my api call:
Which client library are you using? e.g. with requests you just pass the object and it will be converted to JSON:
r = requests.post('http://example.com/post', json={"key": "value"})
And if you give it a string, it will escape the string.

Accessing ISI Web of Science through SOAP

I'm trying to write a python script that retrieves information about publications from ISI Web of Science. I found domoritz's python script wos.py on GitHub. It uses Suds to connect to the ISI Web of Science web service. I've imported it into my python script and I tried this code, following the very brief instructions in the comments:
from wos import *
soap = WokmwsSoapClient()
results = soap.search('Hallam')
I then get an error:
suds.WebFault: Server raised fault: 'line 1:1: unexpected token: Hallam'
I looked through the code in wos.py. Here is the search function:
def search(self, query):
qparams = {
'databaseID' : 'WOS',
'userQuery' : query,
'queryLanguage' : 'en',
'editions' : [{
'collection' : 'WOS',
'edition' : 'SCI',
},{
'collection' : 'WOS',
'edition' : 'SSCI',
}]
}
rparams = {
'count' : 5, # 1-100
'firstRecord' : 1,
'fields' : [{
'name' : 'Relevance',
'sort' : 'D',
}],
}
return self.client['search'].service.search(qparams, rparams)
I thought maybe query can't be just a plain python string, as I saw in the WSDL page that userQuery is actually of type xs:string. But this page says that userQuery "Must be a valid WOKQL query statement. This requirement is enforced internally", which makes it seem like I don't have to pass in a special type. Anyway, I tried appending 'xs:string' to the beginning of query but I got the same error.
Does anybody know the proper way to use this method?

You could try to use the Wos Python Client that can be install with:
pip install wos
And then you can use it like this:
from wos import WosClient
import wos.utils
with WosClient('JohnDoe', '12345') as client:
print(wos.utils.query(client, 'AU=Knuth Donald'))
You will also have a CLI tool to be used like:
wos -u 'JohnDoe' -p '12345' query 'AU=Knuth Donald'
*DISCLAIMER: I don't work for Web of Science but I am the author of the client. You need to have web services access (which is a paid service in addition to the normal WOS access) since Web of Science does not allow web services requests coming from normal users. You should ask your university to provide you with the username and password that WOS gave them. This is not just for my client but for anything that uses WOS web service. *

So apparently passing in a python string was fine, but I needed a string that was more like a search query. I found this example on the website I mentioned before:
<soap:Body>
<woksearch:search xmlns:woksearch="http://woksearch.v3.wokmws.thomsonreuters.com">
<!-- this request has the minimum required elements,
but contains all valid retrieve options
for this operation and databaseId -->
<queryParameters>
<databaseId>WOK</databaseId>
<userQuery>AU=Arce, G*</userQuery>
<queryLanguage>en</queryLanguage>
</queryParameters>
....
So I tried using results = soap.search('AU=Hallam') and that worked. I can now do things like print results.recordsFound and I get correct answers.