We Keep Coding

Encode JSON in order to create it as secret with Python k8s Client

I have a big yaml file that I want to store as a secret in my kubernetes cluster. The following command succeeds:
k create secret generic values --from-file=my-values.yaml
But in my code, I want to use the k8s python client.
So I want to do something like this:
def make_k8s_client(kubeconig):
....
def create_secret(name, data, client_api):
secret = client.V1Secret(
api_version="v1",
kind="Secret",
metadata=client.V1ObjectMeta(name=name),
data=data,
)
client_api.create_namespaced_secret(namespace="default",
body=secret)
k8s_api = make_k8s_client("path-to-kubeconfig")
with open("path-to/my-values.yaml") as f:
values = yaml.load(f)
If I pass the yaml like this:
create_secret("mysecret", values, k8s_api)
I get this error:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Secret in version \"v1\" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 0, error found in #10 byte of ...|pe\": \"abc\", \"def|..., bigger context ...|{\"apiVersion\": \"v1\", \"data\": {\"k8sType\": \"abc\", \"secret\": \"mysecret\", \"type\": \"mytype","reason":"BadRequest","code":400}
If I pass the secret like this:
create_secret("mysecret", base64.urlsafe_b64encode(json.dumps(values).encode()).decode(), k8s_api)
I get this error:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Secret in version \"v1\" cannot be handled as a Secret: v1.Secret.Data: ReadMapCB: expect { or n, but found \", error found in #10 byte of ...| \"data\": \"eyJrOHNUeX|..., bigger context ...|{\"apiVersion\": \"v1\", \"data\": \"eyJrOHNUeXBlIjogImF3cyIsICJnYXJkZW5lclNlY3JldCI6IC|...","reason":"BadRequest","code":400}
How do I have to encode the json file in order to be able to pass it to the python k8s client?

Data contains the secret data. Each key must consist of alphanumeric characters, '-', '_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here (1).
V1Secret(
api_version="v1",
kind="Secret",
metadata=client.V1ObjectMeta(name=name),
data={
'my-values.yaml': base64.b64encode(json.dumps(values).encode()).decode("utf-8")
},
How to create and use a Secret

AioHttp: How can I create a data form for an aiohttp post

This is what the data looks like when I'm using requests and it works fine.
data ={
"srt": srt,
"firstname" : firstname,
"lastname" : lastname,
"Email" : email,
"password" : password,
"promotion" : "true",
"action" : {"name":"EMAIL_REG_FORM_SUBMIT"},
"ri" : "NORU",
"ets" : ets
}
I'm trying to convert it so that it works with aiohttp and this is what I have. I think I'm getting an error because of line: "action" : {"name":"EMAIL_REG_FORM_SUBMIT"},
data = aiohttp.FormData()
data.add_field("srt", srt)
data.add_field("firstname", firstname)
data.add_field("lastname", lastname)
data.add_field("Email", email)
data.add_field("password", password)
data.add_field("promotion", 'true')
data.add_field("action", {"name":"EMAIL_REG_FORM_SUBMIT"})
data.add_field("ri", 'NORU')
data.add_field("ets", ets)
If anyone has any ideas on how to make this work pls leave a comment. Essentially I need an async requests with a session, if you know how to do that pls let me know.

I was able to get a submission of a full dictionary in a field by simply converting it to a JSON string:
data.add_field("action", json.dumps({"name":"EMAIL_REG_FORM_SUBMIT"}))
Depending on the data in the dictionary, you might need to add a serialization class to json.dumps to handle "special" data types that json.dumps default serialization class can't handle or for which you need to serialize to JSON following some special format (for example converting a DateTime with time zone to some special text format the server is expecting)
You can also add files with additional calls to add_field, specifying the name of the file you have in the form field(s) as the name field of your data.add_field() call.
Under the hood, FormData tries to transform all your fields and files in a properly formatted multipart/form-data payload.
(I found out most of this after fighting a whole day with both the aiohttp client docs and the server I was submitting data to that was doing "unsmart" things)

JSON Post body not parsing as expected

I have a python function hosted on aws that takes an event and then does a bunch of things, but essentially the problem lies at parsing the incoming POST request.
My code looks like this:
import json
def lambda_handler(event, context):
# TODO implement
merchantid = json.dumps(event['body'])
return {
'statusCode': 200,
'body': json.dumps(merchantid)
}
To send the post request I am just using the aws API Gateway test that comes with the website, so it shouldn't be any weird encoding.
As I am passing in a JSON object, I expect the merchantid field to be storing a json object.
Instead, it returns this string:
"\"{\\n \\\"merchantEmail\\\": \\\"timcook#gmail.com\\\",\\n \\\"customerEmail\\\": \\\"billgates#gmail.com\\\",\\n \\\"merchantWallet\\\": \\\"mWallet\\\",\\n \\\"transactionTotal\\\": 1.00,\\n \\\"transactionDenomination\\\": \\\"AUD\\\",\\n \\\"customerCurrency\\\": \\\"EUR\\\",\\n \\\"merchantAccess\\\" : {\\n \\\"merchantKey\\\": \\\"key\\\",\\n \\\"merchantSecret\\\": \\\"secret\\\"\\n },\\n \\\"customerAccess\\\" : {\\n \\\"customerKey\\\": \\\"ckey\\\",\\n \\\"customerSecret\\\": \\\"csecret\\\"\\n }\\n}\""
I have never seen a string like this, and I can not seem to get this to return to JSON format.
Does anybody know how I can return this to the orignal JSON format that was submitted in the body?
I should mention that the lambda_handler works perfectly fine with the JSON using the test event, it was only once I started to try and trigger it with the API Gateway that I started having this problem.
edit:
This is the JSON object I am passing as the body of the PUT request:
{
"merchantEmail": "timcook#gmail.com",
"customerEmail": "billgates#gmail.com",
"merchantWallet": "mWallet",
"transactionTotal": 1.00,
"transactionDenomination": "AUD",
"customerCurrency": "EUR",
"merchantAccess" : {
"merchantKey": "key",
"merchantSecret": "secret"
},
"customerAccess" : {
"customerKey": "ckey",
"customerSecret": "csecret"
}
}
edit:
Before I attached the API Gateway I was handling it with
merchantid = event['merchantEmail']
but once I passed it in as the body of a PUT is would return a 502 internal server error

First of all check this what is the datatype of the event params, Is it a string, or Is it dict.
if the datatype of body key value is JSON (dict)
merchantid = event.get('body').get('merchantEmail')
if the datatype of body key value is String (str)
merchantid = json.loads(event.get('body')).get('merchantEmail')

The event argument that you get into the Lambda function is a python dict.
Assuming you know the dict structure - all you need to do is the read the value you are looking for.
Example:
data = event['my_key']

POST request remake from Java to python gets null

I am totally new to python flask and encountered a problem when writing some code using the requests and flask modules.
I am working on a project using the web API offered by the Panther platform. The project provided an example using Apache Java.
The source code is as below (see more for details).
public class TestProject {
public static void main(String args[]) throws Exception {
CloseableHttpClient httpclient = HttpClients.createDefault();
try {
HttpPost httppost = new HttpPost("http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?");
StringBody organism = new StringBody("Homo sapiens", ContentType.TEXT_PLAIN);
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
StringBody enrichmentType = new StringBody("process", ContentType.TEXT_PLAIN);
StringBody testType = new StringBody("FISHER", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("FDR", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("BONFERRONI", ContentType.TEXT_PLAIN);
//StringBody cor = new StringBody("NONE", ContentType.TEXT_PLAIN);
StringBody type = new StringBody("enrichment", ContentType.TEXT_PLAIN);
HttpEntity reqEntity = MultipartEntityBuilder.create()
.addPart("organism", organism)
.addPart("geneList", fileData)
.addPart("enrichmentType", enrichmentType)
.addPart("test_type", testType)
.addPart("type", type)
//.addPart("correction", cor)
.build();
httppost.setEntity(reqEntity);
CloseableHttpResponse response = httpclient.execute(httppost);
try {
//System.out.println("----------------------------------------");
//System.out.println(response.getStatusLine());
HttpEntity resEntity = response.getEntity();
if (resEntity != null) {
System.out.println(IOUtils.toString(resEntity.getContent(), StandardCharsets.UTF_8));
}
EntityUtils.consume(resEntity);
} finally {
response.close();
}
} finally {
httpclient.close();
}
}
}
The part I am mostly interested in is .addPart("organism", organism) and all the other code with similar structures. They will help pass the parameters from a third-party website to the web API offered by Panther.
I remade the JAVA code into python3 using requests. The code is as follows:
uploadTemp = {'file':open('./app/static/data_temp/temp.txt','rb')}
url="http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
params = {"organism":organism,"geneList":pantherName,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
# or params = {"organism":organism,"geneList":uploadTemp,"enrichmentType":"fullGO_process","test_type":"BINOMIAL","type":"enrichment","correction":"BONFERRONI"}
Pantherpost= requests.post(url, params = params)
print(Pantherpost.text)
I am expecting an XML object from the web API including some basic biological information. However, the result I got was null (or \n\n\rnull\n when I print Pantherpost.content)
It seems that the parameters I have got from my own web were not correctly sent to the web API.
In addition to this getting null problem, as a beginner, I am also not quite sure about whether the "geneList" part should be receiving a plain-text object or a file. The manual says it is expecting a file, however, it may have been reformatted into plain-text by this command
FileBody fileData = new FileBody(new File("c:\\data_files\\gene_expression_files\\7_data\\humanEnsembl"), ContentType.TEXT_PLAIN);
Anyway, I did try both interpretations: pantherName is a list with name correctly formatted in plain-text and uploadTemp is a .txt file generated for the project. There must be some extra bugs in my code since it returned null in both cases.
Can someone please help out? Thank you very much.

I've found the following issues with your python code:
One. If you want to POST a file using requests, you should use keyword files=.
Two. Keys in files object should match respective parameters of the request (you're using file instead).
Three. You put your parameters in the wrong place of the request by writing params=params.
Function annotation from requests source code:
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:Request.
In example Java code StringBody is used to create parameters, which implies that parameters should be placed inside the body of HTTP request, not query string. So you should use data= keyword instead. If you use params=, output will be null.
SO article on difference between data and params keywords in requests.
So I've spent some time reading thier manual and made a test script:
import requests
url = "http://pantherdb.org/webservices/garuda/tools/enrichment/VER_2/enrichment.jsp?"
filepath = "C:\\data\\YOUR_DATA.txt" # change to your file location
# all required parameters according to manual, except geneList which is a file (see below)
params = { # using defaults from manual
"type": "enrichment",
"organism": "Homo sapiens",
"enrichmentType": "process",
"test_type": "FISHER",
"correction": "FDR",
}
# note that the key here is the name of paramter: geneList
files = {'geneList': open(filepath, 'rb')}
# it outputs null, when 'params=params' is used
r = requests.post(url, data=params, files=files)
print(r.status_code)
print(r.text)
Output:
200
Id Name GeneId raw P-value FDR

Unknown error when submit a REST request to Liferay json API

I'm writing an script in Python to automatically update the structures in my Liferay portal and I want to do it via the json REST API.
I make a request to get an structure (method getStructure), and it worked.
But when I try to do an structure update in the portal it shows me the following error:
ValueError: Content-Length should be specified for iterable data of type class 'dict' {'serviceContext': "{'prueba'}", 'serviceClassName': 'com.liferay.portlet.journal.service.JournalStructureServiceUtil', 'name': 'FOO', 'xsd': '... THE XSD OBTAINED VIA JSON ...', 'serviceParameters': '[groupId,structureId,parentStructureId,name,description,xsd,serviceContext]', 'description': 'FOO Structure', 'serviceMethodName': 'updateStructure', 'groupId': '10133'}
What I'm doing is the next:
urllib.request.Request(url = URL, data = data_update, headers = headers)
URL is http://localhost:8080/tunnel-web/secure/json
The headers are configured with basic authentication (it works, it is tested with the getStructure method).
Data is:
data_update = { "serviceClassName" : "com.liferay.portlet.journal.service.JournalStructureServiceUtil", "serviceMethodName" : "updateStructure", "serviceParameters" : "[groupId,structureId,parentStructureId,name,description,xsd,serviceContext]", "groupId" : 10133, "name" : FOO, "description" : FOO Structure, "xsd" : ... THE XSD OBTAINED VIA JSON ..., "serviceContext" : "{}" }
Does anybody know the solution? Have I to specify the length for the dictionary and how? Or this is a bug?

This might help you. It seems it is python's bug.