I want to convert this json :
{
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
to this:
"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}".
So that i can send it as part of payload in request in Python.
And i have tried multiple methods for the same. json.dumps doesnt work cause it doesnt escape characters in this case & .replace(""",r"\"") doesnt work cause it creates the string like this :
{\\"rate_limit_by\\": [{\\"type\\": \\"IP\\", \\"extract_from_header\\": \\"X-Forwarded-For\\"}]}
(Below is the example of curl but i want to send the data in specific format using python request.)
My upstream expects data in certain format, as of now am sending data to upstream as below:
curl -i --request POST --data "rule_name=only_ip" \
--data-binary "#data.txt" \
--url http://localhost:8001/plugin/rules
Where data.txt looks like this:
rule={
"rate_limit_by": [
{ "type":"IP", "extract_from_header": "X-Forwarded-For" }
]
}
Am trying to convert it to :
curl -i --request POST -H 'Content-Type: application/json' --data-binary #data.json http://localhost:8001/plugin/rules
Where data.json should like this
{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}
Now the value of "rule" is string with character escape.
This am trying to achieve & am doing post using python.
And below is the code for same:-
import requests
import json
import re
url = 'http://localhost:8001/plugin/rules'
rule = {
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
rule = json.dumps(json.dumps(rule))
print(rule) #this output the data in correct format
obj = {
"rule_name" : "test_ip",
"rule": rule #but when used it here its get wrapped in two \\
}
headers = {'Content-Type': 'application/json', 'Accept': 'text/plain'}
print(obj)
r = requests.post(url, data=obj, headers=headers)
print(r.text)
desired is what you say you need in your something.json file. The following prints True. See https://repl.it/repls/DistantTeemingProtocol.
import json
desired = r'''{
"rule_name" : "test_ip",
"rule":"{\"rate_limit_by\": [{\"type\": \"IP\", \"extract_from_header\": \"X-Forwarded-For\"}]}"
}'''
d = {
"rate_limit_by": [{
"type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
}
s = json.dumps(d)
xxx = json.dumps({"rule_name": "test_ip", "rule": s}, indent=4)
o = json.loads(desired)
yyy = json.dumps(o, indent=4)
print(xxx == yyy)
If you're going to POST using requests, then you should not be posting the string but should instead be posting the dictionary.
I.e.,
r = requests.post(url, json={"rule_name": "test_ip", "rule": s})
Do you mean you want to access the items inside somehow?
You should drop the "[]" because that part doesn't really make sense.
import json
x = str({
"rate_limit_by":
[{ "type": "IP",
"extract_from_header": "X-Forwarded-For"
}]
})
x = x.replace("[","")
x = x.replace("]","")
x = eval(x)
d = json.dumps(x)
l = json.loads(d)
l['rate_limit_by']['type']
This outputs "IP". Now you have a dictionary of all you need called l.
Related
$ curl https://api.goclimate.com/v1/flight_footprint \
-u YOUR_API_KEY: \
-d 'segments[0][origin]=ARN' \
-d 'segments[0][destination]=BCN' \
-d 'segments[1][origin]=BCN' \
-d 'segments[1][destination]=ARN' \
-d 'cabin_class=economy' \
-d 'currencies[]=SEK' \
-d 'currencies[]=USD' \
-G
I have the following input, provided as an example by the creators of the API. This input is meant to be used in the terminal and give output in form of a dictionary. How would it be possible to write the input above in a list or dictionary to use it as part of an Python script? I tried it like below but the response from the API is solely b' '
payload = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
r = requests.get('https://api.goclimate.com/v1/flight_footprint', auth=('my_API_key', ''), data=payload)
print(r.content)
You are making a GET request with requests, but you are trying to pass data, which would be appropriate for making a POST request. Here you want to use params instead:
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Now, what should payload be? It can be a dictionary, but it can't be nested in the way you had it, since it needs to be encoded into the URL as parameters (N.B. this is what your -G option was doing in the curl request).
Looking at the docs and your curl example, I think it should be:
payload = {
"segments[0][origin]": "ARN",
"segments[0][destination]": "BCN",
"segments[1][origin]": "BCN",
"segments[1][destination]": "ARN",
"cabin_class": "economy",
"currencies[]": "SEK", # this will actually be overwritten
"currencies[]": "USD", # since this key is a duplicate (see below)
}
response = requests.get(
"https://api.goclimate.com/v1/flight_footprint",
auth=("my_API_key", ""),
params=payload,
)
print(response.content)
Thinking of how we might parse your original dictionary into this structure:
data = {
"segments" : [
{
"origin" : "ARN",
"destination" : "BCN"
},
{
"origin" : "BCN",
"destination" : "ARN"
}
],
"cabin_class" : "economy",
"currencies" : [
"SEK", "USD"
]
}
payload = {}
for index, segment in enumerate(data["segments"]):
origin = segment["origin"]
destination = segment["destination"]
# python 3.6+ needed:
payload[f"segments[{index}][origin]"] = origin
payload[f"segments[{index}][destination]"] = destination
payload["cabin_class"] = data["cabin_class"]
# requests can handle repeated parameters with the same name this way:
payload["currencies[]"] = data["currencies"]
... should do it.
I make a request to a backend API and get the data back in json format
The response looks something like this. Kindly note that the body key values will be different and there are over 100's of them. The data1.json looks like this
[
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\"}]",
code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
},
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\",\}]",
code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
},
...
]
I need to get rid of the
"\" in all of the body key in the json response
Concatenate the key[body'] into one array
ideally it should look something like this.
[
{"body":"[{"task_ids":[],"accounts":[],"entity_ids":[12814],"guid":"2DFEB337-5F5D-4DF5-84CF-E951D237D448","id":"0034030fb97251b3","subject":"Uploaded Application",]","[{"task_ids":[],"accounts":[],"entity_ids":[12814],"guid":"2DFEB337-5F5D-4DF5-84CF-E951D237D448","id":"0034030fb97251b3","subject":"Uploaded Application",]",..}
]
I have tried replace and a lot of methods but none of them are replacing the \ so I cannot even go to step 2. What I did find out was if I save it to a text file the backslashes are replaced but then I cannot again send the response back as a json object. The code to get the data1.json file so far looks like this.
data = json.loads(r.text)
with open('data1.json', 'w') as outfile:
json.dump(data, outfile, sort_keys = True, indent = 4,
ensure_ascii = False)
Any suggestions on how to achieve the first points as in my desired output? Thanks.
(For starters, what you gave is invalid JSON, and json will either fail to parse it completely or produce something bogus. You need to make sure you extract the response correctly, and if this is really what the response is, have the sending party fix it.)
Now, about the question as asked:
You don't need to do anything special. That's just how JSON represents values that themselves contain JSON's special characters ("escapes" them with a backslash).
If you load the data via a proper JSON parser (e.g. json.loads()), it will undo that escaping, and in e.g. data[0]['body'], you will see proper data.
Of course, since that string is JSON itself, you will need to further parse it with json, too, if you need to split it into its meaningful parts...
The JSON data is incorrectly formatted and invalid JSON (missing quotes in "key" strings (e.g. code": 200,), invalid syntax in second dictionary body object, as mentioned in comment (e.g. "Uploaded Application\",\}]")).
However, after fixing these, a simple str.replace() statement can be used to get the expected JSON format. Then, simply parse the JSON content and build the desired list:
import json
data = '''[
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\"}]",
"code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
},
{
"body": "[{\"task_ids\":[],\"accounts\":[],\"entity_ids\":[12814],\"guid\":\"2DFEB337-5F5D-4DF5-84CF-E951D237D448\",\"id\":\"0034030fb97251b3\",\"subject\":\"Uploaded Application\"}]",
"code": 200,
"headers": {
"Content-Type": "application/json"
},
"msg": "OK",
"name": "0"
}
]'''
r = json.loads(data.replace('\\', '').replace('"[', "[").replace("]\"", "]"))
l = []
for d in r:
l.append(d)
Now inspect the contents of l:
>>> l
[{u'body': [{u'entity_ids': [12814], u'accounts': [], u'task_ids': [], u'guid': u'2DFEB337-5F5D-4DF5-84CF-E951D237D448', u'id': u'0034030fb97251b3', u'subject': u'Uploaded Application'}], u'headers': {u'Content-Type': u'application/json'}, u'code': 200, u'name': u'0', u'msg': u'OK'},
{u'body': [{u'entity_ids': [12814], u'accounts': [], u'task_ids': [], u'guid': u'2DFEB337-5F5D-4DF5-84CF-E951D237D448', u'id': u'0034030fb97251b3', u'subject': u'Uploaded Application'}], u'headers': {u'Content-Type': u'application/json'}, u'code': 200, u'name': u'0', u'msg': u'OK'}]
i want to convert a post request body from json to yaml
the origin json format is like this:
{
"traceId": "1000000316028",
"orderDto": {
"totalPrice": "0.01",
"baseList": [],
"hongbaoSn": "",
"couponSn": "",
"shelfCode": "2SvNXIz56rL0EU5PpSpzwA..",
"additionalPurchaseList": [],
"bindingList": [{
"baseSku": {
"count": 1,
"skuId": 1001088
},
"additionalSku": []
}]
}
}
and i convert it to yaml format like below:
traceId: 1000000316028
orderDto:
totalPrice: 0.01
baseList: []
hongbaoSn:
couponSn:
shelfCode: 2SvNXIz56rL0EU5PpSpzwA..
additionalPurchaseList: []
bindingList:
- baseSku: {count: 1, skuId: 1001088}
addtionalSku: []
and put it as post body in python request, like this:
response = requests.request("POST", url, data=json.dumps(data), headers=headers)
but it's wrong, response code is -1.
i copied from postman, the right body format is like that:
data = "{\n\t\"traceId\": \"1000000316028\",\n\t\"orderDto\": {\n\t\t\"totalPrice\": \"0.01\",\n\t\t\"baseList\": [],\n\t\t\"hongbaoSn\": \"\",\n\t\t\"couponSn\": \"\",\n\t\t\"shelfCode\": \"2SvNXIz56rL0EU5PpSpzwA..\",\n\t\t\"additionalPurchaseList\": [],\n\t\t\"bindingList\": [{\n\t\t\t\"baseSku\": {\n\t\t\t\t\"count\": 1,\n\t\t\t\t\"skuId\": 1001088\n\t\t\t},\n\t\t\t\"additionalSku\": []\n\t\t}]\n\t}\n}"
this can response code 200.
but where is wrong if i want to use yaml format?
thanks very much!
solved.
i modified my yaml format as below:
traceId: '1000000316028'
orderDto:
totalPrice: '0.01'
baseList: []
hongbaoSn: ''
couponSn: ''
shelfCode: 2SvNXIz56rL0EU5PpSpzwA..
additionalPurchaseList: []
bindingList:
- baseSku:
count: 1
skuId: 1001088
additionalSku: []
I am using the Fiware Orion Context Broker and I want to POST some data using a python script. The command line (which works fine) looks like this:
curl -X POST -H "Accept: application/json" -H "Fiware-ServicePath: /orion" -H "Fiware-Service: orion" -H "Content-Type: application/json" -d '{"id": "JetsonTX1", "type": "sensor", "title": {"type": "Text","value": "Init"}, "percentage": { "type": "Text", "value": "0%"}}' "http://141.39.159.63:1026/v2/entities/"
My Python script:
import requests
import json
url = 'http://141.39.159.63:1026/v2/entities/'
data = '''{
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}'''
data_json = json.dumps(data)
headers = {"Accept": "application/json", "Fiware-ServicePath": "/bonseyes", "Fiware-Service": "bonseyes", "Content-Type": "application/json"}
response = requests.post(url, data=data_json, headers=headers)
print(response.json())
This is what response.json() returns:
{u'description': u'Errors found in incoming JSON buffer', u'error': u'ParseError'}
Any ideas how to fix this?
Thank you!
You probably should not pass data as string like so:
data = '''{
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}'''
Pass it as normal dict:
data = {
"title": {
"value": "demo",
"type": "Text"
},
"percentage": {
"type": "Text",
"value": "0%"
}}
The request library will automatically convert this dictionary for you. Also make sure you want to use the data parameter and not the json. Below expert from the documentation should clear out why.
def post(url, data=None, json=None, **kwargs):
r"""Sends a POST request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json data to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
From your comment it seems you should pass your data like so:
response = requests.post(url, json=data_json, headers=headers)
Because your endpoint requires json and not form-encoded bytes
And there is also missing curly brace at the end.
In my opinion you should try using keyValues option for OCB. It will make your payload way shorter. I use similar python program for updating values, therefore PATCH request in my approach:
#Sorting out url and payload for request
data = '{"' + attribute + '":' + value + '}'
headers = {'Content-type': 'application/json'}
url = urlOfOCB + '/v2/entities/' + entityId + '/attrs?options=keyValues'
r = requests.patch(url, (data), headers=headers)
You can read about this option here. As I can see you are not defining any new type for your attributes, therefore it will be "Text" by default, while using keyValues.
Attribute/metadata type may be omitted in requests. When omitted in attribute/metadata creation or in update operations, a default is used for the type depending on the value:
If value is a string, then type Text is used
If value is a number, then type Number is used.
If value is a boolean, then type Boolean is used.
If value is an object or array, then StructuredValue is used.
If value is null, then None is used.
More on those things you can find here.
i am using this python script to feed my data to elasticsearch 6.0. How can i store the variable Value with type float in Elasticsearch?
I can't use the metric options for the visualization in Kibana, because all the data is stored automatically as string
from elasticsearch import Elasticsearch
Device=""
Value=""
for key, value in row.items():
Device = key
Value = value
print("Dev",Device, "Val:", Value)
doc = {'Device':Device, 'Measure':Value , 'Sourcefile':filename}
print(' doc: ', doc)
es.index(index=name, doc_type='trends', body=doc)
Thanks
EDIT:
After the advice of #Saul, i could fix this problem with the following code:
import os,csv
import time
from elasticsearch import Elasticsearch
#import pandas as pd
import requests
Datum = time.strftime("%Y-%m-%d_")
path = '/home/pi/Desktop/Data'
os.chdir(path)
name = 'test'
es = Elasticsearch()
#'Time': time ,
#url = 'http://localhost:9200/index_name?pretty'
doc = {
"mappings": {
"doc": {
"properties": {
"device": { "type": "text" },
"measure": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
#headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
#r = requests.post(url, data=json.dumps(data), headers=headers)
r= es.index(index=name, doc_type='trends', body=doc)
print(r)
You need to send a HTTP Post request using python request, as follows:
url = "http://localhost:9200/index_name?pretty”
data = {
"mappings": {
"doc": {
"properties": {
"title": { "type": "text" },
"name": { "type": "text" },
"age": { "type": "integer" },
"created": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)
Please replace index_name in the URL with the name of the index you are defining in to elasticsearch engine.
If you want to delete the index before creating it again, please do as follows:
url = "http://localhost:9200/index_name”
data = { }
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.delete(url, data=json.dumps(data), headers=headers)
please replace index_name in the URL with your actual index name. After deleting the index, create it again with the first code example above including the mappings that you would need. Enjoy.
Elasticsearch defines field types in the index mapping. It looks like you probably have dynamic mapping enabled, so when you send data to Elasticsearch for the first time, it makes an educated guess about the shape of your data and the field types.
Once those types are set, they are fixed for that index, and Elasticsearch will continue to interpret your data according to those types no matter what you do in your python script.
To fix this you need to either:
Define the index mapping before you load any data. This is the better option as it gives you complete control over how your data is interpreted. https://www.elastic.co/guide/en/elasticsearch/reference/6.0/mapping.html
Make sure that, the first time you send data into the index, you use the correct data types. This will rely dynamic mapping generation, but it will typically do the right thing.
Defining the index mapping is the best option. It's common to do that once off, in Kibana or with curl, or if you create a lot of indices, with a template.
However if you want to use python, you should look at the create or put_mapping functions on IndicesClient