We Keep Coding

How to create a json from the json as function

Need to create a function to extract values from the json. after that i need to parse the so many similar type of json and save in to file.
json is below
j = '''{
"action": "post",
"status": "completed",
"result": {
"country": "usa",
"is_allow": 'true',
"advance details": {
"value": 'true'
}
}
}'''
convert to json
k = eval(json.dumps(m))
{'action': 'post',
'status': 'completed',
'result': {'country': 'usa',
'is_allow': 'true',
'advance details': {'value': 'true'}}}
Expected out
{'is_allow': 'true','value': 'true'}
pseudo code
Disclaimer : please don't do m['result']['is_allow'] because i need as function to pass so many json

When working with json in python, it should be deserialized into a python object, like a dict, or list; it's weird think to create a function that is 'generic' to any key you want to get, it's just the __getitem__ of dict, if you think to export thing into function, be more specific like this:
def get_is_allow_and_value(my_data):
result = {}
result['is_allow'] = my_data['result']['is_allow']
result['value'] = my_data['result']['advance details']['value']
return result
Also don't use eval you can simple use json.loads for this

Modify keys in nested dictionary to match for post

So I run a GET to system A and get this output:
'{
"comment": "A good object",
"number": "1.1",
"extra_stuff": {
"extra_id": {"value": 100},
"extra_name": {"value": "Test"},
}
}'
Before I POST to system B, I need to translate (and omit) certain keys (but keep the value). What I want from this GET and translate is ONLY the nested dictionary keys (keys in extra_stuff) and only one of them.
Translation:
mapping_dict = { 'extra_id': 'id' }
dict_to_post = { '#KEY extra_ID': #VALUE FROM extra_ID# :, 'name' : Test, 'other' : True }
I got a function for translating the dictionary keys from another question put here:
def update_dict_keys(obj, mapping_dict):
if isinstance(obj, dict):
return {mapping_dict[k]: update_dict_keys(v, mapping_dict) for k, v in obj.iteritems()}
else:
return obj
But I am unsure how to send only the nested dictionary into that function, I only seem to put the "main" dictionary in there. Not sure what the best way about it is. I would further need to handle information from the "main" dictionary as well in another post.

Are you sure, there is a , behind {"value": "Test"} bacause this is no valid json and your get response looks like json.
If this is only a mistake in your post here, you could
use json.loads() to get a dictionary from your response string:
import json
res = '{
"comment": "A good object",
"number": "1.1",
"extra_stuff": {
"extra_id": {"value": 100},
"extra_name": {"value": "Test"}
}
}'
# convert to dict
res_dict = json.loads(res)
# do your mapping
dict_to_post = { 'extra_id': res_dict['extra_stuff']['extra_id']['value']}
# dict_to_post is {'extra_id': 100} now
Furthermore I don't understand what you need the mapping_dict for. Do you want to have {'id': 100} as result?

So I figured out that the response I get is already decoded! Problem I had was that the decoded response was a list with a dictionary within a dictionary.

How to insert an already created json-format string to Elasticsearch Bulk

In a python script,
I'm trying for elasticsearch.helpers.bulk to store multiple records.
I will get a json-format string from another software, and I want to attach it in the source part
I got the helpers.bulk format by this answer
part of my code:
def saveES(output,name):
es = Elasticsearch([{'host':'localhost','port':9200}])
output = output.split('\n')
i=0
datas=[]
while i<len(output):
data = {
"_index":"name",
"_type":"typed",
"_id":saveES.counter,
"_source":[[PROBLEM]]
}
i+=1
saveES.counter+=1
datas.append(data)
helpers.bulk(es, datas)
I would like to attach a json-format string in [[PROBLEM]]
How can I attach it in? I have tried hard, but it is not output in the correct..
if I use:
"_source":{
"image_name":'"'+name+'",'+output[i]
}
and print data result is:
{'_type': 'typed', '_id': 0, '_source': {'image_name': '"nginx","features": "os,disk,package", "emit_shortname": "f0b03efe94ec", "timestamp": "2017-08-18T17:25:46+0900", "docker_image_tag": "latest"'}, '_index': 'name'}
This result show that combined into a single string.
but I expect:
{'_type': 'typed', '_id': 0, '_source': {'image_name': 'nginx','features': 'os,disk,package', 'emit_shortname': 'f0b03efe94ec', 'timestamp': '2017-08-18T17:25:46+0900', 'docker_image_tag': 'latest'}, '_index': 'name'}

There is many problems in your code.
You override the value of data in your loop
You don't respect any norms (Pesp8 and stuff)
You are while instead of a comprehension list
You created 2 useless variable
You instantiate your es in your function
Here is your improved code
es = Elasticsearch([{'host':'localhost','port':9200}]) # You don't have to initialise this variable every time you are calling the function but only once.
def save_es(output,es): # Peps8 convention
output = output.split('\n') # you don't need a while loop. A comprehension loop will avoid a lot of trouble
data = [ # Please without s in data
{
"_index": "name",
"_type": "typed",
"_id": index,
"_source": {
"image_name":"name" + name}
}
for index, name in enumerate(output)
]
helpers.bulk(es, data)
save_es(output, es)
Hope this help.

Python JSON nested key value pair parsing

I am trying to make api call which has json as value to one of the keys. It is failing because of extra double quotes in inner json formed.
variable = request.GET.get('name', '{}')
This value of variable is json that is passed from webpage.
Sample data in variable will be like:-
variable= {'name': 'ABC', 'Id': '1'}
Now when I try to form my payload to be sent in post call,
payload = {
'var1': var1,
'variable': variable,
}
Now this "variable" which is added to payload is treated as String and append with double quotes, like this:-
{'var1': '130', 'variable': "{'name': 'ABC', 'Id': '1'}"}
But I want it to be like this:-
{'var1': '130', 'variable': {'name': 'ABC', 'Id': '1'}}
Any suggestions how to make it possible?
this is the code to make the post call with data:-
r = requests.post("URL",data= json.dumps(payload),headers={'Authorization': obj.header, 'Content-type': 'application/json'}, proxies=getProxyDict())

When you get variable from the request it isn't just 'treated' as a string... is is a string.
So you need to:
json.loads(variable)
before adding it to the dict which is later serialized to json.
i.e.
variable = json.loads(request.GET.get('name', '{}'))
payload = {
'var1': var1,
'variable': variable,
}
r = requests.post("URL",data= json.dumps(payload),headers={'Authorization': obj.header, 'Content-type': 'application/json'}, proxies=getProxyDict())

You have presumably already dumped variable to a JSON string. Don't do that; leave it as a dict, and dump the whole thing in one go.

import json
variable = {'name': 'ABC', 'Id': '1'}
payload = {
'var1': 'foo',
'variable': variable,
}
print json.dumps(payload, indent=4)
results to
{
"variable": {
"name": "ABC",
"Id": "1"
},
"var1": "foo"
}
which is perfectly json. But, you may need to set your headers according to the API you're talking to.
request = requests.get('myurl', headers={'accept': 'application/json'})
json_data = request.json()
# manipulate json_data
request = requests.post('myurl', headers={'content-type': 'application/json'}, json=json_data)

Few things I would like to point here.
variable= "{'name': 'ABC', 'Id': '1'}"
This is NOT a valid json string.
variable= '{"name": "ABC", "Id": "1"}'
This is a valid json string.
So first before attaching this variable to payload dict you need to convert it to a python dict like this.
variable = json.loads(variable)
This will create variable something like this.
{u'name': u'ABC', u'Id': u'1'}
Now you can go ahead and add this to your payload dict and continue what you are doing.

String format a JSON string gives KeyError

Why does this code give a KeyError?
output_format = """
{
"File": "{filename}",
"Success": {success},
"ErrorMessage": "{error_msg}",
"LogIdentifier": "{log_identifier}"
}
"""
print output_format.format(filename='My_file_name',
success=True,
error_msg='',
log_identifier='123')
Error message:
KeyError: ' "File"'

You need to double the outer braces; otherwise Python thinks { "File".. is a reference too:
output_format = '{{ "File": "{filename}", "Success": {success}, "ErrorMessage": "{error_msg}", "LogIdentifier": "{log_identifier}" }}'
Result:
>>> print output_format.format(filename='My_file_name',
... success=True,
... error_msg='',
... log_identifier='123')
{ "File": "My_file_name", "Success": True, "ErrorMessage": "", "LogIdentifier": "123" }
If, indicentally, you are producing JSON output, you'd be better off using the json module:
>>> import json
>>> print json.dumps({'File': 'My_file_name',
... 'Success': True,
... 'ErrorMessage': '',
... 'LogIdentifier': '123'})
{"LogIdentifier": "123", "ErrorMessage": "", "Success": true, "File": "My_file_name"}
Note the lowercase true in the output, as required by the JSON standard.

As mentioned by Tudor in a comment to another answer, the Template class was the solution that worked best for me. I'm dealing with nested dictionaries or list of dictionaries and handling those were not as straightforward.
Using Template though the solution is quite simple.
I start with a dictionary that is converted into a string. I then replace all instances of { with ${ which is the Template identifier to substitute a placeholder.
The key point of getting this to work is using the Template method safe_substitute. It will replace all valid placeholders like ${user_id} but ignore any invalid ones that are part of the dictionary structure, like ${'name': 'John', ....
After the substitution is done I remove any leftovers $ and convert the string back to a dictionary.
In the code bellow, resolve_placeholders returns a dictionary where each key matches a placeholder in the payload string and the value is substituted by the Template class.
from string import Template
.
.
.
payload = json.dumps(payload)
payload = payload.replace('{', '${')
replace_values = self.resolve_placeholders(payload)
if replace_values:
string_template = Template(payload)
payload = string_template.safe_substitute(replace_values)
payload = payload.replace('${', '{')
payload = json.loads(payload)

To extend on Martijn Pieters answer and comment:
According to MArtijn' comment, escaping the {..} pairs that are not placeholders is they way to go with nested dictionaries. I haven't succeded in doing that, so I suggest the following method.
For nested dictionaries I tried doubling up on any { and } of the nested dictionaries.
a='{{"names":{{"a":"{name}"}}}}'
a.format(name=123) output:
output: '{"names":{"a":"123"}}'
But this makes using format to change values inside a json string, a over-complex method, so I use a twist on the format command.
I replace ${param_name} in a json string. For example:
My predefined JSON looks like this:
my_json_dict = {
'parameter': [
{
'name': 'product',
'value': '${product}'
},
{
'name': 'suites',
'value': '${suites}'
},
{
'name': 'markers',
'value': '${markers}'
}
]
}
I provide this dictionary as values to replace instead of the parameters
parameters = {
'product': 'spam',
'suites': 'ham',
'markers': 'eggs'
}
And use this code to do the replacment
json_str = json.dumps(my_json_dict)
for parameter_name, parameter_value in parameters.iteritems():
parameter_name = '${'+parameter_name+'}'
json_str = json_str.replace(parameter_name, parameter_value)
json_dict = json.loads(json_str)