I have a json(test.json) file with the below data. I have around 10000 records. I need to convert value from string to float write in the new file(test1.json). How can I do do this from Python?
{
"name":"test001",
"cat":"test",
"loc":"x loc",
"ings":[
{
"name":"rrrrrr",
"value":"13.0"
},
{
"name":"hhhh",
"value":"18.0"
}
],
"nums":[
{
"name":"kkkk",
"value":"82.05"
},
{
"name":"uuuuu",
"value":"53.55"
}
]
},
{
"name":"test002",
"cat":"test1",
"loc":"y loc",
"ings":[
{
"name":"trtrtr",
"value":"11.0"
},
{
"name":"wewew",
"value":"19.0"
}
],
"nums":[
{
"name":"iuyt",
"value":"122.05"
},
{
"name":"oiui",
"value":"15.5"
}
]
}
resulting json file(test1.json) should be like below...
{
"name":"test001",
"cat":"test",
"loc":"x loc",
"ings":[
{
"name":"rrrrrr",
"value":13.0
},
{
"name":"hhhh",
"value":18.0
}
],
"nums":[
{
"name":"kkkk",
"value":82.05
},
{
"name":"uuuuu",
"value":53.55
}
]
},
{
"name":"test002",
"cat":"test1",
"loc":"y loc",
"ings":[
{
"name":"trtrtr",
"value":11.0
},
{
"name":"wewew",
"value":19.0
}
],
"nums":[
{
"name":"iuyt",
"value":122.05
},
{
"name":"oiui",
"value":15.5
}
]
}
You can provide an object_hook to the json.loads method which will allow you to modify any object (dicts) found within the json:
import json
json_data = """
[{
"name":"test001",
"cat":"test",
"loc":"x loc",
"ings":[
{
"name":"rrrrrr",
"value":"13.0"
},
{
"name":"hhhh",
"value":"18.0"
}
],
"nums":[
{
"name":"kkkk",
"value":"82.05"
},
{
"name":"uuuuu",
"value":"53.55"
}
]
},
{
"name":"test002",
"cat":"test1",
"loc":"y loc",
"ings":[
{
"name":"trtrtr",
"value":"11.0"
},
{
"name":"wewew",
"value":"19.0"
}
],
"nums":[
{
"name":"iuyt",
"value":"122.05"
},
{
"name":"oiui",
"value":"15.5"
}
]
}]
"""
def as_float(obj):
"""Checks each dict passed to this function if it contains the key "value"
Args:
obj (dict): The object to decode
Returns:
dict: The new dictionary with changes if necessary
"""
if "value" in obj:
obj["value"] = float(obj["value"])
return obj
if __name__ == '__main__':
l = json.loads(json_data, object_hook=as_float)
print (json.dumps(l, indent=4))
This results in what you want:
[
{
"loc": "x loc",
"ings": [
{
"name": "rrrrrr",
"value": 13.0
},
{
"name": "hhhh",
"value": 18.0
}
],
"name": "test001",
"nums": [
{
"name": "kkkk",
"value": 82.05
},
{
"name": "uuuuu",
"value": 53.55
}
],
"cat": "test"
},
{
"loc": "y loc",
"ings": [
{
"name": "trtrtr",
"value": 11.0
},
{
"name": "wewew",
"value": 19.0
}
],
"name": "test002",
"nums": [
{
"name": "iuyt",
"value": 122.05
},
{
"name": "oiui",
"value": 15.5
}
],
"cat": "test1"
}
]
To write to a file instead:
with open("out.json", "w+") as out:
json.dump(l, out, indent=4)
You would need to recursively traverse the data and convert anything that looks like a float to a float:
def fix_floats(data):
if isinstance(data,list):
iterator = enumerate(data)
elif isinstance(data,dict):
iterator = data.items()
else:
raise TypeError("can only traverse list or dict")
for i,value in iterator:
if isinstance(value,(list,dict)):
fix_floats(value)
elif isinstance(value,str):
try:
data[i] = float(value)
except ValueError:
pass
It should do the trick:
my_data = [
{ "name" : "rrrrrr",
"value" : "13.0" },
{ "name" : "hhhh",
"value" : "18.0" },
]
fix_floats(my_data)
>>> my_data
[{'name': 'rrrrrr', 'value': 13.0}, {'name': 'hhhh', 'value': 18.0}]
If you have a single or specific key value object, you can reiterate the value containing alphabetical strings or numerical strings, then map and check against their type with string.isnumeric():
dict = { 'a':'100', 'b':'200', 'c':'300', 'd':'four_hundred', 'e':'500' }
dict_parse = {k: int(v) if v.isnumeric() else v for k, v in dict.items()}
>>> dict_parse
{ 'a': 100, 'b': 200, 'c': 300, 'd':'four_hundred', 'e':500}
when dealing with float numbers amend the if statement to replace decimal point, you can apply same principal to negative numbers:
dict = { 'a':'10.0', 'b':'20.12', 'c':'300.3', 'd':'four_hundred', 'e':'500' }
dict_parse = {k: float(v) if v.replace(".", "").isnumeric() else v for k, v in dict.items()}
>>> dict_parse
{ 'a': 10.0, 'b': 20.12, 'c': 300.3, 'd':'four_hundred', 'e':500}
Related
Here is the PHP code that I want to write in Python.
<?php
$json = '{
"targeting": [
{
"country": {
"allow": [
"US",
"DE"
]
},
"region" : {
"allow" : {
"US" : [
33
],
"DE" : [
10383
]
}
},
"city": {
"allow": {
"US": [
57
],
"DE": [
3324
]
}
},
"os": {
"allow": [
{
"name": "Android",
"comparison": "GTE",
"version": "2.3.1"
},
{
"name": "Apple TV Software",
"comparison": "EQ",
"version": "4.4"
},
{
"name": "Windows",
"comparison": "EQ",
"version": "Vista"
}
]
},
"isp" : {
"allow" : {
"US" : [
"Att"
],
"DE" : [
"Telekom"
]
}
},
"ip": {
"allow": [
"11.12.13.0-17.18.19.22",
"6.0.0.0",
"10.0.0.0-10.0.0.2",
"11.0.0.0/24"
]
},
"device_type": [
"mobile"
],
"browser": {
"allow": [
"Yandex.Browser for iOS",
"SlimBrowser",
"Edge Mobile"
]
},
"brand": {
"allow": [
"Smartbook Entertainment",
"Walton",
"PIPO"
]
},
"sub": {
"allow": {
"1": [
"A",
"B"
]
},
"deny": {
"2": [
"C",
"D"
]
},
"deny_groups": [
{
"1": ""
},
{
"1": "X",
"2": "Y"
}
]
},
"connection": [
"wi-fi",
"cellular"
],
"block_proxy": true,
"affiliate_id": [
1
],
"url": "http://test-url.com"
}
]
}';
$arr = json_decode($json);
$postData = http_build_query($arr);
//POST SomeURLhere
echo urldecode($arr);
What I need is to send this json in this format
targeting[0][country][allow][]=TR
targeting[0][os][allow][][name]=iOS
targeting[1][country][allow][]=DE
targeting[1][os][allow][][name]=iOS
I guess I need to figure out how to use http_build_query in Python.
with referring this answer I found the solution.
from collections.abc import MutableMapping
from urllib.parse import urlencode, unquote
def flatten(dictionary, parent_key=False, separator='.', separator_suffix=''):
"""
Turn a nested dictionary into a flattened dictionary
:param dictionary: The dictionary to flatten
:param parent_key: The string to prepend to dictionary's keys
:param separator: The string used to separate flattened keys
:return: A flattened dictionary
"""
items = []
for key, value in dictionary.items():
new_key = str(parent_key) + separator + key + separator_suffix if parent_key else key
if isinstance(value, MutableMapping):
items.extend(flatten(value, new_key, separator, separator_suffix).items())
elif isinstance(value, list) or isinstance(value, tuple):
for k, v in enumerate(value):
items.extend(flatten({str(k): v}, new_key, separator, separator_suffix).items())
else:
items.append((new_key, value))
return dict(items)
req = {'check': 'command', 'parameters': ({'parameter': '1', 'description':
'2'}, {'parameter': '3', 'description': '4'})}
req = flatten(req, False, '[', ']')
query = urlencode(req)
query_parsed = unquote(query)
print(query)
print(query_parsed)
And the outputs:
check=command¶meters%5B0%5D%5Bparameter%5D=1¶meters%5B0%5D%5Bdescription%5D=2¶meters%5B1%5D%5Bparameter%5D=3¶meters%5B1%5D%5Bdescription%5D=4
check=command¶meters[0][parameter]=1¶meters[0][description]=2¶meters[1][parameter]=3¶meters[1][description]=4
I m trying to form json object from two list vn1","vn2","vn3"] and [6,4,5] using below yaql expression
yaql> dict(data=>dict(["name","id"].zip(["vn1","vn2","vn3"],[6,4,5])))
{
"data": {
"name": "vn1",
"id": "vn2"
}
}
I would like below output
{
"data": [
{
"name": "vn1",
"id": 6
},
{
"name": "vn2",
"id": 4
},
{
"name": "vn3",
"id": 5
}
] }
Your json object resembles a dictionary of lists:
# dictionary of empty list
json_dict = { 'data': [] }
label = ['name', 'id']
v = ["vn1","vn2","vn3"]
k = [6,4,5]
# for each value
for i in range(len(v)):
# add each list entry as dictionarys keys, values
json_dict['data'].append({ label[0] : v[i], label[1]: k[i] })
check it matches your needed json object:
json_object = { "data": [ { "name": "vn1", "id": 6 }, { "name": "vn2", "id": 4 }, { "name": "vn3", "id": 5 } ] }
>>> json_dict == json_object
True
i want to convert the list data into the json format
list data:
list1 = ['coder', 'cats-dogs', 'ic', 'qwerty', 'ash', 'aish', 'ss', 'ssl', 'messi']
i expect the output:
[
{
"projectname":"coder"
},
{
"projectname":"cats-dogs"
},
{
"projectname":"ic"
},
{
"projectname":"qwerty"
},
{
"projectname":"ash"
},
{
"projectname":"aish"
},
{
"projectname":"ss"
},
{
"projectname":"sl"
},
{
"projectname":"messi"
}
]
The output that you seem to want can be obtained with :
[{"projectname" : key} for key in list1]
I am trying to customize json data using object_hook in Python 3, but do not know how to get started. Any pointers are much appreciated. I am trying to introduce a new key and move existing data into the new key in Python Object.
I am trying to convert below json text:
{
"output": [
{
"Id": "101",
"purpose": "xyz text",
"array": [
{
"data": "abcd"
},
{
"data": "ef gh ij"
}
]
},
{
"Id": "102",
"purpose": "11xyz text",
"array": [
{
"data": "abcd"
},
{
"data": "java"
},
{
"data": "ef gh ij"
}
]
}
]
}
to
{
"output": [
{
"Id": "101",
"mydata": {
"purpose": "xyz text",
"array": [
{
"data": "abcd"
},
{
"data": "ef gh ij"
}
]
}
},
{
"Id": "102",
"mydata": {
"purpose": "11xyz text",
"array": [
{
"data": "abcd"
},
{
"data": "java"
},
{
"data": "ef gh ij"
}
]
}
}
]
}
My Python JSON object hook is defined as:
class JSONObject:
def __init__( self, dict ):
vars(self).update( dict )
def toJSON(self):
return json.dumps(self, default=lambda o: o.__dict__,
sort_keys=True, indent=4)
You can specify a custom object_pairs_hook (input_json is the string with your input JSON).
def mydata_hook(obj):
obj_d = dict(obj)
if 'Id' in obj_d:
return {'Id': obj_d['Id'], 'mydata': {k: v for k, v in obj_d.items() if 'Id' not in k}}
else:
return obj_d
print(json.dumps(json.loads(input_json, object_pairs_hook=mydata_hook), indent=2))
And the output:
{
"output": [
{
"mydata": {
"array": [
{
"data": "abcd"
},
{
"data": "ef gh ij"
}
],
"purpose": "xyz text"
},
"Id": "101"
},
{
"mydata": {
"array": [
{
"data": "abcd"
},
{
"data": "java"
},
{
"data": "ef gh ij"
}
],
"purpose": "11xyz text"
},
"Id": "102"
}
]
}
According to this post, I need to use .copy() on a dictionary, if I want to reference a dictionary which gets updated in a loop (instead of always referencing the same dictionary). However, in my code example below this doesn't seem to work:
main.py:
import collections
import json
nodes_list = ['donald', 'daisy', 'mickey', 'minnie']
edges_list = [('donald', 'daisy', '3'), ('mickey', 'minnie', '3'), ('daisy', 'minnie', '2')]
node_dict, edge_dict = collections.defaultdict(dict), collections.defaultdict(dict)
ultimate_list = []
for n in nodes_list:
node_dict["data"]["id"] = str(n)
ultimate_list.append(node_dict.copy())
for e in edges_list:
edge_dict["data"]["id"] = str(e[2])
edge_dict["data"]["source"] = e[0]
edge_dict["data"]["target"] = e[1]
ultimate_list.append(edge_dict.copy())
print(json.dumps(ultimate_list, indent=2))
As a result here I get the following:
[
{
"data": {
"id": "minnie"
}
},
{
"data": {
"id": "minnie"
}
},
{
"data": {
"id": "minnie"
}
},
{
"data": {
"id": "minnie"
}
},
{
"data": {
"target": "minnie",
"id": "2",
"source": "daysi"
}
},
{
"data": {
"target": "minnie",
"id": "2",
"source": "daysi"
}
},
{
"data": {
"target": "minnie",
"id": "2",
"source": "daysi"
}
}
]
Whereas I would actually expect to get this:
[
{
"data": {
"id": "donald"
}
},
{
"data": {
"id": "daisy"
}
},
{
"data": {
"id": "mickey"
}
},
{
"data": {
"id": "minnie"
}
},
{
"data": {
"target": "donald",
"id": "3",
"source": "daysi"
}
},
{
"data": {
"target": "mickey",
"id": "3",
"source": "minnie"
}
},
{
"data": {
"target": "minnie",
"id": "2",
"source": "daysi"
}
}
]
Can anyone please tell me what I'm doing wrong here?
dict.copy only makes a shallow copy of the dict, the nested dictionaries are never copied, you need deep copies to have those copied over too.
However, you can simply define each new dict at each iteration of the loop and append the new dict at that iteration instead:
for n in nodes_list:
node_dict = collections.defaultdict(dict) # create new instance of data structure
node_dict["data"]["id"] = str(n)
ultimate_list.append(node_dict)
Same applies to the edge_dict:
for e in edges_list:
edge_dict = collections.defaultdict(dict)
...
ultimate_list.append(edge_dict)
Use copy.deepcopy(your_dict): deepcopy.
I see a few things. According to your desired results your edge_list is a bit off.
Change:
('daisy', 'minnie', '2')
To:
('minnie', 'daisy', '2')
To create the data the way you would like in your desired output we can do this with a more basic approach to dicts.
If you are trying to match the desired results in your question then you are calling the wrong index in your for e in edges_list function.
It should be:
"target" : e[0]
"id" : str(e[2])
"source" : e[1]
First I removed
node_dict, edge_dict = collections.defaultdict(dict), collections.defaultdict(dict)
as its not needed for my method.
Next I changed how you are defining the data.
Instead of using pre-defined dictionaries we can just append the results of each set of data to the ultimate_list directly. This shortens the code and is a bit easier to set up.
for n in nodes_list:
ultimate_list.append({"data" : {"id" : str(n)}})
for e in edges_list:
ultimate_list.append({"data" : {"target" : e[0], "id" : str(e[2]), "source" : e[1]}})
print(json.dumps(ultimate_list, indent=2))
So the following code:
import collections
import json
nodes_list = ['donald', 'daisy', 'mickey', 'minnie']
edges_list = [('donald', 'daisy', '3'), ('mickey', 'minnie', '3'), ('minnie', 'daisy', '2')]
ultimate_list = []
for n in nodes_list:
ultimate_list.append({"data" : {"id" : str(n)}})
for e in edges_list:
ultimate_list.append({"data" : {"target" : e[0], "id" : str(e[2]), "source" : e[1]}})
print(json.dumps(ultimate_list, indent=2))
Should result in:
[
{
"data": {
"id": "donald"
}
},
{
"data": {
"id": "daisy"
}
},
{
"data": {
"id": "mickey"
}
},
{
"data": {
"id": "minnie"
}
},
{
"data": {
"target": "donald",
"id": "3",
"source": "daisy"
}
},
{
"data": {
"target": "mickey",
"id": "3",
"source": "minnie"
}
},
{
"data": {
"target": "minnie",
"id": "2",
"source": "daisy"
}
}
]