We Keep Coding

Find the longest group after groupby on normalized json in pandas

My code below groups by values and creates a list of values that were once the length of arrays. But how can I return the id that has the largest sum of each number in the elements:
Original Json read into df (not same data as printed because it was too long)
{
"kind":"admin#reports#activities",
"etag":"\"5g8\"",
"nextPageToken":"A:1651795128914034:-4002873813067783265:151219070090:C02f6wppb",
"items":[
{
"kind":"admin#reports#activity",
"id":{
"time":"2022-05-05T23:59:39.421Z",
"uniqueQualifier":"5526793068617678141",
"applicationName":"token",
"customerId":"cds"
},
"etag":"\"jkYcURYoi8\"",
"actor":{
"email":"blah#blah.net",
"profileId":"1323"
},
"ipAddress":"107.178.193.87",
"events":[
{
"type":"auth",
"name":"activity",
"parameters":[
{
"name":"api_name",
"value":"admin"
},
{
"name":"method_name",
"value":"directory.users.list"
},
{
"name":"client_id",
"value":"722230783769-dsta4bi9fkom72qcu0t34aj3qpcoqloq.apps.googleusercontent.com"
},
{
"name":"num_response_bytes",
"intValue":"7158"
},
{
"name":"product_bucket",
"value":"GSUITE_ADMIN"
},
{
"name":"app_name",
"value":"Untitled project"
},
{
"name":"client_type",
"value":"WEB"
}
]
}
]
},
{
"kind":"admin#reports#activity",
"id":{
"time":"2022-05-05T23:58:48.914Z",
"uniqueQualifier":"-4002873813067783265",
"applicationName":"token",
"customerId":"df"
},
"etag":"\"5T53xK7dpLei95RNoKZd9uz5Xb8LJpBJb72fi2HaNYM/9DTdB8t7uixvUbjo4LUEg53_gf0\"",
"actor":{
"email":"blah.blah#bebe.net",
"profileId":"1324"
},
"ipAddress":"54.80.168.30",
"events":[
{
"type":"auth",
"name":"activity",
"parameters":[
{
"name":"api_name",
"value":"gmail"
},
{
"name":"method_name",
"value":"gmail.users.messages.list"
},
{
"name":"client_id",
"value":"927538837578.apps.googleusercontent.com"
},
{
"name":"num_response_bytes",
"intValue":"2"
},
{
"name":"product_bucket",
"value":"GMAIL"
},
{
"name":"client_type",
"value":"WEB"
}
]
}
]
}
]
}
current code:
df = pd.json_normalize(response['items'])
df['test'] = df.groupby('actor.profileId')['events'].apply(lambda x: [len(x.iloc[i][0]['parameters']) for i in range(len(x))])
output:
ID
1002306 [7, 7, 7, 5]
1234444 [3,5,6]
1222222 [1,3,4,5]
desired output
id total
1002306 26
Sorry had to fill up more space, as there was so much code

There’s no need to construct the intermediate df and do groupby on it. You can use pass the record and meta paths to json_normalize to directly flatten the json data. Then your job seems to be about counting the number of rows per actor.profileId and finding the maximum.
df = pd.json_normalize(response['items'], ['events','parameters'], ['actor'])
df['actor.profileId'] = df['actor'].str['profileId']
out = df.value_counts('actor.profileId').pipe(lambda x: x.iloc[[0]])
Output:
actor.profileId
1323 7
dtype: int64

Yaql expression. How can I make two list into json object

I m trying to form json object from two list vn1","vn2","vn3"] and [6,4,5] using below yaql expression
yaql> dict(data=>dict(["name","id"].zip(["vn1","vn2","vn3"],[6,4,5])))
{
"data": {
"name": "vn1",
"id": "vn2"
}
}
I would like below output
{
"data": [
{
"name": "vn1",
"id": 6
},
{
"name": "vn2",
"id": 4
},
{
"name": "vn3",
"id": 5
}
] }

Your json object resembles a dictionary of lists:
# dictionary of empty list
json_dict = { 'data': [] }
label = ['name', 'id']
v = ["vn1","vn2","vn3"]
k = [6,4,5]
# for each value
for i in range(len(v)):
# add each list entry as dictionarys keys, values
json_dict['data'].append({ label[0] : v[i], label[1]: k[i] })
check it matches your needed json object:
json_object = { "data": [ { "name": "vn1", "id": 6 }, { "name": "vn2", "id": 4 }, { "name": "vn3", "id": 5 } ] }
>>> json_dict == json_object
True

How to replace specific parameter in JSON file with KeyValue in python?

In my JSON file how do I replace specific parameter value with key value pair combination?
In the below JSON I want to replace document and code values with by referring dict json sample.
JSON file:
[
{
"_id": "211123",
"_metadata": {
"version": {
"document": "CUS",
"service": "1"
},
"rider": [
{
"code": "01"
}
]
}
},
{
"_id": "211123",
"_metadata": {
"version": {
"document": "POL",
"service": "1"
},
"rider": [
{
"code": "02"
}
]
}
}
]
Referall JSON:
document:
{
"_metadata.version.document.CUS" : "Customer",
"_metadata.version.document.POL" : "Policy"
}
rider:
{
"rider.code.01" : "RIDER01",
"rider.code.02" : "RIDER02"
}
Example:
In the first JSON record, document has CUS value and it should be replaced with Customer.
If code has 01 as value it should be replaced with RIDER01.

Your question is unclear but if I got it right, here is what you are looking for:
import json
json_text = '''
[
{
"_id": "211123",
"_metadata": {
"version": {
"document": "CUS",
"service": "1"
},
"rider": [
{
"code": "01"
}
]
}
},
{
"_id": "211123",
"_metadata": {
"version": {
"document": "POL",
"service": "1"
},
"rider": [
{
"code": "02"
}
]
}
}
]
'''
documents = {
'CUS': 'Customer',
'POL': 'Policy'
}
riders = {
'01': 'RIDER01',
'02': 'RIDER02'
}
json_dict = json.loads(json_text)
for _id in json_dict:
document = _id['_metadata']['version']['document']
if document in documents:
_id['_metadata']['version']['document'] = documents[document]
for i, rider in enumerate(_id['_metadata']['rider']):
code = rider['code']
if code in riders:
rider['code'] = riders[code]
json_text = json.dumps(json_dict)
If your JSON text is in a file called file.json, you can use the following code instead:
import json
from pathlib import Path
documents = {
'CUS': 'Customer',
'POL': 'Policy'
}
riders = {
'01': 'RIDER01',
'02': 'RIDER02'
}
json_file = Path('file.json')
json_dict = json.loads(json_file.read_text())
for _id in json_dict:
document = _id['_metadata']['version']['document']
if document in documents:
_id['_metadata']['version']['document'] = documents[document]
for i, rider in enumerate(_id['_metadata']['rider']):
code = rider['code']
if code in riders:
rider['code'] = riders[code]
json_file.write_text(json.dumps(json_dict, indent=4))
I hope it helps.

This will be helpful
import json
document = {"_metadata.version.document.CUS" : "Customer","_metadata.version.document.POL" : "Policy" }
jsons = {"rider.code.01" : "RIDER01","rider.code.02" : "RIDER02" }
with open('jsonfile.json','r') as f:
json_input = json.load(f)
dlist = [x.split('.') for x in document.keys()]
jlist = [['_metadata']+k for k in [x.split('.') for x in jsons.keys()]]
for js in json_input:
for d in dlist:
if js['_metadata']['version']['document'] == d[-1]:
js['_metadata']['version']['document']= document['.'.join(d)]
break
for j in jlist:
if js['_metadata']['rider'][0]['code'] == j[-1]:
js['_metadata']['rider'][0]['code'] = jsons['.'.join(j[1:])]
break
with open('output_json.json','w') as f:
json.dump(json_input, f)

How to add json inner tag some values each iteration?

import json
myjson = {'msg':[]}
d = {}
d['phrase'] = "event1"
d['start'] = "11.22.2018"
d['end'] = "11.28.2018"
myjson.get('msg').append({'phrase': d})
output = json.dumps(myjson)
print(output)
I tried this code for my problem.
But there is an error.
{
"msg": [
{
"phrase": {
"phrase": "event1",
"start": "11.22.2018",
"end": "11.28.2018"
}
}
]
}
I want to take phrase value to the outer tag.
{
"msg": [
{
"phrase": "event1"{
[
{
"start": "11.22.2018",
"end": "11.28.2018"
},
{
"start": "11.22.2018",
"end": "11.28.2018"
}
]
}
}
]
}
My data consist some events and events has start-end values more than one.
Can i add 1 start-end values to the event each iteration?
How can i get such a json object below?

Convert float string to float in json

I have a json(test.json) file with the below data. I have around 10000 records. I need to convert value from string to float write in the new file(test1.json). How can I do do this from Python?
{
"name":"test001",
"cat":"test",
"loc":"x loc",
"ings":[
{
"name":"rrrrrr",
"value":"13.0"
},
{
"name":"hhhh",
"value":"18.0"
}
],
"nums":[
{
"name":"kkkk",
"value":"82.05"
},
{
"name":"uuuuu",
"value":"53.55"
}
]
},
{
"name":"test002",
"cat":"test1",
"loc":"y loc",
"ings":[
{
"name":"trtrtr",
"value":"11.0"
},
{
"name":"wewew",
"value":"19.0"
}
],
"nums":[
{
"name":"iuyt",
"value":"122.05"
},
{
"name":"oiui",
"value":"15.5"
}
]
}
resulting json file(test1.json) should be like below...
{
"name":"test001",
"cat":"test",
"loc":"x loc",
"ings":[
{
"name":"rrrrrr",
"value":13.0
},
{
"name":"hhhh",
"value":18.0
}
],
"nums":[
{
"name":"kkkk",
"value":82.05
},
{
"name":"uuuuu",
"value":53.55
}
]
},
{
"name":"test002",
"cat":"test1",
"loc":"y loc",
"ings":[
{
"name":"trtrtr",
"value":11.0
},
{
"name":"wewew",
"value":19.0
}
],
"nums":[
{
"name":"iuyt",
"value":122.05
},
{
"name":"oiui",
"value":15.5
}
]
}

You can provide an object_hook to the json.loads method which will allow you to modify any object (dicts) found within the json:
import json
json_data = """
[{
"name":"test001",
"cat":"test",
"loc":"x loc",
"ings":[
{
"name":"rrrrrr",
"value":"13.0"
},
{
"name":"hhhh",
"value":"18.0"
}
],
"nums":[
{
"name":"kkkk",
"value":"82.05"
},
{
"name":"uuuuu",
"value":"53.55"
}
]
},
{
"name":"test002",
"cat":"test1",
"loc":"y loc",
"ings":[
{
"name":"trtrtr",
"value":"11.0"
},
{
"name":"wewew",
"value":"19.0"
}
],
"nums":[
{
"name":"iuyt",
"value":"122.05"
},
{
"name":"oiui",
"value":"15.5"
}
]
}]
"""
def as_float(obj):
"""Checks each dict passed to this function if it contains the key "value"
Args:
obj (dict): The object to decode
Returns:
dict: The new dictionary with changes if necessary
"""
if "value" in obj:
obj["value"] = float(obj["value"])
return obj
if __name__ == '__main__':
l = json.loads(json_data, object_hook=as_float)
print (json.dumps(l, indent=4))
This results in what you want:
[
{
"loc": "x loc",
"ings": [
{
"name": "rrrrrr",
"value": 13.0
},
{
"name": "hhhh",
"value": 18.0
}
],
"name": "test001",
"nums": [
{
"name": "kkkk",
"value": 82.05
},
{
"name": "uuuuu",
"value": 53.55
}
],
"cat": "test"
},
{
"loc": "y loc",
"ings": [
{
"name": "trtrtr",
"value": 11.0
},
{
"name": "wewew",
"value": 19.0
}
],
"name": "test002",
"nums": [
{
"name": "iuyt",
"value": 122.05
},
{
"name": "oiui",
"value": 15.5
}
],
"cat": "test1"
}
]
To write to a file instead:
with open("out.json", "w+") as out:
json.dump(l, out, indent=4)

You would need to recursively traverse the data and convert anything that looks like a float to a float:
def fix_floats(data):
if isinstance(data,list):
iterator = enumerate(data)
elif isinstance(data,dict):
iterator = data.items()
else:
raise TypeError("can only traverse list or dict")
for i,value in iterator:
if isinstance(value,(list,dict)):
fix_floats(value)
elif isinstance(value,str):
try:
data[i] = float(value)
except ValueError:
pass
It should do the trick:
my_data = [
{ "name" : "rrrrrr",
"value" : "13.0" },
{ "name" : "hhhh",
"value" : "18.0" },
]
fix_floats(my_data)
>>> my_data
[{'name': 'rrrrrr', 'value': 13.0}, {'name': 'hhhh', 'value': 18.0}]

If you have a single or specific key value object, you can reiterate the value containing alphabetical strings or numerical strings, then map and check against their type with string.isnumeric():
dict = { 'a':'100', 'b':'200', 'c':'300', 'd':'four_hundred', 'e':'500' }
dict_parse = {k: int(v) if v.isnumeric() else v for k, v in dict.items()}
>>> dict_parse
{ 'a': 100, 'b': 200, 'c': 300, 'd':'four_hundred', 'e':500}
when dealing with float numbers amend the if statement to replace decimal point, you can apply same principal to negative numbers:
dict = { 'a':'10.0', 'b':'20.12', 'c':'300.3', 'd':'four_hundred', 'e':'500' }
dict_parse = {k: float(v) if v.replace(".", "").isnumeric() else v for k, v in dict.items()}
>>> dict_parse
{ 'a': 10.0, 'b': 20.12, 'c': 300.3, 'd':'four_hundred', 'e':500}