I have a dictionary that I'll be adding data to that looks like this:
mydict = {
'key1': ['First string message', 'Second string message'],
'key2': ['String message']
}
I am iterating through a json object that has two keys - key and message. I want to add every key into mydict and append all messages that have the same key.
Example:
[
{
'key': 'key1',
'message': 'First string message'
},
{
'key': 'key2',
'message': 'String message'
},
{
'key': 'key1',
'message': 'Second string message'
},
{
'key': 'key3',
'message': 'Brand new string message'
}
]
I am having a problem on building my dictionary dynamically though. How can I dynamically create my keys so that I can append additional strings?
One of the ways I've tried is this:
import json
response = [
{
'key': 'key1',
'message': 'First string message'
},
{
'key': 'key2',
'message': 'String message'
},
{
'key': 'key1',
'message': 'Second string message'
},
{
'key': 'key3',
'message': 'Brand new string message'
}
]
mydict = {}
for r in response:
mydict[r['key']] = mydict.get(r['key'], [r['message']])
print(mydict)
{'key1': ['First string message'], 'key2': ['String message'], 'key3': ['Brand new string message']}
This, obviously, fails my expected result because key1 doesn't contain both strings.
I tried this:
for r in response:
mydict[r['key']] = mydict.get(r['key'], [r['message']]) + r['message']
But this fails with TypeError: can only concatenate list (not "str") to list
How can I dynamically create my keys so that I can append additional strings?
You should use either defaultdict or setdefault. I prefer defaultdict:
from collections import defaultdict
response = [
{
'key': 'key1',
'message': 'First string message'
},
{
'key': 'key2',
'message': 'String message'
},
{
'key': 'key1',
'message': 'Second string message'
},
{
'key': 'key3',
'message': 'Brand new string message'
}
]
mydict = defaultdict(list)
for r in response:
mydict[r['key']].append(r['message'])
With setdefault you can make mydict a regular dictionary:
mydict = {}
for r in response:
mydict.setdefault(r['key'], []).append(r['message'])
Related
I got the following output from my API:
{
'Type': 'Notification',
'MessageId': 'xxx',
'TopicArn': 'xxx',
'Subject': 'xxx',
'Message': 'EventType=Delete, FriendlyType=was deleted, '
'Timestamp=2021-11-08T15:30:45Z, UserId=1111, UserName=me#me.com, '
'IPAddr=(empty), AccountId=22222, AccountName=test-account, '
'ProjectId=test-project',
'Timestamp': '2021-11-08T15:30:46.214Z',
'SignatureVersion': '1'
}
Now I want to access the "Message" variable - once I am in, and get the following output (as already visible in the previous mentioned JSON):
EventType=Delete, FriendlyType=was deleted, Timestamp=2021-11-08T15:30:45Z, UserId=1111, UserName=me#me.com, IPAddr=(empty), AccountId=22222, AccountName=test-account, ProjectId=test-project
How can I now access the keys like EventType, FriendlyType, etc.? I assume that I have to convert this output at first to a valid JSON, but I am currently baffled.
In case that you are not able to receive the Message data as a JSON, a way to handle the situation is convert the message string into a dict <key>:<value>:
message_as_dict = dict(map(lambda var: var.strip().split("=") ,message.split(",")))
NOTICE the .strip() in order to remove the spaces on the beginning of the key.
That shoud create a dictionary with the following structure:
{'EventType': 'Delete', 'FriendlyType': 'was deleted', 'Timestamp': '2021-11-08T15:30:45Z', 'UserId': '1111', 'UserName': 'me#me.com', 'IPAddr': '(empty)', 'AccountId': '22222', 'AccountName': 'test-account', 'ProjectId': 'test-project'}
Then you can access to the values with, for example:
print(message_as_dict["UserName"])
> me#me.com
you can parse your string spliting and then use it to create a dict. Maybe it's not the best solution but it's a simple one.
response = {
'Type': 'Notification',
'MessageId': 'xxx',
'TopicArn': 'xxx',
'Subject': 'xxx',
'Message': 'EventType=Delete, FriendlyType=was deleted, Timestamp=2021-11-08T15:30:45Z, UserId=1111, UserName=me#me.com, IPAddr=(empty), AccountId=22222, AccountName=test-account, ProjectId=test-project',
'Timestamp': '2021-11-08T15:30:46.214Z',
'SignatureVersion': '1'
}
keyVals = [el.split('=') for el in response['Message'].split(', ')]
subdict = {}
for key,val in keyVals:
subdict[key] = val
As mentioned in one of the answer, you can parse your Message string, but I too feel it won't be the best solution. What I noticed is that your JSON is not in proper format. See below for proper JSON you should be getting from your API.
{
"Type": "Notification",
"MessageId": "xxx",
"TopicArn": "xxx",
"Subject": "xxx",
"Message": {
"EventType": "Delete",
"FriendlyType": "was deleted",
"Timestamp": "2021-11-08T15:30:45Z",
"UserId": "1111",
"UserName": "me#me.com",
"IPAddr": "(empty)",
"AccountId": "22222",
"AccountName": "test-account",
"ProjectId": "test-project"
},
"SignatureVersion": "1"
}
Once you are able to get this output, you may further access nested objects. For example, to access FriendlyType from Message, you can simply say, body.Message.FriendlyType. body here means your entire JSON object.
You could do it by splitting the 'Message' string up into (key, value) pairs and constructing a dictionary from them:
from pprint import pprint
output = {'Type': 'Notification',
'MessageId': 'xxx',
'TopicArn': 'xxx',
'Subject': 'xxx',
'Message': 'EventType=Delete, FriendlyType=was deleted, '
'Timestamp=2021-11-08T15:30:45Z, UserId=1111, UserName=me#me.com, '
'IPAddr=(empty), AccountId=22222, AccountName=test-account, '
'ProjectId=test-project',
'Timestamp': '2021-11-08T15:30:46.214Z',
'SignatureVersion': '1'}
msg_dict = dict(pair.split('=') for pair in output['Message'].split(', '))
pprint(msg_dict, sort_dicts=False)
Output:
{'EventType': 'Delete',
'FriendlyType': 'was deleted',
'Timestamp': '2021-11-08T15:30:45Z',
'UserId': '1111',
'UserName': 'me#me.com',
'IPAddr': '(empty)',
'AccountId': '22222',
'AccountName': 'test-account',
'ProjectId': 'test-project'}
I wish to filter the list of dictionaries below, to get the pg_nos for all dictionaries whose text contains valid.
test_response = [
{
'title': 'some value',
'pg_no': 1,
'text': 'Data is not valid'
},
{
'title': 'another title',
'pg_no': 2,
'text': 'some random text'
},
{
'title': 'last title',
'pg_no': 3,
'text': 'valid data'
}
]
The following comprehension works:
pg_nos = [d["pg_no"] for d in test_response if "valid" in d["text"]]
# [1, 3]
You can try the following code
for i in test_response:
if "valid" in i['text']:
print(i['pg_no']) #Perform your required operation here I just used print for display
Or you can also use list comprehension like this
[print(i['pg_no']) for i in test_response if "valid" in i['text']]
I have an object in Python 3 of this format:
a = {
'events': [
{
'timestamp': 123,
'message': 'test'
},
{
'timestamp': 456,
'message': 'foo'
},
{
'timestamp': 789,
'message': 'testbar'
},
],
'first': 'abc',
'last': 'def'
}
I want to create a new object of the same format, but filtered by whether the message key's corresponding value contains a certain string, for example filtering by "test":
a = {
'events': [
{
'timestamp': 123,
'message': 'test'
},
{
'timestamp': 789,
'message': 'testbar'
},
],
'first': 'abc',
'last': 'def'
}
Can I use a nested comprehension for this? I know you can do nested list comprehensions like:
[[y*2 for y in x] for x in l]
But is there a neat way for a dict > list > dict situation?
One option would be to create a new copy of the input dict without events, and then set the filtered events as you require, like this:
copy = {k: v for k, v in a.items() if k != 'events'}
copy['events'] = [e for e in a['events'] if 'test' in e['message']]
Or if you don't mind overwriting the original input, simply do this:
a['events'] = [e for e in a['events'] if 'test' in e['message']]
I would go with a list comprehension with an if-statement like the following:
[event for event in a["events"] if event["message"] == "test" ]
Loop through the values of the "events"-key and add them to the list if the value of their "message" key equals "test".
The result is a list of dictionaries that you can assign back to a["events"] or a copy of a if you would like to preserve a["events"].
So - you can use multiple layers of comprehension, but that doesn't mean you should. I think for such an example, you'd produce cleaner code, by running it through a couple of for loops. Having that said, I think the following is technically achieves the outcome you're asking for.
>>> pprint.pprint(a)
{'events': [{'message': 'test', 'timestamp': 123},
{'message': 'foo', 'timestamp': 456},
{'message': 'testbar', 'timestamp': 789}],
'first': 'abc',
'last': 'def'}
>>> aa = copy.deepcopy(a)
>>> aa['beta'] = aa['events']
>>> pprint.pprint({k:[item for item in v if 'test' in item['message']] if isinstance(v, list) else v for k, v in aa.items()})
{'beta': [{'message': 'test', 'timestamp': 123},
{'message': 'testbar', 'timestamp': 789}],
'events': [{'message': 'test', 'timestamp': 123},
{'message': 'testbar', 'timestamp': 789}],
'first': 'abc',
'last': 'def'}
>>> pprint.pprint({k:[item for item in v if 'test' in item['message']] if isinstance(v, list) else v for k, v in a.items()})
{'events': [{'message': 'test', 'timestamp': 123},
{'message': 'testbar', 'timestamp': 789}],
'first': 'abc',
'last': 'def'}
As said, this is something you can do; I would however on behalf of everyone who's had to read other people code in their careers, respectfully request that you don't use this in production code. A couple of for loops might be more LOC, but would in most cases be much more readable and maintainable.
In python3 I need to get a JSON response from an API call,
and parse it so I will get a dictionary That only contains the data I need.
The final dictionary I ecxpt to get is as follows:
{'Severity Rules': ('cc55c459-eb1a-11e8-9db4-0669bdfa776e', ['cc637182-eb1a-11e8-9db4-0669bdfa776e']), 'auto_collector': ('57e9a4ec-21f7-4e0e-88da-f0f1fda4c9d1', ['0ab2470a-451e-11eb-8856-06364196e782'])}
the JSON response returns the following output:
{
'RuleGroups': [{
'Id': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rules',
'Order': 1,
'Enabled': True,
'Rules': [{
'Id': 'cc637182-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rule',
'Description': 'Look for default severity text',
'Enabled': False,
'RuleMatchers': None,
'Rule': '\\b(?P<severity>DEBUG|TRACE|INFO|WARN|ERROR|FATAL|EXCEPTION|[I|i]nfo|[W|w]arn|[E|e]rror|[E|e]xception)\\b',
'SourceField': 'text',
'DestinationField': 'text',
'ReplaceNewVal': '',
'Type': 'extract',
'Order': 21520,
'KeepBlockedLogs': False
}],
'Type': 'user'
}, {
'Id': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c',
'Name': 'auto_collector',
'Order': 4,
'Enabled': True,
'Rules': [{
'Id': '2d6bdc1d-4064-11eb-8856-06364196e782',
'Name': 'auto_collector',
'Description': 'DO NOT CHANGE!! Created via API coralogix-blocker tool',
'Enabled': False,
'RuleMatchers': None,
'Rule': 'AUTODISABLED',
'SourceField': 'subsystemName',
'DestinationField': 'subsystemName',
'ReplaceNewVal': '',
'Type': 'block',
'Order': 1,
'KeepBlockedLogs': False
}],
'Type': 'user'
}]
}
I was able to create a dictionary that contains the name and the RuleGroupsID, like that:
response = requests.get(url,headers=headers)
output = response.json()
outputlist=(output["RuleGroups"])
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
# Create a dictionary of NAME + ID
ruleDic = {}
for key in groupRuleName:
for value in groupRuleID:
ruleDic[key] = value
groupRuleID.remove(value)
break
Which gave me a simple dictionary:
{'Severity Rules': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e', 'Rewrites': 'ddbaa27e-1747-11e9-9db4-0669bdfa776e', 'Extract': '0cb937b6-2354-d23a-5806-4559b1f1e540', 'auto_collector': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c'}
but when I tried to parse it as nested JSON things just didn't work.
In the end, I managed to create a function that returns this dictionary,
I'm doing it by breaking the JSON into 3 lists by the needed elements (which are Name, Id, and Rules from the first nest), and then create another list from the nested JSON ( which listed everything under Rule) which only create a list from the keyword "Id".
Finally creating a dictionary using a zip command on the lists and dictionaries created earlier.
def get_filtered_rules() -> List[dict]:
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
ruleIDList = [li['Rules'] for li in outputlist]
ruleIDListClean = []
ruleClean = []
for sublist in ruleIDList:
try:
lstRule = [item['Rule'] for item in sublist]
ruleClean.append(lstRule)
ruleContent=list(zip(groupRuleName, ruleClean))
ruleContentDictionary = dict(ruleContent)
lstID = [item['Id'] for item in sublist]
ruleIDListClean.append(lstID)
# Create a dictionary of NAME + ID + RuleID
ruleDic = dict(zip(groupRuleName, zip(groupRuleID, ruleIDListClean)))
except Exception as e: print(e)
return ruleDic
I am grabbing sort of a complex MongoDB document with Python (v3.5) and I should update some values in it which are scattered all around the object and have no particular pattern in the structure and save it back to a different MongoDB collection. The object looks like this:
# after json.loads(mongo_db_document) my dict looks like this
notification = {
'_id': '570f934f45213b0d14b1256f',
'key': 'receipt',
'label': 'Delivery Receipt',
'version': '0.0.1',
'active': True,
'children': [
{
'key': 'started',
'label': 'Started',
'children': [
'date',
'time',
'offset'
]
},
{
'key': 'stop',
'label': 'Ended',
'children': [
'date',
'time',
'offset'
]
},
{
'label': '1. Particulars',
'template': 'formGroup',
'children': [
{
'children': [
{
'key': 'name',
'label': '2.1 Name',
'value': '********** THIS SHOULD BE UPDATED **********',
'readonly': 'true'
},
{
'key': 'ims_id',
'label': '2.2 IMS Number',
'value': '********** THIS SHOULD BE UPDATED **********',
'readonly': 'true'
}
]
},
{
'children': [
{
'key': 'type',
'readonly': '********** THIS SHOULD BE UPDATED **********',
'label': '2.3 Type',
'options': [
{
'label': 'Passenger',
'value': 'A37'
},
{
'label': 'Cargo',
'value': 'A35'
},
{
'label': 'Other',
'value': '********** THIS SHOULD BE UPDATED **********'
}
]
}
]
}
]
},
{
'template': 'formGroup',
'key': 'waste',
'label': '3. Waste',
'children': [
{
'label': 'Waste',
'children': [
{
'label': 'Plastics',
'key': 'A',
'inputType': 'number',
'inputAttributes': {
'min': 0
},
'value': '********** THIS SHOULD BE UPDATED **********'
},
{
'label': 'B. Oil',
'key': 'B',
'inputType': 'number',
'inputAttributes': {
'min': 0
},
'value': '********** THIS SHOULD BE UPDATED **********'
},
{
'label': 'C. Operational',
'key': 'C',
'inputType': 'number',
'inputAttributes': {
'min': 0
},
'value': '********** THIS SHOULD BE UPDATED **********'
}
]
}
]
},
{
'template': 'formRow',
'children': [
'empty',
'signature'
]
}
],
'filter': {
'timestamp_of_record': [
'date',
'time',
'offset'
]
}
}
My initial idea was to put placeholders (like $var_name) in places where I need to update values, and load the string with Python's string.Template, but that approach unfortunately breaks lots of stuff to other users of the same MongoDB document for some reason.
Is there a solution to simply modify this kind of object without "hardcoding" path to find the values I need to update?
There's this small script that I had written a couple years ago - I used it to find entries in some very long and unnerving JSONs. Admittedly it's not beautiful, but it might help in your case, perhaps?
You can find the script on Bitbucket, here (and here is the code).
Unfortunately it's not documented; at the time I wasn't really believing other people would use it, I guess.
Anyways, if you'd like to try it, save the script in your working directory and then use something like this:
from RecursiveSearch import Retriever
def alter_data(json_data, key, original, newval):
'''
Alter *all* values of said keys
'''
retr = Retriever(json_data)
for item_no, item in enumerate(retr.__track__(key)): # i.e. all 'value'
# Pick parent objects with a last element False in the __track__() result,
# indicating that `key` is either a dict key or a set element
if not item[-1]:
parent = retr.get_parent(key, item_no)
try:
if parent[key] == original:
parent[key] = newval
except TypeError:
# It's a set, this is not the key you're looking for
pass
if __name__ == '__main__':
alter_data(notification, key='value',
original = '********** THIS SHOULD BE UPDATED **********',
newval = '*UPDATED*')
Unfortunately as I said the script isn't well documented, so if you want to try it and need more info, I'll be glad to provide it.
Not sure if I understood correctly, but this will dynamically find all keys "value" and "readonly" and print out the paths to address the fields.
def findem(data, trail):
if isinstance(data, dict):
for k in data.keys():
if k in ('value', 'readonly'):
print("{}['{}']".format(trail, k))
else:
findem(data[k], "{}['{}']".format(trail, k))
elif isinstance(data, list):
for k in data:
findem(k, '{}[{}]'.format(trail, data.index(k)))
if __name__ == '__main__':
findem(notification, 'notification')
notification['children'][2]['children'][0]['children'][0]['readonly']
notification['children'][2]['children'][0]['children'][0]['value']
notification['children'][2]['children'][0]['children'][1]['readonly']
notification['children'][2]['children'][0]['children'][1]['value']
notification['children'][2]['children'][1]['children'][0]['readonly']
notification['children'][2]['children'][1]['children'][0]['options'][0]['value']
notification['children'][2]['children'][1]['children'][0]['options'][1]['value']
notification['children'][2]['children'][1]['children'][0]['options'][2]['value']
notification['children'][3]['children'][0]['children'][0]['value']
notification['children'][3]['children'][0]['children'][1]['value']
notification['children'][3]['children'][0]['children'][2]['value']
Add another list to the JSON object. Each item in that list would be a list of keys that lead to the values to be changed. An example for one such list is: ['children', 2, 'children', 'children', 0, 'value'].
Then, to access the value you could use a loop:
def change(json, path, newVal):
cur = json
for key in path[:-1]:
cur = cur[key]
cur[path[-1]] = newVal
path = notification['paths'][0]
#path, for example, could be ['children', 2, 'children', 'children', 0, 'value']
newVal = 'what ever you want'
change(notification, path, newVal)