Related
I have a problem. I have a dict my_Dict. This is somewhat nested. However, I would like to 'clean up' the dict my_Dict, by this I mean that I would like to separate all nested ones and also generate a unique ID so that I can later find the corresponding object again.
For example, I have detail: {...}, this nested, should later map an independent dict my_Detail_Dict and in addition, detail should receive a unique ID within my_Dict. Unfortunately, my list that I give out is empty. How can I remove my slaughtered keys and give them an ID?
my_Dict = {
'_key': '1',
'group': 'test',
'data': {},
'type': '',
'code': '007',
'conType': '1',
'flag': None,
'createdAt': '2021',
'currency': 'EUR',
'detail': {
'selector': {
'number': '12312',
'isTrue': True,
'requirements': [{
'type': 'customer',
'requirement': '1'}]
}
}
}
def nested_dict(my_Dict):
my_new_dict_list = []
for key in my_Dict.keys():
#print(f"Looking for {key}")
if isinstance(my_Dict[key], dict):
print(f"{key} is nested")
# Add id to nested stuff
my_Dict[key]["__id"] = 1
my_nested_Dict = my_Dict[key]
# Delete all nested from the key
del my_Dict[key]
# Add id to key, but not the nested stuff
my_Dict[key] = 1
my_new_dict_list.append(my_Dict[key])
my_new_dict_list.append(my_Dict)
return my_new_dict_list
nested_dict(my_Dict)
[OUT] []
# What I want
[my_Dict, my_Details_Dict, my_Data_Dict]
What I have
{'_key': '1',
'group': 'test',
'data': {},
'type': '',
'code': '007',
'conType': '1',
'flag': None,
'createdAt': '2021',
'currency': 'EUR',
'detail': {'selector': {'number': '12312',
'isTrue': True,
'requirements': [{'type': 'customer', 'requirement': '1'}]}}}
What I want
my_Dict = {'_key': '1',
'group': 'test',
'data': 18,
'type': '',
'code': '007',
'conType': '1',
'flag': None,
'createdAt': '2021',
'currency': 'EUR',
'detail': 22}
my_Data_Dict = {'__id': 18}
my_Detail_Dict = {'selector': {'number': '12312',
'isTrue': True,
'requirements': [{'type': 'customer', 'requirement': '1'}]}, '__id': 22}
The following code snippet will solve what you are trying to do:
my_Dict = {
'_key': '1',
'group': 'test',
'data': {},
'type': '',
'code': '007',
'conType': '1',
'flag': None,
'createdAt': '2021',
'currency': 'EUR',
'detail': {
'selector': {
'number': '12312',
'isTrue': True,
'requirements': [{
'type': 'customer',
'requirement': '1'}]
}
}
}
def nested_dict(my_Dict):
# Initializing a dictionary that will store all the nested dictionaries
my_new_dict = {}
idx = 0
for key in my_Dict.keys():
# Checking which keys are nested i.e are dictionaries
if isinstance(my_Dict[key], dict):
# Generating ID
idx += 1
# Adding generated ID as another key
my_Dict[key]["__id"] = idx
# Adding nested key with the ID to the new dictionary
my_new_dict[key] = my_Dict[key]
# Replacing nested key value with the generated ID
my_Dict[key] = idx
# Returning new dictionary containing all nested dictionaries with ID
return my_new_dict
result = nested_dict(my_Dict)
print(my_Dict)
# Iterating through dictionary to get all nested dictionaries
for item in result.items():
print(item)
If I understand you correctly, you wish to automatically make each nested dictionary it's own variable, and remove it from the main dictionary.
Finding the nested dictionaries and removing them from the main dictionary is not so difficult. However, automatically assigning them to a variable is not recommended for various reasons. Instead, what I would do is store all these dictionaries in a list, and then assign them manually to a variable.
# Prepare a list to store data in
inidividual_dicts = []
id_index = 1
for key in my_Dict.keys():
# For each key, we get the current value
value = my_Dict[key]
# Determine if the current value is a dictionary. If so, then it's a nested dict
if isinstance(value, dict):
print(key + " is a nested dict")
# Get the nested dictionary, and replace it with the ID
dict_value = my_Dict[key]
my_Dict[key] = id_index
# Add the id to previously nested dictionary
dict_value['__id'] = id_index
id_index = id_index + 1 # increase for next nested dic
inidividual_dicts.append(dict_value) # store it as a new dictionary
# Manually write out variables names, and assign the nested dictionaries to it.
[my_Details_Dict, my_Data_Dict] = inidividual_dicts
Having a bit of difficulties here with looping through this json object content.
The json file is as such:
[{'archived': False,
'cache_ttl': None,
'collection': {'archived': False,
'authority_level': None,
'color': '#509EE3',
'description': None,
'id': 525,
'location': '/450/',
'name': 'eaf',
'namespace': None,
'personal_owner_id': None,
'slug': 'eaf'},
'collection_id': 525,
'collection_position': None,
'created_at': '2022-01-06T20:51:17.06376Z',
'creator_id': 1,
'database_id': 4,
}, ... ]
And I want to loop through each dict in the list check that the collection is not empty and then for each collection if the location equals '/450/' return append that dict to a list.
My code is as follows.
content = json.loads(res.text)
for q in content:
if q['collection']:
for col in q['collection']:
if col['location'] == '/450/':
data.append(q)
print(data)
Having played around with it I keep either getting ValueError: too many values to unpack (expected 2) OR TypeError: string indices must be integers
Any help with my structure would be much appreciated thanks.
Disclaimer:
I had previously written this as a list comprehension and it worked like a charm however that doesnt work anymore as I now need to check if the collection is empty.
How I wrote it previously:
content = [ x for x in content if x['collection']['location'] == '/450/']
That should work for you:
for q in content:
if q['collection']['location'] == '/450/':
data.append(q)
print(data)
If you go with for loop with for col in q['collection'], you just iterate over keys inside q['collection'], so cols = ['archived', 'authority_level', ...].
From your previous list comprehension, "location" is a key in q["collection"].
When you write
for col in q["collection"]
You are iterating over the keys in q["collection"]. One of these keys is "location". Your for loop seems to iterate more than necessary:
if q['collection'] and "location" in q["collection"] and q["collection"]["location"] == "/450/":
data.append(q)
Your Code Has Way too Iterations Than needed.
The error TypeError: string indices must be integers occurs at the second conditional statement when you check col['location'] = "/450/".
That's because not all tokens in the collection object have sub-objects where you can get data with their key.
Take a look at your old code and the modified code for more in depth understanding.
# Your old json datas
content = [{'archived': False,
'cache_ttl': None,
'collection': {'archived': False,
'authority_level': None,
'color': '#509EE3',
'description': None,
'id': 525,
'location': '/450/',
'name': 'eaf',
'namespace': None,
'personal_owner_id': None,
'slug': 'eaf'},
'collection_id': 525,
'collection_position': None,
'created_at': '2022-01-06T20:51:17.06376Z',
'creator_id': 1,
'database_id': 4,
} ]
data = []
for q in content:
if q['collection']:
for col in q['collection']:
if col['location'] == '/450/': # The first object in collection object is [archived] which is a string, this causes the program to throw error
data.append(q)
print(data)
Here is the modified code
# Your json datas
json_datas = [{'archived': False,
'cache_ttl': None,
'collection': {'archived': False,
'authority_level': None,
'color': '#509EE3',
'description': None,
'id': 525,
'location': '/450/',
'name': 'eaf',
'namespace': None,
'personal_owner_id': None,
'slug': 'eaf'},
'collection_id': 525,
'collection_position': None,
'created_at': '2022-01-06T20:51:17.06376Z',
'creator_id': 1,
'database_id': 4,
} ]
list_data = [] # Your list data in which appends the json data if the location is /450/
for data in json_datas: # Getting each Json data
if len(data["collection"]): # Continue if the length of collection is not 0 [NOTE: 0 = False, 1 or more = True]
if data['collection']['location'] == "/450/": # Check the location
list_data.append(data) # Append if true
print(list_data)
Don't need to iterate over the collection object since it's a dictionary and just need to check the location property.
Also, in case the "collection" or "location" properties are not present then use dict.get(key) function rather than dict[key] since the latter will raise a KeyError exception if key is not found and get() returns None value if key is not found.
content = [{'archived': False,
'cache_ttl': None,
'collection': {'archived': False,
'authority_level': None,
'color': '#509EE3',
'description': None,
'id': 525,
'location': '/450/',
'name': 'eaf',
'namespace': None,
'personal_owner_id': None,
'slug': 'eaf'},
'collection_id': 525,
'collection_position': None,
'created_at': '2022-01-06T20:51:17.06376Z',
'creator_id': 1,
'database_id': 4,
},
{'foo': None}
]
#content = json.loads(res.text)
data = []
for q in content:
c = q.get('collection')
if c and c.get('location') == '/450/':
data.append(q)
print(data)
Output:
[{'archived': False, 'cache_ttl': None, 'collection': { 'location': '/450/', 'name': 'eaf', 'namespace': None }, ...}]
In python3 I need to get a JSON response from an API call,
and parse it so I will get a dictionary That only contains the data I need.
The final dictionary I ecxpt to get is as follows:
{'Severity Rules': ('cc55c459-eb1a-11e8-9db4-0669bdfa776e', ['cc637182-eb1a-11e8-9db4-0669bdfa776e']), 'auto_collector': ('57e9a4ec-21f7-4e0e-88da-f0f1fda4c9d1', ['0ab2470a-451e-11eb-8856-06364196e782'])}
the JSON response returns the following output:
{
'RuleGroups': [{
'Id': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rules',
'Order': 1,
'Enabled': True,
'Rules': [{
'Id': 'cc637182-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rule',
'Description': 'Look for default severity text',
'Enabled': False,
'RuleMatchers': None,
'Rule': '\\b(?P<severity>DEBUG|TRACE|INFO|WARN|ERROR|FATAL|EXCEPTION|[I|i]nfo|[W|w]arn|[E|e]rror|[E|e]xception)\\b',
'SourceField': 'text',
'DestinationField': 'text',
'ReplaceNewVal': '',
'Type': 'extract',
'Order': 21520,
'KeepBlockedLogs': False
}],
'Type': 'user'
}, {
'Id': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c',
'Name': 'auto_collector',
'Order': 4,
'Enabled': True,
'Rules': [{
'Id': '2d6bdc1d-4064-11eb-8856-06364196e782',
'Name': 'auto_collector',
'Description': 'DO NOT CHANGE!! Created via API coralogix-blocker tool',
'Enabled': False,
'RuleMatchers': None,
'Rule': 'AUTODISABLED',
'SourceField': 'subsystemName',
'DestinationField': 'subsystemName',
'ReplaceNewVal': '',
'Type': 'block',
'Order': 1,
'KeepBlockedLogs': False
}],
'Type': 'user'
}]
}
I was able to create a dictionary that contains the name and the RuleGroupsID, like that:
response = requests.get(url,headers=headers)
output = response.json()
outputlist=(output["RuleGroups"])
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
# Create a dictionary of NAME + ID
ruleDic = {}
for key in groupRuleName:
for value in groupRuleID:
ruleDic[key] = value
groupRuleID.remove(value)
break
Which gave me a simple dictionary:
{'Severity Rules': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e', 'Rewrites': 'ddbaa27e-1747-11e9-9db4-0669bdfa776e', 'Extract': '0cb937b6-2354-d23a-5806-4559b1f1e540', 'auto_collector': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c'}
but when I tried to parse it as nested JSON things just didn't work.
In the end, I managed to create a function that returns this dictionary,
I'm doing it by breaking the JSON into 3 lists by the needed elements (which are Name, Id, and Rules from the first nest), and then create another list from the nested JSON ( which listed everything under Rule) which only create a list from the keyword "Id".
Finally creating a dictionary using a zip command on the lists and dictionaries created earlier.
def get_filtered_rules() -> List[dict]:
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
ruleIDList = [li['Rules'] for li in outputlist]
ruleIDListClean = []
ruleClean = []
for sublist in ruleIDList:
try:
lstRule = [item['Rule'] for item in sublist]
ruleClean.append(lstRule)
ruleContent=list(zip(groupRuleName, ruleClean))
ruleContentDictionary = dict(ruleContent)
lstID = [item['Id'] for item in sublist]
ruleIDListClean.append(lstID)
# Create a dictionary of NAME + ID + RuleID
ruleDic = dict(zip(groupRuleName, zip(groupRuleID, ruleIDListClean)))
except Exception as e: print(e)
return ruleDic
I'm trying to learn python. Assuming I have the below two dict().
In the 1st dict, it includes user info and the reporting line structures.
In the 2nd dict, it includes item counts belong to each individual.
I want to compare again these two dict and sum up the total item counts then display the result under name_S. The outcome is shown as follow:
data_set_1 = { 'id': 'mid',
'name': 'name_S',
'directory': [
{
'id': 'eid_A',
'name': 'name_A',
'directory': []
},
{ 'id': 'eid_B',
'name': 'name_B',
'directory': []
},
{ 'id': 'eid_C',
'name': 'name_C',
'directory': [
{'id': 'eid_C1',
'name': 'name_C1',
'directory': []},
{'id': 'eid_C2',
'name': 'name_C2',
'directory': []}]
}]}
data_set_2 = { 'eid_A': 5,
'eid_F': 3,
'eid_G': 0,
'eid_C': 1,
'eid_C1': 10,
'eid_C2': 20
}
Result:
{'name_S': 36}
I'm able to get the result if I did this way:
def combine_compare(data_set_1, data_set_2):
combine_result = dict()
combine_id = data_set_1['id']
combine_name = data_set_1['name']
combine_directory = data_set_1['directory']
if combine_directory:
combine_item_sum = 0
combine_item_count = data_set_2.get(combine_id, 0)
for combine_user in combine_directory:
# Recursion starts
for i in [combine_compare(combine_user, data_set_2)]:
for key, value in i.items():
combine_item_sum += value
combine_result[combine_name] = combine_item_sum + combine_item_count
else:
combine_result[combine_name] = data_set_2.get(combine_id, 0)
return combine_result
Now if I want to include the ids that have item counts in the final result, something like this:
#if there is result and directory is not None under name_S
{'name_S': [36, ('eid_A', 'eid_C', eid_C1', 'eid_C2')]}
#if there is no result and directory is not None under name_S, display a default str
{'name_S': [0, ('Null')]}
#if there is result and directory is None under name_S, display name_S id
{'name_S': [1, ('mid')]}
My original idea is to create a list and append the counts and ids but I'm struggling how I can accomplish this. Here is what I try but the list is not returning the outcome I'm expecting and I'm not sure how I can append the count in the list. Any help would be greatly appreciated.
def combine_compare(data_set_1, data_set_2):
combine_result = dict()
# Creating a new list
combine_list = list()
combine_id = data_set_1['id']
combine_name = data_set_1['name']
combine_directory = data_set_1['directory']
if combine_directory:
combine_item_sum = 0
combine_item_count = data_set_2.get(combine_id, 0)
for combine_user in combine_directory:
# Recursion starts
for i in [combine_compare(combine_user, data_set_2)]:
for key, value in i.items():
combine_item_sum += value
# Trying to append the ids where count > 0
if data_set_2.get(combine_user['id'], 0) != 0:
combine_list.append(combine_user['id'])
combine_result[combine_name] = combine_item_sum + combine_item_count
else:
combine_result[combine_name] = data_set_2.get(combine_id, 0)
# Trying to print the list to see the results
print(combine_list)
return combine_result
So what i understood from your question is that you want to know which ids mentioned in data_set_2 are there in data_set_1 and what is there sum. The following code is my version of solving the above mentioned problem.
data_set_1 = { 'id': 'mid',
'name': 'name_S',
'directory': [
{
'id': 'eid_A',
'name': 'name_A',
'directory': []
},
{ 'id': 'eid_B',
'name': 'name_B',
'directory': []
},
{ 'id': 'eid_C',
'name': 'name_C',
'directory': [
{'id': 'eid_C1',
'name': 'name_C1',
'directory': []},
{'id': 'eid_C2',
'name': 'name_C2',
'directory': []
}
]
}
]
}
data_set_2 = { 'eid_A': 5,
'eid_F': 3,
'eid_G': 0,
'eid_C': 1,
'eid_C1': 10,
'eid_C2': 20
}
value = 0
final_result={}
def compare(d1,d2):
global value
temp_result={}
if d1['name'] not in temp_result:
temp_result[d1['name']]=[]
for items in d1['directory']:
if items['directory']:
temp_value=compare(items,d2)
temp_result[d1['name']].append(temp_value)
result=check_for_value(items,d2)
if result:
value+=result
temp_result[d1['name']].append(items['id'])
return temp_result
def check_for_value(d,d2):
if d['id'] in d2:
return d2[d['id']]
final_result=compare(data_set_1,data_set_2)
final_result['value']=value
print("final_result:",final_result)
Which gives you an output dict which tells you exactly under which name does those ids come under.
The output is as follows:
final_result: {'value': 36, 'name_S': ['eid_A', {'name_C': ['eid_C1', 'eid_C2']}, 'eid_C']}
You can change the structure of the final result to make it exactly how you want it to be. Let me know if you have any trouble understanding the program.
So I have been trying to figure out how I can print out two different formats using one for loop. I would like to provide the code before explaining my issue
fullList = [
{
'url': 'www.randomsite.com/251293',
'numbers': '7.5'
},
{
'url': 'www.randomsite.com/251294',
'numbers': '8'
},
{
'url': 'www.randomsite.com/251295',
'numbers': '8.5'
},
{
'url': 'www.randomsite.com/251296',
'numbers': '9'
},
{
'url': 'www.randomsite.com/251297',
'numbers': '9.5'
}
]
#fullList = [
# {
# 'numbers': '7.5'
# },
# {
# 'numbers': '8'
# },
# {
# 'numbers': '8.5'
# },
# {
# 'numbers': '9'
# },
# {
# 'numbers': '9.5'
# }
#]
try:
numbersList = []
for numbers in fullList:
numbersList.append('{}{}'.format('{}'.format(numbers.get('url') if numbers.get('url') else ''), numbers.get('numbers')))
print(numbersList)
except Exception:
pass
and what I am looking for outcome is:
If url is in the list: print('<url|numbers>') meaning the format would be <url|numbers>
If no url is in the list: print(numbers) and the print here should only give the numbers - I sometimes just want the numbers, meaning that in the list I removed all URL's so it will only remain numbers.
My problem is that I dont know how I can combine these two into one format. So far I am able to print out only numbers with the code I have provided.
Use normal if/else. It will be more readable. And you have only one format.
for numbers in fullList:
if numbers.get('url'):
numbersList.append('{}|{}'.format(numbers.get('url'), numbers.get('numbers'))
else:
numbersList.append(numbers.get('numbers'))
You can solve this problem and it will look more pythonic this way:
fullList = [
{'url': 'www.randomsite.com/251293', 'numbers': '7.5'},
{'url': 'www.randomsite.com/251294', 'numbers': '8'},
{'url': 'www.randomsite.com/251295', 'numbers': '8.5'},
{'url': 'www.randomsite.com/251296', 'numbers': '9'},
{'url': 'www.randomsite.com/251297', 'numbers': '9.5'},
{'numbers': '100'}
]
[(x['url'] + '|' + x['numbers']) if x.get('url') else x['numbers'] for x in fullList ]
You are using list comprehensions, minimizing nesting etc.
One solution is to select all values in each subdict and join them with a custom delimiter. In this way, you don't care if the key/value exist or not.
# Let's consider partial data
fullList = [
{
'url': 'www.randomsite.com/251293',
'numbers': '7.5'
},
{
'url': 'www.randomsite.com/251294',
'numbers': '8'
},
{
'url': 'www.randomsite.com/251295',
'numbers': '8.5'
},
{
'url': 'www.randomsite.com/251296',
},
{
'numbers': '9.5'
}
]
numbersList = []
for element in fullList:
numbersList.append("|".join([element[v] for v in element.keys()]))
print(numbersList)
# ['www.randomsite.com/251293|7.5', 'www.randomsite.com/251294|8',
# 'www.randomsite.com/251295|8.5', 'www.randomsite.com/251296', '9.5']
You can do it in one line with list comprehension:
output = ["|".join([element[v] for v in element.keys()]) for element in fullList]
print(output)
# ['www.randomsite.com/251293|7.5', 'www.randomsite.com/251294|8',
# 'www.randomsite.com/251295|8.5', 'www.randomsite.com/251296', '9.5']
Using list comprehension
Ex.
fullList = [
{'url': 'www.randomsite.com/251293','numbers': '7.5'},
{'url': 'www.randomsite.com/251294','numbers': '8'},
{'url': 'www.randomsite.com/251295','numbers': '8.5'},
{'url': 'www.randomsite.com/251296','numbers': '9'},
{'url': 'www.randomsite.com/251297','numbers': '9.5'}
]
list1 = [ "{0}|{1}".format(x['url'],x['numbers']) for x in fullList ]
print(list1)
O/P:
['www.randomsite.com/251293|7.5', 'www.randomsite.com/251294|8', 'www.randomsite.com/251295|8.5', 'www.randomsite.com/251296|9', 'www.randomsite.com/251297|9.5']
OR
for the updated question, if the dictionary does not contain url
fullList = [
{'url': 'www.randomsite.com/251296','numbers': '9'},
{'numbers': '9.5'}
]
list1 = [ "{0}{1}".format((x.get('url')+'|' if 'url' in x else ''),x.get('numbers','')) for x in fullList ]
print(list1)
O/P:
['www.randomsite.com/251296|9', '9.5']