We Keep Coding

How to reset value of multiple dictionaries elegantly in python

I am working on a code which pulls data from database and based on the different type of tables , store the data in dictionary for further usage.
This code handles around 20-30 different table so there are 20-30 dictionaries and few lists which I have defined as class variables for further usage in code.
for example.
class ImplVars(object):
#dictionary capturing data from Asset-Feed table
general_feed_dict = {}
ports_feed_dict = {}
vulns_feed_dict = {}
app_list = []
...
I want to clear these dictionaries before I add data in it.
Easiest or common way is to use clear() function but this code is repeatable as I will have to write for each dict.
Another option I am exploring is with using dir() function but its returning variable names as string.
Is there any elegant method which will allow me to fetch all these class variables and clear them ?

You can use introspection as you suggest:
for d in filter(dict.__instancecheck__, ImplVars.__dict__.values()):
d.clear()
Or less cryptic, covering lists and dicts:
for obj in ImplVars.__dict__.values():
if isinstance(obj, (list, dict)):
obj.clear()
But I would recommend you choose a bit of a different data structure so you can be more explicit:
class ImplVars(object):
data_dicts = {
"general_feed_dict": {},
"ports_feed_dict": {},
"vulns_feed_dict": {},
}
Now you can explicitly loop over ImplVars.data_dicts.values and still have other class variables that you may not want to clear.

code:
a_dict = {1:2}
b_dict = {2:4}
c_list = [3,6]
vars_copy = vars().copy()
for variable, value in vars_copy.items():
if variable.endswith("_dict"):
vars()[variable] = {}
elif variable.endswith("_list"):
vars()[variable] = []
print(a_dict)
print(b_dict)
print(c_list)
result:
{}
{}
[]

Maybe one of the easier kinds of implementation would be to create a list of dictionaries and lists you want to clear and later make the loop clear them all.
d = [general_feed_dict, ports_feed_dict, vulns_feed_dict, app_list]
for element in d:
element.clear()
You could also use list comprehension for that.

Why does the default dictionary in my code keep expanding?

I have a default dictionary and I run it through a couple of loops to look for certain strings in the dictionary. The loops don't really append anything to the dictionary yet as it turns out, during the loop, new items keep getting appended to the dictionary and the final dictionary ends up bigger than the original one before the loop.
I've been trying to pinpoint the error forever but now it's late and I have no idea what's causing this!
from collections import defaultdict
dummydict = defaultdict(list)
dummydict['Alex'].append('Naomi and I love hotcakes')
dummydict['Benjamin'].append('Hayley and I hate hotcakes')
part = ['Alex', 'Benjamin', 'Hayley', 'Naomi']
emp = []
for var in dummydict:
if 'I' in dummydict[var]:
emp.append(var)
for car in part:
for key in range(len(dummydict)):
print('new len', len(dummydict))
print(key, dummydict)
if car in dummydict[key]:
emp.append(car)
print(emp)
print('why are there new values in the dictionary?!', len(dummydict), dummydict)
I expect the dictionary to remain unchanged.

if car in dummydict[key]:
key being an integer, and your dict being initially filled with only string as keys, this will create a new value in dummydict for each key.

Accessing missing keys as in dummydict[key] will add those keys to the defaultdict. Note that key is an int, not the value at that position, as for key in range(len(dummydict)) iterates indexes, not the dict or its keys.

See the docs:
When each key is encountered for the first time, it is not already in the mapping; so an entry is automatically created using the default_factory function which returns an empty list.
For example, this code will show a dummydict with a value in it, because simply accessing dummydict[key] will add the key to the dict if that key is not already there.
from collections import defaultdict
dummydict = defaultdict(list)
dummydict[1]
print (dummydict)
outputs:
defaultdict(<class 'list'>, {1: []})
Your issue is that in your loop, you do things like dummydict[key] and dummydict[var], which adds those keys.

Creating a "dictionary of sets"

I need to efficiently store data in something that would resemble a "dictionary of sets" e.g. have a dictionary with multiple (unique) values matching each unique key. The source of my data would be a (not very well) structured XML.
My idea is:
I will look through a number of elements and find keys. If the key does not exist, add it to dictionary, if it already exists, just add a new value in the corresponding key.
And the result would be something like:
{
'key1': {'1484', '1487', 1488', ...}
'key2': {'1485', '1486', '1489', ...}
'key3': {'1490', '1491', '1492', ...}
...
}
I need to add new keys on the go.
I need to push unique values into each set.
I need to be able to iterate through the whole dictionary.
I am not sure if this is even feasible, but if anybody could push me in the right direction, I would be more than thankful.

I'm not going to benchmark this but in my experience native dicts are faster
store = {}
for key, value in yoursource:
try:
store[key].add(value)
except KeyError:
store[key] = {value}

from collections import defaultdict
mydict = defaultdict(set)
mydict["key1"] |= {'1484', '1487', '1488'}
Iteration is just like the normal dict.

Using dict.setdefault() to create the key if it doesn't exist, and initialising it with an empty set:
store = {}
for key, value in yoursource:
store.setdefault(key, set()).add(value)

Convert a list of dictionaries into a set of dictionaries

How can i make a set of dictionaries from one list of dictionaries?
Example:
import copy
v1 = {'k01': 'v01', 'k02': {'k03': 'v03', 'k04': {'k05': 'v05'}}}
v2 = {'k11': 'v11', 'k12': {'k13': 'v13', 'k14': {'k15': 'v15'}}}
data = []
N = 5
for i in range(N):
data.append(copy.deepcopy(v1))
data.append(copy.deepcopy(v2))
print data
How would you create a set of dictionaries from the list data?
NS: One dictionary is equal to another when they are structurally the same. That means, they got exactly the same keys and same values (recursively)

A cheap workaround would be to serialize your dicts, for example:
import json
dset = set()
d1 = {'a':1, 'b':{'c':2}}
d2 = {'b':{'c':2}, 'a':1} # the same according to your definition
d3 = {'x': 42}
dset.add(json.dumps(d1, sort_keys=True))
dset.add(json.dumps(d2, sort_keys=True))
dset.add(json.dumps(d3, sort_keys=True))
for p in dset:
print json.loads(p)
In the long run it would make sense to wrap the whole thing in a class like SetOfDicts.

Dictionaries are mutable and therefore not hashable in python.
You could either create a dict-subclass with a __hash__ method. Make sure that the hash of a dictionary does not change while it is in the set (that probably means that you cannot allow modifying the members).
See http://code.activestate.com/recipes/414283-frozen-dictionaries/ for an example implementation of frozendicts.
If you can define a sort order on your (frozen) dictionaries, you could alternatively use a data structure based on a binary tree instead of a set. This boils down to the bisect solution provided in the link below.
See also https://stackoverflow.com/a/18824158/5069869 for an explanation why sets without hash do not make sense.

not exactly what you're looking for as this accounts for lists too but:
def hashable_structure(structure):
if isinstance(structure, dict):
return {k: hashable_structure(v) for k, v in structure.items()}
elif isinstance(structure, list):
return {hashable_structure(elem) for elem in structure)}
else:
return structure

dynamically create the defaultdict(list) inside another defaultdict(tree)

I've been fighting with this for quite a long time and need some help, i have some loops searching something and dynamically create dictionary object for them, for example i am scanning a store then all the buckets and then the fruits inside those buckets.
from collections import defaultdict
def tree(): return defaultdict(tree)
test = tree()
test[Store][bucket][fruits] = defaultdict(list)
test[Store][bucket][fruits].append(1)
print test
"""
desired output
{
'Store':{
'bucket1':{
'fruits':['banana', 'mango', 'apple']
}
},
'bucket2':{
'fruits':['banana', 'mango', 'apple']
}
}
}
"""
this approach throws an error and doesn't work, i also tried many other approaches like creating different dictionaries and then merging them, or creating list and then search or find and blah blah.. but i would like to find out how can i merge defaultdict objects inside each other.
can someone please help me with this.
Thanks.

Given your desired output, I don't think you need to use defaultdict(list) at all.
Rather, you can use an ordinary list:
from collections import defaultdict
def tree():
return defaultdict(tree)
test = tree()
test['Store']['bucket']['fruits'] = []
test['Store']['bucket']['fruits'].append(1)
test['Store']['bucket-13']['fruits'] = ['mango', 'apple', 'banana']
print test
However, if you do want to use defaultdict(list), the reason why it's throwing an error is because it is one level too low in your heirarchy. You want to assign it at the "bucket" level, rather then the "fruits" level:
test = tree()
test['Store']['bucket'] = defaultdict(list)
test['Store']['bucket']['fruit'].append('mango')
test['Store']['bucket']['meat'].append('chicken')
# 'fruit' and 'meat' now default to an empty list