I am working on a code which pulls data from database and based on the different type of tables , store the data in dictionary for further usage.
This code handles around 20-30 different table so there are 20-30 dictionaries and few lists which I have defined as class variables for further usage in code.
for example.
class ImplVars(object):
#dictionary capturing data from Asset-Feed table
general_feed_dict = {}
ports_feed_dict = {}
vulns_feed_dict = {}
app_list = []
...
I want to clear these dictionaries before I add data in it.
Easiest or common way is to use clear() function but this code is repeatable as I will have to write for each dict.
Another option I am exploring is with using dir() function but its returning variable names as string.
Is there any elegant method which will allow me to fetch all these class variables and clear them ?
You can use introspection as you suggest:
for d in filter(dict.__instancecheck__, ImplVars.__dict__.values()):
d.clear()
Or less cryptic, covering lists and dicts:
for obj in ImplVars.__dict__.values():
if isinstance(obj, (list, dict)):
obj.clear()
But I would recommend you choose a bit of a different data structure so you can be more explicit:
class ImplVars(object):
data_dicts = {
"general_feed_dict": {},
"ports_feed_dict": {},
"vulns_feed_dict": {},
}
Now you can explicitly loop over ImplVars.data_dicts.values and still have other class variables that you may not want to clear.
code:
a_dict = {1:2}
b_dict = {2:4}
c_list = [3,6]
vars_copy = vars().copy()
for variable, value in vars_copy.items():
if variable.endswith("_dict"):
vars()[variable] = {}
elif variable.endswith("_list"):
vars()[variable] = []
print(a_dict)
print(b_dict)
print(c_list)
result:
{}
{}
[]
Maybe one of the easier kinds of implementation would be to create a list of dictionaries and lists you want to clear and later make the loop clear them all.
d = [general_feed_dict, ports_feed_dict, vulns_feed_dict, app_list]
for element in d:
element.clear()
You could also use list comprehension for that.
Related
How can i make a set of dictionaries from one list of dictionaries?
Example:
import copy
v1 = {'k01': 'v01', 'k02': {'k03': 'v03', 'k04': {'k05': 'v05'}}}
v2 = {'k11': 'v11', 'k12': {'k13': 'v13', 'k14': {'k15': 'v15'}}}
data = []
N = 5
for i in range(N):
data.append(copy.deepcopy(v1))
data.append(copy.deepcopy(v2))
print data
How would you create a set of dictionaries from the list data?
NS: One dictionary is equal to another when they are structurally the same. That means, they got exactly the same keys and same values (recursively)
A cheap workaround would be to serialize your dicts, for example:
import json
dset = set()
d1 = {'a':1, 'b':{'c':2}}
d2 = {'b':{'c':2}, 'a':1} # the same according to your definition
d3 = {'x': 42}
dset.add(json.dumps(d1, sort_keys=True))
dset.add(json.dumps(d2, sort_keys=True))
dset.add(json.dumps(d3, sort_keys=True))
for p in dset:
print json.loads(p)
In the long run it would make sense to wrap the whole thing in a class like SetOfDicts.
Dictionaries are mutable and therefore not hashable in python.
You could either create a dict-subclass with a __hash__ method. Make sure that the hash of a dictionary does not change while it is in the set (that probably means that you cannot allow modifying the members).
See http://code.activestate.com/recipes/414283-frozen-dictionaries/ for an example implementation of frozendicts.
If you can define a sort order on your (frozen) dictionaries, you could alternatively use a data structure based on a binary tree instead of a set. This boils down to the bisect solution provided in the link below.
See also https://stackoverflow.com/a/18824158/5069869 for an explanation why sets without hash do not make sense.
not exactly what you're looking for as this accounts for lists too but:
def hashable_structure(structure):
if isinstance(structure, dict):
return {k: hashable_structure(v) for k, v in structure.items()}
elif isinstance(structure, list):
return {hashable_structure(elem) for elem in structure)}
else:
return structure
I'm trying to create many dictionaries in a for loop in Python 2.7. I have a list as follows:
sections = ['main', 'errdict', 'excdict']
I want to access these variables, and create new dictionaries with the variable names. I could only access the list sections and store an empty dictionary in the list but not in the respective variables.
for i in enumerate(sections):
sections[i] = dict()
The point of this question is. I'm going to obtain the list sections from a .ini file, and that variable will vary. And I can create an array of dictionaries, but that doesn't work well will the further function requirements. Hence, my doubt.
Robin Spiess answered your question beautifully.
I just want to add the one-liner way:
section_dict = {sec : {} for sec in sections}
For maintaining the order of insertion, you'll need an OrderedDict:
from collections import OrderedDict
section_dict = OrderedDict((sec, {}) for sec in sections)
To clear dictionaries
If the variables in your list are already dictionaries use:
for var in sections:
var.clear()
Note that here var = {} does not work, see Difference between dict.clear() and assigning {} in Python.
To create new dictionaries
As long as you only have a handful of dicts, the best way is probably the easiest one:
main = {} #same meaning as main = dict() but slightly faster
errdict = {}
excdict = {}
sections = [main,errdict,excdict]
The variables need to be declared first before you can put them in a list.
For more dicts I support #dslack's answer in the comments (all credit to him):
sections = [dict() for _ in range(numberOfDictsYouWant)]
If you want to be able to access the dictionaries by name, the easiest way is to make a dictionary of dictionaries:
sectionsdict = {}
for var in sections:
sectionsdict[var] = {}
You might also be interested in: Using a string variable as a variable name
I am very new to python, and I have the following problem. I came up with the following solution. I am wondering whether it is "pythonic" or not. If not, what would be the best solution ?
The problem is :
I have a list of dict
each dict has at least three items
I want to find the position in the list of the dict with specific three values
This is my python example
import collections
import random
# lets build the list, for the example
dicts = []
dicts.append({'idName':'NA','idGroup':'GA','idFamily':'FA'})
dicts.append({'idName':'NA','idGroup':'GA','idFamily':'FB'})
dicts.append({'idName':'NA','idGroup':'GB','idFamily':'FA'})
dicts.append({'idName':'NA','idGroup':'GB','idFamily':'FB'})
dicts.append({'idName':'NB','idGroup':'GA','idFamily':'FA'})
dicts.append({'idName':'NB','idGroup':'GA','idFamily':'FB'})
dicts.append({'idName':'NB','idGroup':'GB','idFamily':'FA'})
dicts.append({'idName':'NB','idGroup':'GB','idFamily':'FB'})
# let's shuffle it, again for example
random.shuffle(dicts)
# now I want to have for each combination the index
# I use a recursive defaultdict definition
# because it permits creating a dict of dict
# even if it is not initialized
def tree(): return collections.defaultdict(tree)
# initiate mapping
mapping = tree()
# fill the mapping
for i,d in enumerate(dicts):
idFamily = d['idFamily']
idGroup = d['idGroup']
idName = d['idName']
mapping[idName][idGroup][idFamily] = i
# I end up with the mapping providing me with the index within
# list of dicts
Looks reasonable to me, but perhaps a little too much. You could instead do:
mapping = {
(d['idName'], d['idGroup'], d['idFamily']) : i
for i, d in enumerate(dicts)
}
Then access it with mapping['NA', 'GA', 'FA'] instead of mapping['NA']['GA']['FA']. But it really depends how you're planning to use the mapping. If you need to be able to take mapping['NA'] and use it as a dictionary then what you have is fine.
I was just wondering if there is a simple way to do this. I have a particular structure that is parsed from a file and the output is a list of a dict of a list of a dict. Currently, I just have a bit of code that looks something like this:
for i in xrange(len(data)):
for j, k in data[i].iteritems():
for l in xrange(len(data[i]['data'])):
for m, n in data[i]['data'][l].iteritems():
dostuff()
I just wanted to know if there was a function that would traverse a structure and internally figure out whether each entry was a list or a dict and if it is a dict, traverse into that dict and so on. I've only been using Python for about a month or so, so I am by no means an expert or even an intermediate user of the language. Thanks in advance for the answers.
EDIT: Even if it's possible to simplify my code at all, it would help.
You never need to iterate through xrange(len(data)). You iterate either through data (for a list) or data.items() (or values()) (for a dict).
Your code should look like this:
for elem in data:
for val in elem.itervalues():
for item in val['data']:
which is quite a bit shorter.
Will, if you're looking to decend an arbitrary structure of array/hash thingies then you can create a function to do that based on the type() function.
def traverse_it(it):
if (isinstance(it, list)):
for item in it:
traverse_it(item)
elif (isinstance(it, dict)):
for key in it.keys():
traverse_it(it[key])
else:
do_something_with_real_value(it)
Note that the average object oriented guru will tell you not to do this, and instead create a class tree where one is based on an array, another on a dict and then have a single function to process each with the same function name (ie, a virtual function) and to call that within each class function. IE, if/else trees based on types are "bad". Functions that can be called on an object to deal with its contents in its own way "good".
I think this is what you're trying to do. There is no need to use xrange() to pull out the index from the list since for iterates over each value of the list. In my example below d1 is therefore a reference to the current data[i].
for d1 in data: # iterate over outer list, d1 is a dictionary
for x in d1: # iterate over keys in d1 (the x var is unused)
for d2 in d1['data']: # iterate over the list
# iterate over (key,value) pairs in inner most dict
for k,v in d2.iteritems():
dostuff()
You're also using the name l twice (intentionally or not), but beware of how the scoping works.
well, question is quite old. however, out of my curiosity, I would like to respond to your question for much better answer which I just tried.
Suppose, dictionary looks like: dict1 = { 'a':5,'b': [1,2,{'a':100,'b':100}], 'dict 2' : {'a':3,'b':5}}
Solution:
dict1 = { 'a':5,'b': [1,2,{'a':100,'b':100}], 'dict 2' : {'a':3,'b':5}}
def recurse(dict):
if type(dict) == type({}):
for key in dict:
recurse(dict[key])
elif type(dict) == type([]):
for element in dict:
if type(element) == type({}):
recurse(element)
else:
print element
else:
print dict
recurse(dict1)
I have a case where I need to construct following structure programmatically (yes I am aware of .setdefault and defaultdict but I can not get what I want)
I basically need a dictionary, with a dictionary of dictionaries created within the loop.
At the beginning the structure is completely blank.
structure sample (please note, I want to create an array that has this structure in the code!)
RULE = {
'hard_failure': {
4514 : {
'f_expr' = 'ABC',
'c_expr' = 'XF0',
}
}
}
pseudo code that needs to create this:
...
self.rules = {}
for row in rows:
a = 'hard_failure'
b = row[0] # 4514
c = row[1] # ABC
d = row[2] # XF0
# Universe collapse right after
self.rules = ????
...
The code above is obviously not working since I dont know how to do it!
Example, that you've posted is not a valid python code, I could only imagine that you're trying to do something like this:
self.rules[a] = [{b:{'f_expr': c, 'c_expr': d}}]
this way self.rules is a dictionary of a list of a dictionary of a dictionary. I bet there is more sane way to do this.
rules = {}
failure = 'hard_failure'
rules[failure] = []
for row in rows:
#this is what people are referring to below. You left out the addition of the dictionary structure to the list.
rules[failure][row[0]] = {}
rules[failure][row[0]]['type 1'] = row[1]
rules[failure][row[0]]['type 2'] = row[2]
This is what I created based on how I understood the questions. I wasn't sure what to call the 'f_expr' and 'c_expr' since you never mention where you get those but I assume they are already know column names in a resultset or structure of some sort.
Just keep adding to the structure as you go.
Your example code doesn't seem to be valid Python. It's not clear if the second level element is supposed to be a list or a dictionary.
However, if you're doing what I think you're doing, and it's a dictionary, you could use a tuple as a key in the top-level dictionary instead of nesting dictionaries:
>>> a = 'hard_failure'
>>> b = 4514
>>> c = "ABC"
>>> d = "XF0"
>>> rules = {}
>>> rules[(a,b)] = {'f_expr' : a,'c_expr' : d}
>>> rules
{('hard_failure', 4514): {'c_expr': 'XF0', 'f_expr': 'hard_failure'}}
My favorite way to deal with nested dictionaries & lists of dictionaries is to use PyYAML. See this response for details.
Well, I apologize for the confusion, I never claimed that code actually compiled, hence (pseudo). Arthur Thomas put me on the right track, here is slightly modified version. (Yes, now its a simply nested dictionary, 3 levels down)
RULE_k = 'hard_failure'
self.rules = {}
for row in rows:
self.rules_compiled.setdefault(RULE_k, {})
self.rules_compiled[RULE_k][row[1]] = {}
self.rules_compiled[RULE_k][row[1]]['f_expr'] = row[0]
self.rules_compiled[RULE_k][row[1]]['c_expr'] = row[1]