We Keep Coding

How to reset value of multiple dictionaries elegantly in python

I am working on a code which pulls data from database and based on the different type of tables , store the data in dictionary for further usage.
This code handles around 20-30 different table so there are 20-30 dictionaries and few lists which I have defined as class variables for further usage in code.
for example.
class ImplVars(object):
#dictionary capturing data from Asset-Feed table
general_feed_dict = {}
ports_feed_dict = {}
vulns_feed_dict = {}
app_list = []
...
I want to clear these dictionaries before I add data in it.
Easiest or common way is to use clear() function but this code is repeatable as I will have to write for each dict.
Another option I am exploring is with using dir() function but its returning variable names as string.
Is there any elegant method which will allow me to fetch all these class variables and clear them ?

You can use introspection as you suggest:
for d in filter(dict.__instancecheck__, ImplVars.__dict__.values()):
d.clear()
Or less cryptic, covering lists and dicts:
for obj in ImplVars.__dict__.values():
if isinstance(obj, (list, dict)):
obj.clear()
But I would recommend you choose a bit of a different data structure so you can be more explicit:
class ImplVars(object):
data_dicts = {
"general_feed_dict": {},
"ports_feed_dict": {},
"vulns_feed_dict": {},
}
Now you can explicitly loop over ImplVars.data_dicts.values and still have other class variables that you may not want to clear.

code:
a_dict = {1:2}
b_dict = {2:4}
c_list = [3,6]
vars_copy = vars().copy()
for variable, value in vars_copy.items():
if variable.endswith("_dict"):
vars()[variable] = {}
elif variable.endswith("_list"):
vars()[variable] = []
print(a_dict)
print(b_dict)
print(c_list)
result:
{}
{}
[]

Maybe one of the easier kinds of implementation would be to create a list of dictionaries and lists you want to clear and later make the loop clear them all.
d = [general_feed_dict, ports_feed_dict, vulns_feed_dict, app_list]
for element in d:
element.clear()
You could also use list comprehension for that.

Can you create dictionaries with just a single function?

As a total beginner I'm quite proud of this function. Although I believe there's probably an easier, more pythonic way of doing the exact same thing:
Genes = ['Gen1', 'Gen2', 'Gen3']
Mutations = ['Gen1.A', 'Gen1.B', 'Gen2.A', 'Gen3.A', 'Gen3.B', 'Gen3.C']
def RawDict(keys, values):
dictKeys = []
dictValues = []
for key in keys:
keyVal = []
for value in values:
if value.find(key) == -1:
pass
else:
keyVal.append(value)
dictKeys.append(key)
dictValues.append(keyVal)
return zip(dictKeys, dictValues)
GenDict = dict(RawDict(Genes, Mutations))
print(GenDict)
The function above is a rather overcomplicated (I think) way of putting several values (mutations) within keys (genes). However I was wondering if I could tweak this so I could get a dictionary by just doing this:
dict(GenDict, Genes, Mutations)
print(GenDict)
My struggle involves that when I use dict within the function, this won't work:
Genes = ['Gen1', 'Gen2', 'Gen3']
Mutations = ['Gen1.A', 'Gen1.B', 'Gen2.A', 'Gen3.A', 'Gen3.B', 'Gen3.C']
def fullDict(dictName, keys, values):
dictKeys = []
dictValues = []
for key in keys:
keyVal = []
for value in values:
if value.find(key) == -1:
pass
else:
keyVal.append(value)
dictKeys.append(key)
dictValues.append(keyVal)
dictName = dict(RawDict(Genes, Mutations))
fullDict(GenDict, Genes, Mutations)
print(GenDict)
The above just won't work as GenDict is not defined.

From what I understand, you want to move from this:
gen_dict = make_dictionary(genes, mutations)
to this:
make_dictionary(gen_dict, genes, mutations)
where the make_dictionary function "creates" the variable gen_dict.
Unfortunately, this isn't really how variables work. If you want to define a variable called GenDict, the way to do this is to use GenDict = .... You could do something like this:
gen_dict = {}
fill_dictionary(gen_dict, genes, mutations)
This creates a variable called gen_dict and assigns it to a new, empty dictionary. Your function would then go through and add things to that dictionary:
def fill_dictionary(d, genes, mutations):
for g in genes:
d[g] = [m for m in mutations if m.startswith(g)]
But calling a function cannot cause a new variable to appear in the caller's scope. (This is not completely true, because of globals(), but for most intents and purposes, it is.)
(By the way, there is a one-liner that will create the dictionary: dictionary = { g : [m for m in mutations if m.startswith(g+".")] for g in genes }. Search for list comprehensions and dictionary comprehensions on Google or StackOverflow -- they are amazing!)

I am assuming that you want the "Gen"s to be stored by the numerical value that it contains.
Genes = ['Gen1', 'Gen2', 'Gen3']
Mutations = ['Gen1.A', 'Gen1.B', 'Gen2.A', 'Gen3.A', 'Gen3.B', 'Gen3.C']
the_dict = {i:[] for i in Genes}
for i in Mutations:
new_val = i.split(".")
the_dict[new_val[0]].append(i)
print(the_dict)
Output:
{'Gen2': ['Gen2.A'], 'Gen3': ['Gen3.A', 'Gen3.B', 'Gen3.C'], 'Gen1': ['Gen1.A', 'Gen1.B']}

I assume you have a background in programming in some other language than Python; a language that lets you change function parameters. Well, Python does not. The problem is not with the use of dict, but rather with the fact that you're assigning to a function parameter. This will not have an effect outside the function. What you want to do is probably this:
def fullDict(keys, values):
return { key: [ value for value in values if key in value] for key in keys }
print(fullDict(Genes, Mutations))

Checking items in a list of dictionaries in python

I have a list of dictionaries=
a = [{"ID":1, "VALUE":2},{"ID":2, "VALUE":2},{"ID":3, "VALUE":4},...]
"ID" is a unique identifier for each dictionary. Considering the list is huge, what is the fastest way of checking if a dictionary with a certain "ID" is in the list, and if not append to it? And then update its "VALUE" ("VALUE" will be updated if the dict is already in list, otherwise a certain value will be written)

You'd not use a list. Use a dictionary instead, mapping ids to nested dictionaries:
a = {
1: {'VALUE': 2, 'foo': 'bar'},
42: {'VALUE': 45, 'spam': 'eggs'},
}
Note that you don't need to include the ID key in the nested dictionary; doing so would be redundant.
Now you can simply look up if a key exists:
if someid in a:
a[someid]['VALUE'] = newvalue
I did make the assumption that your ID keys are not necessarily sequential numbers. I also made the assumption you need to store other information besides VALUE; otherwise just a flat dictionary mapping ID to VALUE values would suffice.
A dictionary lets you look up values by key in O(1) time (constant time independent of the size of the dictionary). Lists let you look up elements in constant time too, but only if you know the index.
If you don't and have to scan through the list, you have a O(N) operation, where N is the number of elements. You need to look at each and every dictionary in your list to see if it matches ID, and if ID is not present, that means you have to search from start to finish. A dictionary will still tell you in O(1) time that the key is not there.

If you can, convert to a dictionary as the other answers suggest, but in case you you have reason* to not change the data structure storing your items, here's what you can do:
items = [{"ID":1, "VALUE":2}, {"ID":2, "VALUE":2}, {"ID":3, "VALUE":4}]
def set_value_by_id(id, value):
# Try to find the item, if it exists
for item in items:
if item["ID"] == id:
break
# Make and append the item if it doesn't exist
else: # Here, `else` means "if the loop terminated not via break"
item = {"ID": id}
items.append(id)
# In either case, set the value
item["VALUE"] = value
* Some valid reasons I can think of include preserving the order of items and allowing duplicate items with the same id. For ways to make dictionaries work with those requirements, you might want to take a look at OrderedDict and this answer about duplicate keys.

Convert your list into a dict and then checking for values is much more efficient.
d = dict((item['ID'], item['VALUE']) for item in a)
for new_key, new_value in new_items:
if new_key not in d:
d[new_key] = new_value

Also need to update on key found:
d = dict((item['ID'], item['VALUE']) for item in a)
for new_key, new_value in new_items:
d.setdefault(new_key, 0)
d[new_key] = new_value

Answering the question you asked, without changing the datastructure around, there's no real faster way of looking without a loop and checking every element and doing a dictionary lookup for each one - but you can push the loop down to the Python runtime instead of using Python's for loop.
I haven't tried if it ends up faster though.
a = [{"ID":1, "VALUE":2},{"ID":2, "VALUE":2},{"ID":3, "VALUE":4}]
id = 2
tmp = filter(lambda d: d['ID']==id, a)
# the filter will either return an empty list, or a list of one item.
if not tmp:
tmp = {"ID":id, "VALUE":"default"}
a.append(tmp)
else:
tmp = tmp[0]
# tmp is bound to the found/new dictionary

python dictionary keys

I'm having a bit of a problem, don't even know if this is doable. I have a number of launchers that each have a interface defined with the expected inputs. The input values come as a dictionary. For example:
dict_key = str(req_input[0]['name'])
data = dict(dict_key = value)
Now req_input[0]['name'] is the key I would like to insert into the dictionary. Now I know what I'm doint here only creates a dictionary of the form {'dict_key' : value} but I was wondering if it is possible to create the dicionary as to be the form {'Actual value of dict_key' : value}
Regards,
Bogdan

The best way to do this is simply using the dict literal {}:
data = {dict_key: value}
Other ways would be
data = dict({dict_key: value})
or
data = dict()
data[dict_key] = value
but these are longer so stay with the first one.

Another way is:
data = {}
data[req_input[0]['name']] = value
This way you can add multiple values to the same dictionary, or loop through req_input if you have multiple parameters there, i.e.:
data = {}
for input in req_input:
data[input['name']] = value

List of dictionaries, in a dictionary - in Python

I have a case where I need to construct following structure programmatically (yes I am aware of .setdefault and defaultdict but I can not get what I want)
I basically need a dictionary, with a dictionary of dictionaries created within the loop.
At the beginning the structure is completely blank.
structure sample (please note, I want to create an array that has this structure in the code!)
RULE = {
'hard_failure': {
4514 : {
'f_expr' = 'ABC',
'c_expr' = 'XF0',
}
}
}
pseudo code that needs to create this:
...
self.rules = {}
for row in rows:
a = 'hard_failure'
b = row[0] # 4514
c = row[1] # ABC
d = row[2] # XF0
# Universe collapse right after
self.rules = ????
...
The code above is obviously not working since I dont know how to do it!

Example, that you've posted is not a valid python code, I could only imagine that you're trying to do something like this:
self.rules[a] = [{b:{'f_expr': c, 'c_expr': d}}]
this way self.rules is a dictionary of a list of a dictionary of a dictionary. I bet there is more sane way to do this.

rules = {}
failure = 'hard_failure'
rules[failure] = []
for row in rows:
#this is what people are referring to below. You left out the addition of the dictionary structure to the list.
rules[failure][row[0]] = {}
rules[failure][row[0]]['type 1'] = row[1]
rules[failure][row[0]]['type 2'] = row[2]
This is what I created based on how I understood the questions. I wasn't sure what to call the 'f_expr' and 'c_expr' since you never mention where you get those but I assume they are already know column names in a resultset or structure of some sort.
Just keep adding to the structure as you go.

Your example code doesn't seem to be valid Python. It's not clear if the second level element is supposed to be a list or a dictionary.
However, if you're doing what I think you're doing, and it's a dictionary, you could use a tuple as a key in the top-level dictionary instead of nesting dictionaries:
>>> a = 'hard_failure'
>>> b = 4514
>>> c = "ABC"
>>> d = "XF0"
>>> rules = {}
>>> rules[(a,b)] = {'f_expr' : a,'c_expr' : d}
>>> rules
{('hard_failure', 4514): {'c_expr': 'XF0', 'f_expr': 'hard_failure'}}

My favorite way to deal with nested dictionaries & lists of dictionaries is to use PyYAML. See this response for details.

Well, I apologize for the confusion, I never claimed that code actually compiled, hence (pseudo). Arthur Thomas put me on the right track, here is slightly modified version. (Yes, now its a simply nested dictionary, 3 levels down)
RULE_k = 'hard_failure'
self.rules = {}
for row in rows:
self.rules_compiled.setdefault(RULE_k, {})
self.rules_compiled[RULE_k][row[1]] = {}
self.rules_compiled[RULE_k][row[1]]['f_expr'] = row[0]
self.rules_compiled[RULE_k][row[1]]['c_expr'] = row[1]