We Keep Coding

LEFT JOIN dictionaries in python based on value

#Input
dict_1 = {"conn": {"ts":15,"uid":"ABC","orig_h":"10.10.210.250"}}
dict_2 = {"conn": {"ts":15,"uid":"ABC","orig_h":"10.10.210.252"}}
#Mapper can be modified as required
mapper = {"10.10.210.250":"black","192.168.2.1":"black"}
I am getting each dict in a loop, in each iteration I need to check a dict against the mapper and append a flag based on match between dict_1.orig_h and mapper.10.10.210.250 . I have the flexibility to define the mapper however I need.
So the desired result would be:
dict_1 = {"conn": {"ts":15,"uid":"ABC","orig_h":"10.10.210.250", "class":"black"}}
dict_2 will remain unchanged since there is no matching value in mapper.
This is kinda what I want, but it works only if orig_h is an int
import collections
result = collections.defaultdict(dict)
for d in dict_1:
result[d[int('orig_h')]].update(d)
for d in mapper:
result[d[int('orig_h')]].update(d)

Not much explaining to be done; if the ip is in the mapper dictionary (if mapper has a key which is that ip) then set the desired attribute of the dict to the value of the key in the mapper dict ('black' here).
def update_dict(dic, mapper):
ip = dic['conn']['orig_h']
if ip in mapper:
dic['conn']['class'] = mapper[ip]
which works exactly as desired:
>>> update_dict(dict_1, mapper)
>>> dict_1
{'conn': {'ts': 15, 'uid': 'ABC', 'orig_h': '10.10.210.250', 'class': 'black'}}
>>> update_dict(dict_2, mapper)
>>> dict_2
{'conn': {'ts': 15, 'uid': 'ABC', 'orig_h': '10.10.210.252'}}

Extracting the conn value for simplicity:
conn_data = dict_1['conn']
conn_data['class'] = mapper[conn_data['orig_h']]

A two liner, extracting class and dict if the 'orig_h' is in the mapper dictionary's keys, if it id, keep it, otherwise don't keep it, then create a new dictionary comprehension inside the list comprehension to add 'class' to the dictionary's 'conn' key's dictionary.
l=[(i,mapper[i['conn']['orig_h']]) for i in (dict_1,dict_2) if i['conn']['orig_h'] in mapper]
print([{'conn':dict(a['conn'],**{'class':b})} for a,b in l])
BTW this answer chooses the dictionaries automatically

What exactly does the type lambda do when used with defaultdict

I remember this example
from collections import defaultdict
d_int = defaultdict(int, a=10, b=12, c=13)
d_int.default_factory = lambda: 1
d_int['d']
1
When we pass an unknown key it returns the default value instead of error,I understand this.
But in this SO question,populating a nested dictionary thing get more complicated.
final = collections.defaultdict(lambda: collections.defaultdict(list))
What does lambda type do in this case?

This kind of default dictionary would be useful for 2 levels of data. Something like this:
{ k11 : { k21 : [...], k22 : [...] }, k12 : { ... } }
Here, each dictionary is defaultdict.
The lambda returns a defaultdict for the second level, when the first level key does not exist:
In [234]: final['k11'] # first level access
Out[234]: defaultdict(list, {})
In [235]: final['k11']['k21'] # second level access
Out[235]: []

defaultdict needs a function (the factory).
In the second case, you need to create a collections.defaultdict(list). For that you need a function, and it's simpler to define this function inline with a lambda that generates a new collections.defaultdict(list) object.

How to merge multiple dictionaries

I have main_dict.
main_dict={'name1':{'key1':'value1', 'key2':'value2'}, 'name2':{'key1':'value3', 'key2':'value8'} ... }
I have 2 other dictionaries which brings some more data to be added in the main_dict.
like,
**age_dict= {{'age':'age_value1', 'name': 'name1'}, {'age':'age_value1', 'name': 'name2'}}
gender_dict= {{'gender':'gen_value1', 'name': 'name1'}, {'gender':'gen_value2', 'name': 'name2'}}**
Now i would like to make some loops and merge these dictionaries such that
it checks for the same name and takes values from age and gender dictionaries and create keys 'age' , 'gender' and add them into main_dict.
For now i have done this, but i think django can help to do this in a single way:
for user in age_dict:
for key, value in main_dict.iteritems():
if key == user['name']:
value['age'] = user['age_value']
for user in gender_dict:
for key, value in main_dict.iteritems():
if key == user['name']:
value['gender'] = user['gen_value']
EDIT: Modified age_dict and gender_dict.

General hint: if you are doing something like
for key, val in some_dict.iteritems():
if key == some_value:
do_something(val)
you are most likely doing it wrong, because you are not using the dictionaries very purpose: accessing elements by their keys. Instead, do
do_something(some_dict[key])
and use exceptions if you can't be sure that somedict[key] exists.
You don't have to interate over dictionaries to find the appropriate key. Just access it directly, that's what dictionaries are for:
main_dict={'name1':{'key1':'value1', 'key2':'value2'}, 'name2':{'key1':'value3', 'key2':'value8'}}
age_dicts = [{'age':'age_value1', 'name': 'name1'}, 'age':'age_value1', 'name': 'name2'}]
gender_dicts = [{'gender':'gen_value1', 'name': 'name1'}, 'gender':'gen_value2', 'name': 'name2'}]
for dct in age_dicts:
main_dict[dct['name']]['age'] = dct['age']
for dct in gender_dicts:
main_dict[dct['name']]['gender'] = dct['gender']
Specific answer to the pre-edit case:
age_dict= {'name1':'age_value1', 'name2':'age_value2'}
gender_dict= {'name1':'gen_value1', 'name2':'gen_value2'}
If you are sure that gender_dict and age_dict provide values for each name, it's as easy as
for name, dct in main_dict.iteritems():
dct['age'] = age_dict[name]
dct['gender'] = gender_dict[name]
If there are names without entries in the other dictionaries, you can use exceptions:
for name, dct in main_dict.iteritems():
try:
dct['age'] = age_dict[name]
except KeyError: # no such name in age_dict
pass
try:
dct['gender'] = gender_dict[name]
except KeyError: # no such name in gender_dict
pass

The setdefault method of dict looks up a key, and returns the value if found. If not found, it returns a default, and also assigns that default to the key.
super_dict = {}
for d in dicts:
for k, v in d.iteritems():
super_dict.setdefault(k, []).append(v)
Also, you might consider using a defaultdict. This just automates setdefault by calling a function to return a default value when a key isn't found.
import collections
super_dict = collections.defaultdict(list)
for d in dicts:
for k, v in d.iteritems():
super_dict[k].append(v)
Also, as Sven Marnach astutely observed, you seem to want no duplication of values in your lists. In that case, set gets you what you want:
import collections
super_dict = collections.defaultdict(set)
for d in dicts:
for k, v in d.iteritems():
super_dict[k].add(v)

So you want use age_dict and gender_dict to enrich the values for the keys in main_dict. Well, given Python guarantees average dict lookup to be constant you are constrained only by the number of keys in main_dict and you can reach the enrichment in O(n) where n is the size of the dictionary:
for user_name, user_info in main_dict.items():
if user_name in gender_dict:
user_info['gender'] = gender_dict[user_name]
if user_name in age_dic:
user_info['age'] = age_dict[user_name]
And a fancy function doing this in a generic way:
def enrich(target, **complements):
for user_name, user_info in target.items():
for complement_key, complemented_users in complements.items():
if user_name in complemented_users:
user_info[complement_key] = complemented_users[user_name]
enrich(main_dict, age=age_dict, gender=gender_dict)
Even if you see two nested loops, it is more likely the number of users in main_dict dominates over the number of complementary dictionaries.

Initializing a dictionary in python with a key value and no corresponding values

I was wondering if there was a way to initialize a dictionary in python with keys but no corresponding values until I set them. Such as:
Definition = {'apple': , 'ball': }
and then later i can set them:
Definition[key] = something
I only want to initialize keys but I don't know the corresponding values until I have to set them later. Basically I know what keys I want to add the values as they are found. Thanks.

Use the fromkeys function to initialize a dictionary with any default value. In your case, you will initialize with None since you don't have a default value in mind.
empty_dict = dict.fromkeys(['apple','ball'])
this will initialize empty_dict as:
empty_dict = {'apple': None, 'ball': None}
As an alternative, if you wanted to initialize the dictionary with some default value other than None, you can do:
default_value = 'xyz'
nonempty_dict = dict.fromkeys(['apple','ball'],default_value)

You could initialize them to None.

you could use a defaultdict. It will let you set dictionary values without worrying if the key already exists. If you access a key that has not been initialized yet it will return a value you specify (in the below example it will return None)
from collections import defaultdict
your_dict = defaultdict(lambda : None)

It would be good to know what your purpose is, why you want to initialize the keys in the first place. I am not sure you need to do that at all.
1) If you want to count the number of occurrences of keys, you can just do:
Definition = {}
# ...
Definition[key] = Definition.get(key, 0) + 1
2) If you want to get None (or some other value) later for keys that you did not encounter, again you can just use the get() method:
Definition.get(key) # returns None if key not stored
Definition.get(key, default_other_than_none)
3) For all other purposes, you can just use a list of the expected keys, and check if the keys found later match those.
For example, if you only want to store values for those keys:
expected_keys = ['apple', 'banana']
# ...
if key_found in expected_keys:
Definition[key_found] = value
Or if you want to make sure all expected keys were found:
assert(all(key in Definition for key in expected_keys))

You can initialize the values as empty strings and fill them in later as they are found.
dictionary = {'one':'','two':''}
dictionary['one']=1
dictionary['two']=2

Comprehension could be also convenient in this case:
# from a list
keys = ["k1", "k2"]
d = {k:None for k in keys}
# or from another dict
d1 = {"k1" : 1, "k2" : 2}
d2 = {k:None for k in d1.keys()}
d2
# {'k1': None, 'k2': None}

q = input("Apple")
w = input("Ball")
Definition = {'apple': q, 'ball': w}

Based on the clarifying comment by #user2989027, I think a good solution is the following:
definition = ['apple', 'ball']
data = {'orange':1, 'pear':2, 'apple':3, 'ball':4}
my_data = {}
for k in definition:
try:
my_data[k]=data[k]
except KeyError:
pass
print my_data
I tried not to do anything fancy here. I setup my data and an empty dictionary. I then loop through a list of strings that represent potential keys in my data dictionary. I copy each value from data to my_data, but consider the case where data may not have the key that I want.

Why does this Python list of dictionary only have the keys of the dictionary?

I wrote a module, which is serving as a config file:
users = list(
dict(
username="x",
password="y"
)
)
However, when I inspect the imported contents of the list, it contains the keys of the dictionary, not a dictionary entry:
>>> import user as userdata
import user as userdata
>>> userdata.users
userdata.users
['username', 'password']
>>>

You are not making a list of dicts, you are making a list of dictionary keys from the dictionary, list(dict(...)) returns a list of keys:
>>> d = dict(username="x", password="y")
>>> list(d)
['username', 'password']
Probably you want to define users this way:
users = [dict(username="x", password="y")]
or
users = [{'username': 'x', 'password': 'y'}]

You are creating the list from a dictionary. The list constructor will iterate over the dict, which just gives the keys.
If you want a list that contains the dict as one element, you can use:
[dict(username='test')]

Try with:
users = [dict(username="x",password="y")]
print users
if you want to have array of dictionaries.

Its because when you call list on a dictionary, it makes a new list of only your keys, which is the same as using the method call of yourdictionary.keys().

I can't understand why would you use dict( ) when you can simply use { }..
You know the dictionary functions? If not I'd recommend you to try the dir() function.
There are some functions in there that you can use:
dictionary.keys(), returns a list of all the keys in the dictionary.
dictionary.values(), returns a list of all the values in the dictionary.
dictionary.items(), returns a list which in each cell there are 2 'variables', [0]=key, [1]=value
Example:
d = { "First":1, "Second":2 }
d.keys()
>> [ "First", "Second" ]
d.values()
>> [ 1, 2 ]
d.items()
>> [ ("First", 1), ("Second", 2) ]