I am making a group chatting app and I have images associated with the users, so whenever they say something, their image is displayed next to it. I wrote the server in python and the client will be an iOS app. I use a dictionary to store all of the message/image pairs. Whenever my iOS app sends a command to the server (msg:<message), the dictionary adds the image and message to the dictionary like so:dictionary[message] = imageName, which is converted to lists then strings to be sent off in a socket. I would like to add the incoming messages to the start of the dictionary, instead of the end. Something like
#When added to end:
dictionary = {"hello":image3.png}
#new message
dictionary = {"hello":image3.png, "i like py":image1.png}
#When added to start:
dictionary = {"hello":image3.png}
#new message
dictionary = {"i like py":image1.png, "hello":image3.png}
Is there any way to add the object to the start of the dictionary?
For the use case described it sounds like a list of tuples would be a better data structure.
However, it has been possible to order a dictionary since Python 3.7. Dictionaries are now ordered by insertion order.
To add an element anywhere other than the end of a dictionary, you need to re-create the dictionary and insert the elements in order. This is pretty simple if you want to add an entry to the start of the dictionary.
# Existing data structure
old_dictionary = {"hello": "image3.png"}
# Create a new dictionary with "I like py" at the start, then
# everything from the old data structure.
new_dictionary = {"i like py": "image1.png"}
new_dictionary.update(old_dictionary)
# new_dictionary is now:
# {'i like py': 'image1.png', 'hello': 'image3.png'}
(python3) good example from Manjeet on geeksforgeeks.org
test_dict = {"Gfg" : 5, "is" : 3, "best" : 10}
updict = {"pre1" : 4, "pre2" : 8}
# ** operator for packing and unpacking items in order
res = {**updict, **test_dict}
First of all it doesn't added the item at the end of dictionary because dictionaries use hash-table to storing their elements and are unordered. if you want to preserve the order you can use collections.OrderedDict.but it will appends the item at the end of your dictionary. One way is appending that item to the fist of your items then convert it to an Orderd:
>>> from collections import OrderedDict
>>> d=OrderedDict()
>>> for i,j in [(1,'a'),(2,'b')]:
... d[i]=j
...
>>> d
OrderedDict([(1, 'a'), (2, 'b')])
>>> d=OrderedDict([(3,'t')]+d.items())
>>> d
OrderedDict([(3, 't'), (1, 'a'), (2, 'b')])
Also as another efficient way if it's not necessary to use a dictionary you can use a deque that allows you to append from both side :
>>> from collections import deque
>>> d=deque()
>>> d.append((1,'a'))
>>> d.append((4,'t'))
>>> d
deque([(1, 'a'), (4, 't')])
>>> d.appendleft((8,'p'))
>>> d
deque([(8, 'p'), (1, 'a'), (4, 't')])
I am not sure that a dictionary is the best data structure for your data, but you may find useful collections.OderedDict. It is basically a dictionary that remembers the order of keys added to the dictionary in a FIFO fashion (which is the opposite of what you need).
If you want to retrieve all your items starting from the most recent one, you can use reversed() to reverse dictionary iterators. You can also use the method popitem() to retrieve (and remove) from the dictionary the key-value pair you last entered.
Link to the docs: https://docs.python.org/2/library/collections.html#collections.OrderedDict
As others have pointed out, there is no concept of "order" in a standard dict. Although you can use an OrderedDict to add ordering behavior, this brings other considerations -- unlike a normal dict, isn't a portable data structure (e.g. dumping to JSON then reloading does not preserve order) -- and isn't available in the standard library for all versions of Python.
You might be better off using a sequential key in a standard dict -- a count-up index, or time stamp -- to keep things simple.
As others have pointed out, there is no "order" in a dictionary. However, if you are like me and just need a temporary (/ hacky) workaround that works for your purposes. There is a way to do this.
You can iterate the dictionary, and append the item at the beginning of the iteration process. This seemed to work for me.
The relevant part is where new_fields is declared. I am including the rest for context.
userprofile_json = Path(__file__).parents[2] / f"data/seed-data/user-profile.json"
with open(userprofile_json) as f:
user_profiles = json.load(f)
for user_profile in user_profiles:
new_fields = {
'user': user_profile['fields']['username'],
}
for k, v in user_profile['fields'].items():
new_fields[k] = v
user_profile['fields'] = new_fields
with open(userprofile_json, 'w') as f:
json.dump(user_profiles, f, indent=4)
Related
Let's say we have a Python dictionary d, and we're iterating over it like so:
for k, v in d.iteritems():
del d[f(k)] # remove some item
d[g(k)] = v # add a new item
(f and g are just some black-box transformations.)
In other words, we try to add/remove items to d while iterating over it using iteritems.
Is this well defined? Could you provide some references to support your answer?
See also How to avoid "RuntimeError: dictionary changed size during iteration" error? for the separate question of how to avoid the problem.
Alex Martelli weighs in on this here.
It may not be safe to change the container (e.g. dict) while looping over the container.
So del d[f(k)] may not be safe. As you know, the workaround is to use d.copy().items() (to loop over an independent copy of the container) instead of d.iteritems() or d.items() (which use the same underlying container).
It is okay to modify the value at an existing index of the dict, but inserting values at new indices (e.g. d[g(k)] = v) may not work.
It is explicitly mentioned on the Python doc page (for Python 2.7) that
Using iteritems() while adding or deleting entries in the dictionary may raise a RuntimeError or fail to iterate over all entries.
Similarly for Python 3.
The same holds for iter(d), d.iterkeys() and d.itervalues(), and I'll go as far as saying that it does for for k, v in d.items(): (I can't remember exactly what for does, but I would not be surprised if the implementation called iter(d)).
You cannot do that, at least with d.iteritems(). I tried it, and Python fails with
RuntimeError: dictionary changed size during iteration
If you instead use d.items(), then it works.
In Python 3, d.items() is a view into the dictionary, like d.iteritems() in Python 2. To do this in Python 3, instead use d.copy().items(). This will similarly allow us to iterate over a copy of the dictionary in order to avoid modifying the data structure we are iterating over.
I have a large dictionary containing Numpy arrays, so the dict.copy().keys() thing suggested by #murgatroid99 was not feasible (though it worked). Instead, I just converted the keys_view to a list and it worked fine (in Python 3.4):
for item in list(dict_d.keys()):
temp = dict_d.pop(item)
dict_d['some_key'] = 1 # Some value
I realize this doesn't dive into the philosophical realm of Python's inner workings like the answers above, but it does provide a practical solution to the stated problem.
The following code shows that this is not well defined:
def f(x):
return x
def g(x):
return x+1
def h(x):
return x+10
try:
d = {1:"a", 2:"b", 3:"c"}
for k, v in d.iteritems():
del d[f(k)]
d[g(k)] = v+"x"
print d
except Exception as e:
print "Exception:", e
try:
d = {1:"a", 2:"b", 3:"c"}
for k, v in d.iteritems():
del d[f(k)]
d[h(k)] = v+"x"
print d
except Exception as e:
print "Exception:", e
The first example calls g(k), and throws an exception (dictionary changed size during iteration).
The second example calls h(k) and throws no exception, but outputs:
{21: 'axx', 22: 'bxx', 23: 'cxx'}
Which, looking at the code, seems wrong - I would have expected something like:
{11: 'ax', 12: 'bx', 13: 'cx'}
Python 3 you should just:
prefix = 'item_'
t = {'f1': 'ffw', 'f2': 'fca'}
t2 = dict()
for k,v in t.items():
t2[k] = prefix + v
or use:
t2 = t1.copy()
You should never modify original dictionary, it leads to confusion as well as potential bugs or RunTimeErrors. Unless you just append to the dictionary with new key names.
This question asks about using an iterator (and funny enough, that Python 2 .iteritems iterator is no longer supported in Python 3) to delete or add items, and it must have a No as its only right answer as you can find it in the accepted answer. Yet: most of the searchers try to find a solution, they will not care how this is done technically, be it an iterator or a recursion, and there is a solution for the problem:
You cannot loop-change a dict without using an additional (recursive) function.
This question should therefore be linked to a question that has a working solution:
How can I remove a key:value pair wherever the chosen key occurs in a deeply nested dictionary? (= "delete")
Also helpful as it shows how to change the items of a dict on the run: How can I replace a key:value pair by its value wherever the chosen key occurs in a deeply nested dictionary? (= "replace").
By the same recursive methods, you will also able to add items as the question asks for as well.
Since my request to link this question was declined, here is a copy of the solution that can delete items from a dict. See How can I remove a key:value pair wherever the chosen key occurs in a deeply nested dictionary? (= "delete") for examples / credits / notes.
import copy
def find_remove(this_dict, target_key, bln_overwrite_dict=False):
if not bln_overwrite_dict:
this_dict = copy.deepcopy(this_dict)
for key in this_dict:
# if the current value is a dict, dive into it
if isinstance(this_dict[key], dict):
if target_key in this_dict[key]:
this_dict[key].pop(target_key)
this_dict[key] = find_remove(this_dict[key], target_key)
return this_dict
dict_nested_new = find_remove(nested_dict, "sub_key2a")
The trick
The trick is to find out in advance whether a target_key is among the next children (= this_dict[key] = the values of the current dict iteration) before you reach the child level recursively. Only then you can still delete a key:value pair of the child level while iterating over a dictionary. Once you have reached the same level as the key to be deleted and then try to delete it from there, you would get the error:
RuntimeError: dictionary changed size during iteration
The recursive solution makes any change only on the next values' sub-level and therefore avoids the error.
I got the same problem and I used following procedure to solve this issue.
Python List can be iterate even if you modify during iterating over it.
so for following code it will print 1's infinitely.
for i in list:
list.append(1)
print 1
So using list and dict collaboratively you can solve this problem.
d_list=[]
d_dict = {}
for k in d_list:
if d_dict[k] is not -1:
d_dict[f(k)] = -1 # rather than deleting it mark it with -1 or other value to specify that it will be not considered further(deleted)
d_dict[g(k)] = v # add a new item
d_list.append(g(k))
Today I had a similar use-case, but instead of simply materializing the keys on the dictionary at the beginning of the loop, I wanted changes to the dict to affect the iteration of the dict, which was an ordered dict.
I ended up building the following routine, which can also be found in jaraco.itertools:
def _mutable_iter(dict):
"""
Iterate over items in the dict, yielding the first one, but allowing
it to be mutated during the process.
>>> d = dict(a=1)
>>> it = _mutable_iter(d)
>>> next(it)
('a', 1)
>>> d
{}
>>> d.update(b=2)
>>> list(it)
[('b', 2)]
"""
while dict:
prev_key = next(iter(dict))
yield prev_key, dict.pop(prev_key)
The docstring illustrates the usage. This function could be used in place of d.iteritems() above to have the desired effect.
I have a dictionary currently setup as
{'name': 'firm', 'name':'firm', etc},
Where keys are analyst names and values are analyst firms.
I am trying to create a new dictionary where the new values are the old k,v pairs and the associated key is simply the index (1, 2, 3, 4, etc).
Current code is below:
num_analysts = len(analysts.keys())
for k,v in analysts.items():
analysts_dict = dict.fromkeys(range(num_analysts), [k,v])
Current result
Each numeric key is getting given the same value (old k,v pair). What is wrong with my expression?
You can enumerate the items and convert them to a dictionary. However, dictionaries, in general, are not ordered. This means that the keys may be assigned essentially randomly.
dict(enumerate(analysts.items(), 1))
#{1: ('name1', 'firm1'), 2: ('name2', 'firm2')}
Enumerate and dictionary comprehension for this
d = {'name1': 'firm1', 'name2': 'firm2'}
d2 = {idx: '{}, {}'.format(item, d[item]) for idx, item in enumerate(d, start = 1)}
{1: 'name1, firm1', 2: 'name2, firm2'}
There are already effective answer posted by others. So I may just put the reason why your own solution does't work properly. It may caused by lazy binding. There are good resource on: http://quickinsights.io/python/python-closures-and-late-binding/
Because late binding will literally pick up the last one in dictionary you created. But this last one is not "virtually last one", it is determined by the OS. (Other people already give some explanation on dict data-structure.)
For each time you run in python command line the result may change. If you put the code in .py file, For each time you run in IDE, the result will be same.(always the last one in dict)
During each iteration, analysts_dict is assigned value based on the result of dict.items().
However, you should use comprehension to generate the final result in one line,
E.g. [{i: e} for i, e in enumerate(analysts.items())]
analysts = {
"a": 13,
"b": 123,
"c": 1234
}
num_analysts = len(analysts.keys())
analysts_dict = [{i: e} for i, e in enumerate(analysts.items())]
print(analysts_dict)
>> [{0: ('a', 13)}, {1: ('b', 123)}, {2: ('c', 1234)}]
This code
for k,v in analysts.items():
analysts_dict = dict.fromkeys(range(num_analysts), [k,v])
loops over the original dict and on each loop iteration it creates a new dict using the range numbers as the keys. By the way, every item in that dict shares a reference to a single [k, v] list object. That's generally a bad idea. You should only use an immutable object (eg None, a number, or a string) as the value arg to the dict.fromkeys method. The purpose of the method is to allow you to create a dict with a simple default value for the keys you supply, you can't use it to make a dict with lists as the values if you want those lists to be separate lists.
The new dict object is bound to the name analysts_dict. On the next loop iteration, a new dict is created and bound to that name, replacing the one just created on the previous loop, and the replaced dict is destroyed.
So you end up with an analysts_dict containing a bunch of references to the final [k, v] pair read from the original dict.
To get your desired result, you should use DYZ's code, which I won't repeat here. Note that it stores the old name & firm info in tuples, which is better than using lists for this application.
For hashable objects inside a dict I could easily pair down duplicate values store in a dict using a set. For example:
a = {'test': 1, 'key': 1, 'other': 2}
b = set(a.values())
print(b)
Would display [1,2]
Problem I have is I am using a dict to store mapping between variable keys in __dict__ and the corresponding processing functions that will be passed to an engine to order and process those functions, some of these functions may be fast some may be slower due to accessing an API. The problem is each function may use multiple variable, therefor need multiple mappings in the dict. I'm wondering if there is a way to do this or if I am stuck writing my own solution?
Ended up building a callable class, since caching could speed things up for me:
from collections.abc import Callable
class RemoveDuplicates(Callable):
input_cache = []
output_cache = []
def __call__(self, in_list):
if list in self.input_cache:
idx = self.input_cache.index(in_list)
return self.output_cache[idx]
else:
self.input_cache.append(in_list)
out_list = self._remove_duplicates(in_list)
self.output_cache.append(out_list)
return out_list
def _remove_duplicates(self, src_list):
result = []
for item in src_list:
if item not in result:
result.append(item)
return result
If the objects can be ordered, you can use itertools.groupby to eliminate the duplicates:
>>> a = {'test': 1, 'key': 1, 'other': 2}
>>> b = [k for k, it in itertools.groupby(sorted(a.values()))]
>>> print(b)
[1, 2]
Is there something simple like a set for un-hashable objects
Not in the standard library but you need to look beyond and search for BTree implementation of dictionary. I googled and found few hits where the first one (BTree)seems promising and interesting
Quoting from the wiki
The BTree-based data structures differ from Python dicts in several
fundamental ways. One of the most important is that while dicts
require that keys support hash codes and equality comparison, the
BTree-based structures don’t use hash codes and require a total
ordering on keys.
Off-course its trivial fact that a set can be implemented as a dictionary where the value is unused.
You could (indirectly) use the bisect module to create sorted collection of your values which would greatly speed-up the insertion of new values and value membership testing in general — which together can be utilized to unsure that only unique values get put into it.
In the code below, I've used un-hashable set values for the sake of illustration.
# see http://code.activestate.com/recipes/577197-sortedcollection
from sortedcollection import SortedCollection
a = {'test': {1}, 'key': {1}, 'other': {2}}
sc = SortedCollection()
for value in a.values():
if value not in sc:
sc.insert(value)
print(list(sc)) # --> [{1}, {2}]
What's the shortest way to get first item of OrderedDict in Python 3?
My best:
list(ordered_dict.items())[0]
Quite long and ugly.
I can think of:
next(iter(ordered_dict.items())) # Fixed, thanks Ashwini
But it's not very self-describing.
Any better suggestions?
Programming Practices for Readabililty
In general, if you feel like code is not self-describing, the usual solution is to factor it out into a well-named function:
def first(s):
'''Return the first element from an ordered collection
or an arbitrary element from an unordered collection.
Raise StopIteration if the collection is empty.
'''
return next(iter(s))
With that helper function, the subsequent code becomes very readable:
>>> extension = {'xml', 'html', 'css', 'php', 'xhmtl'}
>>> one_extension = first(extension)
Patterns for Extracting a Single Value from Collection
The usual ways to get an element from a set, dict, OrderedDict, generator, or other non-indexable collection are:
for value in some_collection:
break
and:
value = next(iter(some_collection))
The latter is nice because the next() function lets you specify a default value if collection is empty or you can choose to let it raise an exception. The next() function is also explicit that it is asking for the next item.
Alternative Approach
If you actually need indexing and slicing and other sequence behaviors (such as indexing multiple elements), it is a simple matter to convert to a list with list(some_collection) or to use [itertools.islice()][2]:
s = list(some_collection)
print(s[0], s[1])
s = list(islice(n, some_collection))
print(s)
Use popitem(last=False), but keep in mind that it removes the entry from the dictionary, i.e. is destructive.
from collections import OrderedDict
o = OrderedDict()
o['first'] = 123
o['second'] = 234
o['third'] = 345
first_item = o.popitem(last=False)
>>> ('first', 123)
For more details, have a look at the manual on collections. It also works with Python 2.x.
Subclassing and adding a method to OrderedDict would be the answer to clarity issues:
>>> o = ExtOrderedDict(('a',1), ('b', 2))
>>> o.first_item()
('a', 1)
The implementation of ExtOrderedDict:
class ExtOrderedDict(OrderedDict):
def first_item(self):
return next(iter(self.items()))
Code that's readable, leaves the OrderedDict unchanged and doesn't needlessly generate a potentially large list just to get the first item:
for item in ordered_dict.items():
return item
If ordered_dict is empty, None would be returned implicitly.
An alternate version for use inside a stretch of code:
for first in ordered_dict.items():
break # Leave the name 'first' bound to the first item
else:
raise IndexError("Empty ordered dict")
The Python 2.x code corresponding to the first example above would need to use iteritems() instead:
for item in ordered_dict.iteritems():
return item
You might want to consider using SortedDict instead of OrderedDict.
It provides SortedDict.peekitem to peek an item.
Runtime complexity: O(log(n))
>>> sd = SortedDict({'a': 1, 'b': 2, 'c': 3})
>>> sd.peekitem(0)
('a', 1)
If you need a one-liner:
ordered_dict[[*ordered_dict.keys()][0]]
It creates a list of dict keys, picks the first and use it as key to access the dictionary value.
First record:
[key for key, value in ordered_dict][0]
Last record:
[key for key, value in ordered_dict][-1]
Ladies and Gents,
I have a question about dictionaries in python. While playing around I noticed something that to me seems strange.
I define a dict like this
stuff={'age':26,'name':'Freddie Mercury', 'ciy':'Vladivostok'}
I then add the word 'first' to stuff like this:
stuff[1]='first'
When I print it out, it's fine
stuff
{1: 'first', 'age': 26, 'name': 'Freddie Mercury', 'city': 'Vladivostok'}
Then I add the word second:
stuff[2]='second'
and that's fine, but when I display the content I get:
stuff
{1: 'first', 'age': 26, 2: 'second', 'name': 'Freddie Mercury', 'city': 'Vladivostok'}
** notice that 2 is now the third element, and not the second (in order) or the first (if elements are added to the beginning) element
And when I add in the third element 'wtf', now all of a sudden everything is back in order and I'm quite confused as to what's going on.
stuff[3]='wtf'
stuff
{1: 'first', 2: 'second', 3: 'wtf', 'name': 'Freddie Mercury', 'age': 26, 'city': 'Vladivostok'}
Could someone please explain to me what's going on here?
The order you get from a dictionary is undefined. You should not rely on it. In this case, it happens to depend on the hash values of the underlying keys, but you shouldn't assume that's always the case.
If order matters to you, use should use an OrderedDict (since Python 2.7):
>>> from collections import OrderedDict
>>> stuff=OrderedDict({'age':26,'name':'Freddie Mercury', 'city':'Vladivostok'})
>>> stuff[1]='first'
>>> print stuff
OrderedDict([('city', 'Vladivostok'), ('age', 26), ('name', 'Freddie Mercury'), (1, 'first')])
>>> stuff[2]='second'
>>> print stuff
OrderedDict([('city', 'Vladivostok'), ('age', 26), ('name', 'Freddie Mercury'), (1, 'first'), (2, 'second')])
>>> stuff[3]='wtf'
>>> print stuff
OrderedDict([('city', 'Vladivostok'), ('age', 26), ('name', 'Freddie Mercury'), (1, 'first'), (2, 'second'), (3, 'wtf')])
Dictionaries are unordered data structures, so you should have no expectations
Learn what a hashtable is: http://en.wikipedia.org/wiki/Hash_table
In short, dict has an internal array, and inserts values at slots chosen through a hash function. The nature of this function is that it spreads the entries around evenly.
There are some (important) guarantees about order. From the docs (Python 2.7.2) http://docs.python.org/library/stdtypes.html#dict.items:
If items(), keys(), values(), iteritems(), iterkeys(), and
itervalues() are called with no intervening modifications to the
dictionary, the lists will directly correspond. This allows the
creation of (value, key) pairs using zip(): pairs = zip(d.values(),
d.keys()). The same relationship holds for the iterkeys() and
itervalues() methods: pairs = zip(d.itervalues(), d.iterkeys())
provides the same value for pairs. Another way to create the same list
is pairs = [(v, k) for (k, v) in d.iteritems()].
In a list, elements are ordered by their index (position in the list).
A dictionary is, for a all intensive purposes, a bag. Things may move around, but you shouldn't be concerned about that. You access items by their keys. You could think of keys as labels, which uniquely identify their values.
stuff = {} # hey python, please create a bag called stuff.
stuff[1]='first' # hey python, please put the string 'first' in my bag called stuff.
# if I ever need to access 'first' from this bag, I will ask for 1
# so please attach the label (key) 1 to the item (value) 'first'
print stuff[1] # hey python, please find the thing in my bag called stuff
# that has a label with 1 attached to it
print stuff[2] # hey python, please find the thing in my bag called stuff
# that has a label called 1 attached to it
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
KeyError: 2 # python says "hey programmer, nothing in your bag called stuff has a label with 2 attached to it
Hope this helps
The build in dict gives no guarantee of order of keys after an insertion or deletion.
If you want to keep insertion order (by time of key insertion/update or by key + value insertion/ update) or sorted by key, use the orderddict package http://anthon.home.xs4all.nl/Python/ordereddict/ that I wrote. It is implemented in C (and thus only works for CPython, but almost as fast as the build in dict).