Updating a MongoDB document with PyMongo - python

I am trying to update a document in a collection:
for bookerName,bookingId in zip(bookerNames,bookingIds):
Person.update_one({'Name': bookerName}, {'$push': {'Bookings': bookingId}})
I am trying to add a new bookingId to the list of Bookings this Person already has.
I receive the following error:
bson.errors.InvalidDocument: Cannot encode object: 0
From what I understand about this error, it means Mongo don't know how to encode a list, it treats it as a custom type.
However, this is strange since I can do:
for bookerName in BookerNames:
personRecord = {"Name":bookerName,"Bookings":[1,2,3,4]}
Person.insert_one(personRecord)
And it works perfectly.
I have also tried to update using $set and even by deleting the old record and trying to insert a new one with the updated list.
Any idea why this is happening or how can I update a list in a document?
I am using Python 3.5, PyMongo 3.0.3 and MongoDB 3.0.7. Could the difference in version between PyMongo and MondoDB be the issue?
EDIT:
for bookerName in bookerNames:
bookingList = bookersToBookings[bookerName]
personRecord = {"Name":bookerName,"Bookings":bookingList}
Person.insert_one(personRecord)
Where bookersToBookings is a dictionary mapping bookers to their bookingIds, as list.
This does not work, although if I make a record with a list like [1,2,3], it works.
EDIT 2:
for bookerName in bookerNames:
personRecord = {"Name":bookerName,"Bookings":[]}
Person.insert_one(personRecord)
for x in Person.find():
print(x)
This works. It prints:
{'Name': 'Alice', '_id': ObjectId('56241ad79e44c71641efa383'), 'Bookings': []}
{'Name': 'John', '_id': ObjectId('56241ad79e44c71641efa384'), 'Bookings': []}
{'Name': 'Jane', '_id': ObjectId('56241ad79e44c71641efa385'), 'Bookings': []}
{'Name': 'Mary', '_id': ObjectId('56241ad79e44c71641efa386'), 'Bookings': []}
{'Name': 'Dan', '_id': ObjectId('56241ad79e44c71641efa387'), 'Bookings': []}
This also works:
for bookerName in bookerNames:
personRecord = {"Name":bookerName,"Bookings":[1,2,3]}
Person.insert_one(personRecord)
Result:
{'Bookings': [1, 2, 3], '_id': ObjectId('56241b029e44c716497b60b2'), 'Name': 'Alice'}
{'Bookings': [1, 2, 3], '_id': ObjectId('56241b029e44c716497b60b3'), 'Name': 'John'}
{'Bookings': [1, 2, 3], '_id': ObjectId('56241b029e44c716497b60b4'), 'Name': 'Jane'}
{'Bookings': [1, 2, 3], '_id': ObjectId('56241b029e44c716497b60b5'), 'Name': 'Mary'}
{'Bookings': [1, 2, 3], '_id': ObjectId('56241b029e44c716497b60b6'), 'Name': 'Dan'}
Even this works:
for bookerName in bookerNames:
y = [1,2,3,4]
personRecord = {"Name":bookerName,"Bookings":[x for x in y]}
Person.insert_one(personRecord)
Result:
{'_id': ObjectId('56241b5a9e44c7165395a0f0'), 'Bookings': [1, 2, 3, 4], 'Name': 'Alice'}
{'_id': ObjectId('56241b5a9e44c7165395a0f1'), 'Bookings': [1, 2, 3, 4], 'Name': 'John'}
{'_id': ObjectId('56241b5a9e44c7165395a0f2'), 'Bookings': [1, 2, 3, 4], 'Name': 'Jane'}
{'_id': ObjectId('56241b5a9e44c7165395a0f3'), 'Bookings': [1, 2, 3, 4], 'Name': 'Mary'}
{'_id': ObjectId('56241b5a9e44c7165395a0f4'), 'Bookings': [1, 2, 3, 4], 'Name': 'Dan'}
This, however, does not work:
for bookerName in bookerNames:
y = bookersToBookings[bookerName]
personRecord = {"Name":bookerName,"Bookings":[x for x in y]}
Person.insert_one(personRecord)
bookersToBookings looks like this:
{'Mary': [5, 9], 'Jane': [4, 7, 8, 13, 14], 'Alice': [0, 6, 10], 'Dan': [12], 'John': [1, 2, 3, 11]}
EDIT 3 - SOLVED:
For $%#^ sake! The list was made of numpy64 ints, not python ints. It works now.
Thanks for the help!

Related

Removing duplicates in nested dictionaries

I am trying to remove all the duplicates and only make the original value remain in a nested dictionary. This is my nested dictionary below. In this dictionary, I am trying to check if two names are the same and if they are same, then remove the second and subsequent duplicates. For example: dictionaries 4 and 5 have the same name 'Sasha', so dictionary 5 should be removed.
dict1 = {
1: {'friends': [2],
'history': [],
'id': 1,
'name': 'Fred',
'date_of_birth': datetime.date(2022, 2, 1)},
2: {'friends': [1],
'history': [],
'id': 2,
'name': 'Jenny',
'date_of_birth': datetime.date(2004, 11, 18)},
3: {'friends': [4],
'history': [],
'id': 3,
'name': 'Jiang',
'date_of_birth': datetime.date(1942, 9, 16)},
4: {'friends': [3],
'history': [],
'id': 4,
'name': 'Sasha',
'date_of_birth': datetime.date(1834, 2, 2)},
5: {'friends': [6],
'history': [],
'id': 5,
'name': 'Sasha',
'date_of_birth': datetime.date(1834, 2, 2)},
6: {'friends': [5],
'history': [],
'id': 6,
'name': 'Amir',
'date_of_birth': datetime.date(1981, 8, 11)}}
I have implemented my solution like this but I don't understand where I am going wrong.
temp = []
res = dict()
for key, val in dict1.items():
if val not in temp:
temp.append(val)
res[key] = val
print(pprint.pformat(res))
It would be great if someone could help me with this.
In your for-loop, val is the inner dictionary and if you look closely, the value of 4 and 5 are different ("friends" and "id" are different), so it's not dropped. However, since you only need the "name" to be the same (not the entire dictionary), you can keep track of the "name" instead and keep only unique names:
temp = []
res = dict()
for key, val in dict1.items():
if val['name'] not in temp:
temp.append(val['name'])
res[key] = val
Edit:
If the goal is to "shift" keys as well, you could approach it a little differently by only storing the non-duplicate values in res, then zip it with the keys of dict1 to create the output dictionary:
temp = set()
res = []
for val in dict1.values():
if val['name'] not in temp:
temp.add(val['name'])
res.append(val)
out = dict(zip(dict1, res))
Output:
{1: {'friends': [2],
'history': [],
'id': 1,
'name': 'Fred',
'date_of_birth': datetime.date(2022, 2, 1)},
2: {'friends': [1],
'history': [],
'id': 2,
'name': 'Jenny',
'date_of_birth': datetime.date(2004, 11, 18)},
3: {'friends': [4],
'history': [],
'id': 3,
'name': 'Jiang',
'date_of_birth': datetime.date(1942, 9, 16)},
4: {'friends': [3],
'history': [],
'id': 4,
'name': 'Sasha',
'date_of_birth': datetime.date(1834, 2, 2)},
5: {'friends': [5],
'history': [],
'id': 6,
'name': 'Amir',
'date_of_birth': datetime.date(1981, 8, 11)}}

Table of dictionaries

I have a table of dicts looking like:
[{
'variant_id': 4126274,
'stock': [
{'stock_id': 6, 'quantity': 86},
{'stock_id': 4, 'quantity': 23},
{'stock_id': 3, 'quantity': 9}
]
}, ...]
My goal is to unzip every piece of stock to look like this:
[{'variant_id': 4126274, 'stock_id': 6, 'quantity':86}
{'variant_id': 4126274, 'stock_id': 4, 'quantity':23}
{'variant_id': 4126274, 'stock_id': 3, 'quantity':9}...]
Is there any fast and optimal way to do this?
You could do something like:
result = [{'variant_id': entry['variant_id'],
'stock_id': stock_entry['stock_id'],
'quantity': stock_entry['quantity']} for entry in table for stock_entry in entry['stock']]
This gives
[{'quantity': 86, 'stock_id': 6, 'variant_id': 4126274},
{'quantity': 23, 'stock_id': 4, 'variant_id': 4126274},
{'quantity': 9, 'stock_id': 3, 'variant_id': 4126274}]
Here are two approaches: one with nested for loops, and one with a list comprehension.
data = [{'variant_id': 4126274, 'stock': [{'stock_id': 6, 'quantity': 86}, {'stock_id': 4, 'quantity': 23}, {'stock_id': 3, 'quantity': 9}]}]
result = []
for entry in data:
for stock in entry['stock']:
result.append({'variant_id': entry['variant_id'], 'stock_id': stock['stock_id'], 'quantity': stock['quantity']})
print(result)
result_list_comprehension = [{'variant_id': entry['variant_id'], 'stock_id': stock['stock_id'], 'quantity': stock['quantity']} for entry in data for stock in entry['stock']]
print(result_list_comprehension)

Removing an element of a list [duplicate]

This question already has answers here:
How to modify list entries during for loop?
(10 answers)
Closed 1 year ago.
I have a list of lists containing dictionaries:
[[{'event_id': 1, 'order_id': 1, 'item_id': 1, 'count': 1, 'return_count': 0, 'status': 'OK'},
{'event_id': 2, 'order_id': 1, 'item_id': 1, 'count': 1, 'return_count': 0, 'status': 'OK'}],
[{'order_id': 2, 'item_id': 1, 'event_id': 1, 'count': 3, 'return_count': 1, 'status': 'OK'},
{'order_id': 2, 'event_id': 2, 'item_id': 1, 'count': 3, 'return_count': 1, 'status': 'OK'},
{'order_id': 2, 'event_id': 1, 'item_id': 2, 'count': 4, 'return_count': 2, 'status': 'OK'}]]
For each item in the given order I only need those dictionaries whose event_id is max. So I wrote the following code:
for el in lst:
for element in el:
if element['event_id'] != max(x['event_id'] for x in el if element['item_id'] == x['item_id']):
el.remove(element)
lst is the initial list.
For some reason, after running the code lst remains unchanged.
This isn't in one line, but it does return dictionaries with the max event id
dictlist = [
[{'event_id': 1, 'order_id': 1, 'item_id': 1, 'count': 1, 'return_count': 0, 'status': 'OK'},
{'event_id': 2, 'order_id': 1, 'item_id': 1, 'count': 1, 'return_count': 0, 'status': 'OK'}],
[{'order_id': 2, 'item_id': 1, 'event_id': 1, 'count': 3, 'return_count': 1, 'status': 'OK'},
{'order_id': 2, 'event_id': 2, 'item_id': 1, 'count': 3, 'return_count': 1, 'status': 'OK'},
{'order_id': 2, 'event_id': 1, 'item_id': 2, 'count': 4, 'return_count': 2, 'status': 'OK'}]]
max = 0
parsed = []
for item in dictlist:
for i in item:
if i['event_id'] > max:
max = i['event_id']
for item in dictlist:
for dic in item:
if dic['event_id'] == max:
parsed.append(dic)
You're trying to remove an element from a list you're iterating over, that won't work. And it's really hard to understand what you're trying to do; my suggestion would be to do something like this:
newlst = []
for el in lst:
max_event_id = max(element['event_id'] for element in el)
max_event_element = next(element for element in el if element['event_id'] == max_event_id)
newlst.append(max_event_element)
The expected result ends up in the newlst variable.
sort on "event_id" and keep only the max (last element):
result = [sorted(l, key=lambda x: x["event_id"])[-1] for l in lst]
If you want to keep all dictionaries with the max "event_id":
lsts = [[x for x in l if x["event_id"]==max(l, key=lambda x: x["event_id"])["event_id"]] for l in lst]
result = [item for sublist in lsts for item in lsts]

Sort list by indexes of another list?

I have a dictionary of dictionaries. Within these subdictionaries, I have two keys - ui_section and section_order - that determine if the value of this subdictionary is shown in a specfic portion of the UI and if so, what order it appears. My dictionary looks like this:
MASTER_DICT = {
'key1': {'ui_section':[1,2],'section_order':1, 'value': 'key1'},
'key2': {'ui_section':[1],'section_order':2, 'value': 'key2'},
'key3': {'ui_section':[1,2],'section_order':3, 'value': 'key3'},
'key4': {'ui_section':[1],'section_order':4, 'value': 'key4'},
'key5': {'ui_section':[1],'section_order':5, 'value': 'key5'},
'key6': {'ui_section':[1],'section_order':6, 'value': 'key6'},
'key7': {'ui_section':[1],'section_order':7, 'value': 'key7'},
'key8': {'ui_section':[1],'section_order':8, 'value': 'key8'},
'key9': {'ui_section':[1],'section_order':9, 'value': 'key9'},
}
The ui_section is a list of possible sections the key can appear in. I determine this via the following code:
def show_section_ui(master_dict, section=None):
if section:
ui_sections = []
# Find the keys that are part of this section
for k in master_dict.keys():
try:
if section in master_dict[k]['ui_section']:
ui_sections.append(master_dict[k])
except AttributeError:
pass
# Order the keys by sort order
ui_sections.sort(key=lambda x: x['section_order'])
return ui_sections
else:
return None
This portion of the code works. The output below shows that order is correct for both sections 1 and 2.
>>> pprint.pprint(show_section_ui(MASTER_DICT, 1))
[{'section_order': 1, 'ui_section': [1,2], 'value': 'key1'},
{'section_order': 2, 'ui_section': [1], 'value': 'key2'},
{'section_order': 3, 'ui_section': [1,2], 'value': 'key3'},
{'section_order': 4, 'ui_section': [1], 'value': 'key4'},
{'section_order': 5, 'ui_section': [1], 'value': 'key5'},
{'section_order': 6, 'ui_section': [1], 'value': 'key6'},
{'section_order': 7, 'ui_section': [1], 'value': 'key7'},
{'section_order': 8, 'ui_section': [1], 'value': 'key8'},
{'section_order': 9, 'ui_section': [1], 'value': 'key9'}]
>>> pprint.pprint(show_section_ui(MASTER_DICT, 2))
[{'section_order': 1, 'ui_section': [1,2], 'value': 'key1'},
{'section_order': 3, 'ui_section': [1,2], 'value': 'key3'}]
My problem is that the section_order needs to have sort orders per ui_section. For example, in the above outputs, in section 2, I'd like key3 to be first. My initial thought was to make section_order a list as well. But, I'm not sure how to adjust this line to properly account for the list (and to select the correct index to sort by then)
ui_sections.sort(key=lambda x: x['section_order'])
My intention was to do something like this:
MASTER_DICT = {
'key1': {'ui_section':[1,2],'section_order':[1,2], 'value': 'key1'},
'key2': {'ui_section':[1],'section_order':[2], 'value': 'key2'},
'key3': {'ui_section':[1,2],'section_order':[3,1], 'value': 'key3'},
}
Getting me output like this:
>>> pprint.pprint(show_section_ui(MASTER_DICT, 2))
[{'section_order': [3,1], 'ui_section': [1,2], 'value': 'key3'},
{'section_order': [1,2], 'ui_section': [1,2], 'value': 'key1'}]
How can I sort by the ui_section and the appropriate index within the key?
I think I understood. If you want to sort a list of items according to the order of another list of items, you could do something like this. Step by step, the last line is what need.
We need itertools.count(), which is an infinite incrementing range, used to apply an index.
import itertools
these should be sorted by 'value'
>>> values = [{"name": "A", "value": 10},
{"name": "B", "value": 8},
{"name": "C", "value": 9}]
these are in the same input order as values, and should be sorted in the the same order
>>> to_sort = [{"name": "A", "payload": "aaa"},
{"name": "B", "payload": "bbb"},
{"name": "C", "payload": "ccc"}]
Zip the values with their indexes. This annotates each item to include its original order. This is a list of pairs of (object, index)
>>> zip(values, itertools.count())
[({'name': 'A', 'value': 10}, 0),
({'name': 'B', 'value': 8}, 1),
({'name': 'C', 'value': 9}, 2)]
Now sort by the key 'value'. The x[0] is to get the first item of the pair (the object).
>>> sorted(zip(values, itertools.count()), key=lambda x: x[0]["value"])
[({'name': 'B', 'value': 8}, 1),
({'name': 'C', 'value': 9}, 2),
({'name': 'A', 'value': 10}, 0)]
Now retrieve the indexes from the pairs to return the new order of the original indexes.
>>> map(lambda x: x[1],
sorted(zip(values, itertools.count()),
key=lambda x: x[0]["value"]))
[1, 2, 0]
Now use those indexes to map over the to_sort list and retrieve the items at those indexes.
>>> map(lambda i: to_sort[i],
map(lambda x: x[1],
sorted(zip(values, itertools.count()),
key=lambda x: x[0]["value"])))
[{'name': 'B', 'payload': 'bbb'},
{'name': 'C', 'payload': 'ccc'},
{'name': 'A', 'payload': 'aaa'}]
I hope that answers your question. It means changing MASTER_DICT to be a list, but I think that's the best representation for it anyway.
I don't have a nice one line code change for you, but you can replace the line you have:
ui_sections.sort(key=lambda x: x['section_order'])
With this:
sort_orders = []
for s in ui_sections:
ndx = s['ui_section'].index(section)
# This next line makes the assumption that you ALWAYS have a section_order
# for every ui_section listed. If not, you'll get an IndexError
sort_orders.append(s['section_order'][ndx])
# Magic happens here
sorted_sections = [x for y, x in sorted(zip(sort_orders,ui_sections))]
return sorted_sections
Output:
>>> pprint.pprint(show_section_ui(MASTER_DICT, 2))
[{'section_order': [3, 1], 'ui_section': [1, 2], 'value': 'key3'},
{'section_order': [1, 2], 'ui_section': [1, 2], 'value': 'key1'}]
>>> pprint.pprint(show_section_ui(MASTER_DICT, 1))
[{'section_order': [1, 2], 'ui_section': [1, 2], 'value': 'key1'},
{'section_order': [2], 'ui_section': [1], 'value': 'key2'},
{'section_order': [3, 1], 'ui_section': [1, 2], 'value': 'key3'},
{'section_order': [4], 'ui_section': [1], 'value': 'key4'},
{'section_order': [5], 'ui_section': [1], 'value': 'key5'},
{'section_order': [6], 'ui_section': [1], 'value': 'key6'},
{'section_order': [7], 'ui_section': [1], 'value': 'key7'},
{'section_order': [8], 'ui_section': [1], 'value': 'key8'},
{'section_order': [9], 'ui_section': [1], 'value': 'key9'}]
Adding key8 to the second ui_section at position 3, and key7 at position 4:
[{'section_order': [3, 1], 'ui_section': [1, 2], 'value': 'key3'},
{'section_order': [1, 2], 'ui_section': [1, 2], 'value': 'key1'},
{'section_order': [8, 3], 'ui_section': [1, 2], 'value': 'key8'},
{'section_order': [7, 4], 'ui_section': [1, 2], 'value': 'key7'}]
This is utilizing this answer. First, though, it finds the index that the section is listed in the ui_section:
ndx = s['ui_section'].index(section)
The value at this location is then added to the sort_orders list. Note that the code provided does not error check that this is valid (ie. if you don't have a value for the second position), and will throw an IndexError if it is not.
sort_orders.append(s['section_order'][ndx])
Next it zips the two lists together so that you have a list of tuples containing the sort order and the section dictionary.
[(3, {'ui_section': [1, 2], 'section_order': [8, 3], 'value': 'key8'}),
(1, {'ui_section': [1, 2], 'section_order': [3, 1], 'value': 'key3'}),
(2, {'ui_section': [1, 2], 'section_order': [1, 2], 'value': 'key1'}),
(4, {'ui_section': [1, 2], 'section_order': [7, 4], 'value': 'key7'})
]
Then we sort that based on the first position in the tuple. Then we unzip it and pull back the sorted information. All of that occurs in this line:
sorted_sections = [x for y, x in sorted(zip(sort_orders,ui_sections))]

Iterating over list of dictionaries

I have a list -myList - where each element is a dictionary. I wish to iterate over this list but I am only interesting in one attribute - 'age' - in each dictionary each time. I am also interested in keeping count of the number of iterations.
I do:
for i, entry in enumerate(myList):
print i;
print entry['age'];
But was wondering is there something more pythonic. Any tips?
You could use a generator to only grab ages.
# Get a dictionary
myList = [{'age':x} for x in range(1,10)]
# Enumerate ages
for i, age in enumerate(d['age'] for d in myList):
print i,age
And, yeah, don't use semicolons.
Very simple way, list of dictionary iterate
>>> my_list
[{'age': 0, 'name': 'A'}, {'age': 1, 'name': 'B'}, {'age': 2, 'name': 'C'}, {'age': 3, 'name': 'D'}, {'age': 4, 'name': 'E'}, {'age': 5, 'name': 'F'}]
>>> ages = [li['age'] for li in my_list]
>>> ages
[0, 1, 2, 3, 4, 5]
For printing, probably what you're doing is just about right. But if you want to store the values, you could use a list comprehension:
>>> d_list = [dict((('age', x), ('foo', 1))) for x in range(10)]
>>> d_list
[{'age': 0, 'foo': 1}, {'age': 1, 'foo': 1}, {'age': 2, 'foo': 1}, {'age': 3, 'foo': 1}, {'age': 4, 'foo': 1}, {'age': 5, 'foo': 1}, {'age': 6, 'foo': 1}, {'age': 7, 'foo': 1}, {'age': 8, 'foo': 1}, {'age': 9, 'foo': 1}]
>>> ages = [d['age'] for d in d_list]
>>> ages
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> len(ages)
10
The semicolons at the end of lines aren't necessary in Python (though you can use them if you want to put multiple statements on the same line). So it would be more pythonic to omit them.
But the actual iteration strategy is easy to follow and pretty explicit about what you're doing. There are other ways to do it. But an explicit for-loop is perfectly pythonic.
(Niklas B.'s answer will not do precisely what you're doing: if you want to do something like that, the format string should be "{0}\n{1}".)

Categories