I have a dynamodb table with an attribute containing a nested map and I would like to update a specific inventory item that is filtered via a filter expression that results in a single item from this map.
How to write an update expression to update the location to "in place three" of the item with name=opel,tags include "x1" (and possibly also f3)?
This should just update the first list elements location attribute.
{
"inventory": [
{
"location": "in place one", # I want to update this
"name": "opel",
"tags": [
"x1",
"f3"
]
},
{
"location": "in place two",
"name": "abc",
"tags": [
"a3",
"f5"
]
}],
"User" :"test"
}
Updated Answer - based on updated question statement
You can update attributes in a nested map using update expressions such that only a part of the item would get updated (ie. DynamoDB would apply the equivalent of a patch to your item) but, because DynamoDB is a document database, all operations (Put, Get, Update, Delete etc.) work on the item as a whole.
So, in your example, assuming User is the partition key and that there is no sort key (I didn't see any attribute that could be a sort key in that example), an Update request might look like this:
table.update_item(
Key={
'User': 'test'
},
UpdateExpression="SET #inv[0].#loc = :locVal",
ExpressionAttributeNames={
'#inv': 'inventory',
'#loc': 'location'
},
ExpressionAttributeValues={
':locVal': 'in place three',
},
)
That said, you do have to know what the item schema looks like and which attributes within the item should be updated exactly.
DynamoDB does NOT have a way to operate on sub-items. Meaning, there is no way to tell Dynamo to execute an operation such as "update item, set 'location' property of elements of the 'inventory' array that have a property of 'name' equal to 'opel'"
This is probably not the answer you were hoping for, but it is what's available today. You may be able to get closer to what you want by changing the schema a bit.
If you need to reference the sub-items by name, perhaps storing something like:
{
"inventory": {
"opel": {
"location": "in place one", # I want to update this
"tags": [ "x1", "f3" ]
},
"abc": {
"location": "in place two",
"tags": [ "a3", "f5" ]
}
},
"User" :"test"
}
Then your query would be:
table.update_item(
Key={
'User': 'test'
},
UpdateExpression="SET #inv.#brand.#loc = :locVal",
ExpressionAttributeNames={
'#inv': 'inventory',
'#loc': 'location',
'#brand': 'opel'
},
ExpressionAttributeValues={
':locVal': 'in place three',
},
)
But YMMV as even this has limitations because you are limited to identifying inventory items by name (ie. you still can't say "update inventory with tag 'x1'"
Ultimately you should carefully consider why you need Dynamo to perform these complex operations for you as opposed to you being specific about what you want to update.
You can update the nested map as follow:
First create and empty item attribute of type map. In the example graph is the empty item attribute.
dynamoTable = dynamodb.Table('abc')
dynamoTable.put_item(
Item={
'email': email_add,
'graph': {},
}
Update nested map as follow:
brand_name = 'opel'
DynamoTable = dynamodb.Table('abc')
dynamoTable.update_item(
Key={
'email': email_add,
},
UpdateExpression="set #Graph.#brand= :name, ",
ExpressionAttributeNames={
'#Graph': 'inventory',
'#brand': str(brand_name),
},
ExpressionAttributeValues = {
':name': {
"location": "in place two",
'tag': {
'graph_type':'a3',
'graph_title': 'f5'
}
}
Updating Mike's answer because that way doesn't work any more (at least for me).
It is working like this now (attention for UpdateExpression and ExpressionAttributeNames):
table.update_item(
Key={
'User': 'test'
},
UpdateExpression="SET inv.#brand.loc = :locVal",
ExpressionAttributeNames={
'#brand': 'opel'
},
ExpressionAttributeValues={
':locVal': 'in place three',
},
)
And whatever goes in Key={}, it is always partition key (and sort key, if any).
EDIT:
Seems like this way only works when with 2 level nested properties. In this case you would only use "ExpressionAttributeNames" for the "middle" property (in this example, that would be #brand: inv.#brand.loc). I'm not yet sure what is the real rule now.
DynamoDB UpdateExpression does not search on the database for matching cases like SQL (where you can update all items that match some condition). To update an item you first need to identify it and get primary key or composite key, if there are many items that match your criteria, you need to update one by one.
then the issue to update nested objects is to define UpdateExpression,ExpressionAttributeValues & ExpressionAttributeNames to pass to Dynamo Update Api .
I use a recursive function to update nested Objects on dynamoDB. You ask for Python but I use javascript, I think is easy to see this code and implents on Python:
https://gist.github.com/crsepulv/4b4a44ccbd165b0abc2b91f76117baa5
/**
* Recursive function to get UpdateExpression,ExpressionAttributeValues & ExpressionAttributeNames to update a nested object on dynamoDB
* All levels of the nested object must exist previously on dynamoDB, this only update the value, does not create the branch.
* Only works with objects of objects, not tested with Arrays.
* #param obj , the object to update.
* #param k , the seed is any value, takes sense on the last iteration.
*/
function getDynamoExpression(obj, k) {
const key = Object.keys(obj);
let UpdateExpression = 'SET ';
let ExpressionAttributeValues = {};
let ExpressionAttributeNames = {};
let response = {
UpdateExpression: ' ',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
};
//https://stackoverflow.com/a/16608074/1210463
/**
* true when input is object, this means on all levels except the last one.
*/
if (((!!obj) && (obj.constructor === Object))) {
response = getDynamoExpression(obj[key[0]], key);
UpdateExpression = 'SET #' + key + '.' + response['UpdateExpression'].substring(4); //substring deletes 'SET ' for the mid level values.
ExpressionAttributeNames = {['#' + key]: key[0], ...response['ExpressionAttributeNames']};
ExpressionAttributeValues = response['ExpressionAttributeValues'];
} else {
UpdateExpression = 'SET = :' + k;
ExpressionAttributeValues = {
[':' + k]: obj
}
}
//removes trailing dot on the last level
if (UpdateExpression.indexOf(". ")) {
UpdateExpression = UpdateExpression.replace(". ", "");
}
return {UpdateExpression, ExpressionAttributeValues, ExpressionAttributeNames};
}
//you can try many levels.
const obj = {
level1: {
level2: {
level3: {
level4: 'value'
}
}
}
}
I had the same need.
Hope this code helps. You only need to invoke compose_update_expression_attr_name_values passing the dictionary containing the new values.
def compose_update_expression_attr_name_values(data: dict) -> (str, dict, dict):
""" Constructs UpdateExpression, ExpressionAttributeNames, and ExpressionAttributeValues for updating an entry of a DynamoDB table.
:param data: the dictionary of attribute_values to be updated
:return: a tuple (UpdateExpression: str, ExpressionAttributeNames: dict(str: str), ExpressionAttributeValues: dict(str: str))
"""
# prepare recursion input
expression_list = []
value_map = {}
name_map = {}
# navigate the dict and fill expressions and dictionaries
_rec_update_expression_attr_name_values(data, "", expression_list, name_map, value_map)
# compose update expression from single paths
expression = "SET " + ", ".join(expression_list)
return expression, name_map, value_map
def _rec_update_expression_attr_name_values(data: dict, path: str, expressions: list, attribute_names: dict,
attribute_values: dict):
""" Recursively navigates the input and inject contents into expressions, names, and attribute_values.
:param data: the data dictionary with updated data
:param path: the navigation path in the original data dictionary to this recursive call
:param expressions: the list of update expressions constructed so far
:param attribute_names: a map associating "expression attribute name identifiers" to their actual names in ``data``
:param attribute_values: a map associating "expression attribute value identifiers" to their actual values in ``data``
:return: None, since ``expressions``, ``attribute_names``, and ``attribute_values`` get updated during the recursion
"""
for k in data.keys():
# generate non-ambiguous identifiers
rdm = random.randrange(0, 1000)
attr_name = f"#k_{rdm}_{k}"
while attr_name in attribute_names.keys():
rdm = random.randrange(0, 1000)
attr_name = f"#k_{rdm}_{k}"
attribute_names[attr_name] = k
_path = f"{path}.{attr_name}"
# recursion
if isinstance(data[k], dict):
# recursive case
_rec_update_expression_attr_name_values(data[k], _path, expressions, attribute_names, attribute_values)
else:
# base case
attr_val = f":v_{rdm}_{k}"
attribute_values[attr_val] = data[k]
expression = f"{_path} = {attr_val}"
# remove the initial "."
expressions.append(expression[1:])
Related
I have a mongo database including the following collection:
"
"_id": {
"$oid": "12345"
},
"id": "333555",
"token": [
{
"access_token": "ac_33bc",
"expires_in": 3737,
"token_type": "bearer",
"expires_at": {
"$date": "2021-07-02T13:37:28.123Z"
}
}
]
}
In the next python script I'm trying to return and print only the access_token but can't figure out how to do so. I've tried various methods which none of the worked.I've given the "id" as a parameter
def con_mongo():
try:
client = pymongo.MongoClient("mongodb:localhost")
#DB name
db = client["db1"]
#Collection
coll = db["coll1"]
#1st method
x = coll.find({"id":"333555"},{"token":"access_token"})
for data in x:
print(x)
#2nd method
x= coll.find({"id":"333555"})
tok=x.distinct("access_token")
#print(x[0])
for data in tok:
print(data)
except Exception:
logging.info(Exception)
It doesn't work this way, although if I replace (or remove) the "access_token" with simply "token" it works but I get back all the informations included in the field "token" where I only need the value of the "access_token".
Since access_token is an array element, you need to qualify it's name with the name of the array, to properly access its value.
Actually you can first extract the whole document and get the desired value through simple list and dict indexing.
So, assuming you are retrieving many documents with that same id:
x = [doc["token"][0]["access_token"] for doc in coll.find({"id":"333555"})]
The above, comprehensively creates a list with the access_tokens of all the documents matching the given id.
If you just need the first (and maybe only) occurrence of a document with that id, you can use find_one() instead:
x = coll.find_one({"id":"333555"})["token"][0]["access_token"]
# returns ac_33bc
token is a list so you have to reference the list element, e.g.
x = coll.find({"id":"333555"},{"token.access_token"})
for data in x:
print(data.get('token')[0].get('access_token'))
prints:
ac_33bc
I have a json object and I'm trying to extract a couple of values from a nested list. Then print them in markup. I'm getting and error - AttributeError: 'list' object has no attribute 'get'
I understand that it's a list and I can't preform a get. I've been searching for the proper method for a few hours now and I'm running out of steam. I'm able to get the Event, but not Value1 and Value2.
This is the json object
{
"resource": {
"data": {
"event": "qwertyuiop",
"eventVersion": "1.05",
"parameters": {
"name": "sometext",
"othername": [
""
],
"thing": {
"something": {
"blah": "whatever"
},
"abc": "123",
"def": {
"xzy": "value"
}
},
"something": [
"else"
]
},
"whatineed": [{
"value1": "text.i.need",
"value2": "text.i.need.also"
}]
}
}
}
And this is my function
def parse_json(json_data: dict) -> Info:
some_data = json_data.get('resource', {})
specific_data = some_data.get('data', {})
whatineed_data = specific_data.get('whatineed', {})
formatted_json = json.dumps(json_data, indent=2)
description = f'''
h3. Details
*Event:* {some_data.get('event')}
*Value1:* {whatineed_data('value1')}
*Value2:* {whatineed_data('value2')}
'''
From the data structure, whatineed is a list with a single item, which in turn is a dictionary. So, one way to access it would be:
whatineed_list = specific_data.get('whatineed', [])
whatineed_dict = whatineed_list[0]
At this point you can do:
value1 = whatineed_dict.get('value1')
value2 = whatineed_dict.get('value2')
You can change your function to the following:
def parse_json(json_data: dict) -> Info:
some_data = json_data.get('resource')
specific_data = some_data.get('data', {})
whatineed_data = specific_data.get('whatineed', {})
formatted_json = json.dumps(json_data, indent=2)
description = '''
h3. Details
*Event:* {}
*Value1:* {}
*Value2:* {}
'''.format(some_data.get('data').get('event'),whatineed_data[0]['value1'], whatineed_data[0]['value2'])
Since whatineed_data is a list, you need to index the element first
Python handles json as strings unless they are coming directly from a file. This could be the source for some of your problems. Also this article might help.
Assuming that "whatineed" attribute is really a list, and it's elements are dicts, you can't call whatineed.get asking for Value1 or Value2 as if they are attributes, because it is a list and it don't have attributes.
So, you have two options:
If whatineed list has a single element ever, you can access this element directly and than access the element attributes:
element = whatineed[0]
v1 = element.get('value1', {})
v2 = element.get('value2', {})
Or, if whatineed list can have more items, so, you will need to iterate over this list and access those elements:
for element in whatineed:
v1 = element.get('value1', {})
v2 = element.get('value2', {})
## Do something with values
How to print the string of "value" only for id: "resolution" ??
Here in this case I want to print the value "Fixed"
customFields: {
string: [
{
id: "device_type",
value: "iPhone 6"
},
{
id: "os_version",
value: "iOS 10.x"
},
{
id: "rabbit_build",
value: "2.11.llyu"
},
{
id: "resolution",
value: "Fixed"
},
My Python code is
for ib in data['documents']:
sid = ib['id']
tit = ib['title']
stat = ib['status']
nstep = ib['next_step']['action']
requester = ib['requesterIdentity']
resolution = ib['customFields']['string']
print(sid, tit, stat, nstep, requester, resolution)
The output prints all the ids and values rather I want to print "value" only of id: "resolution"
Your present code don't do what you want because you are storing in a new variable called "resolution" the whole list in ib['customField']['string'] and print the value of this new variable. (what is 'ib' ?)
There is no relation whatsoever between the name of this new variable and the behavior of the '=' operator.
You are trying to grab a specific element of the list, check if the 'id' contains 'resolution' then print it.
you can use
resolutionList = [it for it in ib['customFields']['string'] if it['id'] == 'resolution']
for element in resolutionList :
print(element['value'])
or (in one line)
print(*[it['value'] for it in ib['customFields']['string'] if it['id'] == 'resolution'])
The * operator is used to deplete the list (print(*[a,b]) <=> print(a,b))
This is pretty straightforward, but should work:
for ib in data['documents']:
sid = ib['id']
tit = ib['title']
stat = ib['status']
nstep = ib['next_step']['action']
requester = ib['requesterIdentity']
for item in ib['customFields']['string']:
if item['id'] == "resolution":
resolution = item['string']
Sure, since "string" is a list of dictionaries you can't get specific values without looping through all dictionaries of this list. I'd also recommend you to change this list into a dictionary, so your json structure will look like this:
string: {
"device_type": iPhone 6".
"os_version": "iOS 10.x",
"rabbit_build": "2.11.llyu",
"resolution": "Fixed",
}
With this kind of structure you could check all the values without loops by straightly getting value of "resolution" key
There is no example code for the Podio API for Python but this is the example code written for Ruby:
Podio::Item.update(210606, {
:fields => {
'title' => 'The API documentation is much more funny',
'business_value' => { :value => 20000, :currency => 'EUR' },
'due_date' => { :start => '2011-05-06 11:27:20', :end =>
5.days.from_now.to_s(:db) }
}
})
I can't for the life of me figure out how to translate this to Python 3. I've tried using dictionaries, dicts inside lists, referencing the field with both their field-id and their names etc. But it never actually updates anything.
This is my failed attempt at translating the above to Python code (with different fields since the fields in my 'Bugs (API example) app' aren't the same as in the example code):
newValues = {'fields':{'title': "This is my title",'description_of_problem':
"the not work"}}
try:
podio.Item.update(629783395, newValues['fields'])
print('updating was successful')
except:
print('updating was not successful')
With podio being:
podio = api.OAuthClient(
client_id,
client_secret,
username,
password,
)
The 'fields' part in my code doesn't make any sense really, but I couldn't figure out what else to do with that part of the Ruby code, I suspect that is the issue. The program always prints 'updating was successful' as if the Item.update method was successfully called, but as I said it doesn't actually update anything in Podio. Can anyone see what's wrong?
I'd just follow the Item update API, and pass in a dictionary that matches the request section there:
{
"revision": The revision of the item that is being updated. This is optional,
"external_id": The new external_id of the item,
"fields": The values for each field, see the create item operation for details,
"file_ids": The list of attachments,
"tags": The list of tags,
"reminder": Optional reminder on this task
{
"remind_delta": Minutes (integer) to remind before the due date
},
"recurrence": The recurrence for the task, if any,
{
"name": The name of the recurrence, "weekly", "monthly" or "yearly",
"config": The configuration for the recurrence, depends on the type
{
"days": List of weekdays ("monday", "tuesday", etc) (for "weekly"),
"repeat_on": When to repeat, "day_of_week" or "day_of_month" (for "monthly")
},
"step": The step size, 1 or more,
"until": The latest date the recurrence should take place
},
"linked_account_id": The linked account to use for meetings,
"ref" The reference of the item
{
"type": The type of reference,
"id": The id of the reference
}
}
The documentation further points to the item creation API for further examples. Note how that object has a "fields" key in the outermost mapping.
All the Ruby documentation does is build that mapping as a Ruby hash (in Python, a dict) with the entries that need updating; :field is an immutable string (called a symbol) that defines a key in that hash pointing to a nested hash. The Python implementation for the update method just converts that dictionary to a JSON post body.
A direct translation of the Ruby code to Python is:
from datetime import datetime, timedelta
podio.item.update(210606, {
'fields': {
'title': 'The API documentation is much more funny',
'business_value': {'value': 20000, 'currency': 'EUR'},
'due_date': {
'start': '2011-05-06 11:27:20',
'end': (datetime.now() + timedelta(days=5)).strftime('%Y-%m-%d %H:%M:%S')}
}
})
What you did wrong in your case is not include the 'fields' key in the outermost dictionary; you unwrapped the outermost dictionary and only posted the nested dictionary under 'fields'. Instead, include that outer dictionary:
newValues = {
'fields': {
'title': "This is my title",
'description_of_problem': "the not work"
}
}
podio.Item.update(629783395, newValues)
I'm trying to figure out if there is a way to use conditionals such as $set to do more advanced updating. This is what I'm trying to do in pseudo code:
# new data to use in a possible update
newData = { 'emailAddress' : $usersEmailAddress,
'keyA' : 'valueA',
'keyB' : None,
'keyC' : '<null>' }
self.request.root.db.users.update(
{ 'emailAddress' : newData['emailAddress'] },
{ '$set': {
"Here, loop through all newData keys/values and if
notNull(newData[key]) == True and is different than the
corresponding key/value in the user
document (or if user document doesn't have that key)
than update with newData key/value"
} }, upsert = False, safe = True )
# The result should be that if the value of keyA (and ONLY key A because the
# others are null) is different in the user document
# than in newData, than the user document should be updated with the new value.
# function to catch any possible None value or equivalent string
def notNull(valueToCheck):
if thingToCheck and thingToCheck != "null" and thingToCheck != 'nil' and thingToCheck != '<null>' and thingToCheck != '' and thingToCheck != ' ':
return True
else:
return False
What's the most efficient way of doing this? Because currently I'm having to pull the entire document with find_one and, I'm told, thats rather expensive. Is there a way to do this with just $set?
No, MongoDB does not support this feature. You can either, as you say, retrieve the document, analyze it in your client-side code, and issue an update based on its contents, or you could issue a series of updates like:
db.users.update({
'emailAddress': newData['emailAddress'],
'$or': [
{ 'keyA': { '$exists': false } },
{ 'keyA': None } ] }
]
}, {
'$set': { 'keyA': newData['keyA'] }
})
The former will be more efficient, of course, since it's a single fetch and a single update. But consider whether you need to guard against multiple MongoDB clients simultaneously fetching and updating the same document.