We Keep Coding

Extracting certain value from MongoDB using Python

I have a mongo database including the following collection:
"
"_id": {
"$oid": "12345"
},
"id": "333555",
"token": [
{
"access_token": "ac_33bc",
"expires_in": 3737,
"token_type": "bearer",
"expires_at": {
"$date": "2021-07-02T13:37:28.123Z"
}
}
]
}
In the next python script I'm trying to return and print only the access_token but can't figure out how to do so. I've tried various methods which none of the worked.I've given the "id" as a parameter
def con_mongo():
try:
client = pymongo.MongoClient("mongodb:localhost")
#DB name
db = client["db1"]
#Collection
coll = db["coll1"]
#1st method
x = coll.find({"id":"333555"},{"token":"access_token"})
for data in x:
print(x)
#2nd method
x= coll.find({"id":"333555"})
tok=x.distinct("access_token")
#print(x[0])
for data in tok:
print(data)
except Exception:
logging.info(Exception)
It doesn't work this way, although if I replace (or remove) the "access_token" with simply "token" it works but I get back all the informations included in the field "token" where I only need the value of the "access_token".

Since access_token is an array element, you need to qualify it's name with the name of the array, to properly access its value.
Actually you can first extract the whole document and get the desired value through simple list and dict indexing.
So, assuming you are retrieving many documents with that same id:
x = [doc["token"][0]["access_token"] for doc in coll.find({"id":"333555"})]
The above, comprehensively creates a list with the access_tokens of all the documents matching the given id.
If you just need the first (and maybe only) occurrence of a document with that id, you can use find_one() instead:
x = coll.find_one({"id":"333555"})["token"][0]["access_token"]
# returns ac_33bc

token is a list so you have to reference the list element, e.g.
x = coll.find({"id":"333555"},{"token.access_token"})
for data in x:
print(data.get('token')[0].get('access_token'))
prints:
ac_33bc

Looking to export JSON in Python using multiple nested blocks

I am trying to build out a JSON export mirrored on this example:
import json
data = {}
data['people'] = []
data['people'].append({
'name': 'Scott',
'website': 'stackabuse.com',
'from': 'Nebraska'
})
with open('data.txt', 'w') as outfile:
json.dump(data, outfile)
However, I wish for there to be more nested {} than just the one above, but cannot seem to figure out how to do that. This is what I'd like the JSON to look like:
{
"people":{
"Citizens":{
"Workers":{
"wage":"34",
"id":"1 of 2"
},
"non-workers":{
"wage":"0",
"id":"2 of 2"
}
}
}
}
Can anyone help me? Thank you!

While the method Brenden Price pointed out works, there is no need for so many statements. The number of statements solely depends on how deep the dictionaries which you are assigning are.
At its shortest:
data = {}
data["people"] = {
"Citizens": {
"Workers": {"wage": "34", "id": "1 of 2"},
"Non-Workers": {"wage": "0", "id": "2 of 2"}
}
}

To create the nested {}, you're adding more dictionaries. I'm not sure the goal of this, but o achieve that exact example you've provided, your code would be something along the lines of this:
data = {}
data['people'] = {} # empty dict of people
data['people']['citizens'] = {} # empty dict of workers inside people
data['people']['citizens']['workers'] = {} # empty dict of workers inside people
data['people']['citizens']['non-workers'] = {} # empty dict of non-workers inside people
data['people']['citizens']['workers']['wage'] = "34" # set workers wage to str "34"
data['people']['citizens']['workers']['id'] = "1 of 2" # set id to str "1 of 2"
data['people']['citizens']['non-workers']['wage'] = "0" # set wage to "0"
data['people']['citizens']['non-workers']['id'] = "2 of 2" # set id to "2 of 2"
Depending on the usage, you could technically do this and create that exact example as well:
data = {
"people":{
"Citizens":{
"Workers":{
"wage":"34",
"id":"1 of 2"
},
"non-workers":{
"wage":"0",
"id":"2 of 2"
}
}
}
}

print dictionary value based on another value?

How to print the string of "value" only for id: "resolution" ??
Here in this case I want to print the value "Fixed"
customFields: {
string: [
{
id: "device_type",
value: "iPhone 6"
},
{
id: "os_version",
value: "iOS 10.x"
},
{
id: "rabbit_build",
value: "2.11.llyu"
},
{
id: "resolution",
value: "Fixed"
},
My Python code is
for ib in data['documents']:
sid = ib['id']
tit = ib['title']
stat = ib['status']
nstep = ib['next_step']['action']
requester = ib['requesterIdentity']
resolution = ib['customFields']['string']
print(sid, tit, stat, nstep, requester, resolution)
The output prints all the ids and values rather I want to print "value" only of id: "resolution"

Your present code don't do what you want because you are storing in a new variable called "resolution" the whole list in ib['customField']['string'] and print the value of this new variable. (what is 'ib' ?)
There is no relation whatsoever between the name of this new variable and the behavior of the '=' operator.
You are trying to grab a specific element of the list, check if the 'id' contains 'resolution' then print it.
you can use
resolutionList = [it for it in ib['customFields']['string'] if it['id'] == 'resolution']
for element in resolutionList :
print(element['value'])
or (in one line)
print(*[it['value'] for it in ib['customFields']['string'] if it['id'] == 'resolution'])
The * operator is used to deplete the list (print(*[a,b]) <=> print(a,b))

This is pretty straightforward, but should work:
for ib in data['documents']:
sid = ib['id']
tit = ib['title']
stat = ib['status']
nstep = ib['next_step']['action']
requester = ib['requesterIdentity']
for item in ib['customFields']['string']:
if item['id'] == "resolution":
resolution = item['string']
Sure, since "string" is a list of dictionaries you can't get specific values without looping through all dictionaries of this list. I'd also recommend you to change this list into a dictionary, so your json structure will look like this:
string: {
"device_type": iPhone 6".
"os_version": "iOS 10.x",
"rabbit_build": "2.11.llyu",
"resolution": "Fixed",
}
With this kind of structure you could check all the values without loops by straightly getting value of "resolution" key

Update nested map dynamodb

I have a dynamodb table with an attribute containing a nested map and I would like to update a specific inventory item that is filtered via a filter expression that results in a single item from this map.
How to write an update expression to update the location to "in place three" of the item with name=opel,tags include "x1" (and possibly also f3)?
This should just update the first list elements location attribute.
{
"inventory": [
{
"location": "in place one", # I want to update this
"name": "opel",
"tags": [
"x1",
"f3"
]
},
{
"location": "in place two",
"name": "abc",
"tags": [
"a3",
"f5"
]
}],
"User" :"test"
}

Updated Answer - based on updated question statement
You can update attributes in a nested map using update expressions such that only a part of the item would get updated (ie. DynamoDB would apply the equivalent of a patch to your item) but, because DynamoDB is a document database, all operations (Put, Get, Update, Delete etc.) work on the item as a whole.
So, in your example, assuming User is the partition key and that there is no sort key (I didn't see any attribute that could be a sort key in that example), an Update request might look like this:
table.update_item(
Key={
'User': 'test'
},
UpdateExpression="SET #inv[0].#loc = :locVal",
ExpressionAttributeNames={
'#inv': 'inventory',
'#loc': 'location'
},
ExpressionAttributeValues={
':locVal': 'in place three',
},
)
That said, you do have to know what the item schema looks like and which attributes within the item should be updated exactly.
DynamoDB does NOT have a way to operate on sub-items. Meaning, there is no way to tell Dynamo to execute an operation such as "update item, set 'location' property of elements of the 'inventory' array that have a property of 'name' equal to 'opel'"
This is probably not the answer you were hoping for, but it is what's available today. You may be able to get closer to what you want by changing the schema a bit.
If you need to reference the sub-items by name, perhaps storing something like:
{
"inventory": {
"opel": {
"location": "in place one", # I want to update this
"tags": [ "x1", "f3" ]
},
"abc": {
"location": "in place two",
"tags": [ "a3", "f5" ]
}
},
"User" :"test"
}
Then your query would be:
table.update_item(
Key={
'User': 'test'
},
UpdateExpression="SET #inv.#brand.#loc = :locVal",
ExpressionAttributeNames={
'#inv': 'inventory',
'#loc': 'location',
'#brand': 'opel'
},
ExpressionAttributeValues={
':locVal': 'in place three',
},
)
But YMMV as even this has limitations because you are limited to identifying inventory items by name (ie. you still can't say "update inventory with tag 'x1'"
Ultimately you should carefully consider why you need Dynamo to perform these complex operations for you as opposed to you being specific about what you want to update.

You can update the nested map as follow:
First create and empty item attribute of type map. In the example graph is the empty item attribute.
dynamoTable = dynamodb.Table('abc')
dynamoTable.put_item(
Item={
'email': email_add,
'graph': {},
}
Update nested map as follow:
brand_name = 'opel'
DynamoTable = dynamodb.Table('abc')
dynamoTable.update_item(
Key={
'email': email_add,
},
UpdateExpression="set #Graph.#brand= :name, ",
ExpressionAttributeNames={
'#Graph': 'inventory',
'#brand': str(brand_name),
},
ExpressionAttributeValues = {
':name': {
"location": "in place two",
'tag': {
'graph_type':'a3',
'graph_title': 'f5'
}
}

Updating Mike's answer because that way doesn't work any more (at least for me).
It is working like this now (attention for UpdateExpression and ExpressionAttributeNames):
table.update_item(
Key={
'User': 'test'
},
UpdateExpression="SET inv.#brand.loc = :locVal",
ExpressionAttributeNames={
'#brand': 'opel'
},
ExpressionAttributeValues={
':locVal': 'in place three',
},
)
And whatever goes in Key={}, it is always partition key (and sort key, if any).
EDIT:
Seems like this way only works when with 2 level nested properties. In this case you would only use "ExpressionAttributeNames" for the "middle" property (in this example, that would be #brand: inv.#brand.loc). I'm not yet sure what is the real rule now.

DynamoDB UpdateExpression does not search on the database for matching cases like SQL (where you can update all items that match some condition). To update an item you first need to identify it and get primary key or composite key, if there are many items that match your criteria, you need to update one by one.
then the issue to update nested objects is to define UpdateExpression,ExpressionAttributeValues & ExpressionAttributeNames to pass to Dynamo Update Api .
I use a recursive function to update nested Objects on dynamoDB. You ask for Python but I use javascript, I think is easy to see this code and implents on Python:
https://gist.github.com/crsepulv/4b4a44ccbd165b0abc2b91f76117baa5
/**
* Recursive function to get UpdateExpression,ExpressionAttributeValues & ExpressionAttributeNames to update a nested object on dynamoDB
* All levels of the nested object must exist previously on dynamoDB, this only update the value, does not create the branch.
* Only works with objects of objects, not tested with Arrays.
* #param obj , the object to update.
* #param k , the seed is any value, takes sense on the last iteration.
*/
function getDynamoExpression(obj, k) {
const key = Object.keys(obj);
let UpdateExpression = 'SET ';
let ExpressionAttributeValues = {};
let ExpressionAttributeNames = {};
let response = {
UpdateExpression: ' ',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
};
//https://stackoverflow.com/a/16608074/1210463
/**
* true when input is object, this means on all levels except the last one.
*/
if (((!!obj) && (obj.constructor === Object))) {
response = getDynamoExpression(obj[key[0]], key);
UpdateExpression = 'SET #' + key + '.' + response['UpdateExpression'].substring(4); //substring deletes 'SET ' for the mid level values.
ExpressionAttributeNames = {['#' + key]: key[0], ...response['ExpressionAttributeNames']};
ExpressionAttributeValues = response['ExpressionAttributeValues'];
} else {
UpdateExpression = 'SET = :' + k;
ExpressionAttributeValues = {
[':' + k]: obj
}
}
//removes trailing dot on the last level
if (UpdateExpression.indexOf(". ")) {
UpdateExpression = UpdateExpression.replace(". ", "");
}
return {UpdateExpression, ExpressionAttributeValues, ExpressionAttributeNames};
}
//you can try many levels.
const obj = {
level1: {
level2: {
level3: {
level4: 'value'
}
}
}
}

I had the same need.
Hope this code helps. You only need to invoke compose_update_expression_attr_name_values passing the dictionary containing the new values.
def compose_update_expression_attr_name_values(data: dict) -> (str, dict, dict):
""" Constructs UpdateExpression, ExpressionAttributeNames, and ExpressionAttributeValues for updating an entry of a DynamoDB table.
:param data: the dictionary of attribute_values to be updated
:return: a tuple (UpdateExpression: str, ExpressionAttributeNames: dict(str: str), ExpressionAttributeValues: dict(str: str))
"""
# prepare recursion input
expression_list = []
value_map = {}
name_map = {}
# navigate the dict and fill expressions and dictionaries
_rec_update_expression_attr_name_values(data, "", expression_list, name_map, value_map)
# compose update expression from single paths
expression = "SET " + ", ".join(expression_list)
return expression, name_map, value_map
def _rec_update_expression_attr_name_values(data: dict, path: str, expressions: list, attribute_names: dict,
attribute_values: dict):
""" Recursively navigates the input and inject contents into expressions, names, and attribute_values.
:param data: the data dictionary with updated data
:param path: the navigation path in the original data dictionary to this recursive call
:param expressions: the list of update expressions constructed so far
:param attribute_names: a map associating "expression attribute name identifiers" to their actual names in ``data``
:param attribute_values: a map associating "expression attribute value identifiers" to their actual values in ``data``
:return: None, since ``expressions``, ``attribute_names``, and ``attribute_values`` get updated during the recursion
"""
for k in data.keys():
# generate non-ambiguous identifiers
rdm = random.randrange(0, 1000)
attr_name = f"#k_{rdm}_{k}"
while attr_name in attribute_names.keys():
rdm = random.randrange(0, 1000)
attr_name = f"#k_{rdm}_{k}"
attribute_names[attr_name] = k
_path = f"{path}.{attr_name}"
# recursion
if isinstance(data[k], dict):
# recursive case
_rec_update_expression_attr_name_values(data[k], _path, expressions, attribute_names, attribute_values)
else:
# base case
attr_val = f":v_{rdm}_{k}"
attribute_values[attr_val] = data[k]
expression = f"{_path} = {attr_val}"
# remove the initial "."
expressions.append(expression[1:])

Comparing value in a JSON using Python

I receive a fairly uncomfortable JSON to work with, which looks as follows:
[
{
"attributes": [
{
"type": "COMMAND",
"name": "COMMAND",
"value": [
"buttonState"
]
},
{
"type": "std_msgs.msg.Bool",
"name": "buttonState",
"value": {
"data": false
}
}
],
"type": "sensor",
"id": "s_2"
}]
And I would like to compare a piece of data (more precisely - value of Button state) but I seem to fail. Tried following:
import requests
import json
yo = 1
switchPost = "http://192.168.0.104:7896/iot/d?k=123456789&i=san_1_switch&d=sw|{}"
robGet = "http://192.168.0.109:10100/robot/sen_2"
r = requests.get(robGet, headers={"content-type":"application/json"})
resp = json.loads(r.text)
for attrs in (resp['attributes']['value']):
if attrs['data'] == false:
yo = 100
break
g = requests.post(switchPost.format(yo), headers={"content-type":"text/plain"})
print(r.text)
Unfortunately, the error I receive is the following:
for attrs in (resp['attributes']['value']):
TypeError: list indices must be integers, not str

In your JSON, the fact that it is wrapped in [ then ] means it is a JSON array, but with just one element.
So, as your error message suggests, resp needs an integer as its index, for which element of the array you want. resp[0] then refers to
{
"attributes": [
{
"type": "COMMAND",
"name": "COMMAND",
"value": [
"buttonState"
]
},
{
"type": "std_msgs.msg.Bool",
"name": "buttonState",
"value": {
"data": false
}
}
],
"type": "sensor",
"id": "s_2"
}
(notice no [] now, so it's a JSON object)
Then you want resp[0]['attributes'] to refer to the single part of this object, 'attributes' which again refers to an array.
Therefore for attribute in resp[0]['attributes'] will allow you to loop through this array.
To get the boolean value you want, you'll then want to find which element of that array has 'name' of 'buttonState' and check the corresponding 'value'.
In all, you're probably looking for something like:
for attribute in resp[0]['attributes']:
if attribute['name'] == 'buttonState' and attribute['value']['data'] is False:
# Do your thing here

resp is a list so, to get first element, access it as resp[0]. Same with resp[0]['attributes']
So you can access it as follows
resp[0]['attributes'][0]['value']
You can restructure your for loop as follows
for d in resp[0]['attributes']:
if isinstance(d['value'], dict) and d['value'].get('data') == false:
yo = 100
break

The answer is in the error message I think:
TypeError: list indices must be integers, not str
The first entry in attributes has a value that is a list, so you can't get 'data' from that.
Since you have a mix of types, you might need to check if 'value' is a list or a dict.
Edit:
Jumped the gun here I think. #dennlinger gives an explanation to your error message. But you'll get it again once you're past that...