We Keep Coding

Updating double nested object in MongoDB with pymongo

I have the following object in MongoDB:
{
"_id":"...",
"username":"XXX",
"keys":[
{
"key":"c443c2cc2754d3",
"ref":"Autos",
"lists":[
{
"list_name":"Toyota",
"key":"c443c2cc2754d3",
"broken_parts":{
"headlights":false,
"bonnet":true,
"interior":{
"dashboard":true,
"electronics":false
}
},
"timestamp":"2023-01-26T13:00:21.803Z",
"status":"parked"
},
{
"list_name":"Nissan",
"key":"c443c2cc2754d3",
"broken_parts":{
"headlights":true,
"bonnet":false,
"interior":{
"dashboard":false,
"electronics":true
}
},
"timestamp":"2023-01-26T13:00:21.803Z",
"status":"garage"
}
]
},
{
"key":"80d54bd834ff60",
"ref":"Trucks",
"lists":[
{
"list_name":"MAN",
"key":"c443c2cc2754d3",
"broken_parts":{
"headlights":false,
"bonnet":false,
"interior":{
"dashboard":false,
"electronics":false
}
},
"timestamp":"2023-01-26T13:00:21.803Z",
"status":"parked"
},
{
"list_name":"Toyota",
"key":"c443c2cc2754d3",
"broken_parts":{
"headlights":true,
"bonnet":false,
"interior":{
"dashboard":true,
"electronics":true
}
},
"timestamp":"2023-01-26T13:00:21.803Z",
"status":"leased"
}
]
}
]
}
and whenever I try to update the status of an item inside a list, it only updates the first object.
If I try to update the status of the 1st list, 2nd object, it will only update the first one:
self.users.update_one({"username": username,
"keys.key": key,
"keys.lists.list_name": listid
},
{"$set": {
"keys.0.lists.$.status": 'repaired'
}
}
)
What am I doing wrong and how can I update the list item I want? or even the 3rd nested object (dashboard, electronics)?

Return empty columns in athena boto response object

When executing a select query against an athena table via boto3, the response object given is in the syntax:
{
"UpdateCount":0,
"ResultSet":{
"Rows":[
{
"Data":[
{
"VarCharValue":"site_name"
},
{
"VarCharValue":"volume_out_capacity"
},
{
"VarCharValue":"region"
},
{
"VarCharValue":"site_ref"
}
]
},
{
"Data":[
{
"VarCharValue":"ASSET 12"
},
{
"VarCharValue":"10"
},
{
"VarCharValue":"NORTH"
},
{
"VarCharValue":"RHW007777000138"
}
]
}
]
}
Is there an additional argument that can be passed so that the response object will contain columns that do not contain values? Something like:
{
"VarCharValue":"xyz"
}
]
},
{
"Data":[
{
"VarCharValue":None
}
I have looked through the documentation extensively but cannot find arguments that can describe how to format the response in get_query_results() or start_query_execution()

I do not see a option to get that data back directly from Athena. Alternatively if you use S3 Select instead of Athena, you'll get back a json object with all the columns whether they have data or are empty.
Sample Data:
name,number,city,state
coin,1,driggs,
michelle,,chicago,
shaniqua,2,,
marcos,3,stlouis,
S3 Select Result:
{"name":"coin","number":"1","city":"driggs","state":""}
{"name":"michelle","number":"","city":"chicago","state":""}
{"name":"shaniqua","number":"2","city":"","state":""}
{"name":"marcos","number":"3","city":"stlouis","state":""}
Code:
import boto3
session = boto3.session.Session(profile_name="<my-profile>")
s3 = session.client('s3')
resp = s3.select_object_content(
Bucket='<my-bucket>',
Key='<my-file>',
ExpressionType='SQL',
Expression="SELECT * FROM s3object",
InputSerialization={'CSV': {"FileHeaderInfo": "Use", "RecordDelimiter": '\r\n'}, 'CompressionType': 'NONE'},
OutputSerialization={'JSON': {}},
)
for event in resp['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
print(records)
elif 'Stats' in event:
statsDetails = event['Stats']['Details']
print("Stats details bytesScanned: ")
print(statsDetails['BytesScanned'])
print("Stats details bytesProcessed: ")
print(statsDetails['BytesProcessed'])
print("Stats details bytesReturned: ")
print(statsDetails['BytesReturned'])

pymongo - Update a data and access the found value

I am trying to update a value of an array stored in a mongodb collection
any_collection: {
{
"_id": "asdw231231"
"values": [
{
"item" : "a"
},
{
"item" : "b"
}
],
"role": "role_one"
},
...many similar
}
the idea is that I want to access values ​​and edit a value with the following code that I found in the mongodb documentation
conn.any_collection.find_one_and_update(
{
"_id": any_id,
"values.item": "b"
},
{
"$set": {
"values.$.item": "new_value" # here the error, ".$."
}
}
)
This should work, but I can't understand what the error is or what is the correct syntax for pymongo. The error is generated when adding "$";

It works fine with my fastAPI.
#app.get("/find/{id}")
async def root(id: int):
db = get_database()
q = {'_id': 'asdw231231','values.item': 'b'}
u = {'$set': {'values.$.item': 'new_value' }}
c = db['any'].find_one_and_update(q, u)
return {"message": c}
mongoplayground

Pymongo: Unable to find record from mongodb

I have a collection containing country records, I need to find particular country with uid and it's countryId
Below is the sample collection data:
{
"uid": 15024,
"countries": [{
"countryId": 123,
"popullation": 45000000
},
{
"countryId": 456,
"poppulation": 9000000000
}
]
},
{
"uid": 15025,
"countries": [{
"countryId": 987,
"popullation": 560000000
},
{
"countryId": 456,
"poppulation": 8900000000
}
]
}
I have tried with below query in in python but unable to find any result:
foundRecord = collection.find_one({"uid" : 15024, "countries.countryId": 456})
but it return None.
Please help and suggest.

I think following will work better :
foundRecord = collection.find_one({"uid" : 15024,
"countries" : {"$elemMatch" : { "countryId" : 456 }})

Are you sure you're using the same Database / Collection source?
Seems that you're saving results on another collection.
I've tried to reproduce your problem and it works on my mongodb ( note that I'm using v4)
EDIT: Would be nice to have the piece of code where you're defining "collection"

Issue parsing JSON file-Python

Have this section in one large JSON file
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "AIDAI2YJINPRUEM3XHKXO",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/citrix-xendesktop-ec2-provisioning"
},
Need to extract AttachedManagedPolicy.Policy name for user
Desired output:
"citrix-xendesktop-ec2-provisioning","AmazonEC2FullAccess"
"citrix-xendesktop-ec2-provisioning","AmazonS3FullAccess"
Some users don't have any policy at all so need some checking mechanism to avoid errors
with open('1.json') as file:
data = json.load(file)
for element in data['UserDetailList']:
s = element['UserName'], element['AttachedManagedPolicies']
print s
And getting
(u'citrix-xendesktop-ec2-provisioning', [{u'PolicyName': u'AmazonEC2FullAccess', u'PolicyArn': u'arn:aws:iam::aws:policy/AmazonEC2FullAccess'}, {u'PolicyName': u'AmazonS3FullAccess', u'PolicyArn': u'arn:aws:iam::aws:policy/AmazonS3FullAccess'}])
When added element['AttachedManagedPolicies']['PolicyName']
got: TypeError: list indices must be integers, not str

You are getting error because element['AttachedManagedPolicies'] is list not dictionary you need to iterate over element['AttachedManagedPolicies'] and then access key as below:
[i['PolicyName'] for i in element['AttachedManagedPolicies']]
this will construct list of values for key PolicyName
As you said you have very large JSON structure you might have empty values or not values and for that you can proceed as below:
d = {
"UserDetailList": [
{
"UserName": "citrix-xendesktop-ec2-provisioning",
"GroupList": [],
"CreateDate": "2017-11-07T14:20:14Z",
"UserId": "AIDAI2YJINPRUEM3XHKXO",
"Path": "/",
"AttachedManagedPolicies": [
{
"PolicyName": "AmazonEC2FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
},
{
"PolicyName": "AmazonS3FullAccess",
"PolicyArn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
],
"Arn": "arn:aws:iam::279052847476:user/citrix-xendesktop-ec2-provisioning"
}
]
}
user_list = d.get("UserDetailList", None) # if unable to fetch key then it will return None
if user_list:
for user_detail in user_list:
username = user_detail.get("UserName", None)
policies = [i.get('PolicyName') for i in user_detail.get('AttachedManagedPolicies', []) if i.get('PolicyName', None)] # empty list constructed if no policy exist
print(username, policies)