We Keep Coding

S3 Select Query JSON for nested value when keys are dynamic

I have a JSON object in S3 which follows this structure:
<code> : {
<client>: <value>
}
For example,
{
"code_abc": {
"client_1": 1,
"client_2": 10
},
"code_def": {
"client_2": 40,
"client_3": 50,
"client_5": 100
},
...
}
I am trying to retrieve the numerical value with an S3 Select query, where the "code" and the "client" are populated dynamically with each query.
So far I have tried:
sql_exp = f"SELECT * from s3object[*][*] s where s.{proc}.{client_name} IS NOT NULL"
sql_exp = f"SELECT * from s3object s where s.{proc}[*].{client_name}[*] IS NOT NULL"
as well as without the asterisk inside the square brackets, but nothing works, I get ClientError: An error occurred (ParseUnexpectedToken) when calling the SelectObjectContent operation: Unexpected token found LITERAL:UNKNOWN at line 1, column X (depending on the length of the query string)
Within the function defining the object, I have:
resp = s3.select_object_content(
Bucket=<bucket>,
Key=<filename>,
ExpressionType="SQL",
Expression=sql_exp,
InputSerialization={'JSON': {"Type": "Document"}},
OutputSerialization={"JSON": {}},
)
Is there something off in the way I define the object serialization? How can I fix the query so I can retrieve the desired numerical value on the fly when I provide ”code” and “client”?

I did some tinkering based on the documentation, and it works!
I need to access the single event in the EventStream (resp) as follows:
event_stream = resp['Payload']
# unpack successful query response
for event in event_stream:
if "Records" in event:
output_str = event["Records"]["Payload"].decode("utf-8") # bytes to string
output_dict = json.loads(output_str) # string to dict
Now the correct SQL expression is:
sql_exp= f"SELECT s['{code}']['{client}'] FROM S3Object s"
where I have gotten (dynamically) my values for code and client beforehand.
For example, based on the dummy JSON structure above, if code = "code_abc" and client = "client_2", I want this S3 Select query to return the value 10.
The f-string resolves to sql_exp = "SELECT s['code_abc']['client_2'] FROM S3Object s", and when we call resp, we retrieve output_dict = {'client_2': 10} (Not sure if there is a clear way to get the value by itself without the client key, this is how it looks like in the documentation as well).
So, the final step is to retrieve value = output_dict['client_2'], which in our case is equal to 10.

How to handle missing JSON nested keys from an API response in python?

Here is the JSON response I get from an API request:
{
"associates": [
{
"name":"DOE",
"fname":"John",
"direct_shares":50,
"direct_shares_details":{
"shares_PP":25,
"shares_NP":25
},
"indirect_shares":50,
"indirect_shares_details": {
"first_type": {
"shares_PP": 25,
"shares_NP": 0
},
"second_type": {
"shares_PP": 25,
"shares_NP": 0
}
}
}
]
}
However, in some occasions, some values will be equal to None. In that case, I handle it in my function for all the values that I know will be integers. But it doesn't work in this scenario for the nested keys inside indirect_shares_details:
{
"associates": [
{
"name":"DOE",
"fname":"John",
"direct_shares":50,
"direct_shares_details":{
"shares_PP":25,
"shares_NP":25
},
"indirect_shares":None,
"indirect_shares_details": None
}
}
]
}
So when I run my function to get the API values and put them in a custom dict, I get an error because the keys are simply inexistant in the response.
def get_shares_data(response):
associate_from_api = []
for i in response["associates"]:
associate_data = {
"PM_shares": round(company["Shares"], 2),
"full_name": i["name"] + " " + ["fname"]
"details": {
"shares_in_PM": i["direct_shares"],
"shares_PP_in_PM": i["direct_shares_details"]["shares_PP"],
"shares_NP_in_PM": i["direct_shares_details"]["shares_NP"],
"shares_directe": i["indirect_shares"],
"shares_indir_PP_1": i["indirect_shares_details"]["first_type"]["shares_PP"],
"shares_indir_NP_1": i["indirect_shares_details"]["first_type"]["shares_NP"],
"shares_indir_PP_2": i["indirect_shares_details"]["second_type"]["shares_PP"],
"shares_indir_NP_2": i["indirect_shares_details"]["second_type"]["shares_NP"],
}
}
for key,value in associate_data["details"].items():
if value != None:
associate_data["details"][key] = value * associate_data["PM_shares"] / 100
else:
associate_data["calculs"][key] = 0.0
associate_from_api.append(associate_data)
return associate_from_api
I've tried conditioning the access of the nested keys only if the parent key wasn't equal to None but I ended up declaring 3 different dictionaries inside if/else conditions and it turned into a mess, is there an efficient way to achieve this?

You can try accessing the values using dict.get('key') instead of accessing them directly, as in dict['key'].
Using the first approach, you will get None instead of KeyError if the key is not there.
EDIT: tested using the dictionary from the question:

You can try pydantic
Install pydantic
pip install pydantic
# OR
conda install pydantic -c conda-forge
Define some models based on your response structure
from pydantic import BaseModel
from typing import List, Optional
# There are some common fields in your json response.
# So you can put them together.
class ShareDetail(BaseModel):
shares_PP: int
shares_NP: int
class IndirectSharesDetails(BaseModel):
first_type: ShareDetail
second_type: ShareDetail
class Associate(BaseModel):
name: str
fname: str
direct_shares: int
direct_shares_details: ShareDetail
indirect_shares: int = 0 # Sets a default value for this field.
indirect_shares_details: Optional[IndirectSharesDetails] = None
class ResponseModel(BaseModel):
associates: List[Associate]
use ResponseModel.parse_xxx functions to parse response.
Here I use parse_file funtion, you can also use parse_json function
See: https://pydantic-docs.helpmanual.io/usage/models/#helper-functions
def main():
res = ResponseModel.parse_file("./NullResponse.json",
content_type="application/json")
print(res.dict())
if __name__ == "__main__":
main()
Then the response can be successfully parsed. And it automatically validates the input.

Python how to pass JSON objects to a function?

I'm parsing a json file that looks like:
my_json:
"DataChangedEntry": {
"CurrentValue": {
"RefId": {
"Value": "aaaaaaa"
So to get "Value" it looks like:
my_json["DataChangedEntry"]["CurrentValue"]["RefId"]["Value"]
I want to send it to a try/except function (because I have a lot of fields to get) but I don't know how to send the json object over.
I've tried:
get_value = my_function(my_json, ["DataChangedEntry"]["CurrentValue"]["RefId"]["Value"])
But I get error:
TypeError: list indices must be integers or slices, not str
The my_function is just
def my_function(json_prefix, json_field):
try:
value = json_prefix[json_field]
return value
except:
logging.exception('Exception: ')

You have to pass each key as a separate argument (or as a list of separate arguments).
def my_function(obj, *fields):
for f in fields:
try:
obj = obj[f]
except KeyError:
logging.exception("Exceptoin: ")
return
return obj
my_function(my_json, "DataChangedEntry", "CurrentValue", ...)

I confess that this idea of sending this data separately to a function is quite different.
But to directly return the value aaaaaaa:
Send the first argument as the JSON object value
Send the second argument as the JSON object name string
Send third argument as key list
Then you can use eval() to convert the union of strings into code:
def my_function(json_full, json_prefix, json_field):
my_json = json_full
my_json_str = json_prefix
key_field = '["' + '"]["'.join(json_field) + '"]'
try:
value = eval(f'{json_prefix}{key_field}')
return value
except Exception as e:
return e
def main():
my_json = {
"DataChangedEntry": {
"CurrentValue": {
"RefId": {
"Value": "aaaaaaa"
},
},
},
}
get_value = my_function(my_json, 'my_json', ["DataChangedEntry","CurrentValue","RefId","Value"])
print(get_value)
if __name__ == "__main__":
main()
Output:
aaaaaaa

Python container troubles

Basically what I am trying to do is generate a json list of SSH keys (public and private) on a server using Python. I am using nested dictionaries and while it does work to an extent, the issue lies with it displaying every other user's keys; I need it to list only the keys that belong to the user for each user.
Below is my code:
def ssh_key_info(key_files):
for f in key_files:
c_time = os.path.getctime(f) # gets the creation time of file (f)
username_list = f.split('/') # splits on the / character
user = username_list[2] # assigns the 2nd field frome the above spilt to the user variable
key_length_cmd = check_output(['ssh-keygen','-l','-f', f]) # Run the ssh-keygen command on the file (f)
attr_dict = {}
attr_dict['Date Created'] = str(datetime.datetime.fromtimestamp(c_time)) # converts file create time to string
attr_dict['Key_Length]'] = key_length_cmd[0:5] # assigns the first 5 characters of the key_length_cmd variable
ssh_user_key_dict[f] = attr_dict
user_dict['SSH_Keys'] = ssh_user_key_dict
main_dict[user] = user_dict
A list containing the absolute path of the keys (/home/user/.ssh/id_rsa for example) is passed to the function. Below is an example of what I receive:
{
"user1": {
"SSH_Keys": {
"/home/user1/.ssh/id_rsa": {
"Date Created": "2017-03-09 01:03:20.995862",
"Key_Length]": "2048 "
},
"/home/user2/.ssh/id_rsa": {
"Date Created": "2017-03-09 01:03:21.457867",
"Key_Length]": "2048 "
},
"/home/user2/.ssh/id_rsa.pub": {
"Date Created": "2017-03-09 01:03:21.423867",
"Key_Length]": "2048 "
},
"/home/user1/.ssh/id_rsa.pub": {
"Date Created": "2017-03-09 01:03:20.956862",
"Key_Length]": "2048 "
}
}
},
As can be seen, user2's key files are included in user1's output. I may be going about this completely wrong, so any pointers are welcomed.

Thanks for the replies, I read up on nested dictionaries and found that the best answer on this post, helped me solve the issue: What is the best way to implement nested dictionaries?
Instead of all the dictionaries, I simplfied the code and just have one dictionary now. This is the working code:
class Vividict(dict):
def __missing__(self, key): # Sets and return a new instance
value = self[key] = type(self)() # retain local pointer to value
return value # faster to return than dict lookup
main_dict = Vividict()
def ssh_key_info(key_files):
for f in key_files:
c_time = os.path.getctime(f)
username_list = f.split('/')
user = username_list[2]
key_bit_cmd = check_output(['ssh-keygen','-l','-f', f])
date_created = str(datetime.datetime.fromtimestamp(c_time))
key_type = key_bit_cmd[-5:-2]
key_bits = key_bit_cmd[0:5]
main_dict[user]['SSH Keys'][f]['Date Created'] = date_created
main_dict[user]['SSH Keys'][f]['Key Type'] = key_type
main_dict[user]['SSH Keys'][f]['Bits'] = key_bits

How I do multiple eq_joins?

I'm trying to do multiple eq_joins. The error I get is:
ReqlQueryLogicError: Primary keys must be either a number, string, bool, pseudotype or array (got type OBJECT):
{
"First name": "John",
"Last name": "Urquhart",
"employers": [
{
"date_hired": "2-Mar-88",
"organization_id": "2a5e2e3d-275a-426e-9ecd-0bd5601bff6b"
}
],
"id": "e70d5350-c1e0-41ee-a1cc-6638c7136d89",
"primary_photo": "http://www.kingcounty.gov/~/media/safety/sheriff/Sheriff_Urquh in:
r.db('public').table(u'police_internal_affairs_allegations').filter(lambda var_24: var_24.coerce_to('string').match(u'(?i).*?Urquhart.*?')).eq_join(u'organization_id', r.db('public').table(u'organizations')).merge(lambda var_25: r.expr({'right': var_25['right'].coerce_to('array').map(lambda var_26: [(r.expr(u'organization_') + var_26[0]), var_26[1]]).coerce_to('object')})).zip().eq_join(u'person_id', r.db('public').table('people')).merge(lambda var_27: r.expr({'right': var_27['right'].coerce_to('array').map(lambda var_28: [(r.expr(u'person_') + var_28[0]), var_28[1]]).coerce_to('object')})).zip()
My code is:
ids_for_other_tables = [field for field in fields if field.endswith('_id')]
modified_joined_data = []
for field in ids_for_other_tables:
special_names = {'person': 'people'}
t = special_names[field[:-3]] if field[:-3] in special_names else field[:-3]+'s'
dbobj = getattr(dbobj, 'eq_join')(field, r.db("public").table(t))
dbobj = dbobj.merge( lambda row: {'right': row['right'].coerce_to('array').map(
lambda pair: [r.expr(field[:-2]) + pair[0], pair[1]]
).coerce_to('object')})
dbobj = dbobj.zip()
The purpose of this code is auto join info from tables for all fields ending in _id

It's hard to say without looking at the data in your table, but it looks like the problem is that one of the fields you're trying to eq_join on has an object in that field instead of an ID. I'd run the part of the query before the eq_join and make sure it has the format you're expecting.