I'm receiving following data from DynamoDB as a response to client scan API call.
Data = [{'id': {'S': '5'},
'FirstName': {'S': 'Prashanth'},
'LastName': {'S': 'Wadeyar'},
'ClientName': {'S': 'Test'}}]
I want to handle this response and get the output as
{'FirstName':'Prashanth', 'LastName': 'Wadeyar', 'ClientName': 'Test'}
I can handle it by separating it like
for field_obj in data:
obj = (field_obj['FirstName'])
but to get value of Firstname, Key 'S' may differ for each object. like boolean string, list etc.
is there a easy way to get the key and value pairs.
If you don't want to bring external dependencies you can use Table class as described here.
import boto3
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users')
response = table.get_item(
Key={
'username': 'janedoe',
'last_name': 'Doe'
}
)
item = response['Item']
print(item)
Expected Output:
{u'username': u'janedoe',
u'first_name': u'Jane',
u'last_name': u'Doe',
u'account_type': u'standard_user',
u'age': Decimal('25')}
But my personal preference everytime I hear about Python and DynamoDB is to use PynamoDB: https://pynamodb.readthedocs.io/en/latest/ which is sort of an ORM for DynamoDB.
Above answer from Mayank seems to work well, but I also wanted to map fields I want to return in a list. Hence I mapped field and then was able to get the result I was looking for
FIELD_MAP = ['firstName', 'lastName',
'clientName']
for field_obj in response['Items']:
for key in FIELD_MAP:
if field_obj.get(key):
obj = (field_obj.get(key))
for i in obj:
value = obj[i]
db_result.append({key: value})
print(db_result)
Something like this could work, where you don't need to care about the internal keys(S in your case):
In [1018]: d = {}
In [1016]: for i in Data:
...: for k,v in i.items():
...: if k != 'id':
...: if isinstance(v, dict):
...: for j, val in v.items():
...: d[k] = val
...:
In [1024]: d
Out[1024]: {'FirstName': 'Prashanth', 'LastName': 'Wadeyar', 'ClientName': 'Test'}
Related
Heres the original dict
uid_coins = {'141632864': {'username': 'Guest130679138', 'coins': 0},
'141632884': {'username': 'Guest130679156', 'coins': 39123441},
'141632886': {'username': 'Guest130679158', 'coins': 44006638}}
What I am trying to get
d = {'uid':[141632864, 141632884, 141632886],
'username': ['Guest130679138', 'Guest130679156', 'Guest130679158'],
'coins': [0, 39123441, 44006638]}
The keys in original dict represent uid.
This is my what I have done:
uid = list(uid_coins.keys())
username = [u_data['username'] for u_data in uid_coins.values()]
coins = [[u_data['coins'] for u_data in uid_coins.values()]]
d = {"uid":uid, "username":username, "coins":coins}
dict((key,d[key]) for d in data for key in d)
But I am rather looking for a generalized approach, to achieve without manually declaring the keys again, just so it should work with new keys in the original data.
Try:
uid_coins = {
"141632864": {"username": "Guest130679138", "coins": 0},
"141632884": {"username": "Guest130679156", "coins": 39123441},
"141632886": {"username": "Guest130679158", "coins": 44006638},
}
out = {}
for k, v in uid_coins.items():
out.setdefault("uid", []).append(k)
out.setdefault("username", []).append(v["username"])
out.setdefault("coins", []).append(v["coins"])
print(out)
Prints:
{'uid': ['141632864', '141632884', '141632886'],
'username': ['Guest130679138', 'Guest130679156', 'Guest130679158'],
'coins': [0, 39123441, 44006638]}
Generalized (based on original version by Andrej Kesely):
d = {}
for k, v in uid_coins.items():
d.setdefault('uid', []).append(k)
for i in v.keys():
d.setdefault(i, []).append(v[i])
A similar version as #Andrej Kesely but with defaultdict
from collections import defaultdict
uid_coins = {
'141632864': {'username': 'Guest130679138', 'coins': 0},
'141632884': {'username': 'Guest130679156', 'coins': 39123441},
'141632886': {'username': 'Guest130679158', 'coins': 44006638}
}
d = defaultdict(list)
for key, value in uid_coins.items():
d['uid'].append(key)
d['username'].append(value['username'])
d['coins'].append(value['coins'])
Output:
defaultdict(<class 'list'>, {'uid': ['141632864', '141632884', '141632886'], 'username': ['Guest130679138', 'Guest130679156', 'Guest130679158'], 'coins': [0, 39123441, 44006638]})
This is how you can get the output in desired format (as mentioned in question) from the given dictionary.
d = {'uid': list(uid_coins.keys()), 'username': [i['username'] for i in list(uid_coins.values())], 'coins': [i['coins'] for i in list(uid_coins.values())]}
list(uid_coins.keys()) will return all your uids as list.
[i['username'] for i in list(uid_coins.values())] will return 'username' values as list.
[i['coins'] for i in list(uid_coins.values())] will return 'coins' values as list.
I have a collection in MongoDB of nested objects (basically a tree structure). I am trying to access certain "id" and "user_id" values in "children".
The collection looks like this:
Image of a tree object in MongoDB
When I query "children" I get, e.g., this as output:
[{'children': [{'children': [{'children': [{'id': 737992252891537408, 'user_id': 3240396143}], 'id': 737991958161940480, 'user_id': 3240396143}], 'id': 737986305481682944, 'user_id': 56377143}], 'id': 737979183599652864, 'user_id': 3240396143}], 'id': 737978059291234304, 'user_id': 3240396143}]}
How do I efficiently access all the "id"'s with the 'user_id' = 56377143? I cannot seem to get all of them when it is nested too deep.
I tried using a for loop like this but it does not output all the 'id's which match the 'user_id's
val= "children"
lst_rt= []
lst_ids = []
def get_value(mydict, keys):
if type(mydict) == dict:
print(mydict[0]['user_id'], mydict[0]['id'], "TEST")
return get_value(mydict[keys], val)
if type(mydict) == list and keys in mydict[0] and mydict[0]['user_id'] == 56377143 :
print(mydict[0]['id'],mydict[0]['user_id'], 'COND')
return get_value(mydict[0][keys], val)
elif mydict[0]['user_id'] == 56377143 and mydict[0]['id'] != None:
print(mydict[0]['id'], mydict[0]['user_id'])
lst_rt.append(int(mydict[0]['id']))
return mydict[0]['id']
for x in root_tweets:
print(get_value(x['children'], val))
I have an API response for listing out information of all Volumes. I want to loop through the response and get the value of the name and assign each one of them dynamically to each url.
This is my main API endpoint which returns the following:
[{'source': None, 'serial': '23432', 'created': '2018-11-
12T04:27:14Z', 'name': 'v001', 'size':
456456}, {'source': None, 'serial': '4364576',
'created': '2018-11-12T04:27:16Z', 'name': 'v002',
'size': 345435}, {'source': None, 'serial':
'6445645', 'created': '2018-11-12T04:27:17Z', 'name': 'v003', 'size':
23432}, {'source': None,
'serial': 'we43235', 'created': '2018-11-12T04:27:20Z',
'name': 'v004', 'size': 35435}]
I'm doing this to get the value of 'name'
test_url = 'https://0.0.0.0/api/1.1/volume'
test_data = json.loads(r.get(test_url, headers=headers,
verify=False).content.decode('UTF-8'))
new_data = [{
'name': value['name']
} for value in test_data]
final_data = [val['name'] for val in new_data]
for k in final_data:
print(k)
k prints out all the values in name, but i'm stuck at where i want to be able to use it in assigning different API endpoints. Now, k returns
v001
v002
v003
v004
I want to assign each one of them to different endpoints like below:
url_v001 = test_url + v001
url_v002 = test_url + v002
url_v003 = test_url + v003
url_v004 = test_url + v004
I want this to be dynamically done, because there may be more than 4 volume names returned by my main API.
It wouldn't be good to do that, but the best way is to use a dictionary:
d={}
for k in final_test:
d['url_'+k] = test_url + k
Or much better in a dictionary comprehension:
d={'url_'+k:test_url + k for k in final_test}
And now:
print(d)
Both reproduce:
{'url_v001': 'https://0.0.0.0/api/1.1/volumev001', 'url_v002': 'https://0.0.0.0/api/1.1/volumev002', 'url_v003': 'https://0.0.0.0/api/1.1/volumev003', 'url_v004': 'https://0.0.0.0/api/1.1/volumev004'}
To use d:
for k,v in d.items():
print(k+',',v)
Outputs:
url_v001, https://0.0.0.0/api/1.1/volumev001
url_v002, https://0.0.0.0/api/1.1/volumev002
url_v003, https://0.0.0.0/api/1.1/volumev003
url_v004, https://0.0.0.0/api/1.1/volumev004
I have a list of dictionaries in python and I would like to override old value with duplicate value. Please let me know how can I do.
{'message': [{'name': 'raghav', 'id': 10}, {'name': 'raghav', 'id': 11}]}
Output should be:
{'message': [ {'name': 'raghav', 'id': 11}]}
I don't know what you mean by "override old value with duplicate value". If you mean just picking the second dict from the list, you could:
print({k: [v[1]] for (k, v) in data.items()})
If the idea is to update the "name" with a newer value of "id" as you move along the list, then maybe:
def merge_records(data):
records = data['message']
users = {}
for record in records:
name = record['name']
id_ = record['id']
users[name] = id_
new_records = []
for name, id_ in users.items():
new_records.append({'name': name, 'id': id_})
return {'message': new_records}
But, if you have any control over how the data is represented, you might reconsider. You probably want a different data structure.
Here you go:
d = {'message': [{'name': 'raghav', 'id': 10}, {'name': 'raghav', 'id': 11}]}
#loop over outer dictionary
for key, value in d.items():
d[key] = [dict([t for k in value for t in k.items()])]
print(d)
Edit:
As per your requirement:
d = {'message': [ {'name': 'raghav', 'id': 11}, {'name': 'krish', 'id': 20}, {'name': 'anu', 'id': 30}]}
for key, value in d.items():
print [dict((k1,v1)) for k1,v1 in dict([tuple(i.items()) for i in value for val in i.items()]).items()]
I have a database schema in Postgres that looks like this (in pseudo code):
users (table):
pk (field, unique)
name (field)
permissions (table):
pk (field, unique)
permission (field, unique)
addresses (table):
pk (field, unique)
address (field, unique)
association1 (table):
user_pk (field, foreign_key)
permission_pk (field, foreign_key)
association2 (table):
user_pk (field, foreign_key)
address_pk (field, foreign_key)
Hopefully this makes intuitive sense. It's a users table that has a many-to-many relationship with a permissions table as well as a many-to-many relationship with an addresses table.
In Python, when I perform the correct SQLAlchemy query incantations, I get back results that look something like this (after converting them to a list of dictionaries in Python):
results = [
{'pk': 1, 'name': 'Joe', 'permission': 'user', 'address': 'home'},
{'pk': 1, 'name': 'Joe', 'permission': 'user', 'address': 'work'},
{'pk': 1, 'name': 'Joe', 'permission': 'admin', 'address': 'home'},
{'pk': 1, 'name': 'Joe', 'permission': 'admin', 'address': 'work'},
{'pk': 2, 'name': 'John', 'permission': 'user', 'address': 'home'},
]
So in this contrived example, Joe is both a user and and an admin. John is only a user. Both Joe's home and work addresses exist in the database. Only John's home address exists.
So the question is, does anybody know the best way to go from these SQL query 'results' to the more compact 'desired_results' below?
desired_results = [
{
'pk': 1,
'name': 'Joe',
'permissions': ['user', 'admin'],
'addresses': ['home', 'work']
},
{
'pk': 2,
'name': 'John',
'permissions': ['user'],
'addresses': ['home']
},
]
Additional information required: Small list of dictionaries describing the 'labels' I would like to use in the desired_results for each of the fields that have many-to-many relationships.
relationships = [
{'label': 'permissions', 'back_populates': 'permission'},
{'label': 'addresses', 'back_populates': 'address'},
]
Final consideration, I've put together a concrete example for the purposes of this question, but in general I'm trying to solve the problem of querying SQL databases in general, assuming an arbitrary amount of relationships. SQLAlchemy ORM solves this problem well, but I'm limited to using SQLAlchemy Core; so am trying to build my own solution.
Update
Here's an answer, but I'm not sure it's the best / most efficient solution. Can anyone come up with something better?
# step 1: generate set of keys that will be replaced by new keys in desired_result
back_populates = set(rel['back_populates'] for rel in relationships)
# step 2: delete from results keys generated in step 1
intermediate_results = [
{k: v for k, v in res.items() if k not in back_populates}
for res in results]
# step 3: eliminate duplicates
intermediate_results = [
dict(t)
for t in set([tuple(ires.items())
for ires in intermediate_results])]
# step 4: add back information from deleted fields but in desired form
for ires in intermediate_results:
for rel in relationships:
ires[rel['label']] = set([
res[rel['back_populates']]
for res in results
if res['pk'] == ires['pk']])
# done
desired_results = intermediate_results
Iterating over the groups of partial entries looks like a job for itertools.groupby.
But first lets put relationships into a format that is easier to use, prehaps a back_populates:label dictionary?
conversions = {d["back_populates"]:d['label'] for d in relationships}
Next because we will be using itertools.groupby it will need a keyfunc to distinguish between the different groups of entries.
So given one entry from the initial results, this function will return a dictionary with only the pairs that will not be condensed/converted
def grouper(entry):
#each group is identified by all key:values that are not identified in conversions
return {k:v for k,v in entry.items() if k not in conversions}
Now we will be able to traverse the results in groups something like this:
for base_info, group in itertools.groupby(old_results, grouper):
#base_info is dict with info unique to all entries in group
for partial in group:
#partial is one entry from results that will contribute to the final result
#but wait, what do we add it too?
The only issue is that if we build our entry from base_info it will confuse groupby so we need to make an entry to work with:
entry = {new_field:set() for new_field in conversions.values()}
entry.update(base_info)
Note that I am using sets here because they are the natural container when all contence are unique,
however because it is not json-compatible we will need to change them into lists at the end.
Now that we have an entry to build we can just iterate through the group to add to each new field from the original
for partial in group:
for original, new in conversions.items():
entry[new].add(partial[original])
then once the final entry is constructed all that is left is to convert the sets back into lists
for new in conversions.values():
entry[new] = list(entry[new])
And that entry is done, now we can either append it to a list called new_results but since this process is essentially generating results it would make more sense to put it into a generator
making the final code look something like this:
import itertools
results = [
{'pk': 1, 'name': 'Joe', 'permission': 'user', 'address': 'home'},
{'pk': 1, 'name': 'Joe', 'permission': 'user', 'address': 'work'},
{'pk': 1, 'name': 'Joe', 'permission': 'admin', 'address': 'home'},
{'pk': 1, 'name': 'Joe', 'permission': 'admin', 'address': 'work'},
{'pk': 2, 'name': 'John', 'permission': 'user', 'address': 'home'},
]
relationships = [
{'label': 'permissions', 'back_populates': 'permission'},
{'label': 'addresses', 'back_populates': 'address'},
]
#first we put the "relationships" in a format that is much easier to use.
conversions = {d["back_populates"]:d['label'] for d in relationships}
def grouper(entry):
#each group is identified by all key:values that are not identified in conversions
return {k:v for k,v in entry.items() if k not in conversions}
def parse_results(old_results, conversions=conversions):
for base_info, group in itertools.groupby(old_results, grouper):
entry = {new_field:set() for new_field in conversions.values()}
entry.update(base_info)
for partial in group: #for each entry in the original results set
for original, new in conversions.items(): #for each field that will be condensed
entry[new].add(partial[original])
#convert sets back to lists so it can be put back into json
for new in conversions.values():
entry[new] = list(entry[new])
yield entry
Then the new_results can be gotten like this:
>>> new_results = list(parse_results(results))
>>> from pprint import pprint #for demo purpose
>>> pprint(new_results,width=50)
[{'addresses': ['home', 'work'],
'name': 'Joe',
'permissions': ['admin', 'user'],
'pk': 1},
{'addresses': ['home'],
'name': 'John',
'permissions': ['user'],
'pk': 2}]