I have such structure:
actions = {
'cat1': {
'visit': {
'id': 1,
'description': 'Desc 1',
'action': 'Act 1',
},
},
'cat2': {
'download': {
'id': 2,
'description': 'Desc 2',
'action': 'Act 2',
},
'click': {
'id': 3,
'description': 'Desc 3',
'action': 'Act 3',
},
...
},
...
}
And following code for generating tuple of tuples for django choice field:
CHOICES = []
for a in actions.values():
for c in a.values():
CHOICES.append((c['id'], c['description']))
Is it possible to write above code in one line nested for loop?
CHOICES = [(c['id'], c['description']) for a in actions.values() for c in a.values()]
With map and reduce:
map(lambda x : [x['id'],x['description']],reduce(lambda x,y:x+y.values(),actions.values(),[]))
Related
Using Python requests I want to grab a piece of JSON from one source and post it to a destination. The structure of the JSON received and the one required by the destination, however, differs a bit so my question is, how do I best map the items from the source structure onto the destination structure?
To illustrate, imagine we get a list of all purchases made by John and Mary. And now we want to post the individual items purchased linking these to the individuals who purchased them (NOTE: The actual use case involves thousands of entries so I am looking for an approach that would scale accordingly):
Source JSON:
{
'Total Results': 2,
'Results': [
{
'Name': 'John',
'Age': 25,
'Purchases': [
{
'Fruits': {
'Type': 'Apple',
'Quantity': 3,
'Color': 'Red'}
},
{
'Veggie': {
'Type': 'Salad',
'Quantity': 2,
'Color': 'Green'
}
}
]
},
{
'Name': 'Mary',
'Age': 20,
'Purchases': [
{
'Fruits': {
'Type': 'Orange',
'Quantity': 2,
'Color': 'Orange'
}
}
]
}
]
}
Destination JSON:
{
[
{
'Purchase': 'Apple',
'Purchased by': 'John',
'Quantity': 3,
'Type': 'Red',
},
{
'Purchase': 'Salad',
'Purchased by': 'John',
'Quantity': 2,
'Type': 'Green',
},
{
'Purchase': 'Orange',
'Purchased by': 'Mary',
'Quantity': 2,
'Type': 'Orange',
}
]
}
Any help on this would be greatly appreciated! Cheers!
Just consider loop through the dict.
res = []
for result in d['Results']:
value = {}
for purchase in result['Purchases']:
item = list(purchase.values())[0]
value['Purchase'] = item['Type']
value['Purchased by'] = result['Name']
value['Quantity'] = item['Quantity']
value['Type'] = item['Color']
res.append(value)
pprint(res)
[{'Purchase': 'Apple', 'Purchased by': 'John', 'Quantity': 3, 'Type': 'Red'},
{'Purchase': 'Salad', 'Purchased by': 'John', 'Quantity': 2, 'Type': 'Green'},
{'Purchase': 'Orange', 'Purchased by': 'Mary', 'Quantity': 2, 'Type': 'Orange'}]
I have source_json data looking like this:
{
'ID': {
'0': 8573273,
'1': 8573277
},
'prediction': {
'0': 4.411029362081518,
'1': 4.411029362081518
},
'feature': {
'0': 0,
'1': 0
}
}
But I need it to be in this form:
[
{
'ID': 8573273,
'prediction': 4.411029362081518,
'feature': 0
},
{
'ID': 8573277,
'prediction': 4.411029362081518,
'feature': 0
}
]
I convert the first view to Pandas dataframe and then convert it to the desirable json.
t = pd.DataFrame(source_json)
proper_jsone = t.to_dict(orient='records')
The question is: Is there a proper way to do this without creating an additional dataframe?
I prefer traditional way:
all_keys = list(source_json.keys())
all_indices = list(source_json[all_keys[0]].keys())
transformed_json = [ {_k:source_json[_k][_idx] for _k in all_keys} for _idx in all_indices]
You can try list comprehension
[{'ID':i, 'prediction':j, 'feature':k}
for i,j,k in zip(*[i.values() for i in d.values()]) ]
output
[{'ID': 8573273, 'prediction': 4.411029362081518, 'feature': 0},
{'ID': 8573277, 'prediction': 4.411029362081518, 'feature': 0}]
I have the following dictionary, which i got when applied to_dict() method on a pandas dataframe.
{
'name' : {
0: 'abc',
1: 'xyz'
},
'email': {
0: 'abc#abc.com',
1: 'xyz#xyz.com',
},
'category': {
0: 'category 1',
1: 'category 2',
}
}
How do I transform it into the following structure?
[
{
'name': 'abc',
'email' : 'abc#abc.com',
'category': 'category 1',
},
{
'name': 'xyz',
'email' : 'xyz#xyz.com',
'category': 'category 2',
}
]
I tried applying many variations of for loop, but came out as bogus code, if anyone could help or point to some links it would be great, python newbie here :|
EDIT: changed the desired structure to a list of dicts, as dicts are not hashble,
The target structure you display is a set of dicts. Since dicts are not hashable, it is not possible to create.
Instead you probably want a list of dicts.
result = [
{k: yourdict[k][n] for k in yourdict} for n in sorted(yourdict['name'])
]
Testing:
[
{'category': 'category 1', 'email': 'abc#abc.com', 'name': 'abc'},
{'category': 'category 2', 'email': 'xyz#xyz.com', 'name': 'xyz'}
]
You could transpose your dataframe before converting it to a dictionary. This will produce a dictionary of dictionaries, where each key is an index value from the original dataframe.
import pandas as pd
pd.DataFrame({
'name' : {
0: 'abc',
1: 'xyz'
},
'email': {
0: 'abc#abc.com',
1: 'xyz#xyz.com',
},
'category': {
0: 'category 1',
1: 'category 2',
}
}).T.to_dict()
Outputs:
{0: {'name': 'abc', 'email': 'abc#abc.com', 'category': 'category 1'},
1: {'name': 'xyz', 'email': 'xyz#xyz.com', 'category': 'category 2'}}
You could just pass 'records' as the desired orientation to to_dict():
df.to_dict('records')
The default orientation 'dict' produces output like {column -> {index -> value}} as can be seen in your example, where as 'records' a list like [{column -> value}, … , {column -> value}], which is your desired output.
first of all i'm new to mongo so I don't know much and i cannot just remove duplicate rows due to some dependencies.
I have following data stored in mongo
{'id': 1, 'key': 'qscderftgbvqscderftgbvqscderftgbvqscderftgbvqscderftgbv', 'name': 'some name', 'country': 'US'},
{'id': 2, 'key': 'qscderftgbvqscderftgbvqscderftgbvqscderftgbvqscderftgbv', 'name': 'some name', 'country': 'US'},
{'id': 3, 'key': 'pehnvosjijipehnvosjijipehnvosjijipehnvosjijipehnvosjiji', 'name': 'some name', 'country': 'IN'},
{'id': 4, 'key': 'pfvvjwovnewpfvvjwovnewpfvvjwovnewpfvvjwovnewpfvvjwovnew', 'name': 'some name', 'country': 'IN'},
{'id': 5, 'key': 'pfvvjwovnewpfvvjwovnewpfvvjwovnewpfvvjwovnewpfvvjwovnew', 'name': 'some name', 'country': 'IN'}
you can see some of the rows are duplicate with different id
as long as it will take to solve this issue from input I must tackle it on output.
I need the data in the following way:
{'id': 1, 'key': 'qscderftgbvqscderftgbvqscderftgbvqscderftgbvqscderftgbv', 'name': 'some name', 'country': 'US'},
{'id': 3, 'key': 'pehnvosjijipehnvosjijipehnvosjijipehnvosjijipehnvosjiji', 'name': 'some name', 'country': 'IN'},
{'id': 4, 'key': 'pfvvjwovnewpfvvjwovnewpfvvjwovnewpfvvjwovnewpfvvjwovnew', 'name': 'some name', 'country': 'IN'}
My query
keys = db.collection.distinct('key', {})
all_data = db.collection.find({'key': {$in: keys}})
As you can see it takes two queries for a same result set Please combine it to one as the database is very large
I might also create a unique key on the key but the value is so long (152 characters) that it will not help me.
Or it will??
You need to use the aggregation framework for this. There are multiple ways to do this, the solution below uses the $$ROOT variable to get the first document for each group:
db.data.aggregate([{
"$sort": {
"_id": 1
}
}, {
"$group": {
"_id": "$key",
"first": {
"$first": "$$ROOT"
}
}
}, {
"$project": {
"_id": 0,
"id":"$first.id",
"key":"$first.key",
"name":"$first.name",
"country":"$first.country"
}
}])
I have this:
items = {{'project':'Project 1','description':'Task description','time':1222222},
{'project':'Project 2','description':'Task description 2','time':1224322},
{'project':'Project 1','description':'Task description 3','time':13222152}}
And I need something like this:
resultitems = {
'project':'Project 1','pritems':{
{'description':'Task description','time':1222222},
{'description':'Task description 3','time':13222152}},
'project':'Project 2',pritems':{
{'description':'Task description 2','time':1224322}},
}
of simply the name of each project as a key
I've tried this approach:
resultitems = {}
resultitems['Project 2'] = {}
resultitems['Project 2'].update(..)
update does not work, since it replaces the previous value
in php, it was easy,
$resultitems['Project 2'][] = array(...)
but don't find the way to do this in Python
result_items = {
'house project': [{'task': 'cleaning', 'hours': 20}, {'task': 'painting', 'hours: 30', etc.],
'garden project': [{'task': 'mowing the lawn', 'hours': 1, etc.
etc.
}
Your variable 'items' is not correct. If it is a list of dictionaries, it should be:
items = [{...}, {...}, {...}]
Please write the source of the data, from where do you get the data. This will determine the way you will fill in the desired dictionary. If you already have the data as in 'items' (i.e. a list of dictionaries), then here is how to converted it:
items = [{'project':'Project 1','description':'Task description','time':1222222},
{'project':'Project 2','description':'Task description 2','time':1224322},
{'project':'Project 1','description':'Task description 3','time':13222152}]
dct = {}
for e in items :
if e['project'] not in dct :
dct[e['project']] = []
dct[e['project']].append(dict([(k, v) for k,v in e.items() if k != 'project']))
print dct
and output is:
{'Project 2': [{'description': 'Task description 2', 'time': 1224322}], 'Project 1': [{'description': 'Task description', 'time': 1222222}, {'description': 'Task description 3', 'time': 13222152}]}
Finally, I used this:
newdata = {}
for data in result['data']:
try:
newdata[data['project']].append({"description":data['description'],"start":data['start'],"time":data['dur']})
except:
newdata[data['project']] = []
newdata[data['project']].append({"description":data['description'],"start":data['start'],"time":data['dur']})
print newdata
And the result has been like this, and this is what I needed:
{
u'Project 1': [
{'start': u'2015-07-09T18:09:41-03:00', 'description': u'Task 1 name', 'time': 1432000},
{'start': u'2015-07-09T17:42:36-03:00', 'description': u'Task 2 name', 'time': 618000}
],
u'Project 2': [
{'start': u'2015-07-09T20:14:16-03:00', 'description': u'Other Task Name', 'time': 4424000}
],
u'Project 3': [
{'start': u'2015-07-09T22:29:51-03:00', 'description': u'another task name for pr3', 'time': 3697000},
{'start': u'2015-07-09T19:38:02-03:00', 'description': u'something more to do', 'time': 59000},
{'start': u'2015-07-09T19:11:49-03:00', 'description': u'Base tests', 'time': 0},
{'start': u'2015-07-09T19:11:29-03:00', 'description': u'Domain', 'time': 0}
],
u'Project something': [
{'start': u'2015-07-09T19:39:30-03:00', 'description': u'Study more', 'time': 2069000},
{'start': u'2015-07-09T15:46:39-03:00', 'description': u'Study more (2)', 'time': 3800000},
{'start': u'2015-07-09T11:46:00-03:00', 'description': u'check forms', 'time': 660000}
]
}
by the way, I was no asking about the structure itself.. instead what I needed was someway to program a "something like this" structure.