We Keep Coding

Remove null from a json data (python)

When I use the Rest API to download data from Firebase, it looks like this.
{
"Dataset1": [
null,
{
"Key1": 1,
"Key2": 2
},
{
"Key1": 3,
"Key2": 4
}
],
"Dataset2": [
null,
{
"Key1": 1,
"Key2": 2
},
{
"Key1": 3,
"Key2": 4
}
]
}
Is it possible to remove the null value before saving the data to a file? I know the null exists because of how I designed my database, but it is too late for me to redesign the data now. I tried is_not but no luck yet.

It looks like you've stored nodes with sequentially incrementing keys in your database (i.e. "1", "2", "3"). When you do this, Firebase interprets it as an array structure, and coerces it to a (zero-based) array when you retrieve it. And since you have no node for index 0, it adds a null there.
To prevent this array coercion, store nodes with non-numeric keys, for example by prefixing each number with a short non-numeric value. Like "key1", "key2", "key3".
Also see:
Best Practices: Arrays in Firebase.

It seems it's just the first element in each list. You could just use a simple dict comprehension for this if so:
{k: v[1:] for k, v in data.items()}
If not you could use this comprehension:
{k: [e for e in v if e != None] for k, v in data.items()}

Try this code:
Dataset2 = list()
for data in Dataset:
if data not null:
Dataset2.append(data)
Dataset = Dataset2
del Dataset2

Accessing collection field inside python dictionary

totalHotelsInTown=hotels.aggregate([ {"$group": {"_id": "$Town", "TotalRestaurantInTown": {"$sum":1}} } ])
NumOfHotelsInTown={}
for item in totalHotelsInTown:
NumOfHotelsInTown[item['_id']]=item['TotalRestaurantInTown']
results = hotels.aggregate(
[{"$match": {"cuisine": cuisine}},
{"$group": {"_id": "$town", "HotelsCount": {"$sum": 1} }}, {"$project": {"HotelsCount":1,"Percent": {"$multiply": [{"$divide": ["$HotelsCount", NumOfHotelsInTown["$_id"]]}, 100]}}}, {"$sort": {"Percent": 1}},
{"$limit": 1}])
I want to pass the value of "_id" field as a key to python dictionary, but the interpreter is taking "$_id" itself as a key instead of its value and giving a KeyError because of that. Any help would be much appreciated. Thanks!
'NumOfHotelsInTown' dictionary has key value pairs of place and number of hotels
When I am trying to retrieve the value from NumOfHotelsInTown dictionary,
I am giving the key dynamically with "$_id".
The exact error I am getting is:
{"$group": {"_id": "$borough", "HotelsCount": {"$sum": 1} }}, {"$project": {"HotelsCount":1,"Percent": {"$multiply": [{"$divide": ["$HotelsCount", NumOfHotlesInTown["$_id"]]}, 100]}}}, {"$sort": {"Percent": 1}},
KeyError: '$_id'

I see what you're trying to do, but you can't dynamically run python code during a MongbDB aggregate.
What you should do instead:
Get the total counts for every borough (which you have already done)
Get the total counts for every borough for a given cuisine (which you have part of)
Use python to compare the 2 totals to produce a list of percentages and not MongoDB
For example:
group_by_borough = {"$group": {"_id": "$borough", "TotalRestaurantInBorough": {"$sum":1}} }
count_of_restaurants_by_borough = my_collection.aggregate([group_by_borough])
restaurant_count_by_borough = {doc["_id"]: doc["TotalRestaurantInBorough"] for doc in count_of_restaurants_by_borough}
count_of_cuisines_by_borough = my_collection.aggregate([{"$match": {"cuisine": cuisine}}, group_by_borough])
cuisine_count_by_borough = {doc["_id"]: doc["TotalRestaurantInBorough"] for doc in count_of_cuisines_by_borough}
percentages = {}
for borough, count in restaurant_count_by_borough.items():
percentages[borough] = cuisine_count_by_borough.get(borough, 0) / float(count) * 100
# And if you wanted it sorted you can use an OrderedDict
from collections import OrderedDict
percentages = OrderedDict(sorted(percentages.items(), key=lambda x: x[1]))

Convert the dataframe to JSON Based on Column name

I have a dataframe which contains like this below, Am just providing one row !
Vessel_id,Org_id,Vessel_name,Good_Laden_Miles_Min,Good_Ballast_Miles_Min,Severe_Laden_Miles_Min,Severe_Ballast_Miles_Min
1,5,"ABC",10,15,25,35
I want to convert the dataframe to json in this format below,
{
Vessel_id:1,
Vessel_name:"ABC",
Org_id:5,
WeatherGood:{
Good_Laden_Miles_Min:10,
Good_Ballast_Miles_Min:15
},
weatherSevere:{
Severe_Laden_Miles_Min:25,
Severe_Ballast_Miles_Min:35
}
}
how to join all those columns starting with good into a WeatherGood and convert to JSON?

You can first convert the dataframe to a dictionary of records, then transform each record to your desired format. Finally, convert the list of records to JSON.
import json
records = df.to_dict('records')
for record in records:
record['WeatherGood'] = {
k: record.pop(k) for k in ('Good_Laden_Miles_Min', 'Good_Ballast_Miles_Min')
}
record['WeatherSevere'] = {
k: record.pop(k) for k in ('Severe_Laden_Miles_Min', 'Severe_Ballast_Miles_Min')
}
>>> json.dumps(records)
'[{"Vessel_id": 1, "Org_id": 5, "Vessel_name": "ABC", "WeatherGood": {"Good_Laden_Miles_Min": 10, "Good_Ballast_Miles_Min": 15}, "WeatherSevere": {"Severe_Laden_Miles_Min": 25, "Severe_Ballast_Miles_Min": 35}}]'

Merging list of python dictionaries by column value

I have data that is a list of python dictionaries, each representing a row in the data, and want to combine several of these into one dictionary.
I need to combine them by a common value in a single column, note the dictionaries to merge may or may not contain similar columns and values should be concatenated, not clobbered.
Here is an example (combining dicts by value in column 'a'):
data = [{ 'a':0, 'b':10, 'c':20 }
{ 'a':2, 'd':30, 'e':40 }
{ 'a':0, 'b':50, 'c':60 }
{ 'a':1, 'd':70, 'c':80 }
{ 'a':1, 'b':90, 'e':100 }]
Desired output is:
new_data = [{ 'a':0, 'b':[10,50], 'c':[20,60] }
{ 'a':1, 'd':[70], 'c':[80], 'b':[90], 'e':[100] }
{ 'a':2, 'd':[30], 'e':[40] }]
I have a simple function that can accomplish this, but need a faster method (Data has approx 1,000,000 rows and 20 columns). My method of finding the dictionaries I want to merge is very expensive.
Here is where I have an issue with computation time:
unique_idx, locations = [], {}
for i, row in enumerate(data):
_id = row['a']
if _id not in unique_idx:
unique_idx.append(_id)
locations[_id] = [i]
else:
locations[_id].append(i)
grouped_data = [data[loc] for loc in locations.values()]
I need a faster method to collect dictionaries that contain the same value in one column. Ideally I want a quick method with plain python, but if this can be done simply with a pandas DataFrame that is good as well.

How do I create a data structure that will be serialized this JSON format in python?

I have a function that accepts a list of date objects and should output the following dictionary in JSON:
{
"2010":{
"1":{
"id":1,
"title":"foo",
"postContent":"bar"
},
"7":{
"id":2,
"title":"foo again",
"postContent":"bar baz boo"
}
},
"2009":{
"6":{
"id":3,
"title":"foo",
"postContent":"bar"
},
"8":{
"id":4,
"title":"foo again",
"postContent":"bar baz boo"
}
}
}
Basically I would like to access my objects by year and month number.
What code can convert a list to this format in python that can be serialized to the dictionary above in json?

Something along the lines of this should work:
from collections import defaultdict
import json
d = defaultdict(dict)
for date in dates:
d[date.year][date.month] = info_for_date(date)
json.dumps(d)
Where info_for_date is a function that returns a dict like those in your question.