We Keep Coding

Problems appending objects to a list. Data is added correctly, but when verified it is not what I added

I need to fill a db with fake data. To do so I'm creating a list of objects. I generate 10 timestamps, and for each timestamp I create an object, add the timestamp and some randomly assigned data, then it is appended to the list 5 times; each time given a unique id ('tag'). This format is quite specific as I am trying to simulate what the real data will be like.
Everytime I append a value, I run a print to show that the data is correct. Then when I am finished the entire operation, I print the entire list to double check. The first print value returns perfect. Yet the second tells me that I have 10 instances of identical data - my tag value is the same for 5 instances, then jumps up by 5, and repeat.
What am I missing?
cultures = [
{
'name': 'Pink Oyster',
'scientific_name': 'Pleurotus djamor'
},
{
'name': 'Brown Oyster',
'scientific_name': 'Pleurotus ostreatus'
},
{
'name': 'Blue Oyster',
'scientific_name': 'Pleurotus Columbinus'
},
{
'name': 'Chestnut',
'scientific_name': 'Pholiota Adiposa'
},
{
'name': 'Shiitake',
'scientific_name': 'Lentinula edodes'
},
{
'name': "Lion's Mane",
'scientific_name': 'Hericium erinaceus'
},
]
times = pd.date_range(start="2022-01-01",end="2022-07-20", periods=10).to_pydatetime().tolist()
data = []
count = 0
for time in times:
raw = {}
for key, value in cultures[randrange(0,5)].items():
raw[key] = value
raw['generation'] = randrange(0, 10)
raw['stage'] = randrange(0, 3)
raw['user_id'] = 1
raw['created_at'] = time
raw['updated_at'] = time
for i in range(5):
raw['tag'] = count
count += 1
data.append(raw)
#shows my data has been appended correctly
print(data[-1])
#???????
print(data)
example first print - tag value increases by 1
{'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 0}
{'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 1}
{'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 2}
{'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 3}
{'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 4}
example second print - tag is stuck at 4
[{'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 4}, {'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 4}, {'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 4}, {'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 4}, {'name': 'Blue Oyster', 'scientific_name': 'Pleurotus Columbinus', 'generation': 5, 'stage': 1, 'user_id': 1, 'created_at': datetime.datetime(2022, 1, 1, 0, 0), 'updated_at': datetime.datetime(2022, 1, 1, 0, 0), 'tag': 4},
How am i changing the values of previous appends when I change the value after I append the data

In python, stuff is passed around by reference to value. It basically means that primitive types (int, str, float) are passed by value, and complex types (list, object, dict) are passed by reference. So if you add raw to the list, you actually add the reference to it. You then change its contents, and add it again.
Instead of doing data.append(raw) try data.append(raw.copy()).

You need a fresh copy of raw each time round the loop:
for i in range(5):
raw1 = raw.copy()
raw1['tag'] = count
count += 1
data.append(raw1)

Summing Values in Dictionary Based on Certain Values

I have a list of dictionaries that state a date as well as a price. It looks like this:
dict = [{'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 50}, {'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 12}, {'Date':datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]
I'd like to create a new list of dictionaries that sum all the Price values that are on the same date. So the output would look like this:
output_dict = [{'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 62}, {'Date':datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]
How could I achieve this?

You can use Counter from collections module:
from collections import Counter
c = Counter()
for v in dict:
c[v['Date']] += v['Price']
output_dict = [{'Date': name, 'Price': count} for name, count in c.items()]
Output:
[{'Date': datetime.datetime(2020, 6, 1, 0, 0), 'Price': 62},
{'Date': datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]
OR, a new way:
You can use Pandas library to solve this:
Install pandas like:
pip install pandas
Then code would be:
import pandas as pd
output_dict = pd.DataFrame(dict).groupby('Date').agg(sum).to_dict()['Price']
Output:
{Timestamp('2020-06-01 00:00:00'): 62, Timestamp('2020-06-02 00:00:00'): 60}

Another solution using itertools.groupby:
import datetime
from itertools import groupby
dct = [{'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 50}, {'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 12}, {'Date':datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]
out = []
for k, g in groupby(dct, lambda k: k['Date']):
out.append({'Date': k, 'Price': sum(v['Price'] for v in g)})
print(out)
Prints:
[{'Date': datetime.datetime(2020, 6, 1, 0, 0), 'Price': 62}, {'Date': datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]

You can use itertools' groupby, although I'd like to believe that defaultdict will be faster :
#sort dicts
dicts = sorted(dicts, key= itemgetter("Date"))
#get the sum via itertools' groupby
result = [{"Date" : key,
"Price" : sum(entry['Price'] for entry in value)}
for key,value in
groupby(dicts, key = itemgetter("Date"))]
print(result)
[{'Date': datetime.datetime(2020, 6, 1, 0, 0), 'Price': 62},
{'Date': datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]

Using defaultdict
import datetime
from collections import defaultdict
dct = [{'Date': datetime.datetime(2020, 6, 1, 0, 0), 'Price': 50},
{'Date': datetime.datetime(2020, 6, 1, 0, 0), 'Price': 12},
{'Date': datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]
sum_up = defaultdict(int)
for v in dct:
sum_up[v['Date']] += v['Price']
print([{"Date": k, "Price": v} for k, v in sum_up.items()])
[{'Date': datetime.datetime(2020, 6, 1, 0, 0), 'Price': 62}, {'Date': datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]

This a good use-case for defaultdict, let's say our dict is my_dict:
import datetime
my_dict = [{'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 50},
{'Date':datetime.datetime(2020, 6, 1, 0, 0), 'Price': 12},
{'Date':datetime.datetime(2020, 6, 2, 0, 0), 'Price': 60}]
We can accumulate prices using a defaultdict like so:
from collections import defaultdict
new_dict = defaultdict(int)
for dict_ in my_dict:
new_dict[dict_['Date']] += dict_['Price']
Then we just reconvert this dict into a list of dicts!:
my_dict = [{'Date': date, 'Price': price} for date, price in new_dict.items()]

how to calculate the average disk_available, based on the hostname hourly , python

result data :
<QuerySet [{'disk_available': 26, 'hostname': '2', 'day': datetime.datetime(2020, 2, 11, 0, 0, tzinfo=<UTC>), 'c': 354}, {'disk_available': 27, 'hostname': '2', 'day': datetime.datetime(2020, 2, 10, 0, 0, tzinfo=<UTC>), 'c': 273}, {'disk_available': 19, 'hostname': '2', 'day': datetime.datetime(2020, 2, 12, 0, 0, tzinfo=<UTC>), 'c': 12}, {'disk_available': 26, 'hostname': '2', 'day': datetime.datetime(2020, 2, 12, 0, 0, tzinfo=<UTC>), 'c': 45}, {'disk_available': 26, 'hostname': 'tes', 'day': datetime.datetime(2020, 2, 11, 0, 0, tzinfo=<UTC>), 'c': 1945}, {'disk_available': 19, 'hostname': 'tes', 'day': datetime.datetime(2020, 2, 12, 0, 0, tzinfo=<UTC>), 'c': 53}, {'disk_available': 1, 'hostname': 'tes', 'day': datetime.datetime(2020, 2, 11, 0, 0, tzinfo=<UTC>), 'c': 1}, {'disk_available': 26, 'hostname': 'tes', 'day': datetime.datetime(2020, 2, 12, 0, 0, tzinfo=<UTC>), 'c': 45}, {'disk_available': 27, 'hostname': 'tes', 'day': datetime.datetime(2020, 2, 10, 0, 0, tzinfo=<UTC>), 'c': 291}]>

SocketClient.objects.annotate(day=TruncDay('create')).values('day').annotate(c=Count('id')).values('day', 'disk_available', 'hostname', 'c').order_by('hostname')
.
from the results above I want to do that , I want to display the average result of hostname hourly

Python - Inserting and Updating python dict simultaneously

so I have a list of dicts that looks like this:
[{
'field': {
'data': 'F1'
},
'value': F1Value1,
'date': datetime.datetime(2019, 3, 1, 0, 0)
}, {
'field': {
'data': 'F2'
},
'value': F2Value1,
'date': datetime.datetime(2019, 2, 5, 0, 0)
}, {
'field': {
'data': 'F2'
},
'value': F2Value2,
'date': datetime.datetime(2019, 2, 7, 0, 0)
}]
And I want an output that looks like this:
[
{
'F1': [
{
'value': F1Value1,
'date': datetime.datetime(2019, 3, 1, 0, 0)
}
]
},
{
'F2': [
{
'value': F2Value1,
'date': datetime.datetime(2019, 2, 5, 0, 0)
},
{
'value': F2Value2,
'date': datetime.datetime(2019, 2, 5, 0, 0)
},
]
}
]
That is, I want every field.data to be the key and have it append the value and date if it belongs to the same field.
Note: I want to do this WITHOUT using a for loop (apart from the loop to iterate through the list). I want to use python dict functions like update() and append() etc.
Any optimized solutions would be really helpful.

You could just use iterate through the list of dicts and use defaultdict from collections to add the items with a unique key,
>>> from collections import defaultdict
>>> d = defaultdict(list)
>>>
>>> for items in x:
... d[items['field']['data']].append({
... 'value': items['value'],
... 'date': items['date']
... })
...
>>>
>>> import pprint
>>> pprint.pprint(x)
[{'date': datetime.datetime(2019, 3, 1, 0, 0),
'field': {'data': 'F1'},
'value': 'F1Value1'},
{'date': datetime.datetime(2019, 2, 5, 0, 0),
'field': {'data': 'F2'},
'value': 'F2Value1'},
{'date': datetime.datetime(2019, 2, 7, 0, 0),
'field': {'data': 'F2'},
'value': 'F2Value2'}]
>>>
>>> pprint.pprint(list(d.items()))
[('F1', [{'date': datetime.datetime(2019, 3, 1, 0, 0), 'value': 'F1Value1'}]),
('F2',
[{'date': datetime.datetime(2019, 2, 5, 0, 0), 'value': 'F2Value1'},
{'date': datetime.datetime(2019, 2, 7, 0, 0), 'value': 'F2Value2'}])]

Use itertools.groupby:
from itertools import groupby
from pprint import pprint
result = [{key: [{k: v for k, v in element.items() if k != 'field'}
for element in group]}
for key, group in groupby(data, lambda element: element['field']['data'])]
pprint(result)
Output:
[{'F1': [{'date': datetime.datetime(2019, 3, 1, 0, 0), 'value': 'F1Value1'}]},
{'F2': [{'date': datetime.datetime(2019, 2, 5, 0, 0), 'value': 'F2Value1'},
{'date': datetime.datetime(2019, 2, 7, 0, 0), 'value': 'F2Value2'}]}]

Only using dict, list, and set:
[
{
field_data :
[
{ k:v for k, v in thing.items() if k != 'field' }
for thing in things if thing['field']['data'] == field_data
]
for field_data in set(thing['field']['data'] for thing in things)
}
]

Sort list of lists that each contain a dictionary

I have this list:
list_users= [[{'points': 9, 'values': 1, 'division': 1, 'user_id': 3}], [{'points': 3, 'values': 0, 'division': 1, 'user_id': 1}], [{'points': 2, 'values': 0, 'division': 1, 'user_id': 4}], [{'points': 9, 'values': 0, 'division': 1, 'user_id': 11}], [{'points': 3, 'values': 0, 'division': 1, 'user_id': 10}], [{'points': 100, 'values': 4, 'division': 1, 'user_id': 2}], [{'points': 77, 'values': 2, 'division': 1, 'user_id': 5}], [{'points': 88, 'values': 3, 'division': 1, 'user_id': 6}], [{'points': 66, 'values': 1, 'division': 1, 'user_id': 7}], [{'points': 2, 'values': 0, 'division': 1, 'user_id': 8}]]
I need to sort the list by points and values.
How can I sort it if dict is inside a list inside the main list?
I generated this list by query and than just append to list_users?

Access the dictionary containing points and values by indexing on the inner list:
list_users_sorted = sorted(list_users, key=lambda x: (x[0]['points'], x[0]['values']))
# ^ ^

Sort using a key function for sorted that builds a tuple of points and values for each dict in each list.
def kf(x):
return (x[0]["points"], x[0]["values"])
s = sorted(list_users, key=kf)
print(s)
Output:
[[{'division': 1, 'points': 2, 'user_id': 4, 'values': 0}],
[{'division': 1, 'points': 2, 'user_id': 8, 'values': 0}],
[{'division': 1, 'points': 3, 'user_id': 1, 'values': 0}],
[{'division': 1, 'points': 3, 'user_id': 10, 'values': 0}],
[{'division': 1, 'points': 9, 'user_id': 11, 'values': 0}],
[{'division': 1, 'points': 9, 'user_id': 3, 'values': 1}],
[{'division': 1, 'points': 66, 'user_id': 7, 'values': 1}],
[{'division': 1, 'points': 77, 'user_id': 5, 'values': 2}],
[{'division': 1, 'points': 88, 'user_id': 6, 'values': 3}],
[{'division': 1, 'points': 100, 'user_id': 2, 'values': 4}]]