Django turn lists into one list - python

I want to put this data into one list so i can sort it by timestamp. I tried itertools chain but that didn't really work.
Thank you for your help :)
I'm very bad at making clear what i want to do so im sorry upfront if this takes some explaning.
If i try a chain i get the value back like this.
I want to display it on the html page like this :
date, name , rating, text (newline)
likes comments
Which would work the way i did it but if i want to sort it by time, it wouldn't work so i tried to think of a way to make it into a sortable list. Which can be displayed. Is that understandable ?
['Eva Simia', 'Peter Alexander', {'scale': 5, 'value': 5}, {'scale': 5, 'value': 5}, 1, 0, 1, 0]
it should look like this:
['Peter Alexander, scale:5, value:5, 1,0]
['Eva Simia, scale:5, value:5, 1,0]
for i in user:
name.append(i['name'])
for i in next_level:
rating_values.append(i['rating'])
for i in comment_values:
comments_count.append(i['count'])
for i in likes_values:
likes_count.append(i['count'])
for s in rating_values:
ratings.append(s['value'])
for s in date:
ratings.append(s['date'])
ab = itertools.chain([name], [rating_values],
[comments_count], [likes_values],
[comment_values], [date])
list(ab)


Updated after clarification:
The problem as I understand it:
You have a dataset that is split into several lists, one list per field.
Every list has the records in the same order. That is, user[x]'s rating value is necessarily rating_values[x].
You need to merge that information into a single list of composite items. You'd use zip() for that:
merged = zip(user, next_level, comment_values, likes_values, rating_values, date)
# merged is now [(user[0], next_level[0], comment_values[0], ...),
# (user[1], next_level[1], comment_values[1], ...),
# ...]
From there, you can simply sort your list using sorted():
result = sorted(merged, key=lambda i: (i[5], i[0]))
The key argument must be a function. It is given each item in the list once, and must return the key that will be used to compare items. Here, we build a short function on the fly, that returns the date and username, effectively telling things will be sorted, first by date and if dates are equal, then by username.
[Past answer about itertools.chain, before the clarification]
ab = list(itertools.chain(
(i['name'] for i in user),
(i['rating'] for i in next_level),
(i['count'] for i in comment_values),
(i['count'] for i in likes_values),
(i['value'] for i in rating_values),
(i['date'] for i in date),
))
The point of using itertools.chain is usually to avoid needless copies and intermediary objects. To do that, you want to pass it iterators.
chain will return an iterator that will iterate through each of the given iterators, one at a time, moving to the next iterator when current stops.
Note every iterator has to be wrapped in parentheses, else python will complain. Do not make it square brackets, at it would cause an intermediary list to be built.


You can join list by simply using +.
l = name + rating_values + comments_count + ...


date, rating_values,likes_values,comment_values,next_level,user = (list(t) for t in zip(*sorted(zip(date, rating_values,likes_values,comment_values,next_level,user))))

Related

Getting an dict by name with in a list with an element key within a variable

data = {
"items" : [{"potion" : 1}, {"potion2" : 1}]
}
print(data["items"][0]["potion"])
So, here's the jiz. I want to get potion2 without providing number like [0] but i can't because some variables has 5 items within the list while another one might have 3 items so providing a number might not giving me what i need. Is there a way to get potion2 without providing that number before it?

I'm assuming you don't want to provide the index because hard coding it will not work in all circumstances.
You can just pull out any items in the list which have that key.
Build a list of any items, which have that key. It might ordinarily be just one, but the container itself does not enforce that only one entry can have that key.
After that you can either iterate over the list or check if the returned value is empty and just take the first element.
>>> data = {'items': [{'potion': 1}, {'potion2': 1}]}
>>> e = filter(lambda i: 'potion' in i, data['items'])
>>> for i in e:
... print(i['potion'])
...
1
Or to pull out only the first element. I realize you said no indices, but this index is applied to the filtered list and we check that its not empty first, so it's a valid thing to do.
>>> if e:
... print(e[0]['potion'])
...
1

Dicts in Python

I have a multidimensionnal dict, I need to return a specific value.
ConsomRatio={"DAP_Local":[],"MAP11_52":[]}
ConsomRatio["DAP_Local"].append({"Ammonia":"0.229", "Amine":"0.0007"})
ConsomRatio["MAP11_52"].append({"Ammonia":"0.138", "Fuel":"0.003"})
print(ConsomRatio["DAP_Local"])
The result of the print is:
[{'Ammonia': '0.229', 'Amine': '0.0007'}]
My question is : Is there a way to return the value of "Ammonia" only, in "DAP_Local" ?
Thank you!

You can get to it like this. You're appending your dict to a list, so you must select the correct index in the list where the dict is located. In this case the first element in the list or index 0.
ConsomRatio["DAP_Local"][0]["Ammonia"]
By the way, depending on what you are trying to achieve you might wanna take a look at the other answers for different implementations of multi-dimensional dicts.

The other answers are of course correct, but have you considered using a "dict of dicts"? i.e.:
ConsomRatio={"DAP_Local":{},"MAP11_52":{}}
ConsomRatio["DAP_Local"].update({"Ammonia":"0.229", "Amine":"0.0007"})
ConsomRatio["MAP11_52"].update({"Ammonia":"0.138", "Fuel":"0.003"})
print ConsomRatio["DAP_Local"]["Ammonia"]
0.229

since print(ConsomRatio["DAP_Local"]) returns an array of length 1, you need to select the index 0, then key off the 'Ammonia' value as above.
if print(ConsomRatio["DAP_Local"]) returned a dict, then no need to have the [0] and print(ConsomRatio["DAP_Local"]['Amomonia']) would have worked

Why are you putting lists in your dict, anyhow? You can just use dicts inside your main dict.
You can have multidimensional dicts also without the lists, e.g.:
ConsomRatio = {}
ConsomRation["DAP_Local"] = {"Ammonia":"0.229", "Amine":"0.0007"}
ConsomRatio["MAP11_52"] = {"Ammonia":"0.138", "Fuel":"0.003"}
print(ConsomRatio["DAP_Local"]["Ammonia"])
will give the desired result without the extra effort with the list.
You can get even shorter in Python:
ConsomRatio = {
"DAP_Local": {"Ammonia":"0.229", "Amine":"0.0007"},
"MAP11_52" : {"Ammonia":"0.138", "Fuel":"0.003"},
}
print(ConsomRatio["DAP_Local"]["Ammonia"])
To also answer your latest question (in your second comment):
to_produce = 'DAP_Local'
ingredience = 'Ammonia'
print('To produce {to_produce} we need {amount} of {ingredience}'.format(
to_produce=to_produce, ingredience=ingredience,
amount=ConsomRatio[to_produce].get(ingredience, '0.0')))
I hope, that helps!
It gets even better:
for product, ingred_list in ConsomRatio.items():
for iname, ivalue in ingred_list.items():
print('To produce {to_produce} we need {amount} of {ingredience}'
.format(to_produce=product, ingredience=iname,
amount=ivalue))

Parse string with three-level delimitation into dictionary

I've found how to split a delimited string into key:value pairs in a dictionary elsewhere, but I have an incoming string that also includes two parameters that amount to dictionaries themselves: parameters with one or three key:value pairs inside:
clientid=b59694bf-c7c1-4a3a-8cd5-6dad69f4abb0&keyid=987654321&userdata=ip:192.168.10.10,deviceid:1234,optdata:75BCD15&md=AMT-Cam:avatar&playbackmode=st&ver=6&sessionid=&mk=PC&junketid=1342177342&version=6.7.8.9012
Obviously these are dummy parameters to obfuscate proprietary code, here. I'd like to dump all this into a dictionary with the userdata and md keys' values being dictionaries themselves:
requestdict {'clientid' : 'b59694bf-c7c1-4a3a-8cd5-6dad69f4abb0', 'keyid' : '987654321', 'userdata' : {'ip' : '192.168.10.10', 'deviceid' : '1234', 'optdata' : '75BCD15'}, 'md' : {'Cam' : 'avatar'}, 'playbackmode' : 'st', 'ver' : '6', 'sessionid' : '', 'mk' : 'PC', 'junketid' : '1342177342', 'version' : '6.7.8.9012'}
Can I take the slick two-level delimitation parsing command that I've found:
requestDict = dict(line.split('=') for line in clientRequest.split('&'))
and add a third level to it to handle & preserve the 2nd-level dictionaries? What would the syntax be? If not, I suppose I'll have to split by & and then check & handle splits that contain : but even then I can't figure out the syntax. Can someone help? Thanks!

I basically took Kyle's answer and made it more future-friendly:
def dictelem(input):
parts = input.split('&')
listing = [part.split('=') for part in parts]
result = {}
for entry in listing:
head, tail = entry[0], ''.join(entry[1:])
if ':' in tail:
entries = tail.split(',')
result.update({ head : dict(e.split(':') for e in entries) })
else:
result.update({head: tail})
return result

Here's a two-liner that does what I think you want:
dictelem = lambda x: x if ':' not in x[1] else [x[0],dict(y.split(':') for y in x[1].split(','))]
a = dict(dictelem(x.split('=')) for x in input.split('&'))

Can I take the slick two-level delimitation parsing command that I've found:
requestDict = dict(line.split('=') for line in clientRequest.split('&'))
and add a third level to it to handle & preserve the 2nd-level dictionaries?
Of course you can, but (a) you probably don't want to, because nested comprehensions beyond two levels tend to get unreadable, and (b) this super-simple syntax won't work for cases like yours, where only some of the data can be turned into a dict.
For example, what should happen with 'PC'? Do you want to make that into {'PC': None}? Or maybe the set {'PC'}? Or the list ['PC']? Or just leave it alone? You have to decide, and write the logic for that, and trying to write it as an expression will make your decision very hard to read.
So, let's put that logic in a separate function:
def parseCommasAndColons(s):
bits = [bit.split(':') for bit in s.split(',')]
try:
return dict(bits)
except ValueError:
return bits
This will return a dict like {'ip': '192.168.10.10', 'deviceid': '1234', 'optdata': '75BCD15'} or {'AMT-Cam': 'avatar'} for cases where each comma-separated component has a colon inside it, but a list like ['1342177342'] for cases where any of them don't.
Even this may be a little too clever; I might make the "is this in dictionary format" check more explicit instead of just trying to convert the list of lists and see what happens.
Either way, how would you put that back into your original comprehension?
Well, you want to call it on the value in the line.split('='). So let's add a function for that:
def parseCommasAndColonsForValue(keyvalue):
if len(keyvalue) == 2:
return keyvalue[0], parseCommasAndColons(keyvalue[1])
else:
return keyvalue
requestDict = dict(parseCommasAndColonsForValue(line.split('='))
for line in clientRequest.split('&'))
One last thing: Unless you need to run on older versions of Python, you shouldn't often be calling dict on a generator expression. If it can be rewritten as a dictionary comprehension, it will almost certainly be clearer that way, and if it can't be rewritten as a dictionary comprehension, it probably shouldn't be a 1-liner expression in the first place.
Of course breaking expressions up into separate expressions, turning some of them into statements or even functions, and naming them does make your code longer—but that doesn't necessarily mean worse. About half of the Zen of Python (import this) is devoted to explaining why. Or one quote from Guido: "Python is a bad language for code golf, on purpose."
If you really want to know what it would look like, let's break it into two steps:
>>> {k: [bit2.split(':') for bit2 in v.split(',')] for k, v in (bit.split('=') for bit in s.split('&'))}
{'clientid': [['b59694bf-c7c1-4a3a-8cd5-6dad69f4abb0']],
'junketid': [['1342177342']],
'keyid': [['987654321']],
'md': [['AMT-Cam', 'avatar']],
'mk': [['PC']],
'playbackmode': [['st']],
'sessionid': [['']],
'userdata': [['ip', '192.168.10.10'],
['deviceid', '1234'],
['optdata', '75BCD15']],
'ver': [['6']],
'version': [['6.7.8.9012']]}
That illustrates why you can't just add a dict call for the inner level—because most of those things aren't actually dictionaries, because they had no colons. If you changed that, then it would just be this:
{k: dict(bit2.split(':') for bit2 in v.split(',')) for k, v in (bit.split('=') for bit in s.split('&'))}
I don't think that's very readable, and I doubt most Python programmers would. Reading it 6 months from now and trying to figure out what I meant would take a lot more effort than writing it did.
And trying to debug it will not be fun. What happens if you run that on your input, with missing colons? ValueError: dictionary update sequence element #0 has length 1; 2 is required. Which sequence? No idea. You have to break it down step by step to see what doesn't work. That's no fun.
So, hopefully that illustrates why you don't want to do this.

How to compare an element of a tuple (int) to determine if it exists in a list

I have the two following lists:
# List of tuples representing the index of resources and their unique properties
# Format of (ID,Name,Prefix)
resource_types=[('0','Group','0'),('1','User','1'),('2','Filter','2'),('3','Agent','3'),('4','Asset','4'),('5','Rule','5'),('6','KBase','6'),('7','Case','7'),('8','Note','8'),('9','Report','9'),('10','ArchivedReport',':'),('11','Scheduled Task',';'),('12','Profile','<'),('13','User Shared Accessible Group','='),('14','User Accessible Group','>'),('15','Database Table Schema','?'),('16','Unassigned Resources Group','#'),('17','File','A'),('18','Snapshot','B'),('19','Data Monitor','C'),('20','Viewer Configuration','D'),('21','Instrument','E'),('22','Dashboard','F'),('23','Destination','G'),('24','Active List','H'),('25','Virtual Root','I'),('26','Vulnerability','J'),('27','Search Group','K'),('28','Pattern','L'),('29','Zone','M'),('30','Asset Range','N'),('31','Asset Category','O'),('32','Partition','P'),('33','Active Channel','Q'),('34','Stage','R'),('35','Customer','S'),('36','Field','T'),('37','Field Set','U'),('38','Scanned Report','V'),('39','Location','W'),('40','Network','X'),('41','Focused Report','Y'),('42','Escalation Level','Z'),('43','Query','['),('44','Report Template ','\\'),('45','Session List',']'),('46','Trend','^'),('47','Package','_'),('48','RESERVED','`'),('49','PROJECT_TEMPLATE','a'),('50','Attachments','b'),('51','Query Viewer','c'),('52','Use Case','d'),('53','Integration Configuration','e'),('54','Integration Command f'),('55','Integration Target','g'),('56','Actor','h'),('57','Category Model','i'),('58','Permission','j')]
# This is a list of resource ID's that we do not want to reference directly, ever.
unwanted_resource_types=[0,1,3,10,11,12,13,14,15,16,18,20,21,23,25,27,28,32,35,38,41,47,48,49,50,57,58]
I'm attempting to compare the two in order to build a third list containing the 'Name' of each unique resource type that currently exists in unwanted_resource_types. e.g. The final result list should be:
result = ['Group','User','Agent','ArchivedReport','ScheduledTask','...','...']
I've tried the following that (I thought) should work:
result = []
for res in resource_types:
if res[0] in unwanted_resource_types:
result.append(res[1])
and when that failed to populate result I also tried:
result = []
for res in resource_types:
for type in unwanted_resource_types:
if res[0] == type:
result.append(res[1])
also to no avail. Is there something i'm missing? I believe this would be the right place to perform list comprehension, but that's still in my grey basket of understanding fully (The Python docs are a bit too succinct for me in this case).
I'm also open to completely rethinking this problem, but I do need to retain the list of tuples as it's used elsewhere in the script. Thank you for any assistance you may provide.

Your resource types are using strings, and your unwanted resources are using ints, so you'll need to do some conversion to make it work.
Try this:
result = []
for res in resource_types:
if int(res[0]) in unwanted_resource_types:
result.append(res[1])
or using a list comprehension:
result = [item[1] for item in resource_types if int(item[0]) in unwanted_resource_types]

The numbers in resource_types are numbers contained within strings, whereas the numbers in unwanted_resource_types are plain numbers, so your comparison is failing. This should work:
result = []
for res in resource_types:
if int( res[0] ) in unwanted_resource_types:
result.append(res[1])

The problem is that your triples contain strings and your unwanted resources contain numbers, change the data to
resource_types=[(0,'Group','0'), ...
or use int() to convert the strings to ints before comparison, and it should work. Your result can be computed with a list comprehension as in
result=[rt[1] for rt in resource_types if int(rt[0]) in unwanted_resource_types]
If you change ('0', ...) into (0, ... you can leave out the int() call.
Additionally, you may change the unwanted_resource_types variable into a set, like
unwanted_resource_types=set([0,1,3, ... ])
to improve speed (if speed is an issue, else it's unimportant).

The one-liner:
result = map(lambda x: dict(map(lambda a: (int(a[0]), a[1]), resource_types))[x], unwanted_resource_types)
without any explicit loop does the job.
Ok - you don't want to use this in production code - but it's fun. ;-)
Comment:
The inner dict(map(lambda a: (int(a[0]), a[1]), resource_types)) creates a dictionary from the input data:
{0: 'Group', 1: 'User', 2: 'Filter', 3: 'Agent', ...
The outer map chooses the names from the dictionary.

Join two CSV files in python using dictreader

I realise the info to answer this question is probably already on here, but as a python newby I've been trying to piece together the info for a few weeks now and I'm hitting some trouble.
this question Python "join" function like unix "join" answers how to do a join on two lists easily, but the problem is that dictreader objects are iterables and not straightforward lists, meaning that there's an added layer of complications.
I basically am looking for an inner join on two CSV files, using the dictreader object. Here's the code I have so far:
def test(dictreader1, dictreader2):
matchedlist = []
for dictline1 in dictreader1:
for dictline2 in dictreader2:
if dictline1['member']=dictline2['member']:
matchedlist.append(dictline1, dictline2)
else: continue
return matchedlist
This is giving me an error at the if statement, but more importantly, I don't seem to be able to access the ['member'] element of the dictionary within the iterable, as it says it has no attribute "getitem".
Does anyone have any thoughts on how to do this? For reference, I need to keep the lists as iterables because each file is too big to fit in memory. The plan is to control this entire function within another for loop that only feeds it a few lines at a time to iterate over. So it will read one line of the left hand file, iterate over the whole second file to find a member field that matches and then join the two lines, similar to an SQL join statement.
Thanks for any help in advance, please forgive any obvious errors on my part.

A few thoughts:
Replace the = with ==. The latter is used for equality tests; the former for assignments.
Add a line a the beginning, dictreader2 = list(dictreader2). That will make it possible to loop over the dictionary entries more than once.
Add a second pair of parenthese to matchedlist.append((dictline1, dictline2)). The list.append method takes just one argument, so you want to create a tuple out of dictline1 and dictline2.
The final else: continue is unnecessary. A for-loop will automatically loop for you.
Use a print statement or somesuch to verify that dictline1 and dictline2 are both dictionary objects that have member as a key. It could be that your function is correct, but is being called with something other than a dictreader object.
Here is a worked out example using a list of dicts as input (similar to what a DictReader would return):
>>> def test(dictreader1, dictreader2):
dictreader2 = list(dictreader2)
matchedlist = []
for dictline1 in dictreader1:
for dictline2 in dictreader2:
if dictline1['member'] == dictline2['member']:
matchedlist.append((dictline1, dictline2))
return matchedlist
>>> dr1 = [{'member': 2, 'value':'abc'}, {'member':3, 'value':'def'}]
>>> dr2 = [{'member': 4, 'tag':'t4'}, {'member':3, 'tag':'t3'}]
>>> test(dr1, dr2)
[({'member': 3, 'value': 'def'}, {'member': 3, 'tag': 't3'})]
A further suggestion is to combine the two dictionaries into a single entry (this is closer to what an SQL inner join would do):
>>> def test(dictreader1, dictreader2):
dictreader2 = list(dictreader2)
matchedlist = []
for dictline1 in dictreader1:
for dictline2 in dictreader2:
if dictline1['member'] == dictline2['member']:
entry = dictline1.copy()
entry.update(dictline2)
matchedlist.append(entry)
return matchedlist
>>> test(dr1, dr2)
[{'member': 3, 'tag': 't3', 'value': 'def'}]
Good luck with your project :-)

Categories