We Keep Coding

Appending dictionaries when in a for loop

Sorry if this has been asked, I've searched a ton and not finding what I'm looking for. Lots of ways to combine dictionaries, but haven't found anything for combining in a loop that applies.
I have a function (masternodes()) that will return a filtered queryset. With each object returned from that queryset, I loop through and query another function (masternodeDetail()) to get some details about that object. The function (masternodeDetail()) returns a dictionary. I'd like to combine the dictionaries returned in to a single dictionary so I can pass that to my template.
Here's my code I've been working with:
def masternodes(request):
user = request.user
mn = {}
queryset = Masternode.objects.filter(created_by=user)
for n in queryset:
mn = masternodeDetail(n.wallet_address, n.coin_symbol)
print(mn)
args = {'masternodes': queryset, 'masternodetail': mn}
return render(request, 'monitor.html', args)
def masternodeDetail(wallet, ticker):
monitorinfo = MonitorInfo.objects.get(coin_symbol=ticker)
coin_symbol = monitorinfo.coin_symbol
daemon = monitorinfo.daemon
rpc_user = monitorinfo.rpc_user
rpc_password = monitorinfo.rpc_password
rpc_port = monitorinfo.rpc_port
coin_name = monitorinfo.coin_name
notes = monitorinfo.notes
masternode_reward = monitorinfo.masternode_reward
coingecko_id = monitorinfo.coingecko_id
json_output = monitorinfo.json_output
headers = {'content-type': 'text/plain'}
payload = {"method": "masternodelist", "params": ['full', wallet], "jsonrpc": "1.0", "id": "MN Monitoring v0.1"}
response = requests.post("http://" + daemon + ":" + rpc_port, json=payload, headers=headers, auth=(rpc_user,rpc_password)).json()
res = response["result"]
if json_output == False:
for k,v in res.items():
list = v.split()
## status protocol payee lastseen activeseconds lastpaidtime lastpaidblock IP ##
status = list[0]
protocol = list[1]
payee = list[2]
lastseen = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(float(list[3])))
activeseconds = datetime.timedelta(seconds=float(list[4]))
lastpaidtime = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(float(list[5])))
lastpaidblock = list[6]
nodeip = list[7].split(':', 1)[0]
masternode_result = {payee: {'coin':coin_symbol, 'coin_name':coin_name, 'notes':notes, 'masternode_reward':masternode_reward,
'coingecko_id':coingecko_id, 'status':status, 'protocol':protocol, 'payee':payee, 'lastseen': lastseen,
'activeseconds': activeseconds, 'lastpaidtime': lastpaidtime, 'lastpaidblock': lastpaidblock, 'nodeip': nodeip}}
masternodes_result = {**masternode_result, **masternode_result}
return masternodes_result
else:
for txid in res:
status = res[txid]['status']
nodeip = res[txid]['address'].split(':', 1)[0]
protocol = res[txid]['protocol']
payee = res[txid]['payee']
lastseen = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(float(res[txid]['lastseen'])))
activeseconds = datetime.timedelta(seconds=float(res[txid]['activeseconds']))
lastpaidtime = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime(float(res[txid]['lastpaidtime'])))
lastpaidblock = res[txid]['lastpaidblock']
masternode_result = {payee: {'coin':coin_symbol, 'coin_name':coin_name, 'notes':notes, 'masternode_reward':masternode_reward,
'coingecko_id':coingecko_id, 'status':status, 'protocol':protocol, 'payee':payee, 'lastseen': lastseen,
'activeseconds': activeseconds, 'lastpaidtime': lastpaidtime, 'lastpaidblock': lastpaidblock, 'nodeip': nodeip}}
masternodes_result = {**masternode_result, **masternode_result}
return masternodes_result
How can I combine this in to a single dictionary or get all the info I need back to my template? You can see one of my many attempts using the {,} method I've used before but since it's not two different named dictionaries, I'guessing that's why it's not working. Right now, the last dictionary returned from my loop is what's getting passed. Maybe I'm approaching this completely wrong and there's a better way to get this info? Suggestions?

I don't think you want a "single dictionary", I can't imagine what that would look like. Instead you probably want a list of dictionaries - in which case you could do:
mn = []
for n in queryset:
mn.append(masternodeDetail(n.wallet_address, n.coin_symbol))
or even shorter:
mn = [masternodeDetail(n.wallet_address, n.coin_symbol) for n in queryset]

Start with an empty dictionary full_dict and use full_dict.update(d) to add dictionary d to full_dict.
Be aware that if there are commons keys between full_dict and d, they will be replaced.
The following example code starts with an empty dict and updates it with each dict passed in a for loop. Hopefully you can adapt this idea into your code:
full_dict = {}
dict_a = {"a": 10, "b": 20}
dict_b = {"b": 100, "c": 120}
for d in (dict_a, dict_b):
full_dict.update(d)
print(full_dict) # {"a": 10, "b": 100, "c": 120}

Python container troubles

Basically what I am trying to do is generate a json list of SSH keys (public and private) on a server using Python. I am using nested dictionaries and while it does work to an extent, the issue lies with it displaying every other user's keys; I need it to list only the keys that belong to the user for each user.
Below is my code:
def ssh_key_info(key_files):
for f in key_files:
c_time = os.path.getctime(f) # gets the creation time of file (f)
username_list = f.split('/') # splits on the / character
user = username_list[2] # assigns the 2nd field frome the above spilt to the user variable
key_length_cmd = check_output(['ssh-keygen','-l','-f', f]) # Run the ssh-keygen command on the file (f)
attr_dict = {}
attr_dict['Date Created'] = str(datetime.datetime.fromtimestamp(c_time)) # converts file create time to string
attr_dict['Key_Length]'] = key_length_cmd[0:5] # assigns the first 5 characters of the key_length_cmd variable
ssh_user_key_dict[f] = attr_dict
user_dict['SSH_Keys'] = ssh_user_key_dict
main_dict[user] = user_dict
A list containing the absolute path of the keys (/home/user/.ssh/id_rsa for example) is passed to the function. Below is an example of what I receive:
{
"user1": {
"SSH_Keys": {
"/home/user1/.ssh/id_rsa": {
"Date Created": "2017-03-09 01:03:20.995862",
"Key_Length]": "2048 "
},
"/home/user2/.ssh/id_rsa": {
"Date Created": "2017-03-09 01:03:21.457867",
"Key_Length]": "2048 "
},
"/home/user2/.ssh/id_rsa.pub": {
"Date Created": "2017-03-09 01:03:21.423867",
"Key_Length]": "2048 "
},
"/home/user1/.ssh/id_rsa.pub": {
"Date Created": "2017-03-09 01:03:20.956862",
"Key_Length]": "2048 "
}
}
},
As can be seen, user2's key files are included in user1's output. I may be going about this completely wrong, so any pointers are welcomed.

Thanks for the replies, I read up on nested dictionaries and found that the best answer on this post, helped me solve the issue: What is the best way to implement nested dictionaries?
Instead of all the dictionaries, I simplfied the code and just have one dictionary now. This is the working code:
class Vividict(dict):
def __missing__(self, key): # Sets and return a new instance
value = self[key] = type(self)() # retain local pointer to value
return value # faster to return than dict lookup
main_dict = Vividict()
def ssh_key_info(key_files):
for f in key_files:
c_time = os.path.getctime(f)
username_list = f.split('/')
user = username_list[2]
key_bit_cmd = check_output(['ssh-keygen','-l','-f', f])
date_created = str(datetime.datetime.fromtimestamp(c_time))
key_type = key_bit_cmd[-5:-2]
key_bits = key_bit_cmd[0:5]
main_dict[user]['SSH Keys'][f]['Date Created'] = date_created
main_dict[user]['SSH Keys'][f]['Key Type'] = key_type
main_dict[user]['SSH Keys'][f]['Bits'] = key_bits

Selecting values from a JSON file in Python

I am getting JIRA data using the following python code,
how do I store the response for more than one key (my example shows only one KEY but in general I get lot of data) and print only the values corresponding to total,key, customfield_12830, summary
import requests
import json
import logging
import datetime
import base64
import urllib
serverURL = 'https://jira-stability-tools.company.com/jira'
user = 'username'
password = 'password'
query = 'project = PROJECTNAME AND "Build Info" ~ BUILDNAME AND assignee=ASSIGNEENAME'
jql = '/rest/api/2/search?jql=%s' % urllib.quote(query)
response = requests.get(serverURL + jql,verify=False,auth=(user, password))
print response.json()
response.json() OUTPUT:-
http://pastebin.com/h8R4QMgB

From the the link you pasted to pastebin and from the json that I saw, its a you issues as list containing key, fields(which holds custom fields), self, id, expand.
You can simply iterate through this response and extract values for keys you want. You can go like.
data = response.json()
issues = data.get('issues', list())
x = list()
for issue in issues:
temp = {
'key': issue['key'],
'customfield': issue['fields']['customfield_12830'],
'total': issue['fields']['progress']['total']
}
x.append(temp)
print(x)
x is list of dictionaries containing the data for fields you mentioned. Let me know if I have been unclear somewhere or what I have given is not what you are looking for.
PS: It is always advisable to use dict.get('keyname', None) to get values as you can always put a default value if key is not found. For this solution I didn't do it as I just wanted to provide approach.
Update: In the comments you(OP) mentioned that it gives attributerror.Try this code
data = response.json()
issues = data.get('issues', list())
x = list()
for issue in issues:
temp = dict()
key = issue.get('key', None)
if key:
temp['key'] = key
fields = issue.get('fields', None)
if fields:
customfield = fields.get('customfield_12830', None)
temp['customfield'] = customfield
progress = fields.get('progress', None)
if progress:
total = progress.get('total', None)
temp['total'] = total
x.append(temp)
print(x)

Recursive function to create hierarchical JSON object?

I'm just not a good enough computer scientist to figure this out by myself :(
I have an API that returns JSON responses that look like this:
// call to /api/get/200
{ id : 200, name : 'France', childNode: [ id: 400, id: 500] }
// call to /api/get/400
{ id : 400, name : 'Paris', childNode: [ id: 882, id: 417] }
// call to /api/get/500
{ id : 500, name : 'Lyon', childNode: [ id: 998, id: 104] }
// etc
I would like to parse it recursively and build a hierarchical JSON object that looks something like this:
{ id: 200,
name: 'France',
children: [
{ id: 400,
name: 'Paris',
children: [...]
},
{ id: 500,
name: 'Lyon',
children: [...]
}
],
}
So far, I have this, which does parse every node of the tree, but doesn't save it into a JSON object. How can I expand this to save it into the JSON object?
hierarchy = {}
def get_child_nodes(node_id):
request = urllib2.Request(ROOT_URL + node_id)
response = json.loads(urllib2.urlopen(request).read())
for childnode in response['childNode']:
temp_obj = {}
temp_obj['id'] = childnode['id']
temp_obj['name'] = childnode['name']
children = get_child_nodes(temp_obj['id'])
// How to save temp_obj into the hierarchy?
get_child_nodes(ROOT_NODE)
This isn't homework, but maybe I need to do some homework to get better at solving this kind of problem :( Thank you for any help.

def get_node(node_id):
request = urllib2.Request(ROOT_URL + node_id)
response = json.loads(urllib2.urlopen(request).read())
temp_obj = {}
temp_obj['id'] = response['id']
temp_obj['name'] = response['name']
temp_obj['children'] = [get_node(child['id']) for child in response['childNode']]
return temp_obj
hierarchy = get_node(ROOT_NODE)

You could use this (a more compact and readable version)
def get_child_nodes(node_id):
request = urllib2.Request(ROOT_URL + node_id)
response = json.loads(urllib2.urlopen(request).read())
return {
"id":response['id'],
"name":response['name'],
"children":map(lambda childId: get_child_nodes(childId), response['childNode'])
}
get_child_nodes(ROOT_NODE)

You're not returning anything from each call to the recursive function. So, it seems like you just want to append each temp_obj dictionary into a list on each iteration of the loop, and return it after the end of the loop. Something like:
def get_child_nodes(node_id):
request = urllib2.Request(ROOT_URL + node_id)
response = json.loads(urllib2.urlopen(request).read())
nodes = []
for childnode in response['childNode']:
temp_obj = {}
temp_obj['id'] = childnode['id']
temp_obj['name'] = childnode['name']
temp_obj['children'] = get_child_nodes(temp_obj['id'])
nodes.append(temp_obj)
return nodes
my_json_obj = json.dumps(get_child_nodes(ROOT_ID))
(BTW, please beware of mixing tabs and spaces as Python isn't very forgiving of that. Best to stick to just spaces.)

I had the same problem this afternoon, and ended up rejigging some code I found online.
I've uploaded the code to Github (https://github.com/abmohan/objectjson) as well as PyPi (https://pypi.python.org/pypi/objectjson/0.1) under the package name 'objectjson'. Here it is below, as well:
Code (objectjson.py)
import json
class ObjectJSON:
def __init__(self, json_data):
self.json_data = ""
if isinstance(json_data, str):
json_data = json.loads(json_data)
self.json_data = json_data
elif isinstance(json_data, dict):
self.json_data = json_data
def __getattr__(self, key):
if key in self.json_data:
if isinstance(self.json_data[key], (list, dict)):
return ObjectJSON(self.json_data[key])
else:
return self.json_data[key]
else:
raise Exception('There is no json_data[\'{key}\'].'.format(key=key))
def __repr__(self):
out = self.__dict__
return '%r' % (out['json_data'])
Sample Usage
from objectjson import ObjectJSON
json_str = '{ "test": {"a":1,"b": {"c":3} } }'
json_obj = ObjectJSON(json_str)
print(json_obj) # {'test': {'b': {'c': 3}, 'a': 1}}
print(json_obj.test) # {'b': {'c': 3}, 'a': 1}
print(json_obj.test.a) # 1
print(json_obj.test.b.c) # 3

Disclaimer : I have no idea what json is about, so you may have to sort out how to write it correctly in your language :p. If the pseudo-code in my example is too pseudo, feel free to ask more details.
You need to return something somewhere. If you never return something in your recursive call, you can't get the reference to your new objects and store it in the objects you have where you called the recursion.
def getChildNodes (node) returns [array of childNodes]
data = getData(fromServer(forThisNode))
new childNodes array
for child in data :
new temp_obj
temp_obj.stores(child.interestingStuff)
for grandchild in getChildNodes(child) :
temp_obj.arrayOfchildren.append(grandchild)
array.append(temp_obj)
return array
Alternatively, you can use an iterator instead of a return, if your language supports it.

Python - is there an elegant way to avoid dozens try/except blocks while getting data out of a json object?

I'm looking for ways to write functions like get_profile(js) but without all the ugly try/excepts.
Each assignment is in a try/except because occasionally the json field doesn't exist. I'd be happy with an elegant solution which defaulted everything to None even though I'm setting some defaults to [] and such, if doing so would make the overall code much nicer.
def get_profile(js):
""" given a json object, return a dict of a subset of the data.
what are some cleaner/terser ways to implement this?
There will be many other get_foo(js), get_bar(js) functions which
need to do the same general type of thing.
"""
d = {}
try:
d['links'] = js['entry']['gd$feedLink']
except:
d['links'] = []
try:
d['statisitcs'] = js['entry']['yt$statistics']
except:
d['statistics'] = {}
try:
d['published'] = js['entry']['published']['$t']
except:
d['published'] = ''
try:
d['updated'] = js['entry']['updated']['$t']
except:
d['updated'] = ''
try:
d['age'] = js['entry']['yt$age']['$t']
except:
d['age'] = 0
try:
d['name'] = js['entry']['author'][0]['name']['$t']
except:
d['name'] = ''
return d

Replace each of your try catch blocks with chained calls to the dictionary get(key [,default]) method. All calls to get before the last call in the chain should have a default value of {} (empty dictionary) so that the later calls can be called on a valid object, Only the last call in the chain should have the default value for the key that you are trying to look up.
See the python documentation for dictionairies http://docs.python.org/library/stdtypes.html#mapping-types-dict
For example:
d['links'] = js.get('entry', {}).get('gd$feedLink', [])
d['published'] = js.get('entry', {}).get('published',{}).get('$t', '')

Use get(key[, default]) method of dictionaries
Code generate this boilerplate code and save yourself even more trouble.

Try something like...
import time
def get_profile(js):
def cas(prev, el):
if hasattr(prev, "get") and prev:
return prev.get(el, prev)
return prev
def getget(default, *elements):
return reduce(cas, elements[1:], js.get(elements[0], default))
d = {}
d['links'] = getget([], 'entry', 'gd$feedLink')
d['statistics'] = getget({}, 'entry', 'yt$statistics')
d['published'] = getget('', 'entry', 'published', '$t')
d['updated'] = getget('', 'entry', 'updated', '$t')
d['age'] = getget(0, 'entry', 'yt$age', '$t')
d['name'] = getget('', 'entry', 'author', 0, 'name' '$t')
return d
print get_profile({
'entry':{
'gd$feedLink':range(4),
'yt$statistics':{'foo':1, 'bar':2},
'published':{
"$t":time.strftime("%x %X"),
},
'updated':{
"$t":time.strftime("%x %X"),
},
'yt$age':{
"$t":"infinity years",
},
'author':{0:{'name':{'$t':"I am a cow"}}},
}
})
It's kind of a leap of faith for me to assume that you've got a dictionary with a key of 0 instead of a list but... You get the idea.

You need to familiarise yourself with dictionary methods Check here for how to handle what you're asking.

Two possible solutions come to mind, without knowing more about how your data is structured:
if k in js['entry']:
something = js['entry'][k]
(though this solution wouldn't really get rid of your redundancy problem, it is more concise than a ton of try/excepts)
or
js['entry'].get(k, []) # or (k, None) depending on what you want to do
A much shorter version is just something like...
for k,v in js['entry']:
d[k] = v
But again, more would have to be said about your data.