We Keep Coding

How to access a dictionary value from within the same dictionary in Python? [duplicate]

I'm new to Python, and am sort of surprised I cannot do this.
dictionary = {
'a' : '123',
'b' : dictionary['a'] + '456'
}
I'm wondering what the Pythonic way to correctly do this in my script, because I feel like I'm not the only one that has tried to do this.
EDIT: Enough people were wondering what I'm doing with this, so here are more details for my use cases. Lets say I want to keep dictionary objects to hold file system paths. The paths are relative to other values in the dictionary. For example, this is what one of my dictionaries may look like.
dictionary = {
'user': 'sholsapp',
'home': '/home/' + dictionary['user']
}
It is important that at any point in time I may change dictionary['user'] and have all of the dictionaries values reflect the change. Again, this is an example of what I'm using it for, so I hope that it conveys my goal.
From my own research I think I will need to implement a class to do this.

No fear of creating new classes -
You can take advantage of Python's string formating capabilities
and simply do:
class MyDict(dict):
def __getitem__(self, item):
return dict.__getitem__(self, item) % self
dictionary = MyDict({
'user' : 'gnucom',
'home' : '/home/%(user)s',
'bin' : '%(home)s/bin'
})
print dictionary["home"]
print dictionary["bin"]

Nearest I came up without doing object:
dictionary = {
'user' : 'gnucom',
'home' : lambda:'/home/'+dictionary['user']
}
print dictionary['home']()
dictionary['user']='tony'
print dictionary['home']()

>>> dictionary = {
... 'a':'123'
... }
>>> dictionary['b'] = dictionary['a'] + '456'
>>> dictionary
{'a': '123', 'b': '123456'}
It works fine but when you're trying to use dictionary it hasn't been defined yet (because it has to evaluate that literal dictionary first).
But be careful because this assigns to the key of 'b' the value referenced by the key of 'a' at the time of assignment and is not going to do the lookup every time. If that is what you are looking for, it's possible but with more work.

What you're describing in your edit is how an INI config file works. Python does have a built in library called ConfigParser which should work for what you're describing.

This is an interesting problem. It seems like Greg has a good solution. But that's no fun ;)
jsbueno as a very elegant solution but that only applies to strings (as you requested).
The trick to a 'general' self referential dictionary is to use a surrogate object. It takes a few (understatement) lines of code to pull off, but the usage is about what you want:
S = SurrogateDict(AdditionSurrogateDictEntry)
d = S.resolve({'user': 'gnucom',
'home': '/home/' + S['user'],
'config': [S['home'] + '/.emacs', S['home'] + '/.bashrc']})
The code to make that happen is not nearly so short. It lives in three classes:
import abc
class SurrogateDictEntry(object):
__metaclass__ = abc.ABCMeta
def __init__(self, key):
"""record the key on the real dictionary that this will resolve to a
value for
"""
self.key = key
def resolve(self, d):
""" return the actual value"""
if hasattr(self, 'op'):
# any operation done on self will store it's name in self.op.
# if this is set, resolve it by calling the appropriate method
# now that we can get self.value out of d
self.value = d[self.key]
return getattr(self, self.op + 'resolve__')()
else:
return d[self.key]
#staticmethod
def make_op(opname):
"""A convience class. This will be the form of all op hooks for subclasses
The actual logic for the op is in __op__resolve__ (e.g. __add__resolve__)
"""
def op(self, other):
self.stored_value = other
self.op = opname
return self
op.__name__ = opname
return op
Next, comes the concrete class. simple enough.
class AdditionSurrogateDictEntry(SurrogateDictEntry):
__add__ = SurrogateDictEntry.make_op('__add__')
__radd__ = SurrogateDictEntry.make_op('__radd__')
def __add__resolve__(self):
return self.value + self.stored_value
def __radd__resolve__(self):
return self.stored_value + self.value
Here's the final class
class SurrogateDict(object):
def __init__(self, EntryClass):
self.EntryClass = EntryClass
def __getitem__(self, key):
"""record the key and return"""
return self.EntryClass(key)
#staticmethod
def resolve(d):
"""I eat generators resolve self references"""
stack = [d]
while stack:
cur = stack.pop()
# This just tries to set it to an appropriate iterable
it = xrange(len(cur)) if not hasattr(cur, 'keys') else cur.keys()
for key in it:
# sorry for being a duche. Just register your class with
# SurrogateDictEntry and you can pass whatever.
while isinstance(cur[key], SurrogateDictEntry):
cur[key] = cur[key].resolve(d)
# I'm just going to check for iter but you can add other
# checks here for items that we should loop over.
if hasattr(cur[key], '__iter__'):
stack.append(cur[key])
return d
In response to gnucoms's question about why I named the classes the way that I did.
The word surrogate is generally associated with standing in for something else so it seemed appropriate because that's what the SurrogateDict class does: an instance replaces the 'self' references in a dictionary literal. That being said, (other than just being straight up stupid sometimes) naming is probably one of the hardest things for me about coding. If you (or anyone else) can suggest a better name, I'm all ears.
I'll provide a brief explanation. Throughout S refers to an instance of SurrogateDict and d is the real dictionary.
A reference S[key] triggers S.__getitem__ and SurrogateDictEntry(key) to be placed in the d.
When S[key] = SurrogateDictEntry(key) is constructed, it stores key. This will be the key into d for the value that this entry of SurrogateDictEntry is acting as a surrogate for.
After S[key] is returned, it is either entered into the d, or has some operation(s) performed on it. If an operation is performed on it, it triggers the relative __op__ method which simple stores the value that the operation is performed on and the name of the operation and then returns itself. We can't actually resolve the operation because d hasn't been constructed yet.
After d is constructed, it is passed to S.resolve. This method loops through d finding any instances of SurrogateDictEntry and replacing them with the result of calling the resolve method on the instance.
The SurrogateDictEntry.resolve method receives the now constructed d as an argument and can use the value of key that it stored at construction time to get the value that it is acting as a surrogate for. If an operation was performed on it after creation, the op attribute will have been set with the name of the operation that was performed. If the class has a __op__ method, then it has a __op__resolve__ method with the actual logic that would normally be in the __op__ method. So now we have the logic (self.op__resolve) and all necessary values (self.value, self.stored_value) to finally get the real value of d[key]. So we return that which step 4 places in the dictionary.
finally the SurrogateDict.resolve method returns d with all references resolved.
That'a a rough sketch. If you have any more questions, feel free to ask.

If you, just like me wandering how to make #jsbueno snippet work with {} style substitutions, below is the example code (which is probably not much efficient though):
import string
class MyDict(dict):
def __init__(self, *args, **kw):
super(MyDict,self).__init__(*args, **kw)
self.itemlist = super(MyDict,self).keys()
self.fmt = string.Formatter()
def __getitem__(self, item):
return self.fmt.vformat(dict.__getitem__(self, item), {}, self)
xs = MyDict({
'user' : 'gnucom',
'home' : '/home/{user}',
'bin' : '{home}/bin'
})
>>> xs["home"]
'/home/gnucom'
>>> xs["bin"]
'/home/gnucom/bin'
I tried to make it work with the simple replacement of % self with .format(**self) but it turns out it wouldn't work for nested expressions (like 'bin' in above listing, which references 'home', which has it's own reference to 'user') because of the evaluation order (** expansion is done before actual format call and it's not delayed like in original % version).

Write a class, maybe something with properties:
class PathInfo(object):
def __init__(self, user):
self.user = user
#property
def home(self):
return '/home/' + self.user
p = PathInfo('thc')
print p.home # /home/thc

As sort of an extended version of #Tony's answer, you could build a dictionary subclass that calls its values if they are callables:
class CallingDict(dict):
"""Returns the result rather than the value of referenced callables.
>>> cd = CallingDict({1: "One", 2: "Two", 'fsh': "Fish",
... "rhyme": lambda d: ' '.join((d[1], d['fsh'],
... d[2], d['fsh']))})
>>> cd["rhyme"]
'One Fish Two Fish'
>>> cd[1] = 'Red'
>>> cd[2] = 'Blue'
>>> cd["rhyme"]
'Red Fish Blue Fish'
"""
def __getitem__(self, item):
it = super(CallingDict, self).__getitem__(item)
if callable(it):
return it(self)
else:
return it
Of course this would only be usable if you're not actually going to store callables as values. If you need to be able to do that, you could wrap the lambda declaration in a function that adds some attribute to the resulting lambda, and check for it in CallingDict.__getitem__, but at that point it's getting complex, and long-winded, enough that it might just be easier to use a class for your data in the first place.

This is very easy in a lazily evaluated language (haskell).
Since Python is strictly evaluated, we can do a little trick to turn things lazy:
Y = lambda f: (lambda x: x(x))(lambda y: f(lambda *args: y(y)(*args)))
d1 = lambda self: lambda: {
'a': lambda: 3,
'b': lambda: self()['a']()
}
# fix the d1, and evaluate it
d2 = Y(d1)()
# to get a
d2['a']() # 3
# to get b
d2['b']() # 3
Syntax wise this is not very nice. That's because of us needing to explicitly construct lazy expressions with lambda: ... and explicitly evaluate lazy expression with ...(). It's the opposite problem in lazy languages needing strictness annotations, here in Python we end up needing lazy annotations.
I think with some more meta-programmming and some more tricks, the above could be made more easy to use.
Note that this is basically how let-rec works in some functional languages.

The jsbueno answer in Python 3 :
class MyDict(dict):
def __getitem__(self, item):
return dict.__getitem__(self, item).format(self)
dictionary = MyDict({
'user' : 'gnucom',
'home' : '/home/{0[user]}',
'bin' : '{0[home]}/bin'
})
print(dictionary["home"])
print(dictionary["bin"])
Her ewe use the python 3 string formatting with curly braces {} and the .format() method.
Documentation : https://docs.python.org/3/library/string.html

How to say in Pythonese - do something unless it causes an error (without resorting to multilevel try/execpt blocks)

This is a little difficult to explain, so let's hope I'm expressing the problem coherently:
Say I have this list:
my_list = ["a string", 45, 0.5]
The critical point to understand in order to see where the question comes from is that my_list is generated by another function; I don't know ahead of time anything about my_list, specifically its length and the datatype of any of its members.
Next, say that every time <my_list> is generated, there is a number of predetermined operations I want to perform on it. For example, I want to:
my_text = my_list[1]+"hello"
some_var = my_list[10]
mini_list = my_list[0].split('s')[1]
my_sum = my_list[7]+2
etc. The important point here is that it's a large number of operations.
Obviously, some of these operations would succeed with any given my_list and some would fail and, importantly, those which fail will do so with an unpredictable Error type; but I need to run all of them on every generation of my_list.
One obvious solution would be to use try/except on each of these operations:
try:
my_text = my_list[1]+"hello"
except:
my_text = "None"
try:
some_var = my_list[10]
except:
some_var = "couldn't do it"
etc.
But with a large number of operations, this gets very cumbersome. I looked into the various questions about multiple try/excepts, but unless I'm missing something, they don't address this.
Based on someone's suggestion (sorry, lost the link), I tried to create a function with a built-in try/except, create another list of these operations, and send each operation to the function. Something along the lines of
def careful(op):
try:
return op
else:
return "None"
And use it with, for example, the first operation:
my_text = careful(my_list[1]+"hello")
The problem is python seems to evaluate the careful() argument before it's sent out to the function and the error is generated before it can be caught...
So I guess I'm looking for a form of a ternary operator that can do something like:
my text = my_list[1]+"hello" if (this doesn't cause any type of error) else "None"
But, if one exist, I couldn't find it...
Any ideas would be welcome and sorry for the long post.

Maybe something like this?
def careful(op, default):
ret = default
try:
ret = computation()
else:
pass
return ret

If you must do this, consider keeping a collection of the operations as strings and calling exec on them in a loop
actions = [
'my_text = my_list[1]+"hello"',
'some_var = my_list[10]',
'mini_list = my_list[0].split("s")[1]',
'my_sum = my_list[7]+2',
]
If you make this collection a dict, you may also assign a default
Note that if an action default (or part of an action string) is meant to be a string, it must be quoted twice. Consider using block-quotes for this if you already have complex escaping, like returning a raw strings or a string representing a regular expression
{
"foo = bar": r"""r'[\w]+baz.*'"""
}
complete example:
>>> actions_defaults = {
... 'my_text = my_list[1]+"hello"': '"None"',
... 'some_var = my_list[10]': '"couldn\'t do it"',
... 'mini_list = my_list[0].split("s")[1]': '"None"',
... 'my_sum = my_list[7]+2': '"None"',
... }
>>>
>>> for action, default in actions_defaults.items():
... try:
... exec(action)
... except Exception: # consider logging error
... exec("{} = {}".format(action.split("=")[0], default))
...
>>> my_text
'None'
>>> some_var
"couldn't do it"
Other notes
this is pretty evil
declaring your vars before running to be their default values is probably better/clearer (sufficient to pass in the except block, as the assignment will fail)
you may run into weird scoping and need to access some vars via locals()

This sounds like an XY Problem
If you can make changes to the source logic, returning a dict may be a much better solution. Then you can determine if a key exists before doing some action, and potentially also look up the action which should be taken if the key exists in another dict.

Using string as literal expression in function argument in Python

Let's say I have a function that can take various kinds of parameter values, but I don't want to (as a constraint) pass arguments explicitly. Instead, I want to pass them as a string.:
def func(param)
return param+param
a = 'param=4'
func(<do something to a>(a))
>>8
Is this possible in python?
I want to use this idea in Django to create Query filters based on GET parameters in a dictionary and then just chain them using their keys.
lookup_dic = {'user': 'author=user',
'draft': 'Q(publish_date_lte=timezone.now())|
Q(publish_date_isnull=True)'}
Based on whether the user and draft keywords are passed in the GET parameters, this would be read out like:
queryset.objects.filter(author=user).filter(Q(publish_date_lte=timezone.now())|
Q(publish_date_isnull=True))
I understand that I can do this by replacing the author=user by Q(author__name=user), but I wanted to know if this string comprehension feature is implemented in python in general?

Use eval
def func(param=0):
return param+param
a = 'param=4'
eval('func(' + a +')')

Are you looking for this?
def func(param):
return param + param
a = 'param=4'
parameter, value = a.split("=")
print(func(**{parameter: int(value)}))
# >> 8

Data structure of memoization in db

What is the best data structure to cache (save/store/memorize) so many function result in database.
Suppose function calc_regress with flowing definition in python:
def calc_regress(ind_key, dep_key, count=30):
independent_list = sql_select_recent_values(count, ind_key)
dependant_list = sql_select_recent_values(count, dep_key)
import scipy.stats as st
return st.linregress(independent_list, dependant_list)
I see answers to What kind of table structure should be used to store memoized function parameters and results in a relational database? but it seem to resolve problem of just one function while I have about 500 function.

Option A
You could use the structure in the linked answer, un-normalized with the number of columns = max number of arguments among the 500 functions. Also need to add a column for the function name.
Then you could do a SELECT * FROM expensive_func_results WHERE func_name = 'calc_regress' AND arg1 = ind_key AND arg2 = dep_key and arg3 = count, etc.
Ofcourse, that's not a very good design to use. For the same function called with fewer parameters, columns with null values/non-matches need to be ignored; otherwise you'll get multiple result rows.
Option B
Create the table/structure as func_name, arguments, result where 'arguments' is always a kwargs dictionary or positional args but not mixed per entry. Even with the kwargs dict stored as a string, order of keys->values in it is not predictable/consistent even if it's the same args. So you'll need to order it before converting to a string and storing it. When you want to query, you'll use SELECT * FROM expensive_func_results WHERE func_name = 'calc_regress' AND arguments = 'str(kwargs_dict)', where str(kwargs_dict) is something you'll set programmatically. It could also be set to the result of inspect.getargspec, (or inspect.getcallargs) though you'll have to check for consistency.
You won't be able to do queries on the argument combos unless you provide all the arguments to the query or partial match with LIKE.
Option C
Normalised all the way: One table func_calls as func_name, args_combo_id, arg_name_idx, arg_value. Each row of the table will store one arg for one combo of that function's calling args. Another table func_results as func_name, args_combo_id, result. You could also normalise further for func_name to be mapped to a func_id.
In this one, the order of keyword args doesn't matter since you'll be doing an Inner join to select each parameter. This query will have to be built programmatically or done via a stored procedure, since the number of joins required to fetch all the parameters is determined by the number of parameters. Your function above has 3 params but you may have another with 10. arg_name_idx is 'argument name or index' so it also works for mixed kwargs + args. Some duplication may occur in cases like calc_regress(ind_key=1, dep_key=2, count=30) and calc_regress(1, 2, 30) (as well as calc_regress(1, 2) with a default value for count <-- this cases should be avoided, the table entry should have all args); since the args_combo_id will be different for both but result will obviously be the same. Again, the inspect module may help in this area.
[Edit] PS: Additionally, for the func_name, you may need to use a fully qualified name to avoid conflicts across modules in your package. And decorators may interfere with that as well; without a deco.__name__ = func.__name__, etc.
PPS: If objects are being passed to functions being memoized in the db, make sure that their __str__ is something useful & repeatable/consistent to store as arg values.
This particular case doesn't require you to re-create objects from the arg values in the db, otherwise, you'd need to make __str__ or __repr__ like the way __repr__ was intended to be (but isn't generally done):
this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment).

I'd use a key value storage here, where the key could be a concatenation of the id of the function object (to guarantee the key uniqness) and its arguments while the value would be the function returned value.
So calc_regress(1, 5, 30) call would produce an example key 139694472779248_1_5_30 where the first part is id(calc_regress). An example key producing function:
>>> def produce_cache_key(fun, *args, **kwargs):
... args_key = '_'.join(str(a) for a in args)
... kwargs_key = '_'.join('%s%s' % (k, v) for k, v in kwargs.items())
... return '%s_%s_%s' % (id(fun), args_key, kwargs_key)
You could keep your results in memory using a dictionary and a decorator:
>>> def cache_result(cache):
... def decorator(fun):
... def wrapper(*args, **kwargs):
... key = produce_cache_key(fun, *args, **kwargs)
... if key not in cache:
... cache[key] = fun(*args, **kwargs)
... return cache[key]
... return wrapper
... return decorator
...
>>>
>>> #cache_result(cache_dict)
... def fx(x, y, z=0):
... print 'Doing some expensive job...'
...
>>> cache = {}
>>> fx(1, 2, z=1)
Doing some expensive job...
>>> fx(1, 2, z=1)
>>>

python convert a string to an operator

Is it possible to convert a string to an operator in python?
I would like to pass a condition to a function
Ideally it would look like this:
def foo(self, attribute, operator_string, right_value):
left_value = getattr(self, attribute)
if left_value get_operator(operator_string) right_value:
return True
else:
return False
bar.x = 10
bar.foo('x', '>', 10)
[out] False
bar.foo('x', '>=', 10)
[out] True
I could make a dictionary where keys are strings and values are functions of the operator module.
I would have to change foo definition slightly:
operator_dict = {'>', operator.lt,
'>=', operator.le}
def foo(self, attribute, operator_string, right_value):
left_value = getattr(self, attribute)
operator_func = operator_dict[operator_string]
if operator_func(left_value, right_value):
return True
else:
return False
This means I have to make this dictionary, but is it really necessary?

You can use eval to dynamically build a piece of Python code and execute it, but apart from that there are no real alternatives. The dictionary-based solution is much more elegant and safe, however.
Apart from that, is it really that bad? Why not shorten it a bit …
return operator_dict[operator_string](left_value, right_value)

The way the problem is specified I don't see why you can't pass operator.le to the function instead of ">=".
If this operator_string coming from a database or file or something or are you passing it around in your code?
bar.foo('x', operator.le , 10)
Are you just looking to have a convenient shorthand? Then you might do something like:
from operator import le
bar.foo('x', le, 10)
If the real problem here is that you have code or business rules coming in from a database or datafile then maybe you actually need to look at writing a little parser that will map your input into these objects and then you could take a look at using a library like pyparsing, ply, codetalker, etc.

#This is very simple to do with eval()
score=1
trigger_conditon=">="
trigger_value=4
eval(f"{score}{trigger_conditon}{trigger_value}")
#luckily fstring also takes care of int/float or relavaent datatype
operator_str="ge"
import operator
eval(f"operator.{operator_str}({score},{trigger_value})")