We Keep Coding

How checking lookup depth into nested dictionary as class attribute?

I created a nested dictionary based on AttrDict found there :
Object-like attribute access for nested dictionary
I modified it to contain str commands in "leaves" that gets executed when the value is requested/written to :
commands = {'root': {'com': {'read': 'READ_CMD', 'write': 'WRITE_CMD'} } }
class AttrTest()
def __init__:
self.__dict__['attr'] = AttrDict(commands)
test = AttrTest()
data = test.attr.root.com.read # data = value read with the command
test.attr.root.com.write = data # data = value written on the com port
While it works beautifully, I'd like to :
Avoid people getting access to attr/root/com as these returns a sub-level dictonary
People accessing attr.root.com directly (through __getattribute__/__setattr__)
Currently, I'm facing the following problems :
As said, when accessing the 'trunk' of the nested dict, I get a partial dict of the 'leaves'
When accessing attr.root.com it returns {'read': 'READ_CMD', 'write': 'WRITE_CMD'}
If detecting a read I do a forward lookup and return the value, but then attr.root.com.read fails
Is it possible to know what is the final level Python will request in the "path" ?
To block access to attr/root
To read/write the value accessing attr.root.com directly (using forward lookup)
To return the needed partial dict only if attr.root.com.read or attr.root.com.write are requested
Currently I've found nothing that allows me to control how deep the lookup is expected to go.
Thanks for your consideration.

For a given attribute lookup you cannot determine how many others will follow; this is how Python works. In order to resolve x.y.z, first the object x.y needs to be retrieved before the subsequent attribute lookup (x.y).z can be performed.
What you can do however, is return a proxy object that represents the (partial) path instead of the actual underlying object which is stored in the dict. So for example if you did test.attr.com then this would return a proxy object which represents the path attr.com to-be-looked up on the test object. Only when you encounter a read or write leaf in the path, you would resolve the path and read/write the data.
The following is a sample implementation which uses an AttrDict based on __getattr__ to provide the Proxy objects (so you don't have to intercept __getattribute__):
from functools import reduce
class AttrDict(dict):
def __getattr__(self, name):
return Proxy(self, (name,))
def _resolve(self, path):
return reduce(lambda d, k: d[k], path, self)
class Proxy:
def __init__(self, obj, path):
object.__setattr__(self, '_obj', obj)
object.__setattr__(self, '_path', path)
def __str__(self):
return f"Path<{'.'.join(self._path)}>"
def __getattr__(self, name):
if name == 'read':
return self._obj._resolve(self._path)[name]
else:
return type(self)(self._obj, (*self._path, name))
def __setattr__(self, name, value):
if name != 'write' or name not in (_dict := self._obj._resolve(self._path)):
raise AttributeError(f'Cannot set attribute {name!r} for {self}')
_dict[name] = value
commands = {'root': {'com': {'read': 'READ_CMD', 'write': 'WRITE_CMD'} } }
test = AttrDict({'attr': commands})
print(f'{test.attr = !s}') # Path<attr>
print(f'{test.attr.root = !s}') # Path<attr.root>
print(f'{test.attr.root.com = !s}') # Path<attr.root.com>
print(f'{test.attr.root.com.read = !s}') # READ_CMD
test.attr.root.com.write = 'test'
test.attr.root.write = 'illegal' # raises AttributeError

Formencode OneOf validator with dynamic list to test against

I'm using formencode 1.3.0a1 (and turbogeras 2.3.4) and run into a problem with the validator OneOf.
I want to validate some input according to a list in the Database.
Here is my validation schema and the method for getting the list:
from formencode import Schema, validators
def getActiveCodes():
codes = DBSession.query(SomeObject.code).all()
codes = [str(x[0]) for x in codes]
return codes
class itemsEditSchema(Schema):
code = validators.OneOf(getActiveCodes())
allow_extra_fields = True
The method "getActiveCodes" is executed just once (I guess during schema init or something like that).
I need it to run every time when I want to check my user input for "code", how can I do that?
Thanks for the help

I don't know any way to make formencode do what you ask. However, since this is Python, there are few limits to what we can do.
You can solve this by wrapping the call to getActiveCodes in a purpose-built class. The wrapper class, RefreshBeforeContainsCheck, will implement the special methods __iter__ and __contains__ to provide the necessary interface to be used as a iterable object:
from formencode import Schema, validators, Invalid
class RefreshBeforeContainsCheck(object):
def __init__(self, func):
self._func = func
self._current_list = None
def __iter__(self):
print '__iter__ was called.'
#return iter(self._func()) # Could have refreshed here too, but ...
return iter(self._current_list)
def __contains__(self, item):
print '__contains__ was called.'
self._current_list = self._func() # Refresh list.
return item in self._current_list
I've added print statements to make it clearer how it behaves during run-time. The RefreshBeforeContainsCheck class can then be used like
class ItemsEditSchema(Schema):
code = validators.OneOf(RefreshBeforeContainsCheck(getActiveCodes))
allow_extra_fields = True
in the validator schema.
The way it is implemented above, the getActiveCodes function will be called each time the OneOf validator performs an item in list test (where our class acts as the list), because that will cause RefreshBeforeContainsCheck.__contains__ to be called. Now, if the validation fails, the OneOf validator generates an error message listing all the elements of list; that case is handled by our __iter__ implementation. To avoid calling the database twice in case of validation errors, I've chosen to cache the "database" result list as self._current_list, but whether or not that's appropriate depends on your needs.
I've created a gist for this: https://gist.github.com/mtr/9719d08f1bbace9ebdf6, basically creating an example of using the above code with the following code.
def getActiveCodes():
# This function could have performed a database lookup.
print 'getActivityCodes() was called.'
codes = map(str, [1, 2, 3, 4])
return codes
def validate_input(schema, value):
print 'Validating: value: {!r}'.format(value)
try:
schema.to_python(value)
except Invalid, error:
print 'error: {!r}'.format(error)
print
def main():
schema = ItemsEditSchema()
validate_input(schema, {'code': '3'})
validate_input(schema, {'code': '4'})
validate_input(schema, {'code': '5'})
The output of the gist is:
Validating: value: {'code': '3'}
__contains__ was called.
getActivityCodes() was called.
Validating: value: {'code': '4'}
__contains__ was called.
getActivityCodes() was called.
Validating: value: {'code': '5'}
__contains__ was called.
getActivityCodes() was called.
__iter__ was called.
error: Invalid("code: Value must be one of: 1; 2; 3; 4 (not '5')",
{'code': '5'}, None, None,
{'code': Invalid(u"Value must be one of: 1; 2; 3; 4 (not '5')",
'5', None, None, None)})

In the end I worte a FancyValidtor instead of using OneOf, here is my code:
class codeCheck(FancyValidator):
def to_python(self, value, state=None):
if value==None:
raise Invalid('missing a value', value, state)
return super(codeCheck,self).to_python(value,state)
def _validate_python(self, value, state):
codes = DBSession.query(Code).all()
if value not in codes:
raise Invalid('wrong code',value, state)
return value
class itemsEditSchema(Schema):
code = codeCheck ()
allow_extra_fields = True

Using function value as kwargs.get() default

I have a factory function for a model with several foreign keys in my unit tests. I would like for that factory function to be variadic, allowing the user to specify the objects to use as foreign keys as keyword arguments, but calling the relevant factory function to spawn a new one for any that are left out.
I originally wrote something like:
def model_factory(i, **kwargs):
"""Create a new Model for testing"""
test_model_data = {
'fk1': kwargs.get('fk1', fk1_factory(i)),
'fk2': kwargs.get('fk2', fk2_factory(i)),
'fk3': kwargs.get('fk3', fk3_factory(i)),
}
return Model.objects.create(**test_model_data)
but this calls the fkN_factory() methods even if the keyword is present, causing a lot of side effects that are interfering with my tests. My question is whether or not there is a simpler way to do what I intended here without resulting in lots of needless function calls, rather than what I have now, which is more like:
def model_factory(i, **kwargs):
"""Create a new Model for testing"""
test_model_data = {
'fk1': kwargs.get('fk1', None),
'fk2': kwargs.get('fk2', None),
}
if kwargs['f1'] is None:
kwargs['f1'] = fk1_factory(i)
if kwargs['f2'] is None:
kwargs['f2'] = fk2_factory(i)

You want to factor out that repeated code in some way. The simplest is:
def get_value(mapping, key, default_func, *args):
try:
return mapping[key]
except KeyError:
return default_func(*args)
# ...
test_model_data = {
'fk1': get_value(kwargs, 'fk1', fk1_factory, i),
'fk2': get_value(kwargs, 'fk2', fk2_factory, i),
# etc.
}
Almost as simple as your original non-working version.
You could take this even farther:
def map_data(mapping, key_factory_map, *args):
return {key: get_value(mapping, key, factory, *args)
for key, factory in key_factory_map.items()}
# …
test_model_data = map_data(kwargs, {
'fk1': fk1_factory,
'fk2': fk2_factory,
# …
}, i)
But I'm not sure that's actually better. (If you have an obvious place to define that key-to-factory mapping out-of-line, it probably is; if not, probably not.)

How to identify a column name by a string? django/python

I am writing a basic function that takes three arguments, request, field, and user_id.
The idea is, when you pass through the info, the function returns the result which would be the column (identified by argument "field"), the row (identified by the argument "user_id").
this is my function:
def get_user_field(request, user_id, field):
result = Users.objects.raw("SELECT id, %s FROM blog_users WHERE id = %s", [field, user_id])[0]
#return result.??????
what I do not know how to do is to replace those question marks with what to return the corresponding column. If i try
return result.field
It will pass a string where "field" is. And of course a string cannot be put there.
So how can i achieve a function that works pretty much exactly like this and returns one result?
Thanks!

This can be done with the getattr Python builtin:
return getattr(result, field)
But it would be better to do it entirely differently:
def get_user_field(request, user_id, field):
return User.objects.filter(id=user_id).values_list(field, flat=True)[0]
Or, to allow for the possibility of the user not existing:
def get_user_field(request, user_id, field):
result = User.objects.filter(id=user_id).values_list(field, flat=True)
if result:
return result[0]
else:
return None # or raise an exception or whatever you like

How to modify ndb.Query object?

Let's assume that we the following ndb model:
class MyModel(ndb.Model):
x = ndb.StringProperty()
y = ndb.StringProperty()
z = ndb.StringProperty(repeated=True)
We have a method that creates a query for the above model, executes it and fetch the results. However, we want this query to be modified my other functions. Specifically, we have the following:
def method_a():
qry = MyModel.query()
values = {'query':qry}
method_b(**values)
entities = qry.fetch()
def method_b(**kwargs):
k = ['a', 'b', 'c']
qry = kwargs['query']
qry.filter(MyModel.z.IN(k))
The problem is that the Query object is immutable, and thus it cannot be modified by method_b. Also, based on the specific architecture of the code, we cannot have method_b to return the new Query to method_a.
Any ideas on how to achieve the aforementioned functionality in another way??
Update: Please check the architecture of my code as presented below:
First, in a configuration file we specify a list of modules and if they are enabled or not. These modules affect the filters of the query we want to execute.
testparams = {
'Test1': True,
'Test2': True,
'Test3': False,
'Test4': True
}
Then, we have a method somewhere in the code that makes a query after the appropriate modules have been executed. Thus, it seems like this:
def my_func():
qry = MyEntity.query()
# modules
query_wrapper = [qry]
values = {'param':'x', 'query_wrapper':query_wrapper} #other values also
execute_modules(**values)
# get query and add some more things, like ordering
entities = query_wrapper[0].fetch()
The execute_modules function is the following:
def execute_modules(**kwargs):
for k in config.testparams:
if config.testparams[k]:
if kwargs['param'] == 'x':
(globals()[k]).x(**kwargs)
elif kwargs['param'] == 'y':
(globals()[k]).y(**kwargs)
Finally, an indicative module is similar to the following:
class Test1():
#classmethod
def x(cls, *args, **kwargs):
qry = kwargs['query_wrapper'][0]
# do some stuff like adding filters
kwargs['query_wrapper'][0] = qry
Any proposals to modify this architecture to a better approach?

I'm not aware of a way to do this without having method_b either return or change a referenced parameter. You should use a technique to pass a variable by reference, like passing a class with parameters.

You can pass in the args in a refrence object such as a dict/list:
def modify_query(kwargs):
kwargs['qry'] = kwargs['qry'].filter(MyModel.z.IN(k))
qry = MyModel.query()
kwargs = {'qry': qry}
modify_query(kwargs)
result = kwargs['qry'].fetch()
It should be noted that this is an extremly dirty way to accomplish what you want to accomplish. Similarly, if you pass in a list with say one object, then you can modify the contents of said list (through assignment) to modify the object:
def modify_query(list_object):
list_object[0] = list_object[0].filter(...)

You can do some hack for replace it object by other. For example:
def f(args):
qry = args[0]
qry_new = qry.filter(Model.a == 2)
args[0] = qry_new
qry = Model.query()
args = [qry]
f(args)
qry = args[0]