Let's assume that we the following ndb model:
class MyModel(ndb.Model):
x = ndb.StringProperty()
y = ndb.StringProperty()
z = ndb.StringProperty(repeated=True)
We have a method that creates a query for the above model, executes it and fetch the results. However, we want this query to be modified my other functions. Specifically, we have the following:
def method_a():
qry = MyModel.query()
values = {'query':qry}
method_b(**values)
entities = qry.fetch()
def method_b(**kwargs):
k = ['a', 'b', 'c']
qry = kwargs['query']
qry.filter(MyModel.z.IN(k))
The problem is that the Query object is immutable, and thus it cannot be modified by method_b. Also, based on the specific architecture of the code, we cannot have method_b to return the new Query to method_a.
Any ideas on how to achieve the aforementioned functionality in another way??
Update: Please check the architecture of my code as presented below:
First, in a configuration file we specify a list of modules and if they are enabled or not. These modules affect the filters of the query we want to execute.
testparams = {
'Test1': True,
'Test2': True,
'Test3': False,
'Test4': True
}
Then, we have a method somewhere in the code that makes a query after the appropriate modules have been executed. Thus, it seems like this:
def my_func():
qry = MyEntity.query()
# modules
query_wrapper = [qry]
values = {'param':'x', 'query_wrapper':query_wrapper} #other values also
execute_modules(**values)
# get query and add some more things, like ordering
entities = query_wrapper[0].fetch()
The execute_modules function is the following:
def execute_modules(**kwargs):
for k in config.testparams:
if config.testparams[k]:
if kwargs['param'] == 'x':
(globals()[k]).x(**kwargs)
elif kwargs['param'] == 'y':
(globals()[k]).y(**kwargs)
Finally, an indicative module is similar to the following:
class Test1():
#classmethod
def x(cls, *args, **kwargs):
qry = kwargs['query_wrapper'][0]
# do some stuff like adding filters
kwargs['query_wrapper'][0] = qry
Any proposals to modify this architecture to a better approach?
I'm not aware of a way to do this without having method_b either return or change a referenced parameter. You should use a technique to pass a variable by reference, like passing a class with parameters.
You can pass in the args in a refrence object such as a dict/list:
def modify_query(kwargs):
kwargs['qry'] = kwargs['qry'].filter(MyModel.z.IN(k))
qry = MyModel.query()
kwargs = {'qry': qry}
modify_query(kwargs)
result = kwargs['qry'].fetch()
It should be noted that this is an extremly dirty way to accomplish what you want to accomplish. Similarly, if you pass in a list with say one object, then you can modify the contents of said list (through assignment) to modify the object:
def modify_query(list_object):
list_object[0] = list_object[0].filter(...)
You can do some hack for replace it object by other. For example:
def f(args):
qry = args[0]
qry_new = qry.filter(Model.a == 2)
args[0] = qry_new
qry = Model.query()
args = [qry]
f(args)
qry = args[0]
Related
I want to achieve the below:
def do_something(request):
company_name = request.get("company_name", DEFAULT_COMPANY)
data = request.get("data")
response = transform_data_according_to(data, company_name)
return response
I did the following for it:
class Transform(ABC):
def __init__(self, data):
self.data = data
#abstractmethod
def transform(self):
pass
class CompanyA(Transform):
def transform(self):
# do_transformation
return transformed_data
def do_something(request):
company_name = request.get("company_name", DEFAULT_COMPANY)
data = request.get("data")
if company_name == CompanyA:
response = CompanyA.transform(data)
return response
Instead i would like to do something like this using correct object oriented principles:
def do_something(request):
company_name = request.get("company_name", DEFAULT_COMPANY)
data = request.get("data")
response = Transform(data, company_name)
return response
I want to know where I might be thinking wrong in terms of the desired approach versus the implemented approach. Is the implemented approach correct, the if else checks can grow quite big in that case.
Thanks to teraflop
The simple, idiomatic way to do this in Python would be to look up the Transform subclass in a dictionary:
transform_classes = {
"CompanyA": CompanyA,
# ...
}
def do_something(request):
company_name = request.get("company_name", DEFAULT_COMPANY)
data = request.get("data")
transformer = transform_classes[company_name](data)
return transformer.transform()
If you prefer to be more rigorously object-oriented, you could wrap the dictionary in an object (e.g. TransformLookupByName) instead of accessing it directly.
There are also various kinds of metaprogramming magic you can use to build the dictionary automatically without having to name each subclass explicitly. For example, this will collect all of the Transform subclasses in the current source file:
transform_classes = {
k:v for k,v in globals().items()
if isinstance(v, type) and issubclass(v, Transform) and v != Transform
}
I have a function (func.py). Structure of which look like this:
database = 'VENUS'
def first_function():
print("do some thing")
def second_function():
print("call third function)
third_function()
def third_function(db = database):
print("do some other thing")
I need to import this function and used the inner defined function. But, I want to use a different key for database. Basically, I want to overwrite database = 'VENUS' and use database = 'MARS' while second function call the third function. is there any way to do this?
Just provide the database name as argument
first_function("MARS")
second_function("MARS")
So the problem here, if I understood correctly, is that the default argument for func.third_function is defined at import time. It doesn't matter if you later modify the func.database variable, since the change will not reflect on the default argument of func.third_function.
One (admittedly hacky) solution is to inject a variable using a closure over the imported function. Example:
file.py:
x = 1
def print_x(xvalue = x)
print(xvalue)
Python console:
>>> import file
>>> file.print_x()
1
>>> file.x = 10
>>> file.print_x() # does not work (as you're probably aware)
1
>>> def inject_var(func_to_inject, var):
def f(*args, **kwargs):
return func_to_inject(var, *args, **kwargs)
return f
>>> file.print_x = inject_var(file.print_x, 10)
>>> file.print_x() # works
10
So using the inject_var as written above, you could probably do:
func.third_function = inject_var(func.third_function, "MARS")
If I'm getting multiple records from a database with peewee, I can convert them to dicts like this:
users = User.select().where(User.attribute == some_value).dicts()
However, often I only want one record (or know that only one record will be returned), so I can do:
one_user = User.get(User.name == some_value)
But I can't call .dicts() on the object which is returned by that.
Is there a way to get the result of that get query in dict form?
At the moment the only thing I can think of is the unpythonic
one_user = User.select().where(User.name == some_value).dicts()[0]
peewee has an extension function model_to_dict, defined in playhouse.shortcuts. From the example:
>>> from playhouse.shortcuts import model_to_dict
>>> user = User.create(username='charlie')
>>> model_to_dict(user)
{'id': 1, 'username': 'charlie'}
You can use ".get()":
one_user = User.select().where(User.name == some_value).dicts().get()
Though you can also add a helper method:
class User(Model):
#classmethod
def get_as_dict(cls, expr):
query = cls.select().where(expr).dicts()
return query.get()
It's python. You can extend it.
in reference to Peewee model to JSON
i think you should implement a str method as defined in the link, and then when you do.
user = User.get(User.name == some_value)
userDict = json.dumps(str(user))
you will get the dictioanry of the user
So I've written a module that contains a bunch of functions to easily interact with a subprocess. This subprocess has a whole bunch of settings that let you change how it formats and behaves. I realized that it'd be nice to have a convenience class that you could use as a handler to store the settings you prefer to use and pass them on to the module level functions. here's the example code I'm doing testing with:
import inspect
class MyHandler(object):
def __init__(self):
self.format_string='class format string'
self.database='class database'
self.mode = "class mode"
def rename(self, *args, **kwargs):
self._pass_to_function(rename, *args, **kwargs)
def _pass_to_function(self, function, *overrided_args, **overrided_kwargs):
# get the function's remaining arguments with the inspect module
functon_kwargs = inspect.getargspec(function)[0][len(overrided_args):]
handler_vars = vars(self)
kwargs_to_pass = {}
for arg in functon_kwargs:
if arg in handler_vars:
kwargs_to_pass[arg] = handler_vars[arg]
for arg in overrided_kwargs:
kwargs_to_pass[arg] = overrided_kwargs[arg]
return function(*overrided_args, **kwargs_to_pass)
def rename(targets, format_string=None, database=None, mode=None,
not_in_class='None'):
print 'targets = {}'.format(targets)
print 'format_string = {}'.format(format_string)
print 'database = {}'.format(database)
print 'mode = {}'.format(mode)
print 'not_in_class = {}\n'.format(not_in_class)
return
The thing I like about this solution is that it uses the attributes stored in the class, but you can easily override them by simply adding them to the method call if you want a one-off with a different setting. To do this I have the _pass_to_function as a kind of wrapper function to parse and fill in the needed settings and overrides. Here's how it looks:
>>> import argstest
>>> argstest.rename('some_file.avi', database='some database')
targets = some_file.avi
format_string = None
database = some database
mode = None
not_in_class = None
>>> tst = argstest.MyHandler()
>>> tst.rename('some_file.avi')
targets = some_file.avi
format_string = class format string
database = class database
mode = class mode
not_in_class = None
>>> tst.rename('some_file.avi', 'one off format string', not_in_class=True)
targets = some_file.avi
format_string = one off format string
database = class database
mode = class mode
not_in_class = True
Now in my real module I have dozens of module-level functions that I want to access from the handler class. Ideally they would generate automatically based on the functions in the module. Seeing as how all the methods are only going to be passing everything to _pass_to_function I get the sense that this shouldn't be very difficult but I'm having a lot of trouble figuring out exactly how.
I've read about using type to generate a meta-class, but I don't see how I would use it in this situation. Am I not seeing how I could use type? Should I use some sort of module level script that adds the functions with setattr? Is what I was doing the better/clearer way to do things?
Any and all advice would be appreciated.
Okay, I think I've answered my own question for now. This is how the module looks:
import inspect
import sys
from types import MethodType
class MyHandler(object):
def __init__(self):
self.format_string = 'class format string'
self.database = 'class database'
self.mode = "class mode"
self._populate_methods()
def _populate_methods(self):
to_add = inspect.getmembers(sys.modules[__name__], inspect.isfunction)
to_add = [x[0] for x in to_add if not x[0].startswith('_')]
for func_name in to_add:
func = getattr(sys.modules[__name__], func_name) # strings to functions
self._add_function_as_method(func_name, func)
def _add_function_as_method(self, func_name, func):
def f(self, *args, **kwargs): # the template for the method we'll add
return self._pass_to_function(func, *args, **kwargs)
setattr(MyHandler, func_name, MethodType(f, None, MyHandler))
def _pass_to_function(self, function, *overrided_args, **overrided_kwargs):
functon_kwargs = inspect.getargspec(function)[0][len(overrided_args):]
handler_vars = vars(self)
kwargs_to_pass = {}
for arg in functon_kwargs:
if arg in handler_vars:
kwargs_to_pass[arg] = handler_vars[arg]
for arg in overrided_kwargs:
kwargs_to_pass[arg] = overrided_kwargs[arg]
return function(*overrided_args, **kwargs_to_pass)
def rename(targets, format_string=None, database=None, mode=None,
not_in_class='None'):
print 'targets = {}'.format(targets)
print 'format_string = {}'.format(format_string)
print 'database = {}'.format(database)
print 'mode = {}'.format(mode)
print 'not_in_class = {}\n'.format(not_in_class)
return
def something_else():
print "this function should become a method"
def _not_a_member():
print "this function should not become a method"
I've added the _populate_methods and the _add_function_as_method member functions. the _populate_methods function gets the name of all "public" functions in the module, de-references them to their function and passes each one though _add_function_as_method. All this method does is use an internal function to capture arguments and sent them to _pass_to_function, and set that function as a method using setattr.
phew
so it works, but I'm still wondering if there isn't a clearer or more straight forward way to get this done. I'd be very grateful if anyone could chime in.
I have what I think is a small misconception with loading some YAML objects. I defined the class below.
What I want to do is load some objects with the overridden loadConfig function for YAMLObjects. Some of these come from my .yaml file, but others should be built out of objects loaded from the YAML file.
For instance, in the class below, I load a member object named "keep" which is a string naming some items to keep in the region. But I want to also parse this into a list and have the list stored as a member object too. And I don't want the user to have to give both the string and list version of this parameter in the YAML.
My current work around has been to override the __getattr__ function inside Region and make it create the defaults if it looks and doesn't find them. But this is clunky and more complicated than needed for just initializing objects.
What convention am I misunderstanding here. Why doesn't the loadConfig method create additional things not found in the YAML?
import yaml, pdb
class Region(yaml.YAMLObject):
yaml_tag = u'!Region'
def __init__(self, name, keep, drop):
self.name = name
self.keep = keep
self.drop = drop
self.keep_list = self.keep.split("+")
self.drop_list = self.drop.split("+")
self.pattern = "+".join(self.keep_list) + "-" + "-".join(self.drop_list)
###
def loadConfig(self, yamlConfig):
yml = yaml.load_all(file(yamlConfig))
for data in yml:
# These get created fine
self.name = data["name"]
self.keep = data["keep"]
self.drop = data["drop"]
# These do not get created.
self.keep_list = self.keep.split("+")
self.drop_list = self.drop.split("+")
self.pattern = "+".join(self.keep_list) + "-" + "-".join(self.drop_list)
###
### End Region
if __name__ == "__main__":
my_yaml = "/home/path/to/test.yaml"
region_iterator = yaml.load_all(file(my_yaml))
# Set a debug breakpoint to play with region_iterator and
# confirm the extra stuff isn't created.
pdb.set_trace()
And here is test.yaml so you can run all of this and see what I mean:
Regions:
# Note: the string conventions below are for an
# existing system. This is a shortened, representative
# example.
Market1:
!Region
name: USAndGB
keep: US+GB
drop: !!null
Market2:
!Region
name: CanadaAndAustralia
keep: CA+AU
drop: !!null
And here, for example, is what it looks like for me when I run this in an IPython shell and explore the loaded object:
In [57]: %run "/home/espears/testWorkspace/testRegions.py"
--Return--
> /home/espears/testWorkspace/testRegions.py(38)<module>()->None
-> pdb.set_trace()
(Pdb) region_iterator
<generator object load_all at 0x1139d820>
(Pdb) tmp = region_iterator.next()
(Pdb) tmp
{'Regions': {'Market2': <__main__.Region object at 0x1f858550>, 'Market1': <__main__.Region object at 0x11a91e50>}}
(Pdb) us = tmp['Regions']['Market1']
(Pdb) us
<__main__.Region object at 0x11a91e50>
(Pdb) us.name
'USAndGB'
(Pdb) us.keep
'US+GB'
(Pdb) us.keep_list
*** AttributeError: 'Region' object has no attribute 'keep_list'
A pattern I have found useful for working with yaml for classes that are basically storage is to have the loader use the constructor so that objects are created in the same way as when you make them normally. If I understand what you are attempting to do correctly, this kind of structure might be useful:
import inspect
import yaml
from collections import OrderedDict
class Serializable(yaml.YAMLObject):
__metaclass__ = yaml.YAMLObjectMetaclass
#property
def _dict(self):
dump_dict = OrderedDict()
for var in inspect.getargspec(self.__init__).args[1:]:
if getattr(self, var, None) is not None:
item = getattr(self, var)
if isinstance(item, np.ndarray) and item.ndim == 1:
item = list(item)
dump_dict[var] = item
return dump_dict
#classmethod
def to_yaml(cls, dumper, data):
return ordered_dump(dumper, '!{0}'.format(data.__class__.__name__),
data._dict)
#classmethod
def from_yaml(cls, loader, node):
fields = loader.construct_mapping(node, deep=True)
return cls(**fields)
def ordered_dump(dumper, tag, data):
value = []
node = yaml.nodes.MappingNode(tag, value)
for key, item in data.iteritems():
node_key = dumper.represent_data(key)
node_value = dumper.represent_data(item)
value.append((node_key, node_value))
return node
You would then want to have your Region class inherit from Serializable, and remove the loadConfig stuff. The code I posted inspects the constructor to see what data to save to the yaml file, and then when loading a yaml file calls the constructor with that same set of data. That way you just have to get the logic right in your constructor and the yaml loading should get it for free.
That code was ripped from one of my projects, apologies in advance if it doesn't quite work. It is also slightly more complicated than it needs to be because I wanted to control the order of output by using OrderedDict. You could replace my ordered_dump function with a call to dumper.represent_dict.