I am querying (via sqlalchemy) my_table with a conditional on a column and then retrieve distinct values in another column. Quite simply
selection_1 = session.query(func.distinct(my_table.col2)).\
filter(my_table.col1 == value1)
I need to do this repeatedly to get distinct values from different columns from my_table.
selection_2 = session.query(func.distinct(my_table.col3)).\
filter(my_table.col1 == value1).\
filter(my_table.col2 == value2)
selection_3 = session.query(func.distinct(my_table.col4)).\
filter(my_table.col1 == value1).\
filter(my_table.col2 == value2).\
filter(my_table.col3 == value3)
The above code works, but as I need to have 6 successive calls it's getting a bit out of hand. I have created a class to handle the method chaining:
class QueryProcessor:
def add_filter(self, my_table_col, value):
filter(my_table_col == value)
return self
def set_distinct_col(self, my_other_table_col):
self.my_other_table_col = my_other_table_col
return session.query(func.distinct(self.my_other_table_col))
Ideally I'd be able to use the class like
selection_1 = QueryProcessor().set_distinct_col(my_table.col2).add_filter(my_table.col1, value1)
selection_2 = selection_1.set_distinct_col(my_table.col3).add_filter(my_table.col2, value2)
selection_3 = selection_2.set_distinct_col(my_table.col4).add_filter(my_table.col3, value3)
but when I run
selection_1 = QueryProcessor().set_distinct_col(my_table.col2).add_filter(my_table.col1, value1)
I get the following error:
Traceback (most recent call last):
File " ... "
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-20-789b26eccbc5>", line 10, in <module>
selection_1 = QueryProcessor().set_distinct_col(my_table.col2).add_filter(my_table.col1, value1)
AttributeError: 'Query' object has no attribute 'add_filter'
Any help will be much welcomed.
You don't really need a special class for this. Your existing code
selection_2 = session.query(func.distinct(my_table.col3)).\
filter(my_table.col1 == value1).\
filter(my_table.col2 == value2)
works because filter is returning a new query based on the original query, but with an additional filter added to it. You can just iterate over the columns and their corresponding values, replacing each old query with its successor.
selection2 = session.query(func.distinct(my_table.col3))
for col, val in zip([my_table.col1, my_table.col2], [value1, value2]):
selection2 = selection2.filter(col == val)
selection_3 = session.query(func.distinct(my_table.col4))
for col, val in zip([mytable.col1, mytable.col2, mytable.col3],
[value1, value2, value3]):
selection_3 = selection_3.filter(col == val)
That said, the problem with your code is that add_filter doesn't actually call the query's filter method, or update the wrapped query.
class QueryProcessor:
def set_distinct_col(self, my_other_table_col):
self.query = session.query(func.distinct(self.my_other_table_col))
return self
def add_filter(self, my_table_col, value):
self.query = self.query.filter(my_table_col == value)
return self
This poses a problem, though: set_distinct_col creates a new query, so it doesn't really make sense in the following
selection_1 = QueryProcessor().set_distinct_col(my_table.col2).add_filter(my_table.col1, value1)
selection_2 = selection_1.set_distinct_col(my_table.col3).add_filter(my_table.col2, value2)
selection_3 = selection_2.set_distinct_col(my_table.col4).add_filter(my_table.col3, value3)
to call set_distinct_col on an existing instance. It can return either a new query or the existing one, but not both (at least, not if you want to do chaining).
Also, note that selection_1 itself is not the query, but selection_1.query.
For your add_filter() function to work as intended, you need your set_distinct_col() function to return a reference to itself (an instance of QueryProcessor).
session.query() returns a Query object which doesn't have an add_filter() method.
Query could have an add_filter method if you did something like Query.add_filter = add_filter, but that's a bad practice because it modifies the Query class, so I don't recommend doing it.
What you're doing is a better option. In order to have access to the query you create with the set_distinct_col() method, you need to store it as an instance variable.
Below, I have done this by storing the query in the instance variable query withself.query = session.query(func.distinct(self.my_other_table_col))
Then, I changed the add_filter() method to return itself to allow for chaining more add_filter() methods.
class QueryProcessor:
def add_filter(self, my_table_col, value):
self.query = self.query.filter(my_table_col == value)
return self
def set_distinct_col(self, my_other_table_col):
self.my_other_table_col = my_other_table_col
self.query = session.query(func.distinct(self.my_other_table_col))
return self
You should also know that you can use multiple filter conditions at a time, so you don't actually need to chain multiple filters together.
session.query(db.users).filter(or_(db.users.name=='Ryan', db.users.country=='England'))
or
session.query(db.users).filter((db.users.name=='Ryan') | (db.users.country=='England'))
Difference between filter and filter_by in SQLAlchemy
P.S. This code has not been tested
Related
Writing a dry function that returns either previous or next instances of a given instance.
This function return previous instances:
def previous(instance):
try:
return Picture.objects.filter(id__lt=instance.id).first()
except Picture.DoesNotExist:
return instance
I want to create an abstracted function which returns either the previous or the next instance using an additional gt_or_lt argument. The problem lies in interpolating that argument into the filter(id__gt_or_lt).
def seek_instance(gt_or_lt, instance):
try:
return Picture.objects.filter(id__gt_or_lt=instance.id).first()
except Picture.DoesNotExist:
return instance
I've tried:
return Picture.objects.filter(id__gt_or_lt = instance.id).first()
seek_instance("gt", instance)
return Picture.objects.filter(id__f"{gt_or_lt}" = instance.id).first()
seek_instance("gt", instance)
return Picture.objects.filter(f"{gt_or_lt}" = instance.id).first()
return Picture.objects.filter(gt_or_lt = instance.id).first()
seek("id__gt", instance)
All fail with their respective errors.
Use a dictionary with kwargs expansion.
return Picture.objects.filter(**{f"id__{gt_or_lt}": instance.id})
You can use dictionary expansion, like #DanielRoseman suggests. But that will still not per se render the previous, or next item. If for example the model has an ordering option [Django-doc], then it is possible that the order is different than on the id. Furthermore, for the previous one, you will need to reverse the ordering.
Furthermore depending on the situation, you might want to prevent that seek_instance can be given a different lookup, like 'in' for example.
We can thus use an if … elif … else here to branch on the item we wish to retrieve, and raise a ValueError in case you use some other lookup:
def seek_instance(lt_or_gt, instance):
try:
if lt_or_gt == 'lt':
return Picture.objects.filter(pk__lt=instance.pk).order_by('-pk').first()
elif lt_or_gt == 'gt':
return Picture.objects.filter(pk__gt=instance.pk).order_by('pk').first()
else:
raise ValueError("Should be 'lt' or 'gt'")
except Picture.DoesNotExist:
return instance
I want to extend an odoo datatype.
First I wanted to create a new datatype from scretch. This wasn't easy since odoo/openerp just don't know how to store that datatype.
So now I want to extend it.
from openerp import fields
class MyDataType(fields.Text):
#classmethod
def browse_my_data_type(self, value1, value2):
result = value1 + value2
if len(result) > 0:
return result
else:
return False
I tried to use it. It's possible untill I want to call the browse_my_data_type method.
from openerp import models, fields, api
import my_new_fields as my_fields
class my_data_type_test(models.Model):
_name = 'my.type.test'
name = fields.Char('Name')
my_data_type = my_fields.MyDataType("My Data Type")
result = fields.Char("Result", compute="_set_result")
#api.one
def _set_result(self):
result = self.my_data_type.browse_my_data_type("valuehere","anotheronehere")
if result:
self.result = result
else:
self.result = ""
I used this code to test the method. Sadly it gives me the error AttributeError: 'unicode' object has no attribute 'browse_my_data_type'
How can I make sure that he knows the method while using it this way? (self.my_data_type.browse_my_data_type("valuehere","anotheronehere")
Try this code may be work:
from openerp import fields
class MyDataType(fields._String):
type = 'text'
def browse_my_data_type(self, value1, value2):
result = value1 + value2
if len(result) > 0:
return result
else:
return False
I have a recursive script that's scraping a JSON file for cars. At each recursive level, it gets a new variable added, and passes that (along with the other values) on to the recursive call, each time getting more and more detailed in the information. I tried to use locals() to dynamically assign a variable, but it remains None even after the call (I recall seeing that sometimes locals() is read only).
I tried using eval() as well, and it gives me the same issue (I know eval is not ideal). I'd ideally like to avoid using a dictionary, because that would require me to load it with values first, which seems like it has some unnecessary steps, but I'm open to anything at this point.
Example:
scraper(manufacturer='Honda') would scrape a JSON file of models, set model='Accord' and then recursively call
scraper(manufacturer='Honda, model='Accord') which scrapes a file of years, set's year=2014 and recursively calls
scraper(manufacturer='Honda', model='Accord', year='2014') which is the base case
def scraper(self, manufacturers, model=None, year=None):
if year:
scrapeurl = '%s&manufacturer=%s&model=%s&year=%s' % (url, manufacturer, model, year)
return someFinalFunction()
elif model:
scrapeurl = '%s&manufacturer=%s&model=%s' % (url, manufacturer, model)
elif manufacturer:
scrapeurl = '%s&manufacturer=%s' % (url, manufacturer)
j = getJSONFromUrl(scrapeurl)
key, values = j.popitems()
for value in values:
locals()[key] = value
return self.scraper(manufacturer, model, year, color)
I'd appreciate any input on how to handle this, I know Python always seems to have some clever ways of doing things, and I'm always learning more about it, so thank you in advance! I'm using Python3 in this example too, if that changes anything
locals()['key'] = value should be locals()[key] = value
Better yet, use **kwargs:
def scraper(self, manufacturer, model=None, year=None):
kwargs = dict(manufacturer=manufacturer, model=model, year=year)
if year:
scrapeurl = '%s&manufacturer=%s&model=%s&year=%s' % (url, manufacturer, model, year)
return someFinalFunction()
elif model:
scrapeurl = '%s&manufacturer=%s&model=%s' % (url, manufacturer, model)
elif manufacturer:
scrapeurl = '%s&manufacturer=%s' % (url, manufacturer)
j = getJSONFromUrl(scrapeurl)
key, values = j.popitems()
for value in values:
kwargs[key] = value
return self.scraper(**kwargs)
It's not entirely clear what you're trying to do, but perhaps this will help:
def scraper(self, **kwargs):
if kwargs.get('year') is not None:
scrapeurl = '{0}&manufacturer={manufacturer}&model={model}&year={year}'
return someFinalFunction() # not sure why this takes no arguments
elif kwargs.get('model') is not None:
scrapeurl = '{0}&manufacturer={manufacturer}&model={model}'
elif kwargs.get('manufacturer') is not None:
scrapeurl = '{0}&manufacturer={manufacturer}'
else:
raise KeyError
j = getJSONFromUrl(scrapeurl.format(url, **kwargs))
key, values = j.popitems()
for value in values:
kwargs[key] = value
return self.scraper(**kwargs)
This uses Python's built-in functionality to treat arbitrary keyword arguments as a dictionary, along with more modern str.format string formatting, to dynamically handle the arguments you're looking for. The only difference is that you now need to call it:
instance.scraper(manufacturer='...')
rather than just
instance.scraper('...')
An example of the string formatting, mixing positional and keyword arguments:
>>> '{0}&manufacturer={manufacturer}'.format('foo', **{'manufacturer': 'bar'})
'foo&manufacturer=bar'
I am writing a basic function that takes three arguments, request, field, and user_id.
The idea is, when you pass through the info, the function returns the result which would be the column (identified by argument "field"), the row (identified by the argument "user_id").
this is my function:
def get_user_field(request, user_id, field):
result = Users.objects.raw("SELECT id, %s FROM blog_users WHERE id = %s", [field, user_id])[0]
#return result.??????
what I do not know how to do is to replace those question marks with what to return the corresponding column. If i try
return result.field
It will pass a string where "field" is. And of course a string cannot be put there.
So how can i achieve a function that works pretty much exactly like this and returns one result?
Thanks!
This can be done with the getattr Python builtin:
return getattr(result, field)
But it would be better to do it entirely differently:
def get_user_field(request, user_id, field):
return User.objects.filter(id=user_id).values_list(field, flat=True)[0]
Or, to allow for the possibility of the user not existing:
def get_user_field(request, user_id, field):
result = User.objects.filter(id=user_id).values_list(field, flat=True)
if result:
return result[0]
else:
return None # or raise an exception or whatever you like
Let's assume that we the following ndb model:
class MyModel(ndb.Model):
x = ndb.StringProperty()
y = ndb.StringProperty()
z = ndb.StringProperty(repeated=True)
We have a method that creates a query for the above model, executes it and fetch the results. However, we want this query to be modified my other functions. Specifically, we have the following:
def method_a():
qry = MyModel.query()
values = {'query':qry}
method_b(**values)
entities = qry.fetch()
def method_b(**kwargs):
k = ['a', 'b', 'c']
qry = kwargs['query']
qry.filter(MyModel.z.IN(k))
The problem is that the Query object is immutable, and thus it cannot be modified by method_b. Also, based on the specific architecture of the code, we cannot have method_b to return the new Query to method_a.
Any ideas on how to achieve the aforementioned functionality in another way??
Update: Please check the architecture of my code as presented below:
First, in a configuration file we specify a list of modules and if they are enabled or not. These modules affect the filters of the query we want to execute.
testparams = {
'Test1': True,
'Test2': True,
'Test3': False,
'Test4': True
}
Then, we have a method somewhere in the code that makes a query after the appropriate modules have been executed. Thus, it seems like this:
def my_func():
qry = MyEntity.query()
# modules
query_wrapper = [qry]
values = {'param':'x', 'query_wrapper':query_wrapper} #other values also
execute_modules(**values)
# get query and add some more things, like ordering
entities = query_wrapper[0].fetch()
The execute_modules function is the following:
def execute_modules(**kwargs):
for k in config.testparams:
if config.testparams[k]:
if kwargs['param'] == 'x':
(globals()[k]).x(**kwargs)
elif kwargs['param'] == 'y':
(globals()[k]).y(**kwargs)
Finally, an indicative module is similar to the following:
class Test1():
#classmethod
def x(cls, *args, **kwargs):
qry = kwargs['query_wrapper'][0]
# do some stuff like adding filters
kwargs['query_wrapper'][0] = qry
Any proposals to modify this architecture to a better approach?
I'm not aware of a way to do this without having method_b either return or change a referenced parameter. You should use a technique to pass a variable by reference, like passing a class with parameters.
You can pass in the args in a refrence object such as a dict/list:
def modify_query(kwargs):
kwargs['qry'] = kwargs['qry'].filter(MyModel.z.IN(k))
qry = MyModel.query()
kwargs = {'qry': qry}
modify_query(kwargs)
result = kwargs['qry'].fetch()
It should be noted that this is an extremly dirty way to accomplish what you want to accomplish. Similarly, if you pass in a list with say one object, then you can modify the contents of said list (through assignment) to modify the object:
def modify_query(list_object):
list_object[0] = list_object[0].filter(...)
You can do some hack for replace it object by other. For example:
def f(args):
qry = args[0]
qry_new = qry.filter(Model.a == 2)
args[0] = qry_new
qry = Model.query()
args = [qry]
f(args)
qry = args[0]