I'm using Haystack and Whoosh to do search with a django site I'm building. I'd like to use an OR operator on search terms (e.g. "Search String" will find objects with text "Search" OR "String" instead of "Search" AND "String")
This seems pretty straight forward as haystack allows you to override the default "AND" operator by setting HAYSTACK_DEFAULT_OPERATOR = 'OR' in your settings.py file.
Unfortunately, adding this to my settings.py has had no effect. I've found a couple of tangential references to this behavior on stackoverflow, but no solution. I've also found an issue posted on github, but it's been there since last year with no comments or classification.
I may be doing something wrong, so figured I'd post here and see if there's a solution. I'm kinda stuck without one!
My haystack settings in my settings.py:
HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
'PATH': os.path.join(os.path.dirname(__file__), 'whoosh_index'),
},
}
HAYSTACK_DEFAULT_OPERATOR = 'OR'
HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'
My view:
from haystack import views as hsviews
def search_test(request):
return hsviews.basic_search(request)
My search_indexes.py file:
import datetime
from haystack import indexes
from myApp.models import MyModel
from django.contrib.auth.models import User
class MyModelIndex(indexes.SearchIndex, indexes.Indexable):
text = indexes.NgramField(document=True, use_template= True)
isPublic = indexes.BooleanField(model_attr='isPubliclyVisible')
brand = indexes.CharField(model_attr='brand')
model = indexes.CharField(model_attr='model')
owner = indexes.CharField(model_attr='owner')
owner_username = indexes.CharField()
obj_type = indexes.CharField()
def get_model(self):
return MyModel
def index_queryset(self, using=None):
"""Used when the entire index for model is updated."""
return self.get_model().objects.filter(isPubliclyVisible = True)
def prepare_owner_username(self, obj):
return obj.owner.user.username
def prepare_obj_type(self,obj):
return 'MyModel'
I did find this workaround (which I haven't tested/thought through for my solution yet), but I figured this warranted its own question in case I/we are doing something wrong.
Instead of using the Haystack built-in basic_search function, I would suggest writing your own view so you would have more control of how the search queries are performed. That way, you can process more complex searches by extending your view or custom search query function, plus it would be easier to test.
For example, you can build separate SearchQuerySet filters to perform each of the keywords you're seaching for, then "OR" them together, like this:
def get_query(request):
"""
This function retrieves any query terms (e.g q=search+term)
from the request object.
:param request: request object
:returns: query terms as a list (split on whitespace)
"""
query = None
qs_keyword = 'q'
if (qs_keyword in request.GET) and request.GET[qs_keyword].strip():
query_string = request.GET[qs_keyword]
query = query_string.split()
return query
def perform_query(request):
"""
This is a helper function to perform the actual query.
You can extend this to handle more complicated searches using AND,
OR, boolean qualifiers, etc.
:param request: request object
:returns: SearchQuerySet results
"""
query = get_query(request)
if not query:
results = EmptySearchQuerySet()
else:
results = SearchQuerySet()
for search_term in query:
# you can use the "|" (or) operator
results |= results.filter(content=search_term)
# or else use "filter_or"
# results = results.filter_or(content=search_term)
return results
def your_search_view(request, *args, **kwargs):
"""
This is your search view to process the query and display your results.
"""
# call "perform_query" to do the actual search
results = perform_query(request)
# do the rest of your view processing ...
return render_to_response(etc.)
Related
There are two models .I want to make query to extract only the app exact app related Adspaces .
models.py
class Appname(models.Model):
user=models.ForeignKey(User,related_name='appname', null=True, default=None,on_delete=models.CASCADE)
name=models.CharField(max_length=150,blank=False,null=False,help_text='Add your new App')
def __str__(self):
return self.name
def get_absolute_url(self):
return reverse("dashapp:space",kwargs={'pk':self.pk})
class Adspace(models.Model):
user=models.ForeignKey(User,related_name='adspace', null=True, default=None,on_delete=models.CASCADE)
ad_space=models.CharField(max_length=150,blank=False,null=False)
app=models.ForeignKey('Appname', related_name='appnames',default=None, on_delete=models.CASCADE)
PID_TYPE = (
('FN','FORMAT_NATIVE'),
('FNB','FORMAT_NATIVE_BANNER'),
('FI','FORMAT_INTERSTITIAL'),
('FB','FORMAT_BANNER'),
('FMR','FORMAT_MEDIUM,RECT'),
('FRV','FORMAT_REWARDED_VIDEO'),
)
format_type=models.CharField(max_length=3,choices=PID_TYPE,default='FN',blank=False, null=False)
def __str__(self):
return self.ad_space
def get_absolute_url(self):
return reverse("dashapp:create",kwargs={'pk':self.pk})
Views.py
SHowing the one where i need to the query
class spacelist(LoginRequiredMixin,ListView):
model=Adspace
template_name='adspace_list.html'
def get_queryset(self):
query_set=super().get_queryset()
return query_set.filter(user=self.request.user)
Here I need to perform One more query so that EACH APP show their own adspaces when clicked right now every app show every show adspaces.
I have the idea what to do as if i compare app_id then it'll show the exact app related adspaces, but i dont know how to write query for the same as i already have one query present.???
You could try using a Q objects: https://docs.djangoproject.com/en/2.1/topics/db/queries/#complex-lookups-with-q-objects
From what I understand you are trying to filter both on the app_id and the request user at the same time, so you could try look something like this:
from django.db.models import Q
...
def get_queryset(self):
query_set=super().get_queryset()
return query_set.filter(Q(user=self.request.user) & Q(app_id=app_id))
...
This lets you do a single filter with both your requirements at the same time (i.e. retrieve the Adspace instances for a specific user with a specific Appname).
You chain another filter at the end like this:
class spacelist(LoginRequiredMixin,ListView):
model=Adspace
template_name='adspace_list.html'
def get_queryset(self):
query_set = super().get_queryset()
query_set = query_set.filter(user=self.request.user)
app_id = [...]
return query_set.filter(app_id=app_id)
The problem left is to find out what is the app_id coming from. How do you know what is the current app? Several options here.
Option 1: From the request
It can come from the current user: self.request.user.appname.all() but that will give you multiple apps, if the user can only have one app, you should change your model Appname.user to a OneToOneField.
Otherwise, I suggest changing your related_name='appnames' to reflect the multiplicity in the reverse relationship.
Option 2: From the URL
It can come from the URL, your space list view should extract an app_id parameter from the URL where it's defined:
url(r'^(?P<app_id>[0-9]+)/spaces/$', spacelist.as_view(), name='space_list'),
And then in the spacelist view, you would get this parameter like this:
app_id = self.kwargs['app_id']
return query_set.filter(app_id=app_id)
Hope that helps
UPDATE:
Also worth noting that QuerySets are lazy, meaning the result will get evaluated as late as possible by Django. Therefore, when you call:
query_set = query_set.filter(user=self.request.user)
The Django ORM doesn't execute any DB queries yet, and you can chain more filters after that:
query_set = query_set.filter(user=self.request.user)
query_set = query_set.filter(app_id=app_id)
Which behind the scenes is extending the query that will be executed when required. But at this point, no query is actually run. To see the query that will get executed you can print out the query attribute of the QuerySet:
print(query_set.query)
Which should log something like:
SELECT "app_adspace"."user_id" ...
FROM
"app_adspace"
WHERE
"app_adspace"."user_id" = 1234 AND "app_adspace"."app_id" = 5678
I have followed the Getting Starting - Django Haystack example, swapping out their model for mine.
In search_indexes.py, the method index_queryset has the comment "Used when the entire index for model is updated." however it is called every time I do a search from the view search/search.html
The method itself gets all the objects from the database and is very slow, so I assume this isn't the intended behavior.
search_indexes.py
import datetime
from haystack import indexes
from article.models import Article
class ArticleIndex(indexes.SearchIndex, indexes.Indexable):
text = indexes.CharField(document=True, use_template=True)
article_id = indexes.IntegerField(model_attr='id')
title = indexes.CharField(model_attr='title')
summary = indexes.CharField(model_attr='summary')
content = indexes.CharField(model_attr='content')
published_at = indexes.DateTimeField(model_attr='published_at')
def get_model(self):
return Article
def index_queryset(self, using=None):
return self.get_model().objects.filter(
published_at__lte=datetime.datetime.now())
I captured the stack trace to see where it was being called from:
python2.7/site-packages/haystack/views.py(53)__call__()
-> return self.create_response()
python2.7/site-packages/haystack/views.py(133)create_response()
-> (paginator, page) = self.build_page()
python2.7/site-packages/haystack/views.py(110)build_page()
-> self.results[start_offset:start_offset + self.results_per_page]
python2.7/site-packages/haystack/query.py(272)__getitem__()
-> self._fill_cache(start, bound)
python2.7/site-packages/haystack/query.py(191)_fill_cache()
-> to_cache = self.post_process_results(results)
python2.7/site-packages/haystack/query.py(214)post_process_results()
-> objects = index.read_queryset(using=self.query._using)
python2.7/site-packages/haystack/indexes.py(144)read_queryset()
-> return self.index_queryset(using=using)
myApplication/article/search_indexes.py(20)index_queryset()
-> return self.get_model().objects.filter(
Note: I am using django version 1.7.10 and django-haystack version 2.4.1
This is intended behaviour - it is the docstring that is wrong. The function index_queryset basically returns the queryset that Haystack will use to obtain the search results (as well as to index documents).
You say:
The method itself gets all the objects from the database and is very slow
Actually it doesn't. All the method does is return a queryset. Querysets are lazy, so the method doesn't hit the database. The database only gets hit when something tried to access the results of the queryset.
This will happen after your search has been executed and Haystack returns the results. At this point the queryset will be further filtered to return the objects that matched the search. If this is slow then it may indicate a more fundamental performance issue with your model structure.
I want to add a search function for a simple django cms homepage with cmsplugin-blog.
But there are only search indexes for use with django-cms-facetsearch. But facetsearch needs solr, and I don`t want to run a solr server only for a few cms-pages and blogentries. I just want to use haystack with whoosh, because it is really simple to configurate.
Are there multilingual search indexes for cmsplugin-blog models? Or do I have to write them myself?
Thank you for helping...
I had the same problem for a setup using haystack, django-cms, cmsplugin-blog and some other apps as well.
I just created a custom search index for cmsplugin-blog and haystack, inspired by the index used for regular CMS pages in django-cms-search. Take a look at it, it may help you to create your own.
from haystack import indexes
from haystack import site
from cmsplugin_blog.models import Entry, EntryTitle
from cms.models.pluginmodel import CMSPlugin
from django.utils.encoding import force_unicode
import re
def _strip_tags(value):
"""
Returns the given HTML with all tags stripped.
This is a copy of django.utils.html.strip_tags, except that it adds some
whitespace in between replaced tags to make sure words are not erroneously
concatenated.
"""
return re.sub(r'<[^>]*?>', ' ', force_unicode(value))
class BlogIndex(indexes.SearchIndex):
text = indexes.CharField(document=True)
url = indexes.CharField(stored=True, indexed=False, model_attr='get_absolute_url')
title = indexes.CharField(stored=True, indexed=False)
pub_date = indexes.DateTimeField(model_attr='pub_date', null=True)
def get_model(self):
return Entry
def index_queryset(self):
"""Used when the entire index for model is updated."""
return self.get_model().objects.filter(is_published=True)
def prepare_title(self, obj):
return EntryTitle.objects.filter(entry=obj)[0]
def prepare_text(self, obj):
title = EntryTitle.objects.filter(entry=obj)[0]
placeholder_plugins = CMSPlugin.objects.filter(placeholder__in=obj.placeholders.all())
text = force_unicode(title)
plugins = list(placeholder_plugins)
for base_plugin in plugins:
instance, plugin_type = base_plugin.get_plugin_instance()
if instance is None:
# this is an empty plugin
continue
if hasattr(instance, 'search_fields'):
text += u' '.join(force_unicode(_strip_tags(getattr(instance, field, ''))) for field in instance.search_fields)
if getattr(instance, 'search_fulltext', False) or getattr(plugin_type, 'search_fulltext', False):
text += _strip_tags(instance.render_plugin(context=RequestContext(request))) + u' '
return text
site.register(Entry, BlogIndex)
I will consider putting a fork of cmsplugin-blog with a bulletproof version of this search index on github later. Feel free to use it wherever helpful.
Pyramid documentation shows us how to use i18n inside views (and templates as well). But how to does one use it outside of views and templates where we have no access to current request (for example, in forms and models)?
#Michael said to pass request to models and forms. But is it right? I mean if form fields defines before __init__() method calls, the same with models. They don't see any parameters from views...
In Pylons we could simply use get_lang() and set_lang() and define preferable language in parent controller and then use ugettext() and ungettext() in any place we want without calling it from request directly every possible time (in views).
How to do that in Pyramid? Note that the language must be set from user's settings (session, cookies, db, etc).
My solution is to create the form class when it's needed with localizer as parameter. For example
forms.py
class FormFactory(object):
def __init__(self, localizer):
self.localizer = localizer
_ = self.localizer
self.required_msg = _(u'This field is required.')
self.invalid_email_msg = _(u'Invalid email address.')
self.password_not_match_msg = _(u'Password must match')
def make_contact_form(self):
_ = self.localizer
class ContactForm(Form):
email = TextField(_(u'Email address'), [
validators.Required(self.required_msg),
validators.Email(self.invalid_email_msg)
])
content = TextAreaField(_(u'Content'), [
validators.Required(self.required_msg)
])
return ContactForm
When you need to use the form
#view_config(route_name='front_pages.contact_us',
renderer='myweb:templates/front_pages/contact_us.genshi')
def contact_us(request):
"""Display contact us form or send mail
"""
_ = get_localizer(request)
factory = FormFactory(_)
ContactForm = factory.make_contact_form()
form = ContactForm(request.params)
return dict(form=form)
As you can see, we get the localizer in the view, and pass it to the FormFactory, then create a form with that factory. By doing that, all messages in the form was replaced with current locale language.
Likewise, you can do the same with model.
Have you found pyramid.18n.get_localizer yet?
http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/narr/i18n.html#using-a-localizer
Actually I had this very same problem. What I ended up doing was to see how the default locale negotiator works - it looks for a LOCALE property on the given request object. So just use a dummy to create the localizer. You may cache this value too, if you want
def my_get_localizer(locale=None):
request = Request({})
request._LOCALE_ = locale
return get_localizer(request)
Alternatively, join the irc channel #pyramid # freenode and pester the guys enough there to split the functionality of get_localizer in 2 separate documented functions (get_localizer and get_localizer_for_locale_name) for us to enjoy ;)
Also, notice that Pyramid TranslationStrings are lazy, so you can translate them as late as you want, e.g.
class MyModel(Base):
description = TranslationString("My model number ${number}")
...
def view(request):
m = MyModel()
localizer = get_localizer(request)
description = localizer.translate(m.description, mapping={'number': 1})
Sidenote: pylons' i18n was the worst can of worms I had opened in ages. The set_lang, get_lang hack was really awful, and pain in the ass as we needed to send emails to users in their native languages and then tried to restore the language back... also, it was IMPOSSIBLE to translate anything outside of a request in a pylons program, as a translator or the registry did not exist then.
You can make a localizer, and then translate a template accordingly.
When making the localizer, you can pass the lang you want (whether you have it from db or else). Hope it can help.
For the sake of clarity, I will set it as 'fr' below
from pyramid.i18n import make_localizer, TranslationStringFactory
from mako.template import Template
from mako.lookup import TemplateLookup
import os
absolute_path = os.path.dirname(os.path.realpath(__file__))
tsf = TranslationStringFactory('your_domain')
mako_lookup = TemplateLookup(directories=['/'])
template = Template(filename=template_path, lookup=mako_lookup)
localizer = make_localizer("fr", [absolute_path + '/../locale/'])
def auto_translate(*args, **kwargs):
return localizer.translate(tsf(*args, **kwargs))
# Pass _ pointer (translate function) to the context
_ = auto_translate
context.update({
"_": _
})
html = template.render(**context)
EDIT
You can also put this logic into a small function
def get_translator(lang):
"""
Useful when need to translate outside of queries (no pointer to request)
:param lang:
:return:
"""
localizer = make_localizer(lang, [absolute_path + '/../locale/'])
def auto_translate(*args, **kwargs):
return localizer.translate(tsf(*args, **kwargs))
_ = auto_translate
return _
So basically, I've got a rather large Django project going. It's a private web portal that allows users to manage various phone-related tasks.
Several pages of the portal provide a listing of Model objects to users, and list all of their attributes in a HTML table (so that users can visually look through a list of these items).
The problem I'm having is: I cannot find a Django-ish or pythonic way to handle the sorting of these Model objects by field name. As an example of what I'm talking about, here is one of my views which lists all Partyline Model objects:
def list_partylines(request):
"""
List all `Partyline`s that we own.
"""
# Figure out which sort term to use.
sort_field = request.REQUEST.get('sortby', 'did').strip()
if sort_field.startswith('-'):
search = sort_field[1:]
sort_toggle = ''
else:
search = sort_field
sort_toggle = '-'
# Check to see if the sort term is valid.
if not (search in Partyline._meta.get_all_field_names()):
sort_field = 'did'
if is_user_type(request.user, ['admin']):
partylines = Partyline.objects.all().order_by(sort_field)
else:
partylines = get_my_partylines(request.user, sort_field)
variables = RequestContext(request, {
'partylines': partylines,
'sort_toggle': sort_toggle
})
return render_to_response('portal/partylines/list.html', variables)
The sorting code basically allows users to specify a /url/?sortby=model_field_name parameter which will then return a sorted listing of objects whenever users click on the HTML table name displayed on the page.
Since I have various views in various apps which all show a listing of Model objects, and require sorting, I'm wondering if there is a generic way to do this sorting so that I don't have to?
I'm sorry if this question is a bit unclear, I'm struggling to find the right way to phrase this question.
Thanks.
The way that I'd look at doing this is through a custom QuerySet. In your model, you can define the class QuerySet and add your sorting there. In order to maintain all the logic in the model object, I'd also move the contents of get_my_partylines into the QuerySet, too.
## This class is used to replicate QuerySet methods into a manager.
## This way: Partyline.objects.for_user(foo) works the same as
## Partyline.objects.filter(date=today).for_user(foo)
class CustomQuerySetManager(models.Manager):
def get_query_set(self):
return self.model.QuerySet(self.model)
def __getattr__(self, attr, *args):
try:
return getattr(self.__class__, attr, *args)
except AttributeError:
return getattr(self.get_query_set(), attr, *args)
class Partyline(models.Model):
## Define fields, blah blah.
objects = CustomQuerySetManager()
class QuerySet(QuerySet):
def sort_for_request(self, request):
sort_field = request.REQUEST.get('sortby', 'did').strip()
reverse_order = False
if sort_field.startswith('-'):
search = sort_field[1:]
else:
search = sort_field
reverse_order = True
# Check to see if the sort term is valid.
if not (search in Partyline._meta.get_all_field_names()):
sort_field = 'did'
partylines = self.all().order_by(sort_field)
if reverse_order:
partylines.reverse()
return partylines
def for_user(self, user):
if is_user_type(request.user, ['admin']):
return self.all()
else:
## Code from get_my_partylines goes here.
return self.all() ## Temporary.
views.py:
def list_partylines(request):
"""
List all `Partyline`s that we own.
"""
partylines = Partylines.objects.for_user(request.user).sort_for_request(request)
There's a great example of how this is done in a generic way in django.contrib.admin.views.main.ChangeList although that does much more than sorting you can browse it's code for some hints and ideas. You may also want to look at django.contrib.admin.options.ModelAdmin the changelist method in particular to get more context.