We Keep Coding

How to improve query accuracy of Easticsearch from Python?

How can you improve the accuracy search results from Elasticsearch using the Python wrapper? My basic example returns results, but the results are very inaccurate.
I'm running Elasticsearch 5.2 on Ubuntu 16, and I start by creating my index and adding a few documents like:
es = Elasticsearch()
# Document A
es.index(
index='my-test-index',
doc_type='text',
body=dict(
search_key='some specific keywords',
weight=1.0,
data='blah1',
),
)
# Document B
es.index(
index='my-test-index',
doc_type='text',
body=dict(
search_key='some other specific keywords',
weight=1.0,
data='blah2',
),
)
# Document C
es.index(
index='my-test-index',
doc_type='text',
body=dict(
search_key='some other very long text that is very different yet mentions the word specific and keywords',
weight=1.0,
data='blah3',
),
)
I then query it with:
es = Elasticsearch()
es.indices.create(index='my-test-index', ignore=400)
query = 'some specific keywords'
results = es.search(
index='my-test-index',
body={
'query':{
"function_score": {
"query": {
"match": {
"search_key": query
}
},
"functions": [{
"script_score": {
"script": "doc['weight'].value"
}
}],
"score_mode": "multiply"
}
},
}
)
And although it returns all results, it returns them in the order of documents B, C, A, whereas I would expect them in the order A, B, C, because although all the documents contain all my keywords, only the first one is an exact match. I would expect C to be last because, even though it contains all my keywords, it also contains a lot of fluff I'm not explicitly searching for.
This problem compounds when I index more entries. The search returns everything that has even a single keyword from my query, and seemingly weights them all identically, causing the search results get less and less accurate the larger my index grows.
This is making Elasticsearch almost useless. Is there anyway I can fix it? Is there a problem with my search() call?

In your query, you can use a match_phrase query instead of a match query so that the order and proximity of the search terms get into the mix. Additionally, you can add a small slop in order to allow the terms to be further apart or in a different order. But documents with terms in the same order and closer apart will be ranked higher than documents with terms out of order and/or further apart. Try it out
"query": {
"match_phrase": {
"search_key": query,
"slop": 10
}
},
Note: slop is a number that indicates how many "swaps" of the search terms you need to perform in order to land on the term configuration present in the document.

Sorry for not reading your question more carefully and for the loaded answer below. I don't want to a stick in the mud but I think it will be clearer if you understand a bit more how Elasticsearch itself works.
Because you index your document without specifying any index and mapping configuration, Elasticsearch will use several defaults that it provides out of the box. The indexing process will first tokenize field values in your document using the standard tokenizer and analyze them using the standard analyzer before storing them in the index. Both the standard tokenizer and analyzer work by splitting your string based on word boundary. So at the end of index time, what you have in your index for the terms in the search_key field are ["some", "specific", "keywords"], not "some specific keywords".
During search time, the match query controls relevance using a similarity algorithm called term frequency/inverse document frequency, or TF/IDF. This algorithm is very popular in text search in general and there is a wikipedia section on it: https://en.wikipedia.org/wiki/Tf%E2%80%93idf. What's important to note here is that the more frequently your term appear in the index, the less important it is in terms of relevance. some, specific, and keywords appear in ALL 3 documents in your index, so as far as elasticsearch is concerned, they contribute very little to the document's relevance in your search result. Since A contains only these terms, it's like having a document containing only the, an, a in an English index. It won't show up as first result even if you search for the, an, a specifically. B ranks higher than C because B is shorter, which yields higher norm value. This norm value is explained in the relevance document. This is a bit of a speculation on my part, but I think it does work out this way if you explain the query using the explain API.
So, back to your need, how to favor exact match over everything else? There is, of course, the match_phrase query as Val pointed out. Another popular method to do it, which I personally prefer, is to index the raw value in a nested field called search_key.raw using the not_analyzed option when defining your mapping: https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html#_index_2 and simply boost this raw value when you search.

Find documents which contain a particular value - Mongo, Python

I'm trying to add a search option to my website but it doesn't work. I looked up solutions but they all refer to using an actual string, whereas in my case I'm using a variable, and I can't make those solutions work. Here is my code:
cursor = source.find({'title': search_term}).limit(25)
for document in cursor:
result_list.append(document)
Unfortunately this only gives back results which match the search_term variable's value exactly. I want it to give back any results where the title contains the search term - regardless what other strings it contains. How can I do it if I want to pass a variable to it, and not an actual string? Thanks.

You can use $regex to do contains searches.
cursor = collection.find({'field': {'$regex':'regular expression'}})
And to make it case insensitive:
cursor = collection.find({'field': {'$regex':'regular expression', '$options'‌​:'i'}})
Please try cursor = source.find({'title': {'$regex':search_term}}).limit(25)

$text
You can perform a text search using $text & $search. You first need to set a text index, then use it:
$ db.docs.createIndex( { title: "text" } )
$ db.docs.find( { $text: { $search: "search_term" } } )
$regex
You may also use $regex, as answered here: https://stackoverflow.com/a/10616781/641627
$ db.users.findOne({"username" : {$regex : ".*son.*"}});
Both solutions compared
Full Text Search vs. Regular Expressions
... The regular expression search takes longer for queries with just a
few results while the full text search gets faster and is clearly
superior in those cases.

OrientDB metadata attributes

I'm trying orientdb with python. I have created a couple of vertices and I noticed that if I prepend their properties name with #, when I search them on the estudio web app, those properties show up in the metadata section.
Thats interesting, so i went and tried to query vertices filtering by a metadata property id that i created.
When doing so:
select from V where #id = somevalue
I get a huge error msg. I couldnt find a way of querying those custom metadata properties.

Your problem is with python/pyorient (I am presuming you are using pyorient based upon your other questions).
Pyorient attempts to map the database fields to object attributes, but python docs say the valid characters for identifiers are the same as in Python 2.x: the uppercase and lowercase letters A through Z, the underscore _ and, except for the first character, the digits 0 through 9. (source). Thus, you cannot use '#' as a field name and expect pyorient to work.
I went and took a quick peek at pyorient source code, and it turns out the problem is even bigger than above...
pyorient/types.py
elif key[0:1] == '#':
# special case dict
# { '#my_class': { 'accommodation': 'hotel' } }
self.__o_class = key[1:]
for _key, _value in content[key].items():
self.__o_storage[_key] = _value
So pyorient presumes any record field that begins with an '#' character is followed by a class/cluster name, and then a dict of fields. I suppose you could post to the pyorient issue queue and suggest that the above elif section should check if content[key] is a dict, or a "simple value". If it were a dict, it should be handled like it currently is, otherwise it should be handled like a field, but have the # stripped from it.
Ultimately, not using the # symbol in your field names will be the easiest solution.

Compare text to template to detect anomalies (reverse template)

I'm looking for an algorithm or even an algorithm space that deals with the problem of validating that short text (email) matches known templates. Coding will probably be python or perl, but that's flexible.
Here's the problem:
Servers with access to production data need to be able to send out email that will reach the Internet:
Dear John Smith,
We received your last payment for $123.45 on 2/4/13. We'd like you to be aware of the following charges:
$12.34 Spuznitz, LLC on 4/1
$43.21 1-800-FLOWERS on 4/2
As always, you can view these transactions in our portal.
Thank you for your business!
Obviously some of the email contents will vary - the salutation ("John Smith"), "$123.45 on 2/4/13", and the lines with transactions printed out. Other parts ("We received your last payment") are very static. I want to be able to match the static portions of the text and quantify that the dynamic portions are within certain reasonable limits (I might know that the most transaction lines to be printed is 5, for example).
Because I am concerned about data exfiltration, I want to make sure email that doesn't match this template never goes out - I want to examine email and quarantine anything that doesn't look like what I expect. So I need to automate this template matching and block any email messages that are far enough away from matching.
So the question is, where do I look for a filtering mechanism? Bayesian filtering tries to verify a sufficient similarity between a specific message and a non-specific corpus, which is kind of the opposite problem. Things like Perl's Template module are a tight match - but for output, not for input or comparison. Simple 'diff' type comparisons won't handle the limited dynamic info very well.
How do I test to see if these outgoing email messages "quack like a duck"?

You could use grammars for a tight matching. It is possible to organize regexps in grammars for easier abstraction: http://www.effectiveperlprogramming.com/blog/1479
Or you could use a dedicated grammar engine Marpa.
If you want a more statistic approach, consider n-grams. First, tokenize text and replace variable chunks by meaningful placeholders, like CURRENCY and DATE. Then, build the n-grams. Now you can use Jaccard index to compare two texts.
Here is a Pure-Perl implementation which works on trigrams:
#!/usr/bin/env perl
use strict;
use utf8;
use warnings;
my $ngram1 = ngram(3, tokenize(<<'TEXT1'));
Dear John Smith,
We received your last payment for $123.45 on 2/4/13. We'd like you to be aware of the following charges:
$12.34 Spuznitz, LLC on 4/1
$43.21 1-800-FLOWERS on 4/2
As always, you can view these transactions in our portal.
Thank you for your business!
TEXT1
my $ngram2 = ngram(3, tokenize(<<'TEXT2'));
Dear Sally Bates,
We received your last payment for $456.78 on 6/9/12. We'd like you to be aware of the following charges:
$123,43 Gnomovision on 10/1
As always, you can view these transactions in our portal.
Thank you for your business!
TEXT2
my %intersection =
map { exists $ngram1->[2]{$_} ? ($_ => 1) : () }
keys %{$ngram2->[2]};
my %union =
map { $_ => 1 }
keys %{$ngram1->[2]}, keys %{$ngram2->[2]};
printf "Jaccard similarity coefficient: %0.3f\n", keys(%intersection) / keys(%union);
sub tokenize {
my #words = split m{\s+}x, lc shift;
for (#words) {
s{\d{1,2}/\d{1,2}(?:/\d{2,4})?}{ DATE }gx;
s{\d+(?:\,\d{3})*\.\d{1,2}}{ FLOAT }gx;
s{\d+}{ INTEGER }gx;
s{\$\s(?:FLOAT|INTEGER)\s}{ CURRENCY }gx;
s{^\W+|\W+$}{}gx;
}
return #words;
}
sub ngram {
my ($size, #words) = #_;
--$size;
my $ngram = [];
for (my $j = 0; $j <= $#words; $j++) {
my $k = $j + $size <= $#words ? $j + $size : $#words;
for (my $l = $j; $l <= $k; $l++) {
my #buf;
for my $w (#words[$j..$l]) {
push #buf, $w;
}
++$ngram->[$#buf]{join(' ', #buf)};
}
}
return $ngram;
}
You can use one text as a template and match it against your emails.
Check String::Trigram for an efficient implementation. Google Ngram Viewer is a nice resource to illustrate the n-gram matching.

If you want to match a pre-existing template with e.g. control flow elements like {% for x in y %} against a supposed output from it, you're going to have to parse the template language - which seems like a lot of work.
On the other hand, if you're prepared to write a second template for validation purposes - something like:
Dear {{customer}},
We received your last payment for {{currency}} on {{full-date}}\. We'd like you to be aware of the following charges:
( {{currency}} {{supplier}} on {{short-date}}
){,5}As always, you can view these transactions in our portal\.
... which is just a simple extension of regex syntax, it's pretty straightforward to hack something together that will validate against that:
import re
FIELDS = {
"customer": r"[\w\s\.-]{,50}",
"supplier": r"[\w\s\.,-]{,30}",
"currency": r"[$€£]\d+\.\d{2}",
"short-date": r"\d{,2}/\d{,2}",
"full-date": r"\d{,2}/\d{,2}/\d{2}",
}
def validate(example, template_file):
with open(template_file) as f:
template = f.read()
for tag, pattern in FIELDS.items():
template = template.replace("{{%s}}" % tag, pattern)
valid = re.compile(template + "$")
return (re.match(valid, example) is not None)
The example above isn't the greatest Python code of all time by any means, but it's enough to get the general idea across.

I would go for the "longest common subsequence". A standard implementation can be found here:
http://en.wikibooks.org/wiki/Algorithm_implementation/Strings/Longest_common_subsequence
If you need a better algorithm and/or lots of additional ideas for inexact matching of strings, THE standard reference is this book:
http://www.amazon.com/Algorithms-Strings-Trees-Sequences-Computational/dp/0521585198
Don't be fooled by the title. Compuatational biology is mostly about matching of large database of long strings (also known as DNA sequences).

How do I use StandardAnalyzer with TermQuery?

I'm trying to produce something similar to what QueryParser in lucene does, but without the parser, i.e. run a string through StandardAnalyzer, tokenize this and use TermQuery:s in a BooleanQuery to produce a query. My problem is that I only get Token:s from StandardAnalyzer, and not Term:s. I can convert a Token to a term by just extracting the string from it with Token.term(), but this is 2.4.x-only and it seems backwards, because I need to add the field a second time. What is the proper way of producing a TermQuery with StandardAnalyzer?
I'm using pylucene, but I guess the answer is the same for Java etc. Here is the code I've come up with:
from lucene import *
def term_match(self, phrase):
query = BooleanQuery()
sa = StandardAnalyzer()
for token in sa.tokenStream("contents", StringReader(phrase)):
term_query = TermQuery(Term("contents", token.term())
query.add(term_query), BooleanClause.Occur.SHOULD)

The established way to get the token text is with token.termText() - that API's been there forever.
And yes, you'll need to specify a field name to both the Analyzer and the Term; I think that's considered normal. 8-)

I've come across the same problem, and, using Lucene 2.9 API and Java, my code snippet looks like this:
final TokenStream tokenStream = new StandardAnalyzer(Version.LUCENE_29)
.tokenStream( fieldName , new StringReader( value ) );
final List< String > result = new ArrayList< String >();
try {
while ( tokenStream.incrementToken() ) {
final TermAttribute term = ( TermAttribute ) tokenStream.getAttribute( TermAttribute.class );
result.add( term.term() );
}