java.lang.Object
org.apache.lucene.queries.mlt.MoreLikeThis
Generate "more like this" similarity queries. Based on this mail:
Lucene does let you access the document frequency of terms, with IndexReader.docFreq().
Term frequencies can be computed by re-tokenizing the text, which, for a single document,
is usually fast enough. But looking up the docFreq() of every term in the document is
probably too slow.
You can use some heuristics to prune the set of terms, to avoid calling docFreq() too much,
or at all. Since you're trying to maximize a tf*idf score, you're probably most interested
in terms with a high tf. Choosing a tf threshold even as low as two or three will radically
reduce the number of terms under consideration. Another heuristic is that terms with a
high idf (i.e., a low df) tend to be longer. So you could threshold the terms by the
number of characters, not selecting anything less than, e.g., six or seven characters.
With these sorts of heuristics you can usually find small set of, e.g., ten or fewer terms
that do a pretty good job of characterizing a document.
It all depends on what you're trying to do. If you're trying to eek out that last percent
of precision and recall regardless of computational difficulty so that you can win a TREC
competition, then the techniques I mention above are useless. But if you're trying to
provide a "more like this" button on a search results page that does a decent job and has
good performance, such techniques might be useful.
An efficient, effective "more-like-this" query generator would be a great contribution, if
anyone's interested. I'd imagine that it would take a Reader or a String (the document's
text), analyzer Analyzer, and return a set of representative terms using heuristics like those
above. The frequency and length thresholds could be parameters, etc.
Doug
Initial Usage
This class has lots of options to try to make it efficient and flexible. The simplest possible
usage is as follows. The bold fragment is specific to this class.
IndexReader ir = ... IndexSearcher is = ... MoreLikeThis mlt = new MoreLikeThis(ir); Reader target = ... // orig source of doc you want to find similarities to Query query = mlt.like( target); TopDocs topDocs = is.search(query, 10); // now the usual iteration thru 'topDocs' - the only thing to watch for is to make sure // you ignore the doc if it matches your 'target' document, as it should be similar to itself
Thus you:
- do your normal, Lucene setup for searching,
- create a MoreLikeThis,
- get the text of the doc you want to find similarities to
- then call one of the like() calls to generate a similarity query
- call the searcher to find the similar docs
More Advanced Usage
You may want to use setFieldNames(...)
so you can examine multiple
fields (e.g. body and title) for similarity.
Depending on the size of your index and the size and makeup of your documents you may want to call the other set methods to control how the similarity queries are generated:
setMinTermFreq(...)
setMinDocFreq(...)
setMaxDocFreq(...)
setMaxDocFreqPct(...)
setMinWordLen(...)
setMaxWordLen(...)
setMaxQueryTerms(...)
setMaxNumTokensParsed(...)
setStopWord(...)
Changes: Mark Harwood 29/02/04 Some bugfixing, some refactoring, some optimisation. - bugfix: retrieveTerms(int docNum) was not working for indexes without a termvector -added missing code - bugfix: No significant terms being created for fields with a termvector - because was only counting one occurrence per term/field pair in calculations(ie not including frequency info from TermVector) - refactor: moved common code into isNoiseWord() - optimise: when no termvector support available - used maxNumTermsParsed to limit amount of tokenization
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
PriorityQueue that orders words by score.private static class
Use for frequencies and to avoid renewing Integers.private static class
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate Analyzer
Analyzer that will be used to parse the doc.private boolean
Should we apply a boost to the Query based on the scores?private float
Boost factor to use when boosting the termsstatic final boolean
Boost terms in query based on score.static final String[]
Default field names.static final int
Ignore words which occur in more than this many docs.static final int
Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support.static final int
Return a Query with no more than this many terms.static final int
Ignore words greater than this length or if 0 then this has no effect.static final int
Ignore words which do not occur in at least this many docs.static final int
Ignore terms with less than this frequency in the source doc.static final int
Ignore words less than this length or if 0 then this has no effect.static final Set
<?> Default set of stopwords.private String[]
Field name we'll analyze.private final IndexReader
IndexReader to useprivate int
Ignore words which occur in more than this many docs.private int
The maximum number of tokens to parse in each example doc field that is not stored with TermVector supportprivate int
Don't return a query longer than this.private int
Ignore words if greater than this len.private int
Ignore words which do not occur in at least this many docs.private int
Ignore words less frequent that this.private int
Ignore words if less than this len.private TFIDFSimilarity
For idf() calculations.private Set
<?> Current set of stop words. -
Constructor Summary
ConstructorsConstructorDescriptionConstructor requiring an IndexReader.MoreLikeThis
(IndexReader ir, TFIDFSimilarity sim) -
Method Summary
Modifier and TypeMethodDescriptionprivate void
addTermFrequencies
(Reader r, Map<String, Map<String, MoreLikeThis.Int>> perFieldTermFrequencies, String fieldName) Adds term frequencies found by tokenizing text from reader into the Map wordsprivate void
addTermFrequencies
(Map<String, Map<String, MoreLikeThis.Int>> field2termFreqMap, Terms vector, String fieldName) Adds terms and frequencies found in vector into the Map termFreqMapprivate Query
Create the More like query from a PriorityQueueprivate PriorityQueue
<MoreLikeThis.ScoreTerm> createQueue
(Map<String, Map<String, MoreLikeThis.Int>> perFieldTermFrequencies) Create a PriorityQueue from a word->tf map.Describe the parameters that control how the "more like this" query is formed.Returns an analyzer that will be used to parse source doc with.float
Returns the boost factor used when boosting termsString[]
Returns the field names that will be used when generating the 'More Like This' query.int
Returns the maximum frequency in which words may still appear.int
int
Returns the maximum number of query terms that will be included in any generated query.int
Returns the maximum word length above which words will be ignored.int
Returns the frequency at which words will be ignored which do not occur in at least this many docs.int
Returns the frequency below which terms will be ignored in the source doc.int
Returns the minimum word length below which words will be ignored.Set
<?> Get the current stop words being used.private int
getTermsCount
(Map<String, Map<String, MoreLikeThis.Int>> perFieldTermFrequencies) boolean
isBoost()
Returns whether to boost terms in query based on "score" or not.private boolean
isNoiseWord
(String term) determines if the passed term is likely to be of interest in "more like" comparisonslike
(int docNum) Return a query that will return docs like the passed lucene document ID.Return a query that will return docs like the passed Readers.like
(Map<String, Collection<Object>> filteredDocument) String[]
retrieveInterestingTerms
(int docNum) String[]
retrieveInterestingTerms
(Reader r, String fieldName) Convenience routine to make it easy to return the most interesting words in a document.private PriorityQueue
<MoreLikeThis.ScoreTerm> retrieveTerms
(int docNum) Find words for a more-like-this query former.private PriorityQueue
<MoreLikeThis.ScoreTerm> retrieveTerms
(Reader r, String fieldName) Find words for a more-like-this query former.private PriorityQueue
<MoreLikeThis.ScoreTerm> retrieveTerms
(Map<String, Collection<Object>> field2fieldValues) void
setAnalyzer
(Analyzer analyzer) Sets the analyzer to use.void
setBoost
(boolean boost) Sets whether to boost terms in query based on "score" or not.void
setBoostFactor
(float boostFactor) Sets the boost factor to use when boosting termsvoid
setFieldNames
(String[] fieldNames) Sets the field names that will be used when generating the 'More Like This' query.void
setMaxDocFreq
(int maxFreq) Set the maximum frequency in which words may still appear.void
setMaxDocFreqPct
(int maxPercentage) Set the maximum percentage in which words may still appear.void
setMaxNumTokensParsed
(int i) void
setMaxQueryTerms
(int maxQueryTerms) Sets the maximum number of query terms that will be included in any generated query.void
setMaxWordLen
(int maxWordLen) Sets the maximum word length above which words will be ignored.void
setMinDocFreq
(int minDocFreq) Sets the frequency at which words will be ignored which do not occur in at least this many docs.void
setMinTermFreq
(int minTermFreq) Sets the frequency below which terms will be ignored in the source doc.void
setMinWordLen
(int minWordLen) Sets the minimum word length below which words will be ignored.void
setSimilarity
(TFIDFSimilarity similarity) void
setStopWords
(Set<?> stopWords) Set the set of stopwords.
-
Field Details
-
DEFAULT_MAX_NUM_TOKENS_PARSED
public static final int DEFAULT_MAX_NUM_TOKENS_PARSEDDefault maximum number of tokens to parse in each example doc field that is not stored with TermVector support.- See Also:
-
DEFAULT_MIN_TERM_FREQ
public static final int DEFAULT_MIN_TERM_FREQIgnore terms with less than this frequency in the source doc.- See Also:
-
DEFAULT_MIN_DOC_FREQ
public static final int DEFAULT_MIN_DOC_FREQIgnore words which do not occur in at least this many docs.- See Also:
-
DEFAULT_MAX_DOC_FREQ
public static final int DEFAULT_MAX_DOC_FREQIgnore words which occur in more than this many docs.- See Also:
-
DEFAULT_BOOST
public static final boolean DEFAULT_BOOSTBoost terms in query based on score.- See Also:
-
DEFAULT_FIELD_NAMES
Default field names. Null is used to specify that the field names should be looked up at runtime from the provided reader. -
DEFAULT_MIN_WORD_LENGTH
public static final int DEFAULT_MIN_WORD_LENGTHIgnore words less than this length or if 0 then this has no effect.- See Also:
-
DEFAULT_MAX_WORD_LENGTH
public static final int DEFAULT_MAX_WORD_LENGTHIgnore words greater than this length or if 0 then this has no effect.- See Also:
-
DEFAULT_STOP_WORDS
Default set of stopwords. If null means to allow stop words.- See Also:
-
stopWords
Current set of stop words. -
DEFAULT_MAX_QUERY_TERMS
public static final int DEFAULT_MAX_QUERY_TERMSReturn a Query with no more than this many terms.- See Also:
-
analyzer
Analyzer that will be used to parse the doc. -
minTermFreq
private int minTermFreqIgnore words less frequent that this. -
minDocFreq
private int minDocFreqIgnore words which do not occur in at least this many docs. -
maxDocFreq
private int maxDocFreqIgnore words which occur in more than this many docs. -
boost
private boolean boostShould we apply a boost to the Query based on the scores? -
fieldNames
Field name we'll analyze. -
maxNumTokensParsed
private int maxNumTokensParsedThe maximum number of tokens to parse in each example doc field that is not stored with TermVector support -
minWordLen
private int minWordLenIgnore words if less than this len. -
maxWordLen
private int maxWordLenIgnore words if greater than this len. -
maxQueryTerms
private int maxQueryTermsDon't return a query longer than this. -
similarity
For idf() calculations. -
ir
IndexReader to use -
boostFactor
private float boostFactorBoost factor to use when boosting the terms
-
-
Constructor Details
-
MoreLikeThis
Constructor requiring an IndexReader.- Throws:
IOException
-
MoreLikeThis
- Throws:
IOException
-
-
Method Details
-
getBoostFactor
public float getBoostFactor()Returns the boost factor used when boosting terms- Returns:
- the boost factor used when boosting terms
- See Also:
-
setBoostFactor
public void setBoostFactor(float boostFactor) Sets the boost factor to use when boosting terms- See Also:
-
getSimilarity
-
setSimilarity
-
getAnalyzer
Returns an analyzer that will be used to parse source doc with. The default analyzer is not set.- Returns:
- the analyzer that will be used to parse source doc with.
-
setAnalyzer
Sets the analyzer to use. An analyzer is not required for generating a query with thelike(int)
method, all other 'like' methods require an analyzer.- Parameters:
analyzer
- the analyzer to use to tokenize text.
-
getMinTermFreq
public int getMinTermFreq()Returns the frequency below which terms will be ignored in the source doc. The default frequency is theDEFAULT_MIN_TERM_FREQ
.- Returns:
- the frequency below which terms will be ignored in the source doc.
-
setMinTermFreq
public void setMinTermFreq(int minTermFreq) Sets the frequency below which terms will be ignored in the source doc.- Parameters:
minTermFreq
- the frequency below which terms will be ignored in the source doc.
-
getMinDocFreq
public int getMinDocFreq()Returns the frequency at which words will be ignored which do not occur in at least this many docs. The default frequency isDEFAULT_MIN_DOC_FREQ
.- Returns:
- the frequency at which words will be ignored which do not occur in at least this many docs.
-
setMinDocFreq
public void setMinDocFreq(int minDocFreq) Sets the frequency at which words will be ignored which do not occur in at least this many docs.- Parameters:
minDocFreq
- the frequency at which words will be ignored which do not occur in at least this many docs.
-
getMaxDocFreq
public int getMaxDocFreq()Returns the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored. The default frequency isDEFAULT_MAX_DOC_FREQ
.- Returns:
- get the maximum frequency at which words are still allowed, words which occur in more docs than this are ignored.
-
setMaxDocFreq
public void setMaxDocFreq(int maxFreq) Set the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored.- Parameters:
maxFreq
- the maximum count of documents that a term may appear in to be still considered relevant
-
setMaxDocFreqPct
public void setMaxDocFreqPct(int maxPercentage) Set the maximum percentage in which words may still appear. Words that appear in more than this many percent of all docs will be ignored.This method calls
setMaxDocFreq(int)
internally (both conditions cannot be used at the same time).- Parameters:
maxPercentage
- the maximum percentage of documents (0-100) that a term may appear in to be still considered relevant.
-
isBoost
public boolean isBoost()Returns whether to boost terms in query based on "score" or not. The default isDEFAULT_BOOST
.- Returns:
- whether to boost terms in query based on "score" or not.
- See Also:
-
setBoost
public void setBoost(boolean boost) Sets whether to boost terms in query based on "score" or not.- Parameters:
boost
- true to boost terms in query based on "score", false otherwise.- See Also:
-
getFieldNames
Returns the field names that will be used when generating the 'More Like This' query. The default field names that will be used isDEFAULT_FIELD_NAMES
.- Returns:
- the field names that will be used when generating the 'More Like This' query.
-
setFieldNames
Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor.- Parameters:
fieldNames
- the field names that will be used when generating the 'More Like This' query.
-
getMinWordLen
public int getMinWordLen()Returns the minimum word length below which words will be ignored. Set this to 0 for no minimum word length. The default isDEFAULT_MIN_WORD_LENGTH
.- Returns:
- the minimum word length below which words will be ignored.
-
setMinWordLen
public void setMinWordLen(int minWordLen) Sets the minimum word length below which words will be ignored.- Parameters:
minWordLen
- the minimum word length below which words will be ignored.
-
getMaxWordLen
public int getMaxWordLen()Returns the maximum word length above which words will be ignored. Set this to 0 for no maximum word length. The default isDEFAULT_MAX_WORD_LENGTH
.- Returns:
- the maximum word length above which words will be ignored.
-
setMaxWordLen
public void setMaxWordLen(int maxWordLen) Sets the maximum word length above which words will be ignored.- Parameters:
maxWordLen
- the maximum word length above which words will be ignored.
-
setStopWords
Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting".- Parameters:
stopWords
- set of stopwords, if null it means to allow stop words- See Also:
-
getStopWords
Get the current stop words being used.- See Also:
-
getMaxQueryTerms
public int getMaxQueryTerms()Returns the maximum number of query terms that will be included in any generated query. The default isDEFAULT_MAX_QUERY_TERMS
.- Returns:
- the maximum number of query terms that will be included in any generated query.
-
setMaxQueryTerms
public void setMaxQueryTerms(int maxQueryTerms) Sets the maximum number of query terms that will be included in any generated query.- Parameters:
maxQueryTerms
- the maximum number of query terms that will be included in any generated query.
-
getMaxNumTokensParsed
public int getMaxNumTokensParsed()- Returns:
- The maximum number of tokens to parse in each example doc field that is not stored with TermVector support
- See Also:
-
setMaxNumTokensParsed
public void setMaxNumTokensParsed(int i) - Parameters:
i
- The maximum number of tokens to parse in each example doc field that is not stored with TermVector support
-
like
Return a query that will return docs like the passed lucene document ID.- Parameters:
docNum
- the documentID of the lucene doc to generate the 'More Like This" query for.- Returns:
- a query that will return docs like the passed lucene document ID.
- Throws:
IOException
-
like
- Parameters:
filteredDocument
- Document with field values extracted for selected fields.- Returns:
- More Like This query for the passed document.
- Throws:
IOException
-
like
Return a query that will return docs like the passed Readers. This was added in order to treat multi-value fields.- Returns:
- a query that will return docs like the passed Readers.
- Throws:
IOException
-
createQuery
Create the More like query from a PriorityQueue -
createQueue
private PriorityQueue<MoreLikeThis.ScoreTerm> createQueue(Map<String, Map<String, throws IOExceptionMoreLikeThis.Int>> perFieldTermFrequencies) Create a PriorityQueue from a word->tf map.- Parameters:
perFieldTermFrequencies
- a per field map of words keyed on the word(String) with Int objects as the values.- Throws:
IOException
-
getTermsCount
-
describeParams
Describe the parameters that control how the "more like this" query is formed. -
retrieveTerms
Find words for a more-like-this query former.- Parameters:
docNum
- the id of the lucene document from which to find terms- Throws:
IOException
-
retrieveTerms
private PriorityQueue<MoreLikeThis.ScoreTerm> retrieveTerms(Map<String, Collection<Object>> field2fieldValues) throws IOException- Throws:
IOException
-
addTermFrequencies
private void addTermFrequencies(Map<String, Map<String, throws IOExceptionMoreLikeThis.Int>> field2termFreqMap, Terms vector, String fieldName) Adds terms and frequencies found in vector into the Map termFreqMap- Parameters:
field2termFreqMap
- a Map of terms and their frequencies per fieldvector
- List of terms and their frequencies for a doc/field- Throws:
IOException
-
addTermFrequencies
private void addTermFrequencies(Reader r, Map<String, Map<String, throws IOExceptionMoreLikeThis.Int>> perFieldTermFrequencies, String fieldName) Adds term frequencies found by tokenizing text from reader into the Map words- Parameters:
r
- a source of text to be tokenizedperFieldTermFrequencies
- a Map of terms and their frequencies per fieldfieldName
- Used by analyzer for any special per-field analysis- Throws:
IOException
-
isNoiseWord
determines if the passed term is likely to be of interest in "more like" comparisons- Parameters:
term
- The word being considered- Returns:
- true if should be ignored, false if should be used in further analysis
-
retrieveTerms
private PriorityQueue<MoreLikeThis.ScoreTerm> retrieveTerms(Reader r, String fieldName) throws IOException Find words for a more-like-this query former. The result is a priority queue of arrays with one entry for every word in the document. Each array has 6 elements. The elements are:- The word (String)
- The top field that this word comes from (String)
- The score for this word (Float)
- The IDF value (Float)
- The frequency of this word in the index (Integer)
- The frequency of this word in the source document (Integer)
retrieveInterestingTerms()
.- Parameters:
r
- the reader that has the content of the documentfieldName
- field passed to the analyzer to use when analyzing the content- Returns:
- the most interesting words in the document ordered by score, with the highest scoring, or best entry, first
- Throws:
IOException
- See Also:
-
retrieveInterestingTerms
- Throws:
IOException
- See Also:
-
retrieveInterestingTerms
Convenience routine to make it easy to return the most interesting words in a document. More advanced users will callretrieveTerms()
directly.- Parameters:
r
- the source documentfieldName
- field passed to analyzer to use when analyzing the content- Returns:
- the most interesting words in the document
- Throws:
IOException
- See Also:
-