org.apache.nutch.scoring.similarity.util

## Class LuceneTokenizer

- java.lang.Object
- - org.apache.nutch.scoring.similarity.util.LuceneTokenizer

-     public class LuceneTokenizer
      extends Object

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class and Description
    --- | ---
    `static class` | `LuceneTokenizer.TokenizerType`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, boolean useStopFilter, LuceneAnalyzerUtil.StemFilterType stemFilterType)` Creates a tokenizer based on param values
    `LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, List<String> stopWords, boolean addToDefault, LuceneAnalyzerUtil.StemFilterType stemFilterType)` Creates a tokenizer based on param values
    `LuceneTokenizer(String content, LuceneTokenizer.TokenizerType tokenizer, LuceneAnalyzerUtil.StemFilterType stemFilterType, int mingram, int maxgram)` Creates a tokenizer for the ngram model based on param values

  - ### Method Summary

    All Methods Instance Methods Concrete Methods
    Modifier and Type | Method and Description
    --- | ---
    `TokenStream` | `getTokenStream()` Returns the tokenStream created by the Tokenizer

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### LuceneTokenizer

          public LuceneTokenizer(String content,
                                 LuceneTokenizer.TokenizerType tokenizer,
                                 boolean useStopFilter,
                                 LuceneAnalyzerUtil.StemFilterType stemFilterType)

      Creates a tokenizer based on param values
      Parameters:
        `content` - - The text to tokenize
        `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
        `useStopFilter` - - if set to true the token stream will be filtered using default Lucene stopset
        `stemFilterType` - - Type of stemming to perform

    - #### LuceneTokenizer

          public LuceneTokenizer(String content,
                                 LuceneTokenizer.TokenizerType tokenizer,
                                 List<String> stopWords,
                                 boolean addToDefault,
                                 LuceneAnalyzerUtil.StemFilterType stemFilterType)

      Creates a tokenizer based on param values
      Parameters:
        `content` - - The text to tokenize
        `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
        `stopWords` - - Provide a set of user defined stop words
        `addToDefault` - - If set to true, the stopSet words will be added to the Lucene default stop set. If false, then only the user provided words will be used as the stop set
        `stemFilterType` -

    - #### LuceneTokenizer

          public LuceneTokenizer(String content,
                                 LuceneTokenizer.TokenizerType tokenizer,
                                 LuceneAnalyzerUtil.StemFilterType stemFilterType,
                                 int mingram,
                                 int maxgram)

      Creates a tokenizer for the ngram model based on param values
      Parameters:
        `content` - - The text to tokenize
        `tokenizer` - - the type of tokenizer to use CLASSIC or DEFAULT
        `stemFilterType` - - Type of stemming to perform
        `mingram` - - Value of mingram for tokenizing
        `maxgram` - - Value of maxgram for tokenizing

  - ### Method Detail

    - #### getTokenStream

          public TokenStream getTokenStream()

      Returns the tokenStream created by the Tokenizer
      Returns:

Copyright © 2020 The Apache Software Foundation