Package org.languagetool.tokenizers

package org.languagetool.tokenizers
  • Class
    Description
    CompoundWordTokenizer
    Interface for components that take compound words and split them into their parts.
    SentenceTokenizer
    Tokenizes text into sentences.
    SimpleSentenceTokenizer
    A very simple sentence tokenizer that splits on [.!?…] followed by whitespace or an uppercase letter.
    SRXSentenceTokenizer
    Class to tokenize sentences using rules from an SRX file.
    Tokenizer
    Interface for classes that tokenize text into smaller units.
    WordTokenizer
    Tokenizes a sentence into words.