Uses of Interface
org.languagetool.tokenizers.Tokenizer

Packages that use Tokenizer

Package	Description
org.languagetool
org.languagetool.noop
org.languagetool.rules.ngrams
org.languagetool.tokenizers

Uses of Tokenizer in org.languagetool

Methods in org.languagetool that return Tokenizer

Modifier and Type	Method	Description
Tokenizer	Language.getWordTokenizer()	Get this language's word tokenizer implementation.

Uses of Tokenizer in org.languagetool.noop

Methods in org.languagetool.noop that return Tokenizer

Modifier and Type	Method	Description
Tokenizer	NoopLanguage.getWordTokenizer()

Uses of Tokenizer in org.languagetool.rules.ngrams

Methods in org.languagetool.rules.ngrams that return Tokenizer

Modifier and Type	Method	Description
protected Tokenizer	NgramProbabilityRule.getGoogleStyleWordTokenizer()

Uses of Tokenizer in org.languagetool.tokenizers

Subinterfaces of Tokenizer in org.languagetool.tokenizers

Modifier and Type	Interface	Description
interface	CompoundWordTokenizer	Interface for components that take compound words and split them into their parts.
interface	SentenceTokenizer	Tokenizes text into sentences.

Classes in org.languagetool.tokenizers that implement Tokenizer

Modifier and Type	Class	Description
class	SimpleSentenceTokenizer	A very simple sentence tokenizer that splits on [.!?…] followed by whitespace or an uppercase letter.
class	SRXSentenceTokenizer	Class to tokenize sentences using rules from an SRX file.
class	WordTokenizer	Tokenizes a sentence into words.