Package org.languagetool.tokenizers.be

Class BelarusianWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.be.BelarusianWordTokenizer

All Implemented Interfaces:

Tokenizer

public class BelarusianWordTokenizer
extends WordTokenizer

Specific to Belarusian: apostrophes (', ’, ʼ) are part of the word.

Field Summary

Fields

Modifier and Type	Field	Description
private final String	tokenizingCharacters

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
BelarusianWordTokenizer()

Method Summary

Modifier and Type	Method	Description
String	getTokenizingCharacters()
List<String>	tokenize(String text)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

tokenizingCharacters

private final String tokenizingCharacters

Constructor Details

BelarusianWordTokenizer

public BelarusianWordTokenizer()

Method Details

getTokenizingCharacters

public String getTokenizingCharacters()

Overrides:

getTokenizingCharacters in class WordTokenizer

Returns:

The string containing the characters used by the tokenizer to tokenize words.

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer