Package org.languagetool.tokenizers.eo

Class EsperantoWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.eo.EsperantoWordTokenizer

All Implemented Interfaces:

Tokenizer

public class EsperantoWordTokenizer
extends WordTokenizer

Field Summary

Fields

Modifier and Type	Field	Description
private static final Pattern	PATTERN_1
private static final Pattern	PATTERN_2

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
EsperantoWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)	Tokenizes just like WordTokenizer with the exception that words such as "dank'" contain an apostrophe within it.

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

PATTERN_1

private static final Pattern PATTERN_1

PATTERN_2

private static final Pattern PATTERN_2

Constructor Details

EsperantoWordTokenizer

public EsperantoWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Tokenizes just like WordTokenizer with the exception that words such as "dank'" contain an apostrophe within it.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - - Text to tokenize

Returns:

List of tokens. Note: a special string

EO@APOS

is used to replace apostrophe during tokenizing.