Package org.languagetool.tokenizers.ro

Class RomanianWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.ro.RomanianWordTokenizer

All Implemented Interfaces:

Tokenizer

public class RomanianWordTokenizer
extends WordTokenizer

Tokenizes a sentence into words. Punctuation and whitespace gets its own token. Like EnglishWordTokenizer except for some characters: eg: "-'

Since:

20.02.2009 19:53:50

Field Summary

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
RomanianWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RomanianWordTokenizer

public RomanianWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer