Package org.languagetool.tokenizers.crh

Class CrimeanTatarWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.crh.CrimeanTatarWordTokenizer

All Implemented Interfaces:

Tokenizer

public class CrimeanTatarWordTokenizer
extends WordTokenizer

Field Summary

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
CrimeanTatarWordTokenizer()

Method Summary

Modifier and Type	Method	Description
String	getTokenizingCharacters()
List<String>	tokenize(String text)	Tokenizes text.

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CrimeanTatarWordTokenizer

public CrimeanTatarWordTokenizer()

Method Details

getTokenizingCharacters

public String getTokenizingCharacters()

Overrides:

getTokenizingCharacters in class WordTokenizer

Returns:

The string containing the characters used by the tokenizer to tokenize words.

tokenize

public List<String> tokenize(String text)

Tokenizes text. The CrimeanTatar tokenizer differs from the standard one in two respects:

1. it does not treat the hyphen as part of the word if the hyphen is at the end of the word;
2. it includes n-dash as a tokenizing character, as it is used without a whitespace in CrimeanTatar.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - String of words to tokenize.