Package org.languagetool.tokenizers.en

Class EnglishWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.en.EnglishWordTokenizer

All Implemented Interfaces:

Tokenizer

public class EnglishWordTokenizer
extends WordTokenizer

Since:

2.5

Field Summary

Fields

Modifier and Type	Field	Description
private static final Pattern	APOSTYPEW
private static final Pattern	APOSTYPOG
private static final Pattern	CURLY_QUOTE
private final String	enTokenizingChars
private static final List<Pattern>	patternList
private static final Pattern	SINGLE_QUOTE
private static final Pattern	SOFT_HYPHEN
private static final Pattern	tokenizerPattern
private static final String	wordCharacters

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
EnglishWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)	Tokenizes text.
private List<String>	wordsToAdd(String s)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

wordCharacters

private static final String wordCharacters

See Also:

• Constant Field Values

tokenizerPattern

private static final Pattern tokenizerPattern

SINGLE_QUOTE

private static final Pattern SINGLE_QUOTE

CURLY_QUOTE

private static final Pattern CURLY_QUOTE

APOSTYPEW

private static final Pattern APOSTYPEW

APOSTYPOG

private static final Pattern APOSTYPOG

SOFT_HYPHEN

private static final Pattern SOFT_HYPHEN

patternList

private static final List<Pattern> patternList

enTokenizingChars

private final String enTokenizingChars

Constructor Details

EnglishWordTokenizer

public EnglishWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Tokenizes text. The English tokenizer differs from the standard one in two respects:

1. it does not treat the hyphen as part of the word if the hyphen is at the end of the word;
2. it includes n-dash as a tokenizing character, as it is used without a whitespace in English.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - String of words to tokenize.

wordsToAdd

private List<String> wordsToAdd(String s)