Package org.languagetool.tokenizers.pt

Class PortugueseWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.pt.PortugueseWordTokenizer

All Implemented Interfaces:

Tokenizer

public class PortugueseWordTokenizer
extends WordTokenizer

Tokenizes a sentence into words. Punctuation and whitespace gets its own token.

Since:

3.6

Field Summary

Fields

Modifier and Type	Field	Description
private static final Pattern	COLON_NUMBERS_PATTERN
private static final String	COLON_NUMBERS_REPL
private static final Pattern	DATE_PATTERN
private static final String	DATE_PATTERN_REPL
private static final Pattern	DECIMAL_COMMA_PATTERN
private static final String	DECIMAL_COMMA_REPL
private static final char	DECIMAL_COMMA_SUBST
private static final Pattern	DECIMAL_SPACE_PATTERN
private static final Pattern	DOTTED_NUMBERS_PATTERN
private static final String	DOTTED_NUMBERS_REPL
private static final Pattern	DOTTED_ORDINALS_PATTERN
private static final String	DOTTED_ORDINALS_REPL
private static final Pattern	HYPHEN_PATTERN
private static final String	HYPHEN_REPL
private static final Pattern	HYPHEN_SUBST
private static final String	HYPHEN_SUBST_TEXT
private static final Pattern	NEARBY_HYPHENS_PATTERN
private static final String	NEARBY_HYPHENS_REPL
private static final char	NON_BREAKING_COLON_SUBST
private static final char	NON_BREAKING_DOT_SUBST
private static final char	NON_BREAKING_SPACE_SUBST
private final PortugueseTagger	tagger
private final String	wordChars
private final String	wordCharsLeftEdge
private final String	wordCharsRightEdge
private final Pattern	wordPattern

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
PortugueseWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)
private List<String>	wordsToAdd(String s)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

tagger

private final PortugueseTagger tagger

DECIMAL_COMMA_SUBST

private static final char DECIMAL_COMMA_SUBST

See Also:

• Constant Field Values

NON_BREAKING_SPACE_SUBST

private static final char NON_BREAKING_SPACE_SUBST

See Also:

• Constant Field Values

NON_BREAKING_DOT_SUBST

private static final char NON_BREAKING_DOT_SUBST

See Also:

• Constant Field Values

NON_BREAKING_COLON_SUBST

private static final char NON_BREAKING_COLON_SUBST

See Also:

• Constant Field Values

HYPHEN_SUBST_TEXT

private static final String HYPHEN_SUBST_TEXT

See Also:

• Constant Field Values

HYPHEN_SUBST

private static final Pattern HYPHEN_SUBST

DECIMAL_COMMA_PATTERN

private static final Pattern DECIMAL_COMMA_PATTERN

DECIMAL_COMMA_REPL

private static final String DECIMAL_COMMA_REPL

See Also:

• Constant Field Values

DECIMAL_SPACE_PATTERN

private static final Pattern DECIMAL_SPACE_PATTERN

DOTTED_NUMBERS_PATTERN

private static final Pattern DOTTED_NUMBERS_PATTERN

DOTTED_NUMBERS_REPL

private static final String DOTTED_NUMBERS_REPL

See Also:

• Constant Field Values

COLON_NUMBERS_PATTERN

private static final Pattern COLON_NUMBERS_PATTERN

COLON_NUMBERS_REPL

private static final String COLON_NUMBERS_REPL

See Also:

• Constant Field Values

DATE_PATTERN

private static final Pattern DATE_PATTERN

DATE_PATTERN_REPL

private static final String DATE_PATTERN_REPL

See Also:

• Constant Field Values

DOTTED_ORDINALS_PATTERN

private static final Pattern DOTTED_ORDINALS_PATTERN

DOTTED_ORDINALS_REPL

private static final String DOTTED_ORDINALS_REPL

See Also:

• Constant Field Values

HYPHEN_PATTERN

private static final Pattern HYPHEN_PATTERN

HYPHEN_REPL

private static final String HYPHEN_REPL

NEARBY_HYPHENS_PATTERN

private static final Pattern NEARBY_HYPHENS_PATTERN

NEARBY_HYPHENS_REPL

private static final String NEARBY_HYPHENS_REPL

wordChars

private final String wordChars

wordCharsLeftEdge

private final String wordCharsLeftEdge

See Also:

• Constant Field Values

wordCharsRightEdge

private final String wordCharsRightEdge

See Also:

• Constant Field Values

wordPattern

private final Pattern wordPattern

Constructor Details

PortugueseWordTokenizer

public PortugueseWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

wordsToAdd

private List<String> wordsToAdd(String s)