SpanishWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.es.SpanishWordTokenizer

All Implemented Interfaces:: Tokenizer

public class SpanishWordTokenizer extends WordTokenizer

Tokenizes a sentence into words. Punctuation and whitespace gets its own token.

Field Summary

Fields

Modifier and Type

Field

Description

private static final Pattern

DECIMAL_COMMA

private static final Pattern

DECIMAL_POINT

private static final Pattern

ORDINAL_POINT

private static final Pattern

PATTERN_1

private static final Pattern

PATTERN_2

private static final Pattern

PATTERN_3

private static final Pattern

SOFT_HYPHEN

private static final Pattern

tokenizerPattern

private static final String

wordCharacters

Fields inherited from class org.languagetool.tokenizers.WordTokenizer
REMOVED_EMOJI
Constructor Summary

Constructors

Constructor

Description

SpanishWordTokenizer()
Method Summary

Modifier and Type

Method

Description

List<String>

tokenize(String text)

private List<String>

wordsToAdd(String s)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer
getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details
- wordCharacters
  
  private static final String wordCharacters
  See Also:
  
  Constant Field Values
- tokenizerPattern
  
  private static final Pattern tokenizerPattern
- DECIMAL_POINT
  
  private static final Pattern DECIMAL_POINT
- DECIMAL_COMMA
  
  private static final Pattern DECIMAL_COMMA
- ORDINAL_POINT
  
  private static final Pattern ORDINAL_POINT
- PATTERN_1
  
  private static final Pattern PATTERN_1
- PATTERN_2
  
  private static final Pattern PATTERN_2
- PATTERN_3
  
  private static final Pattern PATTERN_3
- SOFT_HYPHEN
  
  private static final Pattern SOFT_HYPHEN
Constructor Details
- SpanishWordTokenizer
  
  public SpanishWordTokenizer()
Method Details
- tokenize
  
  public List<String> tokenize(String text)
  
  Specified by:
  
  tokenize in interface Tokenizer
  
  Overrides:
  
  tokenize in class WordTokenizer
- wordsToAdd
  
  private List<String> wordsToAdd(String s)