Package org.languagetool.tokenizers.fr

Class FrenchWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.fr.FrenchWordTokenizer

All Implemented Interfaces:

Tokenizer

public class FrenchWordTokenizer
extends WordTokenizer

Tokenizes a sentence into words. Punctuation and whitespace get its own token. Special treatment for hyphens and apostrophes in French.

Field Summary

Fields

Modifier and Type	Field	Description
private static final Pattern	DECIMAL_COMMA
private static final Pattern	DECIMAL_POINT
private static final List<String>	doNotSplit
private final String	frTokenizingChars
private static final Pattern	HYPHENS
(package private) static final int	maxPatterns
private static final Pattern	NEARBY_HYPHENS
(package private) static final Pattern[]	patterns
private static final Pattern	SPACE_DIGITS
private static final Pattern	SPACE_DIGITS0
private static final Pattern	SPACE_DIGITS2
private static final Pattern	tokenizerPattern
private static final Pattern	TYPEWRITER_APOSTROPHE
private static final Pattern	TYPOGRAPHIC_APOSTROPHE
private static final String	wordCharacters

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor	Description
FrenchWordTokenizer()

Method Summary

Modifier and Type	Method	Description
List<String>	tokenize(String text)
private List<String>	wordsToAdd(String s)

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

wordCharacters

private static final String wordCharacters

See Also:

• Constant Field Values

tokenizerPattern

private static final Pattern tokenizerPattern

TYPEWRITER_APOSTROPHE

private static final Pattern TYPEWRITER_APOSTROPHE

TYPOGRAPHIC_APOSTROPHE

private static final Pattern TYPOGRAPHIC_APOSTROPHE

NEARBY_HYPHENS

private static final Pattern NEARBY_HYPHENS

HYPHENS

private static final Pattern HYPHENS

DECIMAL_POINT

private static final Pattern DECIMAL_POINT

DECIMAL_COMMA

private static final Pattern DECIMAL_COMMA

SPACE_DIGITS0

private static final Pattern SPACE_DIGITS0

SPACE_DIGITS

private static final Pattern SPACE_DIGITS

SPACE_DIGITS2

private static final Pattern SPACE_DIGITS2

doNotSplit

private static final List<String> doNotSplit

frTokenizingChars

private final String frTokenizingChars

maxPatterns

static final int maxPatterns

See Also:

• Constant Field Values

patterns

static final Pattern[] patterns

Constructor Details

FrenchWordTokenizer

public FrenchWordTokenizer()

Method Details

tokenize

public List<String> tokenize(String text)

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - Text to tokenize

Returns:

List of tokens. Note: a special string xxFR_APOSxx is used to replace apostrophes, and xxFR_HYPHENxx to replace hyphens.

wordsToAdd

private List<String> wordsToAdd(String s)