Package org.languagetool.tagging.disambiguation

Class MultiWordChunker

java.lang.Object

org.languagetool.tagging.disambiguation.AbstractDisambiguator

org.languagetool.tagging.disambiguation.MultiWordChunker

All Implemented Interfaces:

Disambiguator

public class MultiWordChunker
extends AbstractDisambiguator

Multiword tagger-chunker.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	MultiWordChunker.Settings

Field Summary

Fields

Modifier and Type	Field	Description
private boolean	addIgnoreSpelling
private static final Map<MultiWordChunker.Settings,MultiWordChunker>	chunkerCache
private static final String	DEFAULT_SEPARATOR
private static final Pattern	GermanLineExpander
private boolean	initialized
private boolean	isRemovePreviousTags
private static final int	MAX_TOKENS_IN_MULTIWORD
private Map<String,AnalyzedToken>	mFullNoSpace
private Map<String,AnalyzedToken>	mFullSpace
private Map<String,Integer>	mStartNoSpace
private Map<String,Integer>	mStartSpace
private String	separator
private final MultiWordChunker.Settings	settings
static String	tagForNotAddingTags

Constructor Summary

Constructors

Modifier	Constructor	Description
private	MultiWordChunker(MultiWordChunker.Settings settings)

Method Summary

Modifier and Type	Method	Description
AnalyzedSentence	disambiguate(AnalyzedSentence input)	If possible, filters out the wrong POS tags.
final AnalyzedSentence	disambiguate(AnalyzedSentence input, JLanguageTool.CheckCancelledCallback checkCanceled)	Implements multiword POS tags, e.g., <ELLIPSIS> for ellipsis (...) start, and </ELLIPSIS> for ellipsis end.
private void	fillMaps(Map<String,Integer> mStartSpace, Map<String,Integer> mStartNoSpace, Map<String,AnalyzedToken> mFullSpace, Map<String,AnalyzedToken> mFullNoSpace)
static MultiWordChunker	getInstance(String filename)
static MultiWordChunker	getInstance(String filename, boolean allowFirstCapitalized, boolean allowAllUppercase, boolean allowTitlecase)
static MultiWordChunker	getInstance(String filename, boolean allowFirstCapitalized, boolean allowAllUppercase, boolean allowTitlecase, String defaultTag)
private AnalyzedToken	getMultiWordAnalyzedToken(AnalyzedTokenReadings[] aTokens, Integer i)
private String	getNextPosTag(String postag)
List<String>	getTokenLettercaseVariants(String originalToken, Map<String,AnalyzedToken> tokenMap)
private boolean	isLowPriorityTag(String tag)
private void	lazyInit()
private List<String>	loadWords(InputStream stream)
private AnalyzedTokenReadings	prepareNewReading(AnalyzedToken at, String token, AnalyzedTokenReadings atrs, boolean isLast)
private AnalyzedTokenReadings[]	removePreviousTags(AnalyzedTokenReadings[] aTokens)
private AnalyzedTokenReadings	setAndAnnotate(AnalyzedTokenReadings oldReading, AnalyzedToken newReading)
void	setIgnoreSpelling(boolean ignoreSpelling)
void	setRemovePreviousTags(boolean removePreviousTags)

Methods inherited from class org.languagetool.tagging.disambiguation.AbstractDisambiguator

preDisambiguate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

chunkerCache

private static final Map<MultiWordChunker.Settings,MultiWordChunker> chunkerCache

settings

@NotNull
private final MultiWordChunker.Settings settings

initialized

private volatile boolean initialized

mStartSpace

private Map<String,Integer> mStartSpace

mStartNoSpace

private Map<String,Integer> mStartNoSpace

mFullSpace

private Map<String,AnalyzedToken> mFullSpace

mFullNoSpace

private Map<String,AnalyzedToken> mFullNoSpace

MAX_TOKENS_IN_MULTIWORD

private static final int MAX_TOKENS_IN_MULTIWORD

See Also:

• Constant Field Values

DEFAULT_SEPARATOR

private static final String DEFAULT_SEPARATOR

See Also:

• Constant Field Values

separator

private String separator

addIgnoreSpelling

private boolean addIgnoreSpelling

isRemovePreviousTags

private boolean isRemovePreviousTags

tagForNotAddingTags

public static String tagForNotAddingTags

GermanLineExpander

private static final Pattern GermanLineExpander

Constructor Details

MultiWordChunker

private MultiWordChunker(@NotNull
 MultiWordChunker.Settings settings)

Method Details

lazyInit

private void lazyInit()

fillMaps

private void fillMaps(Map<String,Integer> mStartSpace,
 Map<String,Integer> mStartNoSpace,
 Map<String,AnalyzedToken> mFullSpace,
 Map<String,AnalyzedToken> mFullNoSpace)

getTokenLettercaseVariants

public List<String> getTokenLettercaseVariants(String originalToken,
 Map<String,AnalyzedToken> tokenMap)

disambiguate

public AnalyzedSentence disambiguate(AnalyzedSentence input)
                              throws IOException

Description copied from interface: Disambiguator

If possible, filters out the wrong POS tags.

Parameters:

input - The sentence with already tagged words. The words are expected to have multiple tags.

Returns:

Analyzed sentence, where each word has only one (possibly the most correct) tag.

Throws:

IOException

disambiguate

public final AnalyzedSentence disambiguate(AnalyzedSentence input,
 @Nullable
 JLanguageTool.CheckCancelledCallback checkCanceled)
                                    throws IOException

Implements multiword POS tags, e.g., <ELLIPSIS> for ellipsis (...) start, and </ELLIPSIS> for ellipsis end.

Parameters:

input - The tokens to be chunked.

Returns:

AnalyzedSentence with additional markers.

Throws:

IOException

prepareNewReading

private AnalyzedTokenReadings prepareNewReading(AnalyzedToken at,
 String token,
 AnalyzedTokenReadings atrs,
 boolean isLast)

setAndAnnotate

private AnalyzedTokenReadings setAndAnnotate(AnalyzedTokenReadings oldReading,
 AnalyzedToken newReading)

loadWords

private List<String> loadWords(InputStream stream)

setIgnoreSpelling

public void setIgnoreSpelling(boolean ignoreSpelling)

setRemovePreviousTags

public void setRemovePreviousTags(boolean removePreviousTags)

removePreviousTags

private AnalyzedTokenReadings[] removePreviousTags(AnalyzedTokenReadings[] aTokens)

getMultiWordAnalyzedToken

private AnalyzedToken getMultiWordAnalyzedToken(AnalyzedTokenReadings[] aTokens,
 Integer i)

getNextPosTag

private String getNextPosTag(String postag)

isLowPriorityTag

private boolean isLowPriorityTag(String tag)

getInstance

@NotNull
public static MultiWordChunker getInstance(@NotNull
 String filename)

Parameters:

filename - file text with multiwords and tags

getInstance

@NotNull
public static MultiWordChunker getInstance(@NotNull
 String filename,
 boolean allowFirstCapitalized,
 boolean allowAllUppercase,
 boolean allowTitlecase)

getInstance

@NotNull
public static MultiWordChunker getInstance(@NotNull
 String filename,
 boolean allowFirstCapitalized,
 boolean allowAllUppercase,
 boolean allowTitlecase,
 @Nullable
 String defaultTag)