Package org.languagetool.chunking

Class EnglishChunker

java.lang.Object

org.languagetool.chunking.EnglishChunker

All Implemented Interfaces:

Chunker

public class EnglishChunker
extends Object
implements Chunker

OpenNLP-based chunker. Also uses the OpenNLP tokenizer and POS tagger and maps the result to our own tokens (we have our own tokenizer), as far as trivially possible.

Since:

2.3

Field Summary

Fields

Modifier and Type	Field	Description
private static final String	CHUNKER_MODEL
private static opennlp.tools.chunker.ChunkerModel	chunkerModel
private final EnglishChunkFilter	chunkFilter
private static final String	POS_TAGGER_MODEL
private static opennlp.tools.postag.POSModel	posModel
private static final String	TOKENIZER_MODEL
private static opennlp.tools.tokenize.TokenizerModel	tokenModel	This needs to be static to save memory: as Language.LANGUAGES is static, any language that is once created there will never be released.

Constructor Summary

Constructors

Constructor	Description
EnglishChunker()

Method Summary

Modifier and Type	Method	Description
void	addChunkTags(List<AnalyzedTokenReadings> tokenReadings)
private void	assignChunksToReadings(List<ChunkTaggedToken> chunkTaggedTokens)
private String[]	chunk(String[] tokens, String[] posTags)
private List<String>	cleanZeroWidthWhitespaces(String[] tokens)
private AnalyzedTokenReadings	getAnalyzedTokenReadingsFor(int startPos, int endPos, List<AnalyzedTokenReadings> tokenReadings)
private List<ChunkTaggedToken>	getChunkTagsForReadings(List<AnalyzedTokenReadings> tokenReadings)
private String	getSentence(List<AnalyzedTokenReadings> sentenceTokens)
private List<ChunkTaggedToken>	getTokensWithTokenReadings(List<AnalyzedTokenReadings> tokenReadings, String[] tokens, String[] chunkTags)
private String[]	posTag(String[] tokens)
(package private) String[]	tokenize(String sentence)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

TOKENIZER_MODEL

private static final String TOKENIZER_MODEL

See Also:

• Constant Field Values

POS_TAGGER_MODEL

private static final String POS_TAGGER_MODEL

See Also:

• Constant Field Values

CHUNKER_MODEL

private static final String CHUNKER_MODEL

See Also:

• Constant Field Values

tokenModel

private static volatile opennlp.tools.tokenize.TokenizerModel tokenModel

This needs to be static to save memory: as Language.LANGUAGES is static, any language that is once created there will never be released. As English has several variants, we'd have as many posModels etc. as we have variants -> huge waste of memory:

posModel

private static volatile opennlp.tools.postag.POSModel posModel

chunkerModel

private static volatile opennlp.tools.chunker.ChunkerModel chunkerModel

chunkFilter

private final EnglishChunkFilter chunkFilter

Constructor Details

EnglishChunker

public EnglishChunker()

Method Details

addChunkTags

public void addChunkTags(List<AnalyzedTokenReadings> tokenReadings)

Specified by:

addChunkTags in interface Chunker

Parameters:

tokenReadings - the analyzed tokens of one sentence

getChunkTagsForReadings

private List<ChunkTaggedToken> getChunkTagsForReadings(List<AnalyzedTokenReadings> tokenReadings)

cleanZeroWidthWhitespaces

@NotNull
private List<String> cleanZeroWidthWhitespaces(String[] tokens)

tokenize

String[] tokenize(String sentence)

posTag

private String[] posTag(String[] tokens)

chunk

private String[] chunk(String[] tokens,
 String[] posTags)

getTokensWithTokenReadings

private List<ChunkTaggedToken> getTokensWithTokenReadings(List<AnalyzedTokenReadings> tokenReadings,
 String[] tokens,
 String[] chunkTags)

assignChunksToReadings

private void assignChunksToReadings(List<ChunkTaggedToken> chunkTaggedTokens)

getSentence

private String getSentence(List<AnalyzedTokenReadings> sentenceTokens)

getAnalyzedTokenReadingsFor

@Nullable
private AnalyzedTokenReadings getAnalyzedTokenReadingsFor(int startPos,
 int endPos,
 List<AnalyzedTokenReadings> tokenReadings)