Package org.languagetool.language.identifier

Class DefaultLanguageIdentifier

java.lang.Object

org.languagetool.language.identifier.LanguageIdentifier

org.languagetool.language.identifier.DefaultLanguageIdentifier

public class DefaultLanguageIdentifier
extends LanguageIdentifier

Identify the language of a text. Note that some languages might never be detected because they are close to another language. Language variants like en-US or en-GB are not detected, the result will be en for those. By default, only the first 1000 characters of a text are considered. Email signatures that use \n-- \n as a delimiter are ignored.

Since:

2.9

Nested Class Summary

Nested classes/interfaces inherited from class org.languagetool.language.identifier.LanguageIdentifier

LanguageIdentifier.ParsedLanguageLists

Field Summary

Fields

Modifier and Type	Field	Description
private static final int	CONSIDER_ONLY_PREFERRED_THRESHOLD
private static final List<String>	externalLangCodes
private static final float	FASTTEXT_CONFIDENCE_THRESHOLD
private FastTextDetector	fastTextDetector
private final AtomicInteger	fasttextInitCounter
private static final List<String>	ignoreLangCodes
private final com.optimaize.langdetect.LanguageDetector	languageDetector
private static final org.slf4j.Logger	logger
private static final double	MINIMAL_CONFIDENCE
private NGramDetector	ngram
private static final int	SHORT_ALGO_THRESHOLD
private final com.optimaize.langdetect.text.TextObjectFactory	textObjectFactory

Fields inherited from class org.languagetool.language.identifier.LanguageIdentifier

COMMON_WORDS_LANG_IDENTIFIER, maxLength, NON_LATIN_CHARS_LANGUAGES, REMOVE_EMAIL_SIGNATURE_FILTER, REMOVE_MENTION_FILTER, REMOVE_NON_BREAKING_SPACES_FILTER, REMOVE_URL_FILTER, SCORE_THRESHOLD, UNICODE_BASED_LANG_IDENTIFIER

Constructor Summary

Constructors

Constructor	Description
DefaultLanguageIdentifier()
DefaultLanguageIdentifier(int maxLength)

Method Summary

Modifier and Type	Method	Description
Language	detectLanguage(String cleanText)
DetectedLanguage	detectLanguage(String cleanText, List<String> noopLangsTmp, List<String> preferredLangsTmp)
DetectedLanguage	detectLanguage(String cleanText, List<String> noopLangsTmp, List<String> preferredLangsTmp, boolean limitOnPreferredLangs)
private Map.Entry<String,Double>	detectLanguageCode(String text, List<String> preferredLangs, boolean limitOnPreferredLangs)
(package private) void	enableFasttext(File fasttextBinary, File fasttextModel)
(package private) void	enableNgrams(File ngramDir)
List<DetectedLanguage>	getDetectedLanguageScores(String cleanText, List<String> noopLangsTmp, List<String> preferredLangsTmp, boolean limitOnPreferredLangs, int count)
AtomicInteger	getFasttextInitCounter()	For test only
private static List<String>	getLanguageCodes()
boolean	isFastTextEnabled()
private List<com.optimaize.langdetect.profiles.LanguageProfile>	loadProfiles(List<String> langCodes)
private void	reinitFasttextAfterFailure(Exception e)
void	setFastTextDetector(FastTextDetector fastTextDetector)	For test only

Methods inherited from class org.languagetool.language.identifier.LanguageIdentifier

cleanAndShortenText, getHighestScoringResult, getOrderedScores, prepareDetectLanguage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

logger

private static final org.slf4j.Logger logger

MINIMAL_CONFIDENCE

private static final double MINIMAL_CONFIDENCE

See Also:

• Constant Field Values

SHORT_ALGO_THRESHOLD

private static final int SHORT_ALGO_THRESHOLD

See Also:

• Constant Field Values

CONSIDER_ONLY_PREFERRED_THRESHOLD

private static final int CONSIDER_ONLY_PREFERRED_THRESHOLD

See Also:

• Constant Field Values

ignoreLangCodes

private static final List<String> ignoreLangCodes

externalLangCodes

private static final List<String> externalLangCodes

FASTTEXT_CONFIDENCE_THRESHOLD

private static final float FASTTEXT_CONFIDENCE_THRESHOLD

See Also:

• Constant Field Values

languageDetector

private final com.optimaize.langdetect.LanguageDetector languageDetector

textObjectFactory

private final com.optimaize.langdetect.text.TextObjectFactory textObjectFactory

fasttextInitCounter

private final AtomicInteger fasttextInitCounter

fastTextDetector

private FastTextDetector fastTextDetector

ngram

private NGramDetector ngram

Constructor Details

DefaultLanguageIdentifier

DefaultLanguageIdentifier()

DefaultLanguageIdentifier

DefaultLanguageIdentifier(int maxLength)

Parameters:

maxLength - the maximum number of characters that will be considered - can help with performance. Don't use values below 100, as this would decrease accuracy.

Throws:

IllegalArgumentException - if maxLength is less than 10

Since:

4.2

Method Details

enableFasttext

void enableFasttext(File fasttextBinary,
 File fasttextModel)

setFastTextDetector

@TestOnly
public void setFastTextDetector(FastTextDetector fastTextDetector)

For test only

getFasttextInitCounter

@TestOnly
public AtomicInteger getFasttextInitCounter()

For test only

Returns:

a counter how often fasttext was already recreated after a failure

isFastTextEnabled

public boolean isFastTextEnabled()

Since:

5.2

enableNgrams

void enableNgrams(File ngramDir)

getLanguageCodes

private static List<String> getLanguageCodes()

loadProfiles

private List<com.optimaize.langdetect.profiles.LanguageProfile> loadProfiles(List<String> langCodes)
                                                                      throws IOException

Throws:

IOException

detectLanguage

@Nullable
public Language detectLanguage(String cleanText)

Specified by:

detectLanguage in class LanguageIdentifier

Parameters:

cleanText - a cleanText as returned by LanguageIdentifier.cleanAndShortenText(String)

Returns:

language or null if language could not be identified

detectLanguage

public DetectedLanguage detectLanguage(String cleanText,
 List<String> noopLangsTmp,
 List<String> preferredLangsTmp)

Specified by:

detectLanguage in class LanguageIdentifier

Parameters:

cleanText - a cleanText as returned by LanguageIdentifier.cleanAndShortenText(String)

noopLangsTmp - list of codes that are detected but will lead to the NoopLanguage that has no rules

Returns:

language or null if language could not be identified

Since:

4.4 (new parameter noopLangs, changed return type to DetectedLanguage)

detectLanguage

@Nullable
public DetectedLanguage detectLanguage(String cleanText,
 List<String> noopLangsTmp,
 List<String> preferredLangsTmp,
 boolean limitOnPreferredLangs)

Specified by:

detectLanguage in class LanguageIdentifier

getDetectedLanguageScores

@NotNull
public List<DetectedLanguage> getDetectedLanguageScores(String cleanText,
 List<String> noopLangsTmp,
 List<String> preferredLangsTmp,
 boolean limitOnPreferredLangs,
 int count)

Specified by:

getDetectedLanguageScores in class LanguageIdentifier

reinitFasttextAfterFailure

private void reinitFasttextAfterFailure(Exception e)

detectLanguageCode

@Nullable
private Map.Entry<String,Double> detectLanguageCode(String text,
 List<String> preferredLangs,
 boolean limitOnPreferredLangs)