org.lionsoul.jcseg.tokenizer

Class NLPSeg

java.lang.Object

org.lionsoul.jcseg.tokenizer.ASegment

org.lionsoul.jcseg.tokenizer.ComplexSeg

org.lionsoul.jcseg.tokenizer.NLPSeg

All Implemented Interfaces:

ISegment

public class NLPSeg
extends ComplexSeg

NLP segmentation implementation And this extends all the properties of the Complex one the rest of them are build for NLP only

Author:

chenxin

Field Summary

Fields inherited from class org.lionsoul.jcseg.tokenizer.ASegment

behindLatin, config, ctrlMask, dic, ialist, idx, isb, reader, wordPool

Fields inherited from interface org.lionsoul.jcseg.tokenizer.core.ISegment

CHECK_CE_MASk, CHECK_CF_MASK, CHECK_EC_MASK, START_SS_MASK

Constructor Summary

Constructors

Constructor and Description
NLPSeg(JcsegTaskConfig config, ADictionary dic)
NLPSeg(Reader input, JcsegTaskConfig config, ADictionary dic)

Method Summary

Modifier and Type	Method and Description
protected IWord	getNextCJKWord(int c, int pos) get the next CJK word from the current position of the input stream
protected IWord	getNextDatetimeWord(IWord word) get and return the next date-time word
protected IWord	getNextTimeMergedWord(IWord word) get and return the next time merged date-time word
IWord	next() Override the next method to add the date-time entity recognition And we also invoke the parent.next method to get the next token
protected IWord	nextLatinWord(int c, int pos) find the letter or digit word from the current position count until the char is whitespace or not letter_digit

Methods inherited from class org.lionsoul.jcseg.tokenizer.ComplexSeg

getBestCJKChunk

Methods inherited from class org.lionsoul.jcseg.tokenizer.ASegment

appendLatinSyn, appendWordFeatures, enSecondSeg, findCHName, findCHName, getConfig, getDict, getNextLatinWord, getNextMatch, getNextMixedWord, getNextPunctuationPairWord, getPairPunctuationText, getStreamPosition, nextCJKSentence, nextCNNumeric, nextLatinString, nextLetterNumber, nextOtherNumber, pushBack, pushBack, readNext, reset, setConfig, setDict

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NLPSeg

public NLPSeg(Reader input,
              JcsegTaskConfig config,
              ADictionary dic)
       throws IOException

Throws:

IOException

NLPSeg

public NLPSeg(JcsegTaskConfig config,
              ADictionary dic)
       throws IOException

Throws:

IOException

Method Detail

next

public IWord next()
           throws IOException

Override the next method to add the date-time entity recognition And we also invoke the parent.next method to get the next token

Specified by:

next in interface ISegment

Overrides:

next in class ASegment

Returns:

IWord

Throws:

IOException

See Also:

ASegment.next()

getNextTimeMergedWord

protected IWord getNextTimeMergedWord(IWord word)
                               throws IOException

get and return the next time merged date-time word

Parameters:

word -

Returns:

IWord

Throws:

IOException

getNextDatetimeWord

protected IWord getNextDatetimeWord(IWord word)
                             throws IOException

get and return the next date-time word

Parameters:

word -

Returns:

IWord

Throws:

IOException

getNextCJKWord

protected IWord getNextCJKWord(int c,
                               int pos)
                        throws IOException

Description copied from class: ASegment

get the next CJK word from the current position of the input stream

Overrides:

getNextCJKWord in class ASegment

Returns:

IWord could be null and that mean we reached a stop word

Throws:

IOException

See Also:

ASegment.getNextCJKWord(int, int)

nextLatinWord

protected IWord nextLatinWord(int c,
                              int pos)
                       throws IOException

find the letter or digit word from the current position count until the char is whitespace or not letter_digit

Overrides:

nextLatinWord in class ASegment

Parameters:

c -

pos -

Returns:

IWord

Throws:

IOException