org.lionsoul.jcseg.tokenizer

Class ASegment

java.lang.Object

org.lionsoul.jcseg.tokenizer.ASegment

All Implemented Interfaces:

ISegment

Direct Known Subclasses:

ComplexSeg, SearchSeg, SimpleSeg

public abstract class ASegment
extends Object
implements ISegment

abstract segmentation super class: 1. implemented the ISegment interface 2. implemented all the common functions that simple, complex, most segmentation algorithm will all shared.

Author:

chenxin

Field Summary

Fields

Modifier and Type	Field and Description
protected String	behindLatin global behind Latin word after the CJK word added at 2016/11/22 for better mixed word implementation
protected JcsegTaskConfig	config
protected int	ctrlMask segmentation runtime function control mask
protected ADictionary	dic the dictionary and task configuration instance
protected IntArrayList	ialist
protected int	idx the index value of the current input stream mainly for track the start position of the token
protected IStringBuffer	isb
protected IPushbackReader	reader
protected LinkedList<IWord>	wordPool CJK word cache pool, Reusable string buffer and the array list for basic integer

Fields inherited from interface org.lionsoul.jcseg.tokenizer.core.ISegment

CHECK_CE_MASk, CHECK_CF_MASK, CHECK_EC_MASK, START_SS_MASK

Constructor Summary

Constructors

Constructor and Description
ASegment(JcsegTaskConfig config, ADictionary dic)
ASegment(Reader input, JcsegTaskConfig config, ADictionary dic) initialize the segment

Method Summary

Modifier and Type	Method and Description
protected void	appendLatinSyn(IWord w) Check and append the synonyms words of specified word included the CJK and basic Latin words All the synonyms words share the same position part of speech, word type with the primitive word
protected void	appendWordFeatures(IWord word) check and append the pinyin and the synonyms words of the specified word
protected IWord	enSecondSeg(IWord w, boolean retfw) Do the secondary split for the specified complex Latin word This will split a complex English, Arabic, punctuation compose word to multiple simple parts Like 'qq2013' will split to 'qq' and '2013'
protected String	findCHName(char[] chars, int index, IChunk chunk) find an Chinese name from the current position of the input chars
boolean	findCHName(IWord w, IChunk chunk) Deprecated.
protected abstract IChunk	getBestCJKChunk(char[] chars, int index) an abstract method to gain a CJK word from the current position.
JcsegTaskConfig	getConfig() get the current task configuration instance.
ADictionary	getDict() get the current dictionary instance.
protected IWord	getNextCJKWord(int c, int pos) get the next CJK word from the current position of the input stream
protected IWord	getNextLatinWord(int c, int pos) get the next Latin word from the current position of the input stream
protected IWord[]	getNextMatch(char[] chars, int index) match the next CJK word in the dictionary
protected IWord	getNextMixedWord(char[] chars, int cjkidx) get the next mixed word, CJK-English or CJK-English-CJK or whatever
protected IWord	getNextPunctuationPairWord(int c, int pos) get the next punctuation pair word from the current position of the input stream.
protected String	getPairPunctuationText(int c) find pair punctuation of the given punctuation char the purpose is to get the text between them
int	getStreamPosition() get the current length of the stream
IWord	next() segment a word from a char array from a specified position.
protected char[]	nextCJKSentence(int c) load a CJK char list from the stream start from the current position till the char is not a CJK char
protected String	nextCNNumeric(char[] chars, int index) find the Chinese number from the current position count until the char in the specified position is not a other number or whitespace
protected String	nextLatinString(int c) the simple version of the next basic Latin fetch logic Just return the next Latin string with the keep punctuation after it
protected IWord	nextLatinWord(int c, int pos) find the letter or digit word from the current position count until the char is whitespace or not letter_digit
protected String	nextLetterNumber(int c) find the next other letter from the current position find the letter number from the current position count until the char in the specified position is not a letter number or whitespace
protected String	nextOtherNumber(int c) find the other number from the current position count until the char in the specified position is not a other number or whitespace
protected void	pushBack(int data) push back the data to the stream.
protected void	pushBack(String str) push back a string to the stream
protected int	readNext() read the next char from the current position
void	reset(Reader input) input stream and reader reset.
void	setConfig(JcsegTaskConfig config) set the current task configuration instance.
void	setDict(ADictionary dic) set the current dictionary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

idx

protected int idx

the index value of the current input stream mainly for track the start position of the token

reader

protected IPushbackReader reader

wordPool

protected LinkedList<IWord> wordPool

CJK word cache pool, Reusable string buffer and the array list for basic integer

isb

protected IStringBuffer isb

ialist

protected IntArrayList ialist

behindLatin

protected String behindLatin

global behind Latin word after the CJK word added at 2016/11/22 for better mixed word implementation

ctrlMask

protected int ctrlMask

segmentation runtime function control mask

dic

protected ADictionary dic

the dictionary and task configuration instance

config

protected JcsegTaskConfig config

Constructor Detail

ASegment

public ASegment(Reader input,
                JcsegTaskConfig config,
                ADictionary dic)
         throws IOException

initialize the segment

Parameters:

input -

config - Jcseg task configuration instance

dic - Jcseg dictionary instance

Throws:

IOException

ASegment

public ASegment(JcsegTaskConfig config,
                ADictionary dic)
         throws IOException

Throws:

IOException

See Also:

ASegment(Reader, JcsegTaskConfig, ADictionary)

Method Detail

reset

public void reset(Reader input)
           throws IOException

input stream and reader reset.

Specified by:

reset in interface ISegment

Parameters:

input -

Throws:

IOException

readNext

protected int readNext()
                throws IOException

read the next char from the current position

Throws:

IOException

pushBack

protected void pushBack(int data)
                 throws IOException

push back the data to the stream.

Parameters:

data -

Throws:

IOException

pushBack

protected void pushBack(String str)

push back a string to the stream

Parameters:

data -

getStreamPosition

public int getStreamPosition()

Description copied from interface: ISegment

get the current length of the stream

Specified by:

getStreamPosition in interface ISegment

Returns:

int

setDict

public void setDict(ADictionary dic)

set the current dictionary

Parameters:

dic -

getDict

public ADictionary getDict()

get the current dictionary instance.

Returns:

ADictionary

setConfig

public void setConfig(JcsegTaskConfig config)

set the current task configuration instance.

Parameters:

config -

getConfig

public JcsegTaskConfig getConfig()

get the current task configuration instance.

Returns:

JcsegTaskConfig

next

public IWord next()
           throws IOException

Description copied from interface: ISegment

segment a word from a char array from a specified position.

Specified by:

next in interface ISegment

Returns:

IWord

Throws:

IOException

See Also:

ISegment.next()

getNextCJKWord

protected IWord getNextCJKWord(int c,
                               int pos)
                        throws IOException

get the next CJK word from the current position of the input stream

Parameters:

c -

pos -

Returns:

IWord could be null and that mean we reached a stop word

Throws:

IOException

getNextLatinWord

protected IWord getNextLatinWord(int c,
                                 int pos)
                          throws IOException

get the next Latin word from the current position of the input stream

Parameters:

c -

pos -

Returns:

IWord could be null and that mean we reached a stop word

Throws:

IOException

getNextMixedWord

protected IWord getNextMixedWord(char[] chars,
                                 int cjkidx)
                          throws IOException

get the next mixed word, CJK-English or CJK-English-CJK or whatever

Parameters:

chars -

cjkidx -

Returns:

IWord or null for nothing found

Throws:

IOException

getNextPunctuationPairWord

protected IWord getNextPunctuationPairWord(int c,
                                           int pos)
                                    throws IOException

get the next punctuation pair word from the current position of the input stream.

Parameters:

c -

pos -

Returns:

IWord could be null and that mean we reached a stop word

Throws:

IOException

appendWordFeatures

protected void appendWordFeatures(IWord word)

check and append the pinyin and the synonyms words of the specified word

Parameters:

word -

appendLatinSyn

protected void appendLatinSyn(IWord w)

Check and append the synonyms words of specified word included the CJK and basic Latin words All the synonyms words share the same position part of speech, word type with the primitive word

Parameters:

w -

enSecondSeg

protected IWord enSecondSeg(IWord w,
                            boolean retfw)

Do the secondary split for the specified complex Latin word This will split a complex English, Arabic, punctuation compose word to multiple simple parts Like 'qq2013' will split to 'qq' and '2013'

And all the sub words share the same type and part of speech with the primitive word You should check the config.EN_SECOND_SEG before invoke this method

Parameters:

w -

retfw - whether to return the fword.

Returns:

IWord the first sub token for the secondary segment.

getNextMatch

protected IWord[] getNextMatch(char[] chars,
                               int index)

match the next CJK word in the dictionary

Parameters:

chars -

index -

Returns:

IWord[]

findCHName

protected String findCHName(char[] chars,
                            int index,
                            IChunk chunk)

find an Chinese name from the current position of the input chars

Parameters:

chars -

index -

chunk -

Returns:

IWord

findCHName

@Deprecated
public boolean findCHName(IWord w,
                                      IChunk chunk)

Deprecated.

find the Chinese double name: when the last name and the first char of the name make up a word

Parameters:

chunk - the best chunk.

Returns:

boolean

nextCJKSentence

protected char[] nextCJKSentence(int c)
                          throws IOException

load a CJK char list from the stream start from the current position till the char is not a CJK char

Parameters:

c -

Returns:

char[]

Throws:

IOException

nextLatinWord

protected IWord nextLatinWord(int c,
                              int pos)
                       throws IOException

find the letter or digit word from the current position count until the char is whitespace or not letter_digit

Parameters:

c -

pos -

Returns:

IWord

Throws:

IOException

nextLatinString

protected String nextLatinString(int c)
                          throws IOException

the simple version of the next basic Latin fetch logic Just return the next Latin string with the keep punctuation after it

Parameters:

int - c

Returns:

String

Throws:

IOException

nextLetterNumber

protected String nextLetterNumber(int c)
                           throws IOException

find the next other letter from the current position find the letter number from the current position count until the char in the specified position is not a letter number or whitespace

Parameters:

c -

Returns:

String

Throws:

IOException

nextOtherNumber

protected String nextOtherNumber(int c)
                          throws IOException

find the other number from the current position count until the char in the specified position is not a other number or whitespace

Parameters:

c -

Returns:

String

Throws:

IOException

nextCNNumeric

protected String nextCNNumeric(char[] chars,
                               int index)
                        throws IOException

find the Chinese number from the current position count until the char in the specified position is not a other number or whitespace

Parameters:

chars - char array of CJK items

index -

Returns:

String[]

Throws:

IOException

getPairPunctuationText

protected String getPairPunctuationText(int c)
                                 throws IOException

find pair punctuation of the given punctuation char the purpose is to get the text between them

Parameters:

c -

Throws:

IOException

getBestCJKChunk

protected abstract IChunk getBestCJKChunk(char[] chars,
                                          int index)
                                   throws IOException

an abstract method to gain a CJK word from the current position. simpleSeg and ComplexSeg is different to deal this, so make it a abstract method here

Parameters:

chars -

index -

Returns:

IChunk

Throws:

IOException