Package snap.parse

Class Tokenizer

java.lang.Object

snap.parse.Tokenizer

Direct Known Subclasses:

CodeTokenizer, XMLParser.XMLTokenizer

public class Tokenizer
extends Object

A class to extract tokens from a char sequence.

Field Summary

Fields

Modifier and Type	Field	Description
protected int	_charIndex
protected int	_lineIndex
protected int	_lineStart
static final String	MULTI_LINE_COMMENT
static final String	SINGLE_LINE_COMMENT

Constructor Summary

Constructors

Constructor	Description
Tokenizer()	Constructor.

Method Summary

Modifier and Type	Method	Description
void	addPattern(String aName, String aPattern)	Adds a pattern.
void	addPatternsForRule(ParseRule aRule)	Adds patterns to this tokenizer for given rule.
final char	charAt(int anIndex)	CharSequence method.
protected ParseToken	createToken(String aName, String aPattern, int aStart, int anEnd, ParseToken aSpecialToken)	Creates a new token.
final char	eatChar()	Returns the char at the current index plus offset.
final char	getChar()	Returns the current parse char.
final char	getChar(int anOffset)	Returns the char at the current index plus offset.
final int	getCharIndex()	Returns the current parse char location.
final String	getChars(int aValue)	Returns the next given number of chars as a string.
final int	getColumnIndex()	Returns the current column index in the current line.
CharSequence	getInput()	Returns the current tokenizer input.
CharSequence	getInput(int aStart, int anEnd)	Returns the input subsequence for the given range of characters in input.
final int	getLineIndex()	Returns the current line index.
final int	getLineStart()	Returns the current line start index.
ParseToken	getNextSpecialToken()	Processes and returns a special token if found.
ParseToken	getNextToken()	Returns the next token.
protected Regex[]	getRegexes()	Returns the array of regexes (creating it if missing).
Regex[]	getRegexesForStartChar(char aChar)	Returns list of Regex for a starting char.
final boolean	hasChar()	Returns whether another char is available.
final boolean	hasChars(int aVal)	Returns whether another given number of chars is available.
final int	length()	CharSequence method.
void	setCharIndex(int aValue)	Sets the current parse char location.
void	setInput(CharSequence anInput)	Sets the current tokenizer input.
final void	setLineIndex(int aValue)	Returns the current line index.
protected void	skipWhiteSpace()	Gobble input characters until next non-whitespace or input end.
protected ParseToken	tokenizerFailed(String nextChars)	Called when next chars don't conform to any known token pattern.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

_charIndex

protected int _charIndex

_lineIndex

protected int _lineIndex

_lineStart

protected int _lineStart

SINGLE_LINE_COMMENT

public static final String SINGLE_LINE_COMMENT

See Also:

• Constant Field Values

MULTI_LINE_COMMENT

public static final String MULTI_LINE_COMMENT

See Also:

• Constant Field Values

Constructor Details

Tokenizer

public Tokenizer()

Constructor.

Method Details

getInput

public CharSequence getInput()

Returns the current tokenizer input.

setInput

public void setInput(CharSequence anInput)

Sets the current tokenizer input.

getInput

public CharSequence getInput(int aStart,
 int anEnd)

Returns the input subsequence for the given range of characters in input.

length

public final int length()

CharSequence method.

charAt

public final char charAt(int anIndex)

CharSequence method.

hasChar

public final boolean hasChar()

Returns whether another char is available.

hasChars

public final boolean hasChars(int aVal)

Returns whether another given number of chars is available.

getChar

public final char getChar()

Returns the current parse char.

getChar

public final char getChar(int anOffset)

Returns the char at the current index plus offset.

eatChar

public final char eatChar()

Returns the char at the current index plus offset.

getChars

public final String getChars(int aValue)

Returns the next given number of chars as a string.

getCharIndex

public final int getCharIndex()

Returns the current parse char location.

setCharIndex

public void setCharIndex(int aValue)

Sets the current parse char location.

addPattern

public void addPattern(String aName,
 String aPattern)

Adds a pattern.

addPatternsForRule

public void addPatternsForRule(ParseRule aRule)

Adds patterns to this tokenizer for given rule.

getRegexes

protected Regex[] getRegexes()

Returns the array of regexes (creating it if missing).

getLineIndex

public final int getLineIndex()

Returns the current line index.

setLineIndex

public final void setLineIndex(int aValue)

Returns the current line index.

getLineStart

public final int getLineStart()

Returns the current line start index.

getColumnIndex

public final int getColumnIndex()

Returns the current column index in the current line.

getNextToken

public ParseToken getNextToken()

Returns the next token.

getRegexesForStartChar

public Regex[] getRegexesForStartChar(char aChar)

Returns list of Regex for a starting char.

createToken

protected ParseToken createToken(String aName,
 String aPattern,
 int aStart,
 int anEnd,
 ParseToken aSpecialToken)

Creates a new token.

getNextSpecialToken

public ParseToken getNextSpecialToken()

Processes and returns a special token if found. This version just skips whitespace.

skipWhiteSpace

protected void skipWhiteSpace()

Gobble input characters until next non-whitespace or input end.

tokenizerFailed

protected ParseToken tokenizerFailed(String nextChars)

Called when next chars don't conform to any known token pattern. Default implementation just throws ParseExcpetion.