Package snap.parse

Class CodeTokenizer

java.lang.Object

snap.parse.Tokenizer

snap.parse.CodeTokenizer

public class CodeTokenizer
extends Tokenizer

This Tokenizer subclass supports Java/C single-line and multi-line comments.

Field Summary

Fields inherited from class snap.parse.Tokenizer

_charIndex, _tokenLine, MULTI_LINE_COMMENT, SINGLE_LINE_COMMENT

Constructor Summary

Constructors

Constructor	Description
CodeTokenizer()	Constructor.

Method Summary

Modifier and Type	Method	Description
ParseToken	getMultiLineCommentToken()	Process and return a multi-line comment if next up in input.
ParseToken	getMultiLineCommentTokenMore()	Returns a token from the current char to multi-line comment termination or input end.
ParseToken	getNextSpecialToken()	Processes and returns a special token if found.
protected ParseToken	getNextSpecialTokenImpl()	Processes and returns next special token.
ParseToken	getNextSpecialTokenOrToken()	Returns next special token or token.
protected ParseToken	getSingleLineCommentToken()	Processes and returns a single line comment token if next up in input.
boolean	isReadMultiLineComments()	Returns whether tokenizer reads standard Java multiple line comments.
boolean	isReadSingleLineComments()	Returns whether tokenizer reads standard Java single line comments.
void	setReadMultiLineComments(boolean aValue)	Sets whether tokenizer reads standard Java multiple line comments.
void	setReadSingleLineComments(boolean aValue)	Sets whether tokenizer reads standard Java single line comments.

Methods inherited from class snap.parse.Tokenizer

charAt, createTokenForProps, eatChar, getChar, getChar, getCharIndex, getInput, getNextToken, getRegexes, getTokenDoc, getTokenLine, hasChar, hasChars, length, setCharIndex, setInput, setRegexes, setRegexesForPatternRulesInRule, skipWhiteSpace, tokenizerFailed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CodeTokenizer

public CodeTokenizer()

Constructor.

Method Details

isReadSingleLineComments

public boolean isReadSingleLineComments()

Returns whether tokenizer reads standard Java single line comments.

setReadSingleLineComments

public void setReadSingleLineComments(boolean aValue)

Sets whether tokenizer reads standard Java single line comments.

isReadMultiLineComments

public boolean isReadMultiLineComments()

Returns whether tokenizer reads standard Java multiple line comments.

setReadMultiLineComments

public void setReadMultiLineComments(boolean aValue)

Sets whether tokenizer reads standard Java multiple line comments.

getNextSpecialTokenOrToken

public ParseToken getNextSpecialTokenOrToken()

Returns next special token or token.

getNextSpecialToken

public ParseToken getNextSpecialToken()

Processes and returns a special token if found. If more than one in a row, returns last one, which points to previous ones.

Overrides:

getNextSpecialToken in class Tokenizer

getNextSpecialTokenImpl

protected ParseToken getNextSpecialTokenImpl()

Processes and returns next special token.

getSingleLineCommentToken

protected ParseToken getSingleLineCommentToken()

Processes and returns a single line comment token if next up in input.

getMultiLineCommentToken

public ParseToken getMultiLineCommentToken()

Process and return a multi-line comment if next up in input.

getMultiLineCommentTokenMore

public ParseToken getMultiLineCommentTokenMore()

Returns a token from the current char to multi-line comment termination or input end.