Package snap.parse

Class Tokenizer

java.lang.Object
snap.parse.Tokenizer
Direct Known Subclasses:
XMLParser.XMLTokenizer
public class Tokenizer extends Object
A class to extract tokens from a char sequence.

  • Field Details

  • Constructor Details

    • Tokenizer

      public Tokenizer()
      Constructor.

  • Method Details

    • getRegexes

      public Regex[] getRegexes()
      Returns the array of regexes.

    • setRegexes

      public void setRegexes(Regex[] theRegexes)
      Sets the array of regexes.

    • setRegexesForGrammar

      public void setRegexesForGrammar(Grammar aGrammar)
      Sets the regexes for given grammar.

    • getInput

      public CharSequence getInput()
      Returns the current tokenizer input.

    • setInput

      public void setInput(CharSequence anInput)
      Sets the current tokenizer input.

    • length

      public final int length()
      CharSequence method.

    • charAt

      public final char charAt(int anIndex)
      CharSequence method.

    • hasChar

      public final boolean hasChar()
      Returns whether another char is available.

    • hasChars

      public final boolean hasChars(int aVal)
      Returns whether another given number of chars is available.

    • nextChar

      public final char nextChar()
      Returns the next parse char.

    • nextCharAt

      public final char nextCharAt(int anOffset)
      Returns the char at the current index plus offset.

    • nextCharEquals

      public boolean nextCharEquals(char aChar)
      Returns whether next char is equal to given char.

    • nextCharsStartWith

      public boolean nextCharsStartWith(CharSequence startChars)
      Returns whether next chars start with given string.

    • eatChar

      public final char eatChar()
      Returns the char at the current index plus offset.

    • eatChars

      public void eatChars(int charCount)
      Advances charIndex by given char count.

    • eatCharsTillLineEnd

      public void eatCharsTillLineEnd()
      Eats the chars till line end.

    • getCharIndex

      public final int getCharIndex()
      Returns the current parse char location.

    • setCharIndex

      public void setCharIndex(int aValue)
      Sets the current parse char location.

    • getLastToken

      public ParseToken getLastToken()
      Returns the last token.

    • setLastToken

      public void setLastToken(ParseToken aToken)
      Sets the last token.

    • getNextToken

      public ParseToken getNextToken()
      Returns the next token.

    • getNextTokenImpl

      protected ParseToken getNextTokenImpl()
      Returns the next token.

    • getTokenLine

      public TokenLine getTokenLine()
      Returns the current token line.

    • getTokenDoc

      public TokenDoc getTokenDoc()
      Returns the current token doc.

    • createTokenForProps

      public ParseToken createTokenForProps(String aName, String aPattern, int aStart, int anEnd)
      Creates a new token for given properties.

    • tokenizerFailed

      protected ParseToken tokenizerFailed(String nextChars)
      Called when next chars don't conform to any known token pattern. Default implementation just throws ParseExcpetion.

    • enableCodeComments

      public void enableCodeComments()
      Turns on Java style comments.

    • enableSpecialTokens

      public void enableSpecialTokens()
      Turns on special tokens.