Package snap.parse

Class Tokenizer

java.lang.Object
snap.parse.Tokenizer
Direct Known Subclasses:
CodeTokenizer, XMLParser.XMLTokenizer
public class Tokenizer extends Object
A class to extract tokens from a char sequence.

  • Field Details

  • Constructor Details

    • Tokenizer

      public Tokenizer()
      Constructor.

  • Method Details

    • getRegexes

      public Regex[] getRegexes()
      Returns the array of regexes.

    • setRegexes

      public void setRegexes(Regex[] theRegexes)
      Sets the array of regexes.

    • setRegexesForPatternRulesInRule

      public void setRegexesForPatternRulesInRule(ParseRule aRule)
      Sets regexes for pattern rules in given rule.

    • getInput

      public CharSequence getInput()
      Returns the current tokenizer input.

    • setInput

      public void setInput(CharSequence anInput)
      Sets the current tokenizer input.

    • length

      public final int length()
      CharSequence method.

    • charAt

      public final char charAt(int anIndex)
      CharSequence method.

    • hasChar

      public final boolean hasChar()
      Returns whether another char is available.

    • hasChars

      public final boolean hasChars(int aVal)
      Returns whether another given number of chars is available.

    • getChar

      public final char getChar()
      Returns the current parse char.

    • getChar

      public final char getChar(int anOffset)
      Returns the char at the current index plus offset.

    • eatChar

      public final char eatChar()
      Returns the char at the current index plus offset.

    • getCharIndex

      public final int getCharIndex()
      Returns the current parse char location.

    • setCharIndex

      public void setCharIndex(int aValue)
      Sets the current parse char location.

    • getNextToken

      public ParseToken getNextToken()
      Returns the next token.

    • getTokenLine

      public TokenLine getTokenLine()
      Returns the current token line.

    • getTokenDoc

      public TokenDoc getTokenDoc()
      Returns the current token doc.

    • createTokenForProps

      public ParseToken createTokenForProps(String aName, String aPattern, int aStart, int anEnd)
      Creates a new token for given properties.

    • getNextSpecialToken

      public ParseToken getNextSpecialToken()
      Processes and returns a special token if found. This version just skips whitespace.

    • skipWhiteSpace

      protected void skipWhiteSpace()
      Gobble input characters until next non-whitespace or input end.

    • tokenizerFailed

      protected ParseToken tokenizerFailed(String nextChars)
      Called when next chars don't conform to any known token pattern. Default implementation just throws ParseExcpetion.