This class performs lexical analysis of a specified input stream and breaks the input into tokens. It can be extremely useful when writing simple parsers. nextToken( ) returns the next token in the stream; this is either one of the constants defined by the class (which represent end-of-file, end-of-line, a parsed floating-point number, and a parsed word) or a character value. pushBack( ) pushes the token back onto the stream, so that it is returned by the next call to nextToken( ). The public variables sval and nval contain the string and numeric values (if applicable) of the most recently read token. They are applicable when the returned token is TT_WORD or TT_NUMBER. lineno( ) returns the current line number.

The remaining methods allow you to specify how tokens are recognized. wordChars( ) specifies a range of characters that should be treated as parts of words. whitespaceChars( ) specifies a range of characters that serve to delimit tokens. ordinaryChars( ) and ordinaryChar( ) specify characters that are never part of tokens and should be returned as-is. resetSyntax( ) makes all characters ordinary. eolIsSignificant( ) specifies whether end-of-line is significant. If so, the TT_EOL constant is returned for end-of-lines; otherwise, they are treated as whitespace. commentChar( ) specifies a character that begins a comment that lasts until the end of the line. No characters in the comment are returned. slashStarComments( ) and slashSlashComments( ) specify whether the StreamTokenizer should recognize C- and C++-style comments. If so, no part of the comment is returned as a token. quoteChar( ) specifies a character used to delimit strings. When a string token is parsed, the quote character is returned as the token value, and the body of the string is stored in the sval variable. lowerCaseMode( ) specifies whether TT_WORD tokens should be converted to all lowercase characters before being stored in sval. parseNumbers( ) specifies that the StreamTokenizer should recognize and return double-precision floating-point number tokens.

public class StreamTokenizer {
// Public Constructors
#    public StreamTokenizer(InputStream is);  
1.1  public StreamTokenizer(Reader r);  
// Public Constants
     public static final int TT_EOF;       =-1
     public static final int TT_EOL;       =10
     public static final int TT_NUMBER;    =-2
     public static final int TT_WORD;      =-3
// Public Instance Methods
     public void commentChar(int ch);  
     public void eolIsSignificant(boolean flag);  
     public int lineno( );  
     public void lowerCaseMode(boolean fl);  
     public int nextToken( ) throws IOException;  
     public void ordinaryChar(int ch);  
     public void ordinaryChars(int low, int hi);  
     public void parseNumbers( );  
     public void pushBack( );  
     public void quoteChar(int ch);  
     public void resetSyntax( );  
     public void slashSlashComments(boolean flag);  
     public void slashStarComments(boolean flag);  
     public void whitespaceChars(int low, int hi);  
     public void wordChars(int low, int hi);  
// Public Methods Overriding Object
     public String toString( );  
// Public Instance Fields
     public double nval;  
     public String sval;  
     public int ttype;  
}