9.4 The `StringTokenizer` Class

As we have already mentioned, Java's Application Programming Interface (API) is a collection of hundreds of classes designed to add power and flexibility to the language. The StringTokenizer class is one of these; it provides a number of methods for dealing with strings that are composed of sequences of tokens, separated by delimiters. The tokens are usually words and the delimiters are usually blanks. As an example, consider the following string:
String quote = "I can resist anything except temptation";
If we consider blanks to be delimiters, then this string contains six tokens: "I", "can", ... , and "temptation".
We can use the StringTokenizer class to extract the tokens from such a string, as illustrated in the following example.

Example 1

The following fragment prints the words of the string quote with one word per line. It does so by first creating a new StringTokenizer object and then, as long as the hasMoreTokens method indicates that there are still more words to be processed, it uses the nextToken method to extract the next word.
StringTokenizer st = new StringTokenizer(quote); while (st.hasMoreTokens()) System.out.println(st.nextToken()) ;

The output from the fragment would be

`StringTokenizer` Constructors

The class has three constructors, each of which has one String parameter that is used to create a StringTokenizer object. They differ in their treatment of delimiters .

public StringTokenizer (String s)
This constructor, as we showed in Example 1, creates a String-Tokenizer object for the string s. As we noted there, it uses blanks as delimiters. It also uses any other white space characters (such as newlines, tabs, carriage returns, or form feeds) as delimiters .

public StringTokenizer (String s, String d)
This constructs a StringTokenizer object for the string s using any of the characters in the string d as delimiters. The characters in d are not treated as tokens .

public StringTokenizer (String s, String d, boolean flag)
Like the previous constructor, this one constructs a StringTokenizer object for the string s using any of the characters in the string d as delimiters. Now, however, if the value of flag is true, then the delimiter characters are also returned as tokens. (Each delimiter is returned as a string of length one.) If flag is false, the constructor behaves like the second one with characters in d not treated as tokens.

Example 2

The statement
StringTokenizer st1 = new StringTokenizer("12*(345+6789)","*/+-()",true);
would create a StringTokenizer object st1 using the given string. The delimiters are any of the characters in the string "*/+-()". Because the value of the third argument is true, the delimiters are treated as tokens. The resulting tokens of st1 are
"12" "*" "(" "345" "+" "6789" ")"

The statement
StringTokenizer st2 = new StringTokenizer("http://www.java.sun.com",":/'.");
would create a StringTokenizer object st2 with tokens
"http" "www" "java" "sun" "com"
The statement
StringTokenizer st3 = new StringTokenizer(s," \t\n\r\f",false);
would have the same effect as writing
StringTokenizer st3 = new StringTokenizer(s);
because the delimiters specified in the first constructor are the default whitespace delimiters and the fact that the flag in the first constructor is set to false means that the delimiters themselves will not be returned as tokens.

`StringTokenizer` Methods

We have already seen two methods of the class in Example 1. Here we will take a closer look at those plus one other. Collectively, the methods allow us to scan through a string from left to right, one token at a time, extracting tokens as we proceed.

public String nextToken ()
This returns the next token from the implicit string tokenizer object. The first time that this method is called, it returns the leftmost token in the string. Each call causes it to move along the string to the position of the next token. If there are no more tokens, the method throws a NoSuchElementException.

public boolean hasMoreTokens ()
This method checks to see if there are any more tokens available from the implicit string tokenizer object. It returns true if and only if there is at least one token in the string after the current position.

public int countTokens ()
This method returns the number of tokens remaining in the implicit string tokenizer object.
      Almost all the classes that we have been working with up to now have been in the core package called java.lang. The class StringTokenizer, on the other hand, is in the package java.util. If a program is to use a method from a class outside java.lang, we must inform the compiler of this. One way to do so is to use an import statement at the beginning of a program. If we want to use the StringTokenizer class in a program, we can precede the program with the statement
   import java.util.StringTokenizer;
As an alternative, we can write the import statement in the form
   import java.util.*;
This allows us to use any method in any of the classes in the java.util package. This form is useful if more than one class in a package is required in a program.

Exercise 9.4

Assuming that the string s contains sentences that consist of words, blanks, and punctuation marks (period, comma, exclamation mark, question mark, colon, and semi-colon), construct a StringTokenizer object whose tokens consist only of the words in s.

Write a class method wordCount that has a single String parameter, s. Assuming that s consists of words separated by whitespace, the method should return the number of words in s.

Write a class method value that will return, as an int, the value of a simple arithmetic expression contained in a string. The string should contain two integers surrounding one of the operators *, /, %, +, or -. The string may also contain blanks. As an example, given that
s = " 13 + 2"
the method should return the value 15. The method should assume that its argument contains a valid expression.

9.4 The StringTokenizer Class