Tools
Class StringTokenizer

java.lang.Object
  Tools.StringTokenizer

All Implemented Interfaces:

java.util.Enumeration, java.util.Iterator

public class StringTokenizer
extends java.lang.Object
implements java.util.Enumeration, java.util.Iterator

The string tokenizer class allows an application to break a string into tokens. More information about this class is available from ostermiller.org.

The tokenization method is much simpler than the one used by the StreamTokenizer class. The StringTokenizer methods do not distinguish among identifiers, numbers, and quoted strings, nor do they recognize and skip comments.

The set of delimiters (the characters that separate tokens) may be specified either at creation time or on a per-token basis.

There are two kinds of delimiters: token delimiters and nontoken delimiters. A token is either one token delimiter character, or a maximal sequence of consecutive characters that are not delimiters.

A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.

The implementation is not thread safe; if a StringTokenizer object is intended to be used in multiple threads, an appropriate wrapper must be provided.

The following is one example of the use of the tokenizer. It also demonstrates the usefulness of having both token and nontoken delimiters in one StringTokenizer.

The code:

String s = " ( aaa \t * (b+c1 ))";
StringTokenizer st = new StringTokenizer(s, " \t\n\r\f", "()+*");
while (st.hasMoreTokens()) {
System.out.println(st.nextToken());
};

prints the following output:

(
aaa
*
(
b
+
c1
)
)

Compatibility with java.util.StringTokenizer

In the original version of java.util.StringTokenizer, the method nextToken() left the current position after the returned token, and the method hasMoreTokens() moved (as a side effect) the current position before the beginning of the next token. Thus, the code:

String s = "x=a,b,c";
java.util.StringTokenizer st = new java.util.StringTokenizer(s,"=");
System.out.println(st.nextToken());
while (st.hasMoreTokens()) {
System.out.println(st.nextToken(","));
};

prints the following output:

x
a
b
c

The Java SDK 1.3 implementation removed the undesired side effect of hasMoreTokens method: now, it does not advance current position. However, after these changes the output of the above code was:

x
=a
b
c

and there was no good way to produce a second token without "=".

To solve the problem, this implementation introduces a new method skipDelimiters(). To produce the original output, the above code should be modified as:

String s = "x=a,b,c";
StringTokenizer st = new StringTokenizer(s,"=");
System.out.println(st.nextToken());
st.skipDelimiters();
while (st.hasMoreTokens()) {
System.out.println(st.nextToken(","));
};

Since:

ostermillerutils 1.00.00

Author:

Stephen Ostermiller http://ostermiller.org/contact.pl?regarding=Java+Utilities

Field Summary

protected int	delimsChangedPosition Indicates at which position the delimiters last changed.
protected boolean	emptyReturned One of two variables used to maintain state through the tokenizing process.
protected char	maxDelimChar Stores the value of the delimiter character with the highest value.
protected java.lang.String	nontokenDelims The set of nontoken delimiters.
protected int	position One of two variables used to maintain state through the tokenizing process.
protected boolean	returnEmptyTokens Whether empty tokens should be returned.
protected int	strLength The length of the text.
protected java.lang.String	text The string to be tokenized.
protected int	tokenCount A cache of the token count.
protected java.lang.String	tokenDelims The set of token delimiters.

Constructor Summary

StringTokenizer(java.lang.String text)
Constructs a string tokenizer for the specified string.

StringTokenizer(java.lang.String text, java.lang.String nontokenDelims)
Constructs a string tokenizer for the specified string.

StringTokenizer(java.lang.String text, java.lang.String delims, boolean delimsAreTokens)
Constructs a string tokenizer for the specified string.

StringTokenizer(java.lang.String text, java.lang.String nontokenDelims, java.lang.String tokenDelims)
Constructs a string tokenizer for the specified string.

StringTokenizer(java.lang.String text, java.lang.String nontokenDelims, java.lang.String tokenDelims, boolean returnEmptyTokens)
Constructs a string tokenizer for the specified string.

Method Summary

private boolean	advancePosition() Advances the state of the tokenizer to the next token or delimiter.
int	countTokens() Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception.
int	countTokens(java.lang.String delims) Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of (nontoken) delimiters.
int	countTokens(java.lang.String delims, boolean delimsAreTokens) Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of delimiters.
int	countTokens(java.lang.String nontokenDelims, java.lang.String tokenDelims) Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of delimiters.
int	countTokens(java.lang.String nontokenDelims, java.lang.String tokenDelims, boolean returnEmptyTokens) Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of delimiters.
int	getCurrentPosition() Get the the index of the character immediately following the end of the last token.
boolean	hasMoreElements() Returns the same value as the hasMoreTokens() method.
boolean	hasMoreTokens() Tests if there are more tokens available from this tokenizer's string.
boolean	hasNext() Returns the same value as the hasMoreTokens() method.
private int	indexOfNextDelimiter(int start) Similar to String.indexOf(int, String) but will look for any character from string rather than the entire string.
java.lang.Object	next() Returns the same value as the nextToken() method, except that its declared return value is Object rather than String.
java.lang.Object	nextElement() Returns the same value as the nextToken() method, except that its declared return value is Object rather than String.
java.lang.String	nextToken() Returns the next token from this string tokenizer.
java.lang.String	nextToken(java.lang.String nontokenDelims) Returns the next token in this string tokenizer's string.
java.lang.String	nextToken(java.lang.String delims, boolean delimsAreTokens) Returns the next token in this string tokenizer's string.
java.lang.String	nextToken(java.lang.String nontokenDelims, java.lang.String tokenDelims) Returns the next token in this string tokenizer's string.
java.lang.String	nextToken(java.lang.String nontokenDelims, java.lang.String tokenDelims, boolean returnEmptyTokens) Returns the next token in this string tokenizer's string.
java.lang.String	peek() Returns the same value as nextToken() but does not alter the internal state of the Tokenizer.
void	remove() This implementation always throws UnsupportedOperationException.
java.lang.String	restOfText() Retrieves the rest of the text as a single token.
void	setDelimiters(java.lang.String delims) Set the delimiters used to this set of (nontoken) delimiters.
void	setDelimiters(java.lang.String delims, boolean delimsAreTokens) Set the delimiters used to this set of delimiters.
void	setDelimiters(java.lang.String nontokenDelims, java.lang.String tokenDelims) Set the delimiters used to this set of delimiters.
void	setDelimiters(java.lang.String nontokenDelims, java.lang.String tokenDelims, boolean returnEmptyTokens) Set the delimiters used to this set of delimiters.
private void	setDelims(java.lang.String nontokenDelims, java.lang.String tokenDelims) Set the delimiters for this StringTokenizer.
void	setReturnEmptyTokens(boolean returnEmptyTokens) Set whether empty tokens should be returned from this point in in the tokenizing process onward.
void	setText(java.lang.String text) Set the text to be tokenized in this StringTokenizer.
boolean	skipDelimiters() Advances the current position so it is before the next token.
java.lang.String[]	toArray() Retrieve all of the remaining tokens in a String array.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

text

protected java.lang.String text

The string to be tokenized. The code relies on this to never be null.

Since:

ostermillerutils 1.00.00

strLength

protected int strLength

The length of the text. Cached for performance. This should be set whenever the string we are working with is changed.

Since:

ostermillerutils 1.00.00

nontokenDelims

protected java.lang.String nontokenDelims

The set of nontoken delimiters.

Since:

ostermillerutils 1.00.00

tokenDelims

protected java.lang.String tokenDelims

The set of token delimiters.

Since:

ostermillerutils 1.00.00

position

protected int position

One of two variables used to maintain state through the tokenizing process.

Represents the position at which we should start looking for the next token(the position of the character immediately following the end of the last token, or 0 to start), or -1 if the entire string has been examined.

Since:

ostermillerutils 1.00.00

emptyReturned

protected boolean emptyReturned

One of two variables used to maintain state through the tokenizing process.

true if and only if is found that an empty token should be returned or if empty token was the last thing returned.

If returnEmptyTokens in false, then this variable will always be false.

Since:

ostermillerutils 1.00.00

maxDelimChar

protected char maxDelimChar

Stores the value of the delimiter character with the highest value. It is used to optimize the detection of delimiter characters. The common case will be that the int values of delimiters will be less than that of most characters in the string (, or space less than any letter for example). Given this, we can check easily check to see if a character is not a delimiter by comparing it to the max delimiter. If it is greater than the max delimiter, then it is no a delimiter otherwise we have to do some more in depth analysis. (ie search the delimiter string.) This will reduce the running time of the algorithm not to depend on the length of the delimiter string for the common case.

Since:

ostermillerutils 1.00.00

returnEmptyTokens

protected boolean returnEmptyTokens

Whether empty tokens should be returned. ie if "" should be returned when text starts with a delim, has two delims next to each other, or ends with a delim.

Since:

ostermillerutils 1.00.00

delimsChangedPosition

protected int delimsChangedPosition

Indicates at which position the delimiters last changed. This will effect how null tokens are returned. Any time that delimiters are changed, the string will be treated as if it is being parsed from position zero, ie, null strings are possible at the very beginning.

Since:

ostermillerutils 1.00.00

tokenCount

protected int tokenCount

A cache of the token count. This variable should be -1 if the token have not yet been counted. It should be greater than or equal to zero if the tokens have been counted.

Since:

ostermillerutils 1.00.00

Constructor Detail

StringTokenizer

public StringTokenizer(java.lang.String text,
                       java.lang.String nontokenDelims,
                       java.lang.String tokenDelims)

Constructs a string tokenizer for the specified string. Both token and nontoken delimiters are specified.

The current position is set at the beginning of the string.

Parameters:

text - a string to be parsed.

nontokenDelims - the nontoken delimiters, i.e. the delimiters that only separate tokens and are not returned as separate tokens.

tokenDelims - the token delimiters, i.e. delimiters that both separate tokens, and are themselves returned as tokens.

Throws:

java.lang.NullPointerException - if text is null.

Since:

ostermillerutils 1.00.00

StringTokenizer

public StringTokenizer(java.lang.String text,
                       java.lang.String nontokenDelims,
                       java.lang.String tokenDelims,
                       boolean returnEmptyTokens)

Constructs a string tokenizer for the specified string. Both token and nontoken delimiters are specified and whether or not empty tokens are returned is specified.

Empty tokens are tokens that are between consecutive delimiters.

It is a primary constructor (i.e. all other constructors are defined in terms of it.)

The current position is set at the beginning of the string.

Parameters:

text - a string to be parsed.

nontokenDelims - the nontoken delimiters, i.e. the delimiters that only separate tokens and are not returned as separate tokens.

tokenDelims - the token delimiters, i.e. delimiters that both separate tokens, and are themselves returned as tokens.

returnEmptyTokens - true if empty tokens may be returned; false otherwise.

Throws:

java.lang.NullPointerException - if text is null.

Since:

ostermillerutils 1.00.00

StringTokenizer

public StringTokenizer(java.lang.String text,
                       java.lang.String delims,
                       boolean delimsAreTokens)

Constructs a string tokenizer for the specified string. Either token or nontoken delimiters are specified.

Is equivalent to:

• If the third parameter is false -- StringTokenizer(text,delims, null)
• If the third parameter is true -- StringTokenizer(text, null ,delims)

Parameters:

text - a string to be parsed.

delims - the delimiters.

delimsAreTokens - flag indicating whether the second parameter specifies token or nontoken delimiters: false -- the second parameter specifies nontoken delimiters, the set of token delimiters is empty; true -- the second parameter specifies token delimiters, the set of nontoken delimiters is empty.

Throws:

java.lang.NullPointerException - if text is null.

Since:

ostermillerutils 1.00.00

StringTokenizer

public StringTokenizer(java.lang.String text,
                       java.lang.String nontokenDelims)

Constructs a string tokenizer for the specified string. The characters in the nontokenDelims argument are the delimiters for separating tokens. Delimiter characters themselves will not be treated as tokens.

Is equivalent to StringTokenizer(text,nontokenDelims, null).

Parameters:

text - a string to be parsed.

nontokenDelims - the nontoken delimiters.

Throws:

java.lang.NullPointerException - if text is null.

Since:

ostermillerutils 1.00.00

StringTokenizer

public StringTokenizer(java.lang.String text)

Constructs a string tokenizer for the specified string. The tokenizer uses " \t\n\r\f" as a delimiter set of nontoken delimiters, and an empty token delimiter set.

Is equivalent to StringTokenizer(text, " \t\n\r\f", null);

Parameters:

text - a string to be parsed.

Throws:

java.lang.NullPointerException - if text is null.

Since:

ostermillerutils 1.00.00

Method Detail

setText

public void setText(java.lang.String text)

Set the text to be tokenized in this StringTokenizer.

This is useful when for StringTokenizer re-use so that new string tokenizers do no have to be created for each string you want to tokenizer.

The string will be tokenized from the beginning of the string.

Parameters:

text - a string to be parsed.

Throws:

java.lang.NullPointerException - if text is null.

Since:

ostermillerutils 1.00.00

setDelims

private void setDelims(java.lang.String nontokenDelims,
                       java.lang.String tokenDelims)

Set the delimiters for this StringTokenizer. The position must be initialized before this method is used. (setText does this and it is called from the constructor)

Parameters:

nontokenDelims - delimiters that should not be returned as tokens.

tokenDelims - delimiters that should be returned as tokens.

Since:

ostermillerutils 1.00.00

hasMoreTokens

public boolean hasMoreTokens()

Tests if there are more tokens available from this tokenizer's string. If this method returns true, then a subsequent call to nextToken with no argument will successfully return a token.

The current position is not changed.

Returns:

true if and only if there is at least one token in the string after the current position; false otherwise.

Since:

ostermillerutils 1.00.00

nextToken

public java.lang.String nextToken()

Returns the next token from this string tokenizer.

The current position is set after the token returned.

Returns:

the next token from this string tokenizer.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

skipDelimiters

public boolean skipDelimiters()

Advances the current position so it is before the next token.

This method skips nontoken delimiters but does not skip token delimiters.

This method is useful when switching to the new delimiter sets (see the second example in the class comment.)

Returns:

true if there are more tokens, false otherwise.

Since:

ostermillerutils 1.00.00

countTokens

public int countTokens()

Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception. The current position is not advanced.

Returns:

the number of tokens remaining in the string using the current delimiter set.

Since:

ostermillerutils 1.00.00

See Also:

nextToken()

setDelimiters

public void setDelimiters(java.lang.String delims)

Set the delimiters used to this set of (nontoken) delimiters.

Parameters:

delims - the new set of nontoken delimiters (the set of token delimiters will be empty).

Since:

ostermillerutils 1.00.00

setDelimiters

public void setDelimiters(java.lang.String delims,
                          boolean delimsAreTokens)

Set the delimiters used to this set of delimiters.

Parameters:

delims - the new set of delimiters.

delimsAreTokens - flag indicating whether the first parameter specifies token or nontoken delimiters: false -- the first parameter specifies nontoken delimiters, the set of token delimiters is empty; true -- the first parameter specifies token delimiters, the set of nontoken delimiters is empty.

Since:

ostermillerutils 1.00.00

setDelimiters

public void setDelimiters(java.lang.String nontokenDelims,
                          java.lang.String tokenDelims)

Set the delimiters used to this set of delimiters.

Parameters:

nontokenDelims - the new set of nontoken delimiters.

tokenDelims - the new set of token delimiters.

Since:

ostermillerutils 1.00.00

setDelimiters

public void setDelimiters(java.lang.String nontokenDelims,
                          java.lang.String tokenDelims,
                          boolean returnEmptyTokens)

Set the delimiters used to this set of delimiters.

Parameters:

nontokenDelims - the new set of nontoken delimiters.

tokenDelims - the new set of token delimiters.

returnEmptyTokens - true if empty tokens may be returned; false otherwise.

Since:

ostermillerutils 1.00.00

countTokens

public int countTokens(java.lang.String delims)

Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of (nontoken) delimiters. The delimiters given will be used for future calls to nextToken() unless new delimiters are given. The current position is not advanced.

Parameters:

delims - the new set of nontoken delimiters (the set of token delimiters will be empty).

Returns:

the number of tokens remaining in the string using the new delimiter set.

Since:

ostermillerutils 1.00.00

See Also:

countTokens()

countTokens

public int countTokens(java.lang.String delims,
                       boolean delimsAreTokens)

Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of delimiters. The delimiters given will be used for future calls to nextToken() unless new delimiters are given. The current position is not advanced.

Parameters:

delims - the new set of delimiters.

delimsAreTokens - flag indicating whether the first parameter specifies token or nontoken delimiters: false -- the first parameter specifies nontoken delimiters, the set of token delimiters is empty; true -- the first parameter specifies token delimiters, the set of nontoken delimiters is empty.

Returns:

the number of tokens remaining in the string using the new delimiter set.

Since:

ostermillerutils 1.00.00

See Also:

countTokens()

countTokens

public int countTokens(java.lang.String nontokenDelims,
                       java.lang.String tokenDelims)

Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of delimiters. The delimiters given will be used for future calls to nextToken() unless new delimiters are given. The current position is not advanced.

Parameters:

nontokenDelims - the new set of nontoken delimiters.

tokenDelims - the new set of token delimiters.

Returns:

the number of tokens remaining in the string using the new delimiter set.

Since:

ostermillerutils 1.00.00

See Also:

countTokens()

countTokens

public int countTokens(java.lang.String nontokenDelims,
                       java.lang.String tokenDelims,
                       boolean returnEmptyTokens)

Calculates the number of times that this tokenizer's nextToken method can be called before it generates an exception using the given set of delimiters. The delimiters given will be used for future calls to nextToken() unless new delimiters are given. The current position is not advanced.

Parameters:

nontokenDelims - the new set of nontoken delimiters.

tokenDelims - the new set of token delimiters.

returnEmptyTokens - true if empty tokens may be returned; false otherwise.

Returns:

the number of tokens remaining in the string using the new delimiter set.

Since:

ostermillerutils 1.00.00

See Also:

countTokens()

advancePosition

private boolean advancePosition()

Advances the state of the tokenizer to the next token or delimiter. This method only modifies the class variables position, and emptyReturned. The type of token that should be emitted can be deduced by examining the changes to these two variables. If there are no more tokens, the state of these variables does not change at all.

Returns:

true if we are at a juncture at which a token may be emitted, false otherwise.

Since:

ostermillerutils 1.00.00

nextToken

public java.lang.String nextToken(java.lang.String nontokenDelims,
                                  java.lang.String tokenDelims)

Returns the next token in this string tokenizer's string.

First, the sets of token and nontoken delimiters are changed to be the tokenDelims and nontokenDelims, respectively. Then the next token (with respect to new delimiters) in the string after the current position is returned.

The current position is set after the token returned.

The new delimiter sets remains the used ones after this call.

Parameters:

nontokenDelims - the new set of nontoken delimiters.

tokenDelims - the new set of token delimiters.

Returns:

the next token, after switching to the new delimiter set.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

See Also:

nextToken()

nextToken

public java.lang.String nextToken(java.lang.String nontokenDelims,
                                  java.lang.String tokenDelims,
                                  boolean returnEmptyTokens)

Returns the next token in this string tokenizer's string.

First, the sets of token and nontoken delimiters are changed to be the tokenDelims and nontokenDelims, respectively; and whether or not to return empty tokens is set. Then the next token (with respect to new delimiters) in the string after the current position is returned.

The current position is set after the token returned.

The new delimiter set remains the one used for this call and empty tokens are returned in the future as they are in this call.

Parameters:

nontokenDelims - the new set of nontoken delimiters.

tokenDelims - the new set of token delimiters.

returnEmptyTokens - true if empty tokens may be returned; false otherwise.

Returns:

the next token, after switching to the new delimiter set.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

See Also:

nextToken()

nextToken

public java.lang.String nextToken(java.lang.String delims,
                                  boolean delimsAreTokens)

Returns the next token in this string tokenizer's string.

Is equivalent to:

• If the second parameter is false -- nextToken(delims, null)
• If the second parameter is true -- nextToken(null ,delims)

Parameters:

delims - the new set of token or nontoken delimiters.

delimsAreTokens - flag indicating whether the first parameter specifies token or nontoken delimiters: false -- the first parameter specifies nontoken delimiters, the set of token delimiters is empty; true -- the first parameter specifies token delimiters, the set of nontoken delimiters is empty.

Returns:

the next token, after switching to the new delimiter set.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

See Also:

nextToken(String,String)

nextToken

public java.lang.String nextToken(java.lang.String nontokenDelims)

Returns the next token in this string tokenizer's string.

Is equivalent to nextToken(delims, null).

Parameters:

nontokenDelims - the new set of nontoken delimiters (the set of token delimiters will be empty).

Returns:

the next token, after switching to the new delimiter set.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

See Also:

nextToken(String,String)

indexOfNextDelimiter

private int indexOfNextDelimiter(int start)

Similar to String.indexOf(int, String) but will look for any character from string rather than the entire string.

Parameters:

start - index in text at which to begin the search

Returns:

index of the first delimiter from the start index (inclusive), or -1 if there are no more delimiters in the string

Since:

ostermillerutils 1.00.00

hasMoreElements

public boolean hasMoreElements()

Returns the same value as the hasMoreTokens() method. It exists so that this class can implement the Enumeration interface.

Specified by:

hasMoreElements in interface java.util.Enumeration

Returns:

true if there are more tokens; false otherwise.

Since:

ostermillerutils 1.00.00

See Also:

Enumeration, hasMoreTokens()

nextElement

public java.lang.Object nextElement()

Returns the same value as the nextToken() method, except that its declared return value is Object rather than String. It exists so that this class can implement the Enumeration interface.

Specified by:

nextElement in interface java.util.Enumeration

Returns:

the next token in the string.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

See Also:

Enumeration, nextToken()

hasNext

public boolean hasNext()

Returns the same value as the hasMoreTokens() method. It exists so that this class can implement the Iterator interface.

Specified by:

hasNext in interface java.util.Iterator

Returns:

true if there are more tokens; false otherwise.

Since:

ostermillerutils 1.00.00

See Also:

Iterator, hasMoreTokens()

next

public java.lang.Object next()

Returns the same value as the nextToken() method, except that its declared return value is Object rather than String. It exists so that this class can implement the Iterator interface.

Specified by:

next in interface java.util.Iterator

Returns:

the next token in the string.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00

See Also:

Iterator, nextToken()

remove

public void remove()

This implementation always throws UnsupportedOperationException. It exists so that this class can implement the Iterator interface.

Specified by:

remove in interface java.util.Iterator

Throws:

java.lang.UnsupportedOperationException - always is thrown.

Since:

ostermillerutils 1.00.00

See Also:

Iterator

setReturnEmptyTokens

public void setReturnEmptyTokens(boolean returnEmptyTokens)

Set whether empty tokens should be returned from this point in in the tokenizing process onward.

Empty tokens occur when two delimiters are next to each other or a delimiter occurs at the beginning or end of a string. If empty tokens are set to be returned, and a comma is the non token delimiter, the following table shows how many tokens are in each string.

String		Number of tokens
"one,two"		2 - normal case with no empty tokens.
"one,,three"		3 including the empty token in the middle.
"one,"		2 including the empty token at the end.
",two"		2 including the empty token at the beginning.
","		2 including the empty tokens at the beginning and the ends.
""		1 - all strings will have at least one token if empty tokens are returned.

Parameters:

returnEmptyTokens - true iff empty tokens should be returned.

Since:

ostermillerutils 1.00.00

getCurrentPosition

public int getCurrentPosition()

Get the the index of the character immediately following the end of the last token. This is the position at which this tokenizer will begin looking for the next token when a nextToken() method is invoked.

Returns:

the current position or -1 if the entire string has been tokenized.

Since:

ostermillerutils 1.00.00

toArray

public java.lang.String[] toArray()

Retrieve all of the remaining tokens in a String array. This method uses the options that are currently set for the tokenizer and will advance the state of the tokenizer such that hasMoreTokens() will return false.

Returns:

an array of tokens from this tokenizer.

Since:

ostermillerutils 1.00.00

restOfText

public java.lang.String restOfText()

Retrieves the rest of the text as a single token. After calling this method hasMoreTokens() will always return false.

Returns:

any part of the text that has not yet been tokenized.

Since:

ostermillerutils 1.00.00

peek

public java.lang.String peek()

Returns the same value as nextToken() but does not alter the internal state of the Tokenizer. Subsequent calls to peek() or a call to nextToken() will return the same token again.

Returns:

the next token from this string tokenizer.

Throws:

NoSuchElementException - if there are no more tokens in this tokenizer's string.

Since:

ostermillerutils 1.00.00