Word Tokenizer Algorithm (Java)

Word Tokenizer is used to tokenize and filter out words and characters in TI and AB fields from citations. The algorithm used in the Java version is slice different than the Lisp version. Please see TI report and AB report for details.

The procedures and creteria are described as follows: