NAME¶

Plucene::Analysis::Tokenizer - base class for tokenizers

SYNOPSIS¶

        my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                ->new({ reader => $reader });

DESCRIPTION¶

This is an abstract base class for tokenizers.

A Tokenizer is a TokenStream whose input is a Reader.

METHODS¶

        my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                ->new({ reader => $reader });