Class s.c.m.t.Tokenizer:
Part of
spamfighter
.
core
.
model
.
thomas
View Source
View In Hierarchy
A simple regex-based whitespace tokenizer. It expects a string and can return all tokens lower-cased or in their existing case.
Method
__init__
Undocumented
Method
tokenize
Undocumented
def __init__(self, lower=True):
(source)
Undocumented
def tokenize(self, obj):
(source)
Undocumented
API Documentation
for SpamFighter, generated by
pydoctor
at 2009-02-27 11:58:37.