docs for maze-dataset v1.3.1
View Source on GitHub

maze_dataset.tokenization.modular

implements ModularMazeTokenizer and related code


 1"""implements `ModularMazeTokenizer` and related code"""
 2
 3__all__ = [
 4	# modules
 5	"all_instances",
 6	"all_tokenizers",
 7	"element_base",
 8	"elements",
 9	"fst_load",
10	"fst",
11	"hashing",
12	"maze_tokenizer_modular",
13	"save_hashes",
14]