maze_dataset.tokenization.modular
implements ModularMazeTokenizer
and related code
1"""implements `ModularMazeTokenizer` and related code""" 2 3__all__ = [ 4 # modules 5 "all_instances", 6 "all_tokenizers", 7 "element_base", 8 "elements", 9 "fst_load", 10 "fst", 11 "hashing", 12 "maze_tokenizer_modular", 13 "save_hashes", 14]