Token

This module implements token.

class Token

The token API.

Implemented by SingleToken and MultiToken.

__init__()

Initialize the token.

strategy_data: Any

A data store reserved for use by strategies.

__len__() int

Return how many tokens are in the tree.

__iter__() Iterator[Optional[super_collator.token.Token[super_collator.token.TT]]]

Iterate the tokens in the tree.

__getitem__(n) Optional[super_collator.token.Token[super_collator.token.TT]]

Return the nth token in the iteration.

__str__() str

Return a string represenation of the token tree.

class SingleToken(data: super_collator.token.TT, user_data: Optional[Any] = None)

Represents one input token.

__init__(data: super_collator.token.TT, user_data: Optional[Any] = None)

Initialize the token.

Parameters

user_data (Any) – user data that will be round-tripped unchanged.

data: super_collator.token.TT

The data to align.

user_data: Any

A data store reserved for user data.

__len__() int

Return how many tokens are in the tree.

__iter__() Iterator[Optional[super_collator.token.Token[super_collator.token.TT]]]

Iterate the tokens in the tree.

__getitem__(n: int) Optional[super_collator.token.Token[super_collator.token.TT]]

Return the nth token in the iteration.

__str__() str

Return a string represenation of the token tree.

class MultiToken(*args: Optional[super_collator.token.Token[super_collator.token.TT]])

A container for one or two tokens.

A MultiToken is a node in a binary tree that mimics the order in which multiple aligments were done.

An alignment produces a sequence of MultiToken trees. Each tree contains all the tokens that were aligned at this position plus any gaps represented as None.

__init__(*args: Optional[super_collator.token.Token[super_collator.token.TT]])

Initialize the token.

__len__() int

Return how many tokens are in the tree.

__iter__() Iterator[Optional[super_collator.token.Token[super_collator.token.TT]]]

Iterate the tokens in the tree.

__getitem__(n: int) Optional[super_collator.token.Token[super_collator.token.TT]]

Return the nth token in the iteration.

__str__() str

Return a string represenation of the token tree.