tokens
Dict of ``{'state': [(regex, tokentype, new_state), ...], ...}``
The initial state is 'root'. ``new_state`` can be omitted to signify
no state transition. If it is a string, the state is pushed on the stack
and changed. If it is a tuple of strings, all states are pushed on the
stack and the current state will be the topmost. It can also be
``combined('state1', 'state2', ...)`` to signify a new, anonymous state
combined from the rules of two or more existing ones. Furthermore, it can
be '#pop' to signify going back one step in the state stack, or '#push'
to push the current state on the stack again.
The tuple can also be replaced with ``include('state')``, in which
case the rules from the state named by the string are included in the
current one.
- Value:
{ ' comments ' : [ ( ' (\\s*#.*) ' , Token.Comment) ] ,
' objList ' : [ ( ' \\s*\\) ' , Token.Text, ' #pop ' ) , 'object'] ,
' object ' : [ ( ' \\s*\\[ ' , Token.Text, ' predObj ' ) ,
( ' \\s*<[^> ]*> ' , Token.Name.Attribute) ,
( ' \\s*("""(?:.|\\n)*?""")(\\@[a-z]{2-4}|\\^\\^<?[a-zA-Z0-9
\\-\\:_#/\\.]*>?)?\\s* ' ,
<function callback at 0x850533c>) ,
( ' \\s*".*?[^\\\\]"(?:\\@[a-z]{2-4}|\\^\\^<?[a-zA-Z0-9\\-\\
...
|
|