mistletoe.block_tokenizer

module documentation
(source)

Block-level tokenizer for mistletoe.

Class FileWrapper Undocumented
Class ParseBuffer A wrapper around builtin list, so that setattr(list, 'loose') is legal.
Function make_tokens Takes a list of pairs (token_type, read_result) and applies token_type(read_result).
Function tokenize Searches for token_types in iterable.
Function tokenize_block Returns a list of pairs (token_type, read_result).
def make_tokens(parse_buffer): (source)

Takes a list of pairs (token_type, read_result) and applies token_type(read_result).

Footnotes are already parsed before this point, and span-level parsing is started here.

def tokenize(iterable, token_types): (source)

Searches for token_types in iterable.

Parameters
iterable:listuser input lines to be parsed.
token_types:lista list of block-level token constructors.
Returns
block-level token instances.
def tokenize_block(iterable, token_types): (source)

Returns a list of pairs (token_type, read_result).

Footnotes are parsed here, but span-level parsing has not started yet.