module documentation

Built-in span-level token classes.

Class AutoLink Autolink token. ("<http://www.google.com>") This is an inline token with a single child of type RawText.
Class CoreTokens Represents core tokens (Strong, Emphasis, Image, Link) during the early stage of parsing. Replaced with objects of the proper classes in the final stage of parsing.
Class Emphasis Emphasis token. ("some text") This is an inline token. Its children are inline (span) tokens. One of the core tokens.
Class EscapeSequence Escape sequence token. ("*") This is an inline token with a single child of type RawText.
Class HTMLSpan Span-level HTML token. This is an inline token without children.
Class Image Image token. ("![alt](src "title")") This is an inline token. Its children are inline (span) tokens holding the image description. One of the core tokens.
Class InlineCode Inline code token. ("some code") This is an inline token with a single child of type RawText.
Class LineBreak Line break token. Hard or soft. This is an inline token without children.
Class Link Link token. ("[name](target)") This is an inline token. Its children are inline (span) tokens holding the link text. One of the core tokens.
Class RawText Raw text token. This is an inline token without children.
Class SpanToken Undocumented
Class Strikethrough Strikethrough token. ("~~some text~~") This is an inline token. Its children are inline (span) tokens.
Class Strong Strong token. ("some text") This is an inline token. Its children are inline (span) tokens. One of the core tokens.
Class XWikiBlockMacroEnd A "block" macro closing tag. ("<onlySpacesAllowed>{{/macroName}}")
Class XWikiBlockMacroStart A "block" macro opening tag. ("{{macroName<optionalParams>}}<newLine>")
Function add_token Allows external manipulation of the parsing process. This function is called in BaseRenderer.__enter__.
Function remove_token Allows external manipulation of the parsing process. This function is called in BaseRenderer.__exit__.
Function reset_tokens Resets global _token_types to all token classes in __all__.
Function tokenize_inner A wrapper around span_tokenizer.tokenize. Pass in all span-level token constructors as arguments to span_tokenizer.tokenize.
Variable _attrs Undocumented
Variable _cdata Undocumented
Variable _closing_tag Undocumented
Variable _comment Undocumented
Variable _declaration Undocumented
Variable _instruction Undocumented
Variable _open_tag Undocumented
Variable _tag Undocumented
Variable _tags Undocumented
Variable _token_types Undocumented
def add_token(token_cls, position=1): (source)

Allows external manipulation of the parsing process. This function is called in BaseRenderer.__enter__.

Parameters
token_cls:SpanTokentoken to be included in the parsing process.
positionUndocumented
def remove_token(token_cls): (source)

Allows external manipulation of the parsing process. This function is called in BaseRenderer.__exit__.

Parameters
token_cls:SpanTokentoken to be removed from the parsing process.
def reset_tokens(): (source)

Resets global _token_types to all token classes in __all__.

def tokenize_inner(content): (source)

A wrapper around span_tokenizer.tokenize. Pass in all span-level token constructors as arguments to span_tokenizer.tokenize.

Doing so (instead of importing span_token module in span_tokenizer) avoids cyclic dependency issues, and allows for future injections of custom token classes.

_token_types variable is at the bottom of this module.

See also: span_tokenizer.tokenize, block_token.tokenize.

Undocumented

Undocumented

_closing_tag = (source)

Undocumented

_comment: str = (source)

Undocumented

_declaration: str = (source)

Undocumented

_instruction: str = (source)

Undocumented

_open_tag = (source)

Undocumented

Undocumented

Undocumented

_token_types: list = (source)

Undocumented