class documentation
Recreate a token stream formatted with the `RawTokenFormatter`. Additional options accepted: `compress` If set to ``"gz"`` or ``"bz2"``, decompress the token stream with the given compression algorithm before lexing (default: ``""``).
Method | __init__ |
Undocumented |
Method | get |
Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. |
Method | get |
Return an iterable of (index, tokentype, value) pairs where "index" is the starting position of the token within the input text. |
Class Variable | aliases |
Undocumented |
Class Variable | filenames |
Undocumented |
Class Variable | mimetypes |
Undocumented |
Class Variable | name |
Undocumented |
Instance Variable | compress |
Undocumented |
Inherited from Lexer
:
Method | __repr__ |
Undocumented |
Method | add |
Add a new stream filter to this lexer. |
Method | analyse |
Has to return a float between ``0`` and ``1`` that indicates if a lexer wants to highlight this text. Used by ``guess_lexer``. If this method returns ``0`` it won't highlight it in any case, if it returns ``1`` highlighting with this lexer is guaranteed. |
Class Variable | alias |
Undocumented |
Class Variable | priority |
Undocumented |
Class Variable | url |
Undocumented |
Instance Variable | encoding |
Undocumented |
Instance Variable | ensurenl |
Undocumented |
Instance Variable | filters |
Undocumented |
Instance Variable | options |
Undocumented |
Instance Variable | stripall |
Undocumented |
Instance Variable | stripnl |
Undocumented |
Instance Variable | tabsize |
Undocumented |
overrides
pygments.lexer.Lexer.get_tokens
Return an iterable of (tokentype, value) pairs generated from `text`. If `unfiltered` is set to `True`, the filtering mechanism is bypassed even if filters are defined. Also preprocess the text, i.e. expand tabs and strip it if wanted and applies registered filters.