babel.messages.jslexer

module documentation

(source)

babel.messages.jslexer ~~~~~~~~~~~~~~~~~~~~~~ A simple JavaScript 1.5 lexer which is used for the JavaScript extractor. :copyright: (c) 2013-2023 by the Babel Team. :license: BSD, see LICENSE for more details.

Class	`Token`	Undocumented
Function	`get_rules`	Get a tokenization rule list given the passed syntax options.
Function	`indicates_division`	A helper function that helps the tokenizer to decide if the current token may be followed by a division operator.
Function	`tokenize`	Tokenize JavaScript/JSX source. Returns a generator of tokens.
Function	`unquote_string`	Unquote a string with JavaScript rules. The string has to start with string delimiters (``'``, ``"`` or the back-tick/grave accent (for template strings).)
Variable	`division_re`	Undocumented
Variable	`dotted_name_re`	Undocumented
Variable	`escapes`	Undocumented
Variable	`hex_escape_re`	Undocumented
Variable	`line_join_re`	Undocumented
Variable	`line_re`	Undocumented
Variable	`name_re`	Undocumented
Variable	`operators`	Undocumented
Variable	`regex_re`	Undocumented
Variable	`uni_escape_re`	Undocumented
Variable	`_rules`	Undocumented

def get_rules(jsx: bool, dotted: bool, template_string: bool) -> list[tuple[str|None, re.Pattern[str]]]: (source) ¶

Get a tokenization rule list given the passed syntax options. Internal to this module.

def indicates_division(token: Token) -> bool: (source) ¶

A helper function that helps the tokenizer to decide if the current token may be followed by a division operator.

def tokenize(source: str, jsx: bool = True, dotted: bool = True, template_string: bool = True, lineno: int = 1) -> Generator[Token, None, None]: (source) ¶

Tokenize JavaScript/JSX source. Returns a generator of tokens. :param jsx: Enable (limited) JSX parsing. :param dotted: Read dotted names as single name token. :param template_string: Support ES6 template strings :param lineno: starting line number (optional)

def unquote_string(string: str) -> str: (source) ¶

Unquote a string with JavaScript rules. The string has to start with string delimiters (``'``, ``"`` or the back-tick/grave accent (for template strings).)

division_re = (source) ¶

Undocumented

dotted_name_re = (source) ¶

Undocumented

escapes: dict[str, str] = (source) ¶

Undocumented

hex_escape_re = (source) ¶

Undocumented

line_join_re = (source) ¶

Undocumented

line_re = (source) ¶

Undocumented