mypy.build

module documentation
(source)

Facilities to analyze entire programs, including imported modules. Parse and analyze the source files of a program in the correct order (based on file dependencies), and collect the results. This module only directs a build, which is performed in multiple passes per file. The individual passes are implemented in separate modules. The function build() is the main interface to this module.

Class BuildManager This class holds shared state for building a mypy program.
Class BuildResult The result of a successful build.
Class CacheMeta Undocumented
Class NodeInfo Some info about a node in the graph of SCCs.
Class State The state for a module.
Exception ModuleNotFound Control flow exception to signal that a module was not found.
Function add_catch_all_gitignore Add catch-all .gitignore to an existing directory.
Function build Analyze a program.
Function cache_meta_from_dict Build a CacheMeta object from a json metadata dictionary
Function compute_hash Undocumented
Function create_metastore Create the appropriate metadata store.
Function default_data_dir Returns directory containing typeshed directory.
Function delete_cache Delete cache files for a module.
Function deps_filtered Filter dependencies for id with pri < pri_max.
Function deps_to_json Undocumented
Function dispatch Undocumented
Function dump_graph Dump the graph as a JSON string to stdout.
Function dump_line_checking_stats Dump per-line expression type checking stats.
Function dump_timing_stats Dump timing stats for each file in the given graph.
Function exclude_from_backups Exclude the directory from various archives and backups supporting CACHEDIR.TAG.
Function exist_added_packages Find if there are any newly added packages that were previously suppressed.
Function find_cache_meta Find cache data for a module.
Function find_config_file_line_number Return the approximate location of setting_name within mypy config file.
Function find_module_and_diagnose Find a module by name, respecting follow_imports and producing diagnostics.
Function find_module_simple Find a filesystem path for module `id` or `None` if not found.
Function find_module_with_reason Find a filesystem path for module `id` or the reason it can't be found.
Function generate_deps_for_cache Generate fine-grained dependencies into a form suitable for serializing.
Function get_cache_names Return the file names for the cache files.
Function import_priority Compute import priority from an import node.
Function in_partial_package Check if a missing module can potentially be a part of a package.
Function invert_deps Splits fine-grained dependencies based on the module of the trigger.
Function is_silent_import_module Undocumented
Function json_dumps Undocumented
Function load_graph Given some source files, load the full dependency graph.
Function load_plugins Load all configured plugins.
Function load_plugins_from_config Load all configured plugins.
Function log_configuration Output useful configuration information to LOG and TRACE
Function missing_stubs_file Undocumented
Function module_not_found Undocumented
Function normpath Convert path to absolute; but to relative in bazel mode.
Function order_ascc Come up with the ideal processing order within an SCC.
Function process_fresh_modules Process the modules in one group of modules from their cached data.
Function process_graph Process everything in dependency order.
Function process_stale_scc Process the modules in one SCC from source code.
Function read_deps_cache Read and validate the fine-grained dependencies cache.
Function read_plugins_snapshot Read cached snapshot of versions and hashes of plugins from previous run.
Function read_quickstart_file Undocumented
Function record_missing_stub_packages Write a file containing missing stub packages.
Function skipping_ancestor Produce an error for an ancestor ignored due to --follow_imports=error
Function skipping_module Produce an error for an import ignored due to --follow_imports=error
Function sorted_components Return the graph's SCCs, topologically sorted by dependencies.
Function strongly_connected_components Compute Strongly Connected Components of a directed graph.
Function take_module_snapshot Take plugin module snapshot by recording its version and hash.
Function topsort Topological sort.
Function validate_meta Checks whether the cached AST of this module can be used.
Function write_cache Write cache files for a module.
Function write_deps_cache Write cache files for fine-grained dependencies.
Function write_plugins_snapshot Write snapshot of versions and hashes of currently active plugins.
Constant CORE_BUILTIN_MODULES Undocumented
Constant DEBUG_FINE_GRAINED Undocumented
Constant DEPS_META_FILE Undocumented
Constant DEPS_ROOT_FILE Undocumented
Constant FAKE_ROOT_MODULE Undocumented
Constant PLUGIN_SNAPSHOT_FILE Undocumented
Constant PRI_ALL Undocumented
Constant PRI_HIGH Undocumented
Constant PRI_INDIRECT Undocumented
Constant PRI_LOW Undocumented
Constant PRI_MED Undocumented
Constant PRI_MYPY Undocumented
Type Variable T Undocumented
Type Alias Graph Undocumented
Variable FgDepMeta Undocumented
Function _build Undocumented
Function _cache_dir_prefix Get current cache directory (or file if id is given).
Function _load_json_file A simple helper to read a JSON file with logging.
def add_catch_all_gitignore(target_dir: str): (source)

Add catch-all .gitignore to an existing directory. No-op if the .gitignore already exists.

def build(sources: list[BuildSource], options: Options, alt_lib_path: str|None = None, flush_errors: Callable[[list[str], bool], None]|None = None, fscache: FileSystemCache|None = None, stdout: TextIO|None = None, stderr: TextIO|None = None, extra_plugins: Sequence[Plugin]|None = None) -> BuildResult: (source)

Analyze a program. A single call to build performs parsing, semantic analysis and optionally type checking for the program *and* all imported modules, recursively. Return BuildResult if successful or only non-blocking errors were found; otherwise raise CompileError. If a flush_errors callback is provided, all error messages will be passed to it and the errors and messages fields of BuildResult and CompileError (respectively) will be empty. Otherwise those fields will report any error messages. Args: sources: list of sources to build options: build options alt_lib_path: an additional directory for looking up library modules (takes precedence over other directories) flush_errors: optional function to flush errors after a file is processed fscache: optionally a file-system cacher

def cache_meta_from_dict(meta: dict[str, Any], data_json: str) -> CacheMeta: (source)

Build a CacheMeta object from a json metadata dictionary Args: meta: JSON metadata read from the metadata cache file data_json: Path to the .data.json file containing the AST trees

def compute_hash(text: str) -> str: (source)

Undocumented

def create_metastore(options: Options) -> MetadataStore: (source)

Create the appropriate metadata store.

def default_data_dir() -> str: (source)

Returns directory containing typeshed directory.

def delete_cache(id: str, path: str, manager: BuildManager): (source)

Delete cache files for a module. The cache files for a module are deleted when mypy finds errors there. This avoids inconsistent states with cache files from different mypy runs, see #4043 for an example.

def deps_filtered(graph: Graph, vertices: AbstractSet[str], id: str, pri_max: int) -> list[str]: (source)

Filter dependencies for id with pri < pri_max.

def deps_to_json(x: dict[str, set[str]]) -> str: (source)

Undocumented

def dispatch(sources: list[BuildSource], manager: BuildManager, stdout: TextIO) -> Graph: (source)

Undocumented

def dump_graph(graph: Graph, stdout: TextIO|None = None): (source)

Dump the graph as a JSON string to stdout. This copies some of the work by process_graph() (sorted_components() and order_ascc()).

def dump_line_checking_stats(path: str, graph: Graph): (source)

Dump per-line expression type checking stats.

def dump_timing_stats(path: str, graph: Graph): (source)

Dump timing stats for each file in the given graph.

def exclude_from_backups(target_dir: str): (source)

Exclude the directory from various archives and backups supporting CACHEDIR.TAG. If the CACHEDIR.TAG file exists the function is a no-op.

def exist_added_packages(suppressed: list[str], manager: BuildManager, options: Options) -> bool: (source)

Find if there are any newly added packages that were previously suppressed. Exclude everything not in build for follow-imports=skip.

def find_cache_meta(id: str, path: str, manager: BuildManager) -> CacheMeta|None: (source)

Find cache data for a module. Args: id: module ID path: module path manager: the build manager (for pyversion, log/trace, and build options) Returns: A CacheMeta instance if the cache data was found and appears valid; otherwise None.

def find_config_file_line_number(path: str, section: str, setting_name: str) -> int: (source)

Return the approximate location of setting_name within mypy config file. Return -1 if can't determine the line unambiguously.

def find_module_and_diagnose(manager: BuildManager, id: str, options: Options, caller_state: State|None = None, caller_line: int = 0, ancestor_for: State|None = None, root_source: bool = False, skip_diagnose: bool = False) -> tuple[str, str]: (source)

Find a module by name, respecting follow_imports and producing diagnostics. If the module is not found, then the ModuleNotFound exception is raised. Args: id: module to find options: the options for the module being loaded caller_state: the state of the importing module, if applicable caller_line: the line number of the import ancestor_for: the child module this is an ancestor of, if applicable root_source: whether this source was specified on the command line skip_diagnose: skip any error diagnosis and reporting (but ModuleNotFound is still raised if the module is missing) The specified value of follow_imports for a module can be overridden if the module is specified on the command line or if it is a stub, so we compute and return the "effective" follow_imports of the module. Returns a tuple containing (file path, target's effective follow_imports setting)

def find_module_simple(id: str, manager: BuildManager) -> str|None: (source)

Find a filesystem path for module `id` or `None` if not found.

def find_module_with_reason(id: str, manager: BuildManager) -> ModuleSearchResult: (source)

Find a filesystem path for module `id` or the reason it can't be found.

def generate_deps_for_cache(manager: BuildManager, graph: Graph) -> dict[str, dict[str, set[str]]]: (source)

Generate fine-grained dependencies into a form suitable for serializing. This does a couple things: 1. Splits fine-grained deps based on the module of the trigger 2. For each module we generated fine-grained deps for, load any previous deps and merge them in. Returns a dictionary from module ids to all dependencies on that module. Dependencies not associated with a module in the build will be associated with the nearest parent module that is in the build, or the fake module FAKE_ROOT_MODULE if none are.

def get_cache_names(id: str, path: str, options: Options) -> tuple[str, str, str|None]: (source)

Return the file names for the cache files. Args: id: module ID path: module path cache_dir: cache directory pyversion: Python version (major, minor) Returns: A tuple with the file names to be used for the meta JSON, the data JSON, and the fine-grained deps JSON, respectively.

def import_priority(imp: ImportBase, toplevel_priority: int) -> int: (source)

Compute import priority from an import node.

def in_partial_package(id: str, manager: BuildManager) -> bool: (source)

Check if a missing module can potentially be a part of a package. This checks if there is any existing parent __init__.pyi stub that defines a module-level __getattr__ (a.k.a. partial stub package).

def invert_deps(deps: dict[str, set[str]], graph: Graph) -> dict[str, dict[str, set[str]]]: (source)

Splits fine-grained dependencies based on the module of the trigger. Returns a dictionary from module ids to all dependencies on that module. Dependencies not associated with a module in the build will be associated with the nearest parent module that is in the build, or the fake module FAKE_ROOT_MODULE if none are.

def is_silent_import_module(manager: BuildManager, path: str) -> bool: (source)

Undocumented

def json_dumps(obj: Any, debug_cache: bool) -> str: (source)

Undocumented

def load_graph(sources: list[BuildSource], manager: BuildManager, old_graph: Graph|None = None, new_modules: list[State]|None = None) -> Graph: (source)

Given some source files, load the full dependency graph. If an old_graph is passed in, it is used as the starting point and modified during graph loading. If a new_modules is passed in, any modules that are loaded are added to the list. This is an argument and not a return value so that the caller can access it even if load_graph fails. As this may need to parse files, this can raise CompileError in case there are syntax errors.

def load_plugins(options: Options, errors: Errors, stdout: TextIO, extra_plugins: Sequence[Plugin]) -> tuple[Plugin, dict[str, str]]: (source)

Load all configured plugins. Return a plugin that encapsulates all plugins chained together. Always at least include the default plugin (it's last in the chain). The second return value is a snapshot of versions/hashes of loaded user plugins (for cache validation).

def load_plugins_from_config(options: Options, errors: Errors, stdout: TextIO) -> tuple[list[Plugin], dict[str, str]]: (source)

Load all configured plugins. Return a list of all the loaded plugins from the config file. The second return value is a snapshot of versions/hashes of loaded user plugins (for cache validation).

def log_configuration(manager: BuildManager, sources: list[BuildSource]): (source)

Output useful configuration information to LOG and TRACE

def missing_stubs_file(cache_dir: str) -> str: (source)

Undocumented

def module_not_found(manager: BuildManager, line: int, caller_state: State, target: str, reason: ModuleNotFoundReason): (source)

Undocumented

def normpath(path: str, options: Options) -> str: (source)

Convert path to absolute; but to relative in bazel mode. (Bazel's distributed cache doesn't like filesystem metadata to end up in output files.)

def order_ascc(graph: Graph, ascc: AbstractSet[str], pri_max: int = PRI_ALL) -> list[str]: (source)

Come up with the ideal processing order within an SCC. Using the priorities assigned by all_imported_modules_in_file(), try to reduce the cycle to a DAG, by omitting arcs representing dependencies of lower priority. In the simplest case, if we have A <--> B where A has a top-level "import B" (medium priority) but B only has the reverse "import A" inside a function (low priority), we turn the cycle into a DAG by dropping the B --> A arc, which leaves only A --> B. If all arcs have the same priority, we fall back to sorting by reverse global order (the order in which modules were first encountered). The algorithm is recursive, as follows: when as arcs of different priorities are present, drop all arcs of the lowest priority, identify SCCs in the resulting graph, and apply the algorithm to each SCC thus found. The recursion is bounded because at each recursion the spread in priorities is (at least) one less. In practice there are only a few priority levels (less than a dozen) and in the worst case we just carry out the same algorithm for finding SCCs N times. Thus the complexity is no worse than the complexity of the original SCC-finding algorithm -- see strongly_connected_components() below for a reference.

def process_fresh_modules(graph: Graph, modules: list[str], manager: BuildManager): (source)

Process the modules in one group of modules from their cached data. This can be used to process an SCC of modules This involves loading the tree from JSON and then doing various cleanups.

def process_graph(graph: Graph, manager: BuildManager): (source)

Process everything in dependency order.

def process_stale_scc(graph: Graph, scc: list[str], manager: BuildManager): (source)

Process the modules in one SCC from source code. Exception: If quick_and_dirty is set, use the cache for fresh modules.

def read_deps_cache(manager: BuildManager, graph: Graph) -> dict[str, FgDepMeta]|None: (source)

Read and validate the fine-grained dependencies cache. See the write_deps_cache documentation for more information on the details of the cache. Returns None if the cache was invalid in some way.

def read_plugins_snapshot(manager: BuildManager) -> dict[str, str]|None: (source)

Read cached snapshot of versions and hashes of plugins from previous run.

def read_quickstart_file(options: Options, stdout: TextIO) -> dict[str, tuple[float, int, str]]|None: (source)

Undocumented

def record_missing_stub_packages(cache_dir: str, missing_stub_packages: set[str]): (source)

Write a file containing missing stub packages. This allows a subsequent "mypy --install-types" run (without other arguments) to install missing stub packages.

def skipping_ancestor(manager: BuildManager, id: str, path: str, ancestor_for: State): (source)

Produce an error for an ancestor ignored due to --follow_imports=error

def skipping_module(manager: BuildManager, line: int, caller_state: State|None, id: str, path: str): (source)

Produce an error for an import ignored due to --follow_imports=error

def sorted_components(graph: Graph, vertices: AbstractSet[str]|None = None, pri_max: int = PRI_ALL) -> list[AbstractSet[str]]: (source)

Return the graph's SCCs, topologically sorted by dependencies. The sort order is from leaves (nodes without dependencies) to roots (nodes on which no other nodes depend). This works for a subset of the full dependency graph too; dependencies that aren't present in graph.keys() are ignored.

def strongly_connected_components(vertices: AbstractSet[str], edges: dict[str, list[str]]) -> Iterator[set[str]]: (source)

Compute Strongly Connected Components of a directed graph. Args: vertices: the labels for the vertices edges: for each vertex, gives the target vertices of its outgoing edges Returns: An iterator yielding strongly connected components, each represented as a set of vertices. Each input vertex will occur exactly once; vertices not part of a SCC are returned as singleton sets. From http://code.activestate.com/recipes/578507/.

def take_module_snapshot(module: types.ModuleType) -> str: (source)

Take plugin module snapshot by recording its version and hash. We record _both_ hash and the version to detect more possible changes (e.g. if there is a change in modules imported by a plugin).

def topsort(data: dict[T, set[T]]) -> Iterable[set[T]]: (source)

Topological sort. Args: data: A map from vertices to all vertices that it has an edge connecting it to. NOTE: This data structure is modified in place -- for normalization purposes, self-dependencies are removed and entries representing orphans are added. Returns: An iterator yielding sets of vertices that have an equivalent ordering. Example: Suppose the input has the following structure: {A: {B, C}, B: {D}, C: {D}} This is normalized to: {A: {B, C}, B: {D}, C: {D}, D: {}} The algorithm will yield the following values: {D} {B, C} {A} From http://code.activestate.com/recipes/577413/.

def validate_meta(meta: CacheMeta|None, id: str, path: str|None, ignore_all: bool, manager: BuildManager) -> CacheMeta|None: (source)

Checks whether the cached AST of this module can be used. Returns: None, if the cached AST is unusable. Original meta, if mtime/size matched. Meta with mtime updated to match source file, if hash/size matched but mtime/path didn't.

def write_cache(id: str, path: str, tree: MypyFile, dependencies: list[str], suppressed: list[str], dep_prios: list[int], dep_lines: list[int], old_interface_hash: str, source_hash: str, ignore_all: bool, manager: BuildManager) -> tuple[str, CacheMeta|None]: (source)

Write cache files for a module. Note that this mypy's behavior is still correct when any given write_cache() call is replaced with a no-op, so error handling code that bails without writing anything is okay. Args: id: module ID path: module path tree: the fully checked module data dependencies: module IDs on which this module depends suppressed: module IDs which were suppressed as dependencies dep_prios: priorities (parallel array to dependencies) dep_lines: import line locations (parallel array to dependencies) old_interface_hash: the hash from the previous version of the data cache file source_hash: the hash of the source code ignore_all: the ignore_all flag for this module manager: the build manager (for pyversion, log/trace) Returns: A tuple containing the interface hash and CacheMeta corresponding to the metadata that was written (the latter may be None if the cache could not be written).

def write_deps_cache(rdeps: dict[str, dict[str, set[str]]], manager: BuildManager, graph: Graph): (source)

Write cache files for fine-grained dependencies. Serialize fine-grained dependencies map for fine grained mode. Dependencies on some module 'm' is stored in the dependency cache file m.deps.json. This entails some spooky action at a distance: if module 'n' depends on 'm', that produces entries in m.deps.json. When there is a dependency on a module that does not exist in the build, it is stored with its first existing parent module. If no such module exists, it is stored with the fake module FAKE_ROOT_MODULE. This means that the validity of the fine-grained dependency caches are a global property, so we store validity checking information for fine-grained dependencies in a global cache file: * We take a snapshot of current sources to later check consistency between the fine-grained dependency cache and module cache metadata * We store the mtime of all of the dependency files to verify they haven't changed

def write_plugins_snapshot(manager: BuildManager): (source)

Write snapshot of versions and hashes of currently active plugins.

CORE_BUILTIN_MODULES: set[str] = (source)

Undocumented

Value
set(['builtins',
     'typing',
     'types',
     'typing_extensions',
     'mypy_extensions',
     '_importlib_modulespec',
     'sys',
...
DEBUG_FINE_GRAINED: bool = (source)

Undocumented

Value
False
DEPS_META_FILE: str = (source)

Undocumented

Value
'@deps.meta.json'
DEPS_ROOT_FILE: str = (source)

Undocumented

Value
'@root.deps.json'
FAKE_ROOT_MODULE: str = (source)

Undocumented

Value
'@root'
PLUGIN_SNAPSHOT_FILE: str = (source)

Undocumented

Value
'@plugins_snapshot.json'
PRI_ALL: int = (source)

Undocumented

Value
99
PRI_HIGH: int = (source)

Undocumented

Value
5
PRI_INDIRECT: int = (source)

Undocumented

Value
30
PRI_LOW: int = (source)

Undocumented

Value
20
PRI_MED: int = (source)

Undocumented

Value
10
PRI_MYPY: int = (source)

Undocumented

Value
25
T = (source)

Undocumented

Value
TypeVar('T')
Graph: _TypeAlias = (source)

Undocumented

Value
Dict[str, State]
FgDepMeta = (source)

Undocumented

def _build(sources: list[BuildSource], options: Options, alt_lib_path: str|None, flush_errors: Callable[[list[str], bool], None], fscache: FileSystemCache|None, stdout: TextIO, stderr: TextIO, extra_plugins: Sequence[Plugin]) -> BuildResult: (source)

Undocumented

def _cache_dir_prefix(options: Options) -> str: (source)

Get current cache directory (or file if id is given).

def _load_json_file(file: str, manager: BuildManager, log_success: str, log_error: str) -> dict[str, Any]|None: (source)

A simple helper to read a JSON file with logging.