Life of an mdtest

Red Knot uses mdtests extensively to test the type checker.

The ‘red_knot_test’ crate is the harness that parses a Markdown file, extracts all of test cases in the file, and runs them.

The ‘red_knot_python_semantic’ crate has an ‘mdtest’ test suite that uses this harness to run all of the tests in every markdown file in the ‘resources/mdtest’ directory. (It uses the ‘dir-test’ crate to handle the “run a Rust test for each file in a directory” part.)

High-level steps

For each file in the test:

• Parse the file via the ‘ruff_db’ and ‘ruff_python_parser’ crates, returning a ‘ruff_python_ast::ModModule’, which corresponds to the ‘Module’ variant of the ‘mod’ rule in the Python abstract grammar.

• Check the types of everything in the module via the ‘red_knot_python_semantic::check_types’ function.

• The above two steps produce a list of diagnostics, which are the warning and errors messages that would be printed out by the CLI type checker. If there are any assertion comments in the test file, use the ‘red_knot_test::matcher’ module to verify that the diagnostics match the assertions.

Checking types

To “check the types” of a module:

• Generate a “semantic index” for the file, which is a collection of all of the scopes in the file, along with some useful information about each one. (Primarily a symbol table and a use/def map)

• For each scope in the file, infer the types of everything in the scope. This checks the types along the way, generating diagnostics for any warnings or errors.

Note that we don't pass the parsed representation of the file into ‘check_types’. The ‘semantic_index’ function calls ‘parsed_module’ again, relying on salsa caching to return the already parsed AST.