Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Architecture Observations

Observations about rsconstruct’s high-level structure — the shapes that determine how the system behaves when you try to change or extend it. Kept separate from suggestions.md (which is tactical features and bugs) because these are about how the code is put together, not about what it does.

Each entry has:

  • A short title naming the pattern or tension.
  • What the current code does.
  • What that implies for changes / extensions / users.
  • Load-bearing: how much of the system this shape dictates. High = touching it ripples everywhere. Low = localized quirk.

The entries are roughly ordered by how much they shape the rest of the codebase.

The central four

1. The graph is the universal coupling point

Every phase — discovery, analysis, classification, execution — reads and/or mutates the BuildGraph. Processors receive &mut BuildGraph in their discover() method and are trusted to add products correctly. There’s no invariant enforcement at insertion time: empty inputs are allowed, bad dep references are allowed, duplicate outputs are caught but duplicate inputs aren’t. Cycles are only detected during topological sort, late.

The graph’s shape also leaks into the executor: the executor knows about output_dirs (creators), variant (multi-format generators), config_hash (cache keys), and product IDs. Adding a new product category (say, a “phantom” product that exists for scheduling but produces no outputs) requires touching both graph and executor.

Implication: the graph is the lingua franca. Any architectural change that touches the product model — adding fields, changing what counts as a dependency, supporting alternate execution orders — ripples into every consumer. A healthy graph layer would have validation (reject ill-formed products at insertion), opaque access (consumers see a trait-shaped view, not the struct), and observer hooks (something watching mutations so --graph-stats and graph show don’t duplicate traversal logic).

Load-bearing: very high.

Every processor and analyzer submits an inventory::submit! entry. The registry is populated at binary link time, and enumeration is a runtime iteration over those entries. This is elegant for modularity — adding a processor means adding one file, no central list to update — but it has consequences:

  • No compile-time enumeration: you can’t write a match statement over all processor names, so the processor-count gets rediscovered on every run, and static checks (e.g. “every processor has a corresponding config struct”) have to be runtime assertions.
  • Lua plugins are second-class: they arrive at runtime after the static registry is frozen. The registry API has to tolerate two populations (static + dynamic) in parallel, which is why find_registry_entry and find_analyzer_plugin have to fall through both.
  • Ordering is alphabetical everywhere: because inventory doesn’t preserve submission order, every code path that touches plugins has to sort by name. This is a minor tax but it’s baked in everywhere.
  • Testing requires the whole binary: you can’t instantiate a stripped-down registry for tests; they pull the full set. Most tests don’t mind, but ones that want a controlled plugin set have to filter rather than inject.

Implication: the registration model favors modularity over introspectability. If rsconstruct ever wants a “declarative build” representation (think Bazel’s static action graph) the plugin layer will have to expose more schema information than it does today.

Load-bearing: high.

3. Config defaults are scattered, not composed — PARTIALLY ADDRESSED

Three sources of defaults apply in sequence:

  1. Per-processor defaults (e.g. ruff → command = "ruff") in a giant match-or-registry lookup.
  2. Scan defaults (src_dirs, src_extensions) via a separate mechanism (ScanDefaultsData).
  3. User TOML overrides both.

The order matters, but it’s encoded across apply_processor_defaults, apply_scan_defaults, and the serde deserialization.

Update: config provenance tracking (src/config/provenance.rs) now records where each field came from (UserToml { line }, ProcessorDefault, ScanDefault, OutputDirDefault, SerdeDefault). rsconstruct config show annotates every field with its source. The defaults pipeline still applies layers across multiple functions, but the provenance map makes it possible to answer “where did this value come from?” without tracing the code.

The remaining gap: adding a new defaults layer (env-derived, user-global) still means inserting into the existing function chain rather than a declarative resolver.

Load-bearing: medium.

4. The executor owns too much policy — RESOLVED

Update: a BuildPolicy trait has been extracted to src/executor/policy.rs. classify_products now delegates per-product decisions to &dyn BuildPolicy. IncrementalPolicy implements the current skip/restore/rebuild logic. Alternate policies (dry-run, always-rebuild, time-windowed) are now a single trait implementation away — no executor changes needed.

Load-bearing: very high, but the tension is resolved.

Structural tensions

5. Processor trait assumes StandardConfig, but allows bypass

The Processor trait has a scan_config() -> &StandardConfig method that every processor must implement. The default implementations of discover(), auto_detect(), and supports_batch() use this config. But processors with richer configs (e.g. ClippyConfig, CcConfig) don’t expose those richer fields through the trait — they store them privately and access them internally. The outside world only sees StandardConfig.

Implication: there’s no way to ask “what config does processor X accept?” through the trait. Introspection goes through the registry (known_fields, must_fields, field_descriptions) instead, which means the processor has to register the metadata separately from implementing the trait. The two representations can drift: someone adds a field to ClippyConfig and forgets to add it to known_fields.

A healthier shape would have one source of truth per processor — the config struct itself — with a derive macro or trait-based reflection generating the known_fields list. Or go the other direction: make the trait parameterized (Processor<Config>) so introspection goes through the type system.

Load-bearing: medium. Doesn’t break anything today but is the root cause of several “remembered to update both places?” bugs we’ve fixed.

6. Analyzers are inputs-only; they can’t add products

DepAnalyzer::analyze() walks existing products and adds inputs to them. It cannot:

  • Create new products (the cpp analyzer can’t spawn a product for a header it discovered).
  • Remove products.
  • Change processor assignments.

This is a deliberate simplification — analyzers run in a single pass after discovery and don’t need fixed-point semantics of their own. But it means the “dependency graph” isn’t really discovered by analyzers; it’s refined by them. The actual discovery of what exists lives entirely in processors.

Implication: if a use case arises where an analyzer legitimately needs to produce a product — e.g. “for every .proto import I find, ensure there’s a product for generating the .pb.cc” — the analyzer interface doesn’t support it. You’d have to turn the analyzer into a processor, or add a “synthesize” callback. The asymmetry between processors (can add products) and analyzers (can only add inputs) is currently invisible but will bite eventually.

Load-bearing: medium. Not a bug, but a limitation that shapes what kinds of features are easy vs. hard.

7. Processor instance ↔ typed processor mapping is one-way — PARTIALLY ADDRESSED

A ProcessorInstance in the config holds (type_name, instance_name, config_toml). Builder::create_processors() deserializes the TOML and produces a Box<dyn Processor>. Afterwards, the TOML blob is discarded.

Update: ProcessorInstance now carries a provenance: ProvenanceMap that records where each field came from (user TOML with line number, processor default, scan default, etc.). This means config show can annotate fields with their source without reparsing TOML, and smart commands can distinguish user-set from defaulted fields.

The remaining gap: a running Box<dyn Processor> still can’t navigate back to its ProcessorInstance or the originating TOML section. The provenance lives on the config side, not the runtime processor side.

Load-bearing: medium.

8. Global state in the processor runtime — RESOLVED

Update: all mutable process globals have been moved into BuildContext (src/build_context.rs):

  • The three processor globals (INTERRUPTED, RUNTIME, INTERRUPT_SENDER) are replaced and deleted. run_command takes &BuildContext explicitly.
  • The three checksum globals (CACHE, MTIME_DB, MTIME_ENABLED) are moved into BuildContext. combined_input_checksum and checksum_fast take &BuildContext.

Remaining process-wide state is all immutable or correctly scoped:

  • RuntimeFlags — immutable after startup, doesn’t vary between contexts.
  • DECLARED_TOOLS — thread_local!, debug-only.
  • Compiled regexes — LazyLock<Regex>, stateless.

Load-bearing: resolved. Multiple BuildContext instances can now run independently (daemon mode, LSP, testing).

Broader patterns

9. Supply-driven model everywhere

The whole pipeline — discover, classify, execute — walks every product unconditionally. There’s no demand-driven path (like make foo which visits only the subgraph producing foo). The --target <glob> flag filters after discovery; it doesn’t trim the work that discovery itself does.

This is a deliberate design — rsconstruct’s typical workload is “build everything incrementally,” and supply-driven matches that well. But it means a user asking “just build X” still pays the cost of discovering all 5000 other products.

Implication: for projects at a certain scale, or for tooling that wants to quickly answer “which products would I run for this file?” (IDE integration, pre-commit hooks), the supply-driven model becomes a bottleneck. A demand-driven shortcut would require either pre-built reverse indexes (input path → product) persisted between runs, or an analytical model of each processor’s output paths (hard — processor output is computed procedurally).

Load-bearing: very high. Changing this means a fundamentally different build-system shape.

10. “Run on every build” is the default stance

Every configured processor discovers and classifies on every invocation. There’s no concept of “processor X is slow, only run when asked.” The -p/-x mechanism works per-invocation but not as a declarative property. See suggestions.md for the proposed build_by_default = false pattern — that’s a tactical fix. The architectural observation is that rsconstruct’s model biases hard toward “all processors together,” whereas the user mental model often has lifecycle phases (lint vs. package vs. deploy).

Implication: adding a “goals” layer (cargo-style subcommands, or npm-style named scripts) is a natural extension direction. It would introduce a new concept — a goal is a named selection of processors — and likely requires CLI reorganization. Bigger than it sounds.

Load-bearing: medium. Shapes the CLI surface and user mental model.

11. Object store as a multi-responsibility module — RESOLVED

Update: ObjectStore has been decomposed into focused submodules:

  • blobs.rs — content-addressed blob storage (store, read, restore, checksum)
  • descriptors.rs — cache descriptor CRUD (store_marker, store_blob, store_tree)
  • restore.rs — cache query and restoration (restore_from_descriptor, needs_rebuild, can_restore, explain)
  • management.rs — cache management (size, trim, remove_stale, list, stats)
  • operations.rs — remote cache push/fetch
  • config_diff.rs — processor config change tracking

mod.rs went from ~664 to ~223 lines (struct definition, types, constructor). Each concern is now a focused 100–150 line file.

Load-bearing: very high, but the monolith is resolved.

What’s absent that one might expect

12. No abstraction for “tool invocation”

Every processor that shells out to a subprocess rolls its own Command building: env vars, arg construction, timeout, output capture, error classification. Shared helpers (run_command, check_command_output) exist but are minimal. Processor implementations still have to know about:

  • How to pass files (positional args vs. --file=X vs. stdin vs. response file when argv is too long).
  • How to interpret exit codes (some tools return 1 for “found issues”, some return 0 and print to stderr, some return 2 for config errors).
  • How to parse output for structured errors.

Implication: processor implementations have roughly 30-80 lines of boilerplate each, and they’re inconsistent. A ToolInvocation abstraction with pluggable arg-passing strategies would shrink most processors to a few lines of declaration. This also makes adding a new processor harder than it needs to be.

Load-bearing: medium.

13. No pluggable reporting / event stream

Today reporting is hardcoded: println! during execution, colored summary at the end, --json mode emits structured events, --trace emits Chrome tracing format. Each reporting path is a separate code path threading through the executor.

Implication: adding a new output format (JUnit XML for CI, GitHub Actions annotations, custom Slack webhook) means threading another code path through the executor. A proper event-bus model — executor emits events, subscribers render them — would make this a two-file change (subscribe + format).

Load-bearing: medium.

14. No formal dry-run execution

There’s --stop-after classify, which stops after classification, and there’s dry_run() (different from --dry-run which is a flag on build), and there’s --explain which annotates per-product decisions. Three partially-overlapping mechanisms. The user-facing story is “to see what would happen, use X or Y or Z depending on what you want.”

Implication: these evolved separately. A unified “simulation mode” that fully runs the classify pipeline and outputs what would happen — including what cache entries would be produced — would subsume the three. Likely a small refactor, but requires aligning on the output shape.

Load-bearing: low-medium.

Summary of architectural recommendations

All four highest-leverage refactors are now complete:

  1. Extract a BuildPolicy trait from the executor — done. classify_products delegates per-product skip/restore/rebuild decisions to a &dyn BuildPolicy. IncrementalPolicy implements the current logic. Future policies (dry-run, always-rebuild, time-windowed) are a single trait impl. See src/executor/policy.rs.
  2. Decompose ObjectStore — done. mod.rs split from 664 → 223 lines into focused submodules: blobs.rs (content-addressed storage), descriptors.rs (cache descriptor CRUD), restore.rs (restore/needs_rebuild/can_restore/explain). Existing management.rs, operations.rs, config_diff.rs unchanged.
  3. Consolidate config resolution with provenance tracking — done. Config fields now carry FieldProvenance (user TOML with line number, processor default, scan default, serde default). config show annotates every field with its source. See src/config/provenance.rs.
  4. Introduce a BuildContext struct replacing process globals — done. The three process globals (INTERRUPTED, RUNTIME, INTERRUPT_SENDER) are replaced by a BuildContext struct threaded through the Processor trait, executor, analyzers, and remote cache. See src/build_context.rs.

Entries 3, 7, and 8 are partially addressed — the core issues are resolved but minor gaps remain (see individual entries above).

Entries 1, 2, 5, 6, 9, 10, 12, 13, 14 are observations about the code’s shape — not necessarily problems to fix, but constraints a new contributor should understand before making structural changes.

The technical observations (code duplication in discovery helpers, dead fields in ProcessorPlugin, scattered error handling) are recorded in suggestions.md as tactical items.