Continuous benchmarking (Bencher)

rag-rat tracks performance over time with Bencher. Two lightweight performance signals are recorded on every push to main and gated on every pull request; a search-quality signal (commit-replay recall) is tracked on main only; and a heavyweight Linux-kernel indexing signal runs only on a published release:

The push/PR harnesses run on different corpus sizes on purpose — see “Corpus” below.

Benches

Both benches live in crates/rag-rat-core/benches/ and share a corpus harness (benches/shared/mod.rs, kept in a subdirectory so cargo doesn’t auto-detect it as a bench):

• rag_pipeline.rs (iai-callgrind, harness = false) — three #[library_benchmark]s: index (rebuild), query_cold (open + search from disk), query_warm (search on an open db).
• index_time.rs (criterion, harness = false) — one full_rebuild benchmark.

Run them locally:

# Instruction counts (needs valgrind + iai-callgrind-runner on PATH — see below)
cargo bench --no-default-features --bench rag_pipeline

# Wall-clock full-index time
cargo bench --no-default-features --bench index_time

--no-default-features builds the hash embedder only — no ONNX / model downloads, no network, fully deterministic. The benches assert nothing about embedding quality; they measure the indexing and retrieval machinery, which must stay model-agnostic to be reproducible in CI.

Local prerequisites for the iai bench

iai-callgrind runs the benched code under valgrind/callgrind and needs a matching runner binary:

sudo apt-get install -y valgrind                     # or your distro's package
cargo install --locked [email protected]   # MUST match the iai-callgrind lib version in Cargo.toml

The criterion bench has no special prerequisites.

Corpus

Both benches index a pinned snapshot of an external repo so the workload is realistic and stable across runs. The corpus is not vendored into git; the harness shallow-clones it on first use and caches it under target/bench-corpus/:

• Repo: rust-lang/cargo, pinned by commit SHA (tag 0.97.1).
• Shallow fetch of just that commit (git init + fetch --depth 1 origin <sha>), so the clone is small and the content can never drift.
• Override the location with RAG_RAT_BENCH_CORPUS=/path/to/checkout to point at an existing checkout (useful offline).

The iai bench indexes a small subtree (src/cargo/core/resolver, ~10 files) because callgrind’s ~50x slowdown applies even to the uncounted setup index builds — a large subtree would time the job out. The criterion bench indexes the whole checkout (~1.3k .rs files) — the realistic “index this repo” workload rag-rat index actually performs — since it runs at native speed; it reports the indexed file_count_by_language and files/sec throughput so the output shows real usage. If you bump the pinned SHA, update the cache key (bench-corpus-cargo-<version>) in all three workflow files too.

Search-quality: commit-replay recall

bench.yml’s eval_recall job tracks how well search retrieves the right code over time, using the commit-replay eval (#120): each recent commit becomes a case (commit message = query, the commit’s changed paths = recall gold), scored against the hybrid (lexical + semantic) index at HEAD. Three measures go to Bencher under the search_replay benchmark:

• recall_at_3 — the headline: |gold ∩ top-3| / |gold|, “did the right chunk land in the first 3 reads”. It carries a statistical lower-boundary t-test threshold (a recall regression is a drop, unlike the latency/instruction signals, which alert on an increase).
• recall_at_10, mrr_at_10 — tracked without a threshold (context for a recall@3 move).

Values are fractions in [0, 1] — the same numbers eval prints — so a Bencher plot reads identically to a local run.

The job builds an embedded index at HEAD (index --discover → models install sentence-transformers/all-MiniLM-L6-v2 → reconcile), then rag-rat --json eval --replay --replay-max-cases 200 → tools/eval-report-bmf.py → bencher run --adapter json. Two footguns it has to handle:

• Off-head ⇒ bogus 0.000. reconcile only embeds at-head chunks, so the index must be built + embedded at HEAD before scoring; a stale/absent index makes every recall read 0.000.
• Shallow checkout ⇒ no cases. The replay walks git history, so the job checks out with fetch-depth: 0 — the default depth-1 checkout would leave it with only HEAD and zero cases.

Main-only, no --err, by design. At ~8–10 min over 200 cases it’s too slow for the PR critical path, and the metric is noisy — the gold is recent commit paths, so a single PR’s recall delta is usually below the per-case noise floor (≥200 cases are needed for signal). So it’s tracked + alerted on main (the statistical threshold flags a real regression in Bencher) rather than hard-failing a run — the same “headline signal, not a gate” stance as the iai/criterion series. Reset the baseline after an embedder or replay-parameter change with Actions → bench → Run workflow (the workflow_dispatch reruns --thresholds-reset).

Run it locally (needs the eval feature + an embedded at-head index):

cargo build --release -p rag-rat --features eval
target/release/rag-rat --json eval --replay --replay-max-cases 200 > eval.json
python3 tools/eval-report-bmf.py eval.json > eval_bmf.json

CI workflows

Four workflows under .github/workflows/:

The pull_request → workflow_run split is Bencher’s documented fork-safe pattern: untrusted fork code runs without secrets, and only the trusted base-repo workflow (which never executes fork code) holds the API token. PR runs use --start-point main --start-point-clone-thresholds --start-point-reset so each PR is measured against a fresh clone of main’s baseline + thresholds.

Release headline: indexing the Linux kernel

tools/bench-kernel.sh (run by bench-release.yml) is the “indexes the Linux kernel in X seconds” benchmark. It shallow-clones a pinned kernel tag (KERNEL_TAG, default v7.0), indexes its C/H sources once with the release rag-rat binary (index --full, model = "none", built --no-default-features so there is no model download), and writes a Bencher Metric Format JSON file with nine measures:

• latency — wall-clock seconds to index, in nanoseconds (Bencher’s built-in Latency measure).
• throughput — indexed files per second.
• memory — peak resident set size in bytes, from getrusage(RUSAGE_CHILDREN).ru_maxrss (the kernel-true peak, catches sub-sample spikes).
• memory_sampled — peak of a 1 Hz /proc/<pid>/status VmRSS time-series (also written to kernel_rss_samples.csv as the before/after curve).
• symbols, edges, resolved_edges, chunks — extracted-fact counts, so a run is comparable per fact and not just per file (the unresolved-edge taxonomy goes to kernel_unresolved_by_kind.csv).
• db_size — on-disk index size in bytes, post-WAL-checkpoint. The structure-only half of the size split (#78): no vectors.

Setting RAG_RAT_KERNEL_VECTORS=1 runs a second pass that installs the hash embedder and embeds every chunk the embedding policy admits (not every chunk yields a vector — the policy skips e.g. generated and low-signal chunks), adding two more measures — db_size_with_vectors and db_size_vectors_delta (the marginal bytes the embedding tier costs). Off by default: at kernel scale it sweeps ~1.6M chunks, and hash-embedder vectors over the kernel are plumbing-validation, not retrieval value. When the pass doesn’t run, both measures are omitted rather than reported as zero. See benchmarks.md.

It’s a single cold rebuild, not a criterion loop: the whole kernel is ~63k C/H files and takes tens of minutes, so criterion’s 10-sample minimum (×10) is a non-starter. One run is also exactly the number a user sees.

Run it locally or trigger it manually before relying on it for a release:

cargo build --release --no-default-features --bin rag-rat
RAG_RAT_BIN=target/release/rag-rat bash tools/bench-kernel.sh   # whole tree (~tens of min, several GB RAM)

# Bound the scope (faster, less memory) while iterating:
RAG_RAT_KERNEL_SUBDIRS="kernel mm fs net lib" RAG_RAT_BIN=target/release/rag-rat bash tools/bench-kernel.sh

Or in CI: Actions → bench-release → Run workflow (the workflow_dispatch input sets the subtree).

Runner sizing. This job runs on the self-hosted [self-hosted, bigmem] runner, not a hosted one — the whole-tree index peaks at ~5.5 GiB (see benchmarks.md), which plus the OS page cache over a ~10 GB index DB is borderline for a hosted runner and, more to the point, would give a noisy, contended wall-clock. If a run OOMs or overruns timeout-minutes on a constrained box, bound RAG_RAT_KERNEL_SUBDIRS to the core subsystems. Validate with a manual dispatch first.

Persistent runner cache retention

CI, evaluation, and PR benchmark jobs finish with runner-cache-cleanup, after tests and artifact uploads. On Hetzner it caps the job’s configured Cargo target at 20 GB using cargo sweep; it never sweeps a sibling runner. Coverage uses its shared cycle lock, and cleanup respects Cargo’s profile locks. Busy locks produce an explicit warning and defer cleanup to the next job. Cleanup failures fail the step. Hosted runners skip this action.

The six-hour runner-maintenance workflow remains a backstop. A busy sibling can make it skip a cache, so a successful maintenance run does not imply every target is below the cap. The per-job step keeps each busy runner bounded between completed jobs without waiting for a host-wide idle window. Hard termination can bypass that step; the scheduled sweep remains necessary. A single build can still grow beyond the target cap while it runs.

Source: docs/bencher.md · synced from cq27-dev/rag-rat · updated 2026-09-14