📖 How to Read the Results
Status Indicators
| Status | Meaning | Threshold |
|---|---|---|
| ✅ pass | Performance within expected bounds | Below warning threshold |
| ⚠️ warn | Performance elevated; investigate on regression | Small repos: >⚙️small_repo_warning_ms ms | Large repos: >⚙️large_repo_warning_ms ms | openclaw: >⚙️openclaw_warning_ms ms |
| ❌ critical | Performance regression requiring immediate attention | Small repos: >⚙️small_repo_critical_ms ms | Large repos: >⚙️large_repo_critical_ms ms | openclaw: >⚙️openclaw_critical_ms ms |
Small repo: repos where mean < large-repo warning threshold.
Large repo: repos where mean ≥ large-repo warning threshold.
openclaw has its own thresholds (~14k files, significantly larger than the rest).
🔬 Measurement Methodology
Timing
- Iterations: ⚙️timing_iterations per-repo timing runs, preceded by ⚙️warmup_iterations warmup runs (discarded)
- Binary:
release-with-debugprofile (same optimisation as release, debug symbols for profiling) - Command:
fettl check <repo-root>- full check pass including file I/O and AST parsing - Wall-clock timing: Measured in the benchmark script; includes process startup overhead
Outlier Removal
- Method: ⚙️outlier_detection_method
- Pass 1 threshold: ⚙️outlier_detection_threshold_pass1× MAD - removes extreme outliers (kernel jitter, thermal throttling)
- Pass 2 threshold: ⚙️outlier_detection_threshold_pass2× MAD - removes moderate outliers from the already-cleaned data
- Primary metric: Median (less affected by remaining outliers than the mean; mean also shown for reference)
Corpus
The real-world corpus contains 14 open-source repositories of varying sizes:
small agent frameworks (~50 files) through large codebases like
openclaw (~14,000 files). This range tests both startup-dominated
and I/O-dominated performance profiles.
🔥 Reading Flamegraphs
- Width of each box = total time spent in that function (including callees)
- Y-axis = call stack depth; bottom = on-CPU functions
- Click any box to zoom into that subtree
- Hover to see the full function name and percentage
- Frames are filtered to keep only those containing
fettlin their name, plus all frames beneath them - startup and OS noise are excluded
Flamegraphs are generated using cargo-flamegraph with Linux perf
and inferno. The profiling binary uses release-with-debug profile
for accurate symbol resolution.
📈 CV% (Coefficient of Variation)
CV% = (stdev / mean) × 100. A low CV% indicates stable, consistent
measurements. High CV% (>10%) may indicate:
- System load variation during the benchmark run
- Thermal throttling on sustained workloads
- I/O variability (page cache effects)
Results with high CV% should be treated with caution; re-running benchmarks on a quieter system is recommended.