Benchmarks¶
The client recomputes diagnostics on every change, so the measure is how long the editor waits, against two thresholds: 16 ms (one frame) and 100 ms (the limit of “reacting instantly”). The number to read is the budget: how many records a document can hold and still stay under 100 ms.
Method¶
benches/bench_diagnostics.py times compute_diagnostics on valid
documents of 1, 25, 100 and 500 records and on a malformed document, and
times the on-demand calls (completion, hover, pain001.corpus.list):
python benches/bench_diagnostics.py # the table
python benches/bench_diagnostics.py --json # machine-readable
python benches/bench_diagnostics.py --quick # what CI runs
Nothing asserts a threshold: wall-clock time is not comparable between
machines. CI runs --quick so the benchmark keeps working against the
current API.
Results¶
Recorded for 0.0.64 (2026-08-29): 0.08 ms per record, a budget of roughly 1,200 records.
Run on 2026-10-21 for 0.0.74 (Apple A18 Pro, Python 3.12.15, pain001 0.0.72) on a machine with a load average above 30, so the absolute numbers are inflated; the shape is what carries over:
Records |
KiB |
Valid document |
Malformed document |
|---|---|---|---|
1 |
0.8 |
0.27 ms |
0.01 ms |
25 |
19.2 |
4.85 ms |
0.01 ms |
100 |
76.9 |
16.91 ms |
0.01 ms |
500 |
384.3 |
113.85 ms |
0.01 ms |
On-demand call |
Time |
|---|---|
|
0.013 ms |
|
under 0.001 ms |
|
16.2 ms |
Cost grows linearly with the number of records, and a document that does not parse returns at once, which matters because that is the state a document is in for most of the time someone is typing.