Benchmarks

This server is a thin shell over the acmt001 library: it validates arguments, calls the library, and marshals a result. The library is benchmarked in its own repository, so re-measuring schema validation here would only measure that again. What belongs here is the shape an agent experiences — which tool costs what, and how that changes with the size of the batch it was handed.

Running it

python benches/bench_tool_dispatch.py           # full run
python benches/bench_tool_dispatch.py --quick   # what CI runs
python benches/bench_tool_dispatch.py --json    # machine-readable

CI runs --quick. That is not a timing gate — wall-clock is not comparable between runners, and a flaky performance gate teaches people to ignore red. It exists so a benchmark that has stopped compiling against the current API fails the build rather than rotting into a file that reads as verified and is not.

What it measures

The dispatch floor. list_message_types touches no records, so its cost is pure shell — argument handling and return marshalling. It lands around a microsecond, which is the price of every call before any work happens.

Metadata lookups. get_required_fields and get_input_schema are what an agent calls while working out how to build a request. Both land in the low hundreds of microseconds.

The two batch tools, side by side. validate_records and generate_message on the same growing input. Reading them together is the point, because they do not scale alike.

The asymmetry worth knowing

validate_records checks every record against the schema, so it is linear in batch size. generate_message, for a single-account message type like the default acmt.007.001.05, renders only the first record — twenty-seven of the thirty-four templates work this way.

So an agent that validates a hundred records and then generates from them pays for a hundred validations and receives one message. On a 2026 laptop that is roughly 260 ms of validation against 6 ms of generation.

This is correct ISO 20022 behaviour — an Account Opening Request describes one account — but it is a genuine trap when batching. The benchmark prints output size beside the timings for exactly this reason: flat bytes across growing input is what tells you the rest of the batch was not rendered.

If you need one message per account, call generate_message once per record. If you want a message type that genuinely batches, list_message_types marks the seven that iterate.

Not measured

verify_lei_online makes a network call. Timing somebody else’s DNS and TLS handshake would say nothing about this code, so it is left out.