Methodology

How every number on this site is produced, summarized and checked. The full reference lives in the repository; this page combines it with the parameters recorded in the published data.

Goals

  1. Production-shaped conditions. Services run under the resource limits of a small production pod, against a real database, with realistic request mixes, not only at saturation.
  2. Same work for everyone. Every implementation passes the same functional checks, and the work it does per request is measured, not assumed.
  3. Transparent and reproducible. Every number comes with the machine, the exact versions, the configuration and the per-pass raw values.
  4. Honest uncertainty. Medians with confidence intervals, and quality flags instead of silent filtering.

Environment and resource limits

Each implementation runs alone in its own container with fixed limits. Supporting services (the environment, such as a database) and the load generator run in separate containers on separate cores. All containers share one network namespace and talk over loopback, so no NAT or bridge sits in the request path.

Backend services

Every implementation serves the same API (JSON, single and multiple queries, updates, a list endpoint and a JSON POST) against the same PostgreSQL 18 database. The service gets 2 CPU cores and 1 GiB of memory, like a small production pod; the database and the load generator run on separate cores. Results cover throughput, latency at fixed load, CPU per request, memory, cold start and warm-up.

CPU limit
2 cores
Memory limit
1 GiB
port
8080
Scenarios
12
Metrics
42

CPU layout

The layout is computed from the machine's topology at the start of every run and stored with the results:

  1. The service gets one logical CPU per physical core for as many cores as its CPU limit. The SMT siblings of those cores stay idle, so the service never shares a core's execution units with other work.
  2. The remaining physical cores are split between the environment and the load generator; the load generator gets the larger half because it must never be the bottleneck.
  3. On machines too small for this, roles share CPUs and the run carries a warning in its quality flags.
  1. CPU 0: Service under test
  2. CPU 1: Service under test
  3. CPU 2: Environment (database)
  4. CPU 3: Environment (database)
  5. CPU 4: Load generator
  6. CPU 5: Load generator
  7. CPU 6: Kept idle
  8. CPU 7: Kept idle
  9. CPU 8: Environment (database)
  10. CPU 9: Environment (database)
  11. CPU 10: Load generator
  12. CPU 11: Load generator
  • Service under test 0, 1
  • Environment (database) 2, 8, 3, 9
  • Load generator 4, 10, 5, 11
  • Kept idle 6, 7

SMT is on. The service gets one logical CPU per physical core; the SMT siblings of those cores stay idle.

Backend services, default run on Intel Xeon W-2133 · 6 cores / 12 threads · bare metal.

Scenarios

Scenarios are declared as files in the repository; the list below comes straight from the dataset. Closed-loop scenarios keep a fixed number of connections busy to find the maximum throughput; fixed-rate scenarios send the same request rate to every implementation and measure latency with coordinated-omission correction.

Backend services
Scenario Kind What it does
Cold start startup Start-up Starts a fresh container several times and measures the time until GET /json first succeeds. Also records memory at idle and the image size. GET /json
Cold load cold Cold load GET /db at full load for 10 seconds right after start, without any warm-up. Shows how much a runtime needs to warm up (JIT, caches, pools) after scaling out. GET /db128 connections10s measured1 SELECT per requestcompared with Single query
JSON json Closed loop GET /json returns a small JSON object. Pure framework and serialization overhead, no I/O. GET /json
Single query db Closed loop GET /db reads one random row by primary key and returns it as JSON. GET /db1 SELECT per request
20 queries queries Closed loop GET /queries runs 20 sequential primary-key reads on one connection and returns them as a JSON array. Measures driver and async efficiency. GET /queries20 SELECT per request
Updates updates Closed loop GET /updates reads 5 random rows, changes each value and writes it back with 5 UPDATE statements. GET /updates5 SELECT + 5 UPDATE per request
List list Closed loop GET /list reads a range of 100 rows, adds a computed field to each and serializes the list, like a typical paginated REST endpoint. GET /list1 SELECT per request
JSON POST orders Closed loop POST /orders sends a 2.5 KB order with 50 line items. The service parses, validates and totals it and returns a summary. No database. POST /ordersbody orders.json
Single query, 16 connections db-c16 Closed loop GET /db with only 16 concurrent connections: behaviour at moderate concurrency, where latency rather than queueing dominates. GET /db16 connections1 SELECT per request
Single query, 512 connections db-c512 Closed loop GET /db with 512 concurrent connections: many idle keep-alive connections and deep queues on 2 cores. GET /db512 connections1 SELECT per request
Fixed load (5,000 req/s) rate Fixed rate GET /db at a fixed 5,000 requests per second for every implementation (open loop, latency corrected for coordinated omission). Shows latency at normal production load instead of at saturation. GET /db5,000 req/s fixed64 connections20s measured1 SELECT per request
Soak (5 minutes) soak Soak GET /updates at a fixed 2,000 requests per second for 5 minutes in 30-second slices. Reports p99 drift and memory growth. Optional: runs only when selected with --scenario. GET /updates2,000 req/s fixed64 connections10 × 30s5 SELECT + 5 UPDATE per requestoptional

Metrics

The metric registry defines every value an implementation can be measured on: label, unit, which direction is better and how prominently the site shows it. primary metrics are shown first, secondary ones on request and diagnostic ones mainly explain measurement quality.

Summary 3 metrics
Throughput index index primary
higher is better

Geometric mean of throughput over the six workload scenarios, each relative to the fastest implementation in that scenario of the same run. 100 means fastest in all six; the leader of a run scores below 100 when it does not lead every workload.

Computed: Geometric mean of “Throughput” over JSON, Single query, 20 queries, Updates, List, JSON POST, each relative to the best implementation in that scenario (best = 100). Only an implementation that is best in every one of them reaches 100, and only implementations with a usable result in every one of them get an index.

throughput_index
CPU efficiency index index primary
higher is better

Geometric mean of CPU time per request over the six workload scenarios, each relative to the most efficient implementation in that scenario of the same run. 100 means most efficient in all six.

Computed: Geometric mean of “CPU per request” over JSON, Single query, 20 queries, Updates, List, JSON POST, each relative to the best implementation in that scenario (best = 100). Only an implementation that is best in every one of them reaches 100, and only implementations with a usable result in every one of them get an index.

efficiency_index
Peak memory MiB primary
lower is better

Highest PSS in any scenario of the run.

Computed: Highest median “Memory under load (PSS)” in any scenario.

peak_memory_mib
Throughput 3 metrics
Throughput req/s primary
higher is better

Successful (2xx/3xx) responses per second in the measured window. Failed requests are counted as errors, not as throughput.

rps
Errors requests secondary
lower is better

Socket errors, timeouts and non-2xx/3xx responses. Any error is flagged next to the result.

errors
Error rate ratio diagnostic
lower is better error_rate
Latency 9 metrics
Latency p50 ms secondary
lower is better latency_p50_ms
Latency p75 ms diagnostic
lower is better latency_p75_ms
Latency p90 ms secondary
lower is better latency_p90_ms
Latency p99 ms primary
lower is better latency_p99_ms
Latency p99.9 ms secondary
lower is better latency_p99_9_ms
Latency p99.99 ms diagnostic
lower is better latency_p99_99_ms
Latency mean ms diagnostic
lower is better latency_mean_ms
Latency std. dev. ms diagnostic
lower is better latency_stdev_ms
Latency max ms diagnostic
lower is better latency_max_ms
CPU 4 metrics
CPU per request µs primary
lower is better

Service CPU time per request, read exactly from the container's cgroup. Less hardware-dependent than throughput.

cpu_us_per_req
CPU used cores secondary
context only (neither higher nor lower is better)

Average number of cores the service kept busy (limit: 2).

cpu_cores
User-space CPU share ratio diagnostic
context only (neither higher nor lower is better)

Share of service CPU time spent in user space; the rest is kernel time (network, syscalls).

cpu_user_share
CPU throttling ms diagnostic
lower is better

Time the service was throttled by its CPU quota during the window.

throttled_ms
Memory 4 metrics
Memory under load (PSS) MiB primary
lower is better

Highest proportional set size of all service processes, sampled every second.

mem_pss_peak_mib
Anonymous memory peak MiB diagnostic
lower is better

Heap, stacks and other anonymous memory of the container cgroup (no file cache).

mem_anon_peak_mib
Memory at idle MiB secondary
lower is better

PSS five seconds after start, before any load.

idle_pss_mib
Anonymous memory at idle MiB diagnostic
lower is better idle_anon_mib
Database 3 metrics
SELECTs per request queries diagnostic
context only (neither higher nor lower is better)

Measured with pg_stat_statements. Must match the scenario's specification, otherwise the result is flagged.

db_select_per_req
UPDATEs per request queries diagnostic
context only (neither higher nor lower is better) db_update_per_req
Database CPU per request µs secondary
lower is better

PostgreSQL CPU time per request. Shows how efficiently an implementation talks to the database.

db_cpu_us_per_req
Start-up 4 metrics
Cold start ms primary
lower is better

Median time from container start to the first successful HTTP response.

startup_ms
Cold start (fastest) ms diagnostic
lower is better startup_min_ms
Cold start (slowest) ms diagnostic
lower is better startup_max_ms
Cold vs. warm throughput ratio secondary
higher is better

Throughput in the first seconds after start divided by warm throughput on the same endpoint. 1.0 = no warm-up penalty.

warm_ratio
Footprint 1 metric
Image size MB secondary
lower is better

Size of the container image as the local Docker image store reports it. Depending on the Docker version this is the unpacked or the compressed size, so compare it within one run only. It depends a lot on the chosen base image.

image_mb
Stability 2 metrics
p99 drift % secondary
lower is better

Change of p99 latency between the first and last minutes of a long run at fixed load.

p99_drift_pct
Memory growth MiB/min secondary
lower is better

Slope of PSS over a long run at fixed load. Persistent growth hints at a leak.

mem_growth_mib_per_min
Energy 2 metrics
CPU package energy J diagnostic
context only (neither higher nor lower is better)

Energy of the whole CPU package (RAPL) during the measurement window. Includes every process on the machine. It is a total for the window, not per request: an implementation that does less work in the window uses less, so it says nothing about efficiency on its own. Use energy per request for that.

package_energy_j
Package energy per request mJ diagnostic
lower is better

CPU package energy (RAPL) divided by requests. Includes the database and load generator, so compare only within one run.

energy_mj_per_req
Network 1 metric
Bytes per response B diagnostic
context only (neither higher nor lower is better)

Average response size including headers, as seen by the load generator.

bytes_per_resp
Disk I/O 1 metric
Disk writes KiB diagnostic
lower is better io_write_kib
Measurement quality 5 metrics
Hypervisor steal % diagnostic
lower is better

CPU time taken by the hypervisor from this machine during the measurement. Above 2 % the measurement is repeated.

steal_pct
Host load runnable tasks diagnostic
lower is better

1-minute load average of the machine right before the measurement.

host_load
CPU clock MHz diagnostic
context only (neither higher nor lower is better)

Average current frequency of the CPUs used by the service during the measurement.

cpu_mhz
Database CPU ratio diagnostic
context only (neither higher nor lower is better)

Busy share of the database's CPUs. Above 0.9 the database limits the throughput, so fast implementations end up close together.

db_cpu_util
Load generator CPU ratio diagnostic
lower is better

Busy share of the load generator's CPUs. Above 0.9 the throughput is a lower bound.

loadgen_cpu_util

Statistics

Median of passes

A run repeats every measurement in several passes (five by default). Each metric is summarized by its median, which a single disturbed pass cannot move far.

95 % bootstrap interval

The uncertainty of the median is a percentile bootstrap interval with 2,000 resamples and a fixed seed, so summaries are reproducible. With five passes it is the range from the lowest to the highest pass, which covers the true median with 93.75 % probability. With fewer than five passes there is no interval, and the site says so.

Outliers are flagged

A pass whose value has a robust z-score above 3.5 (based on the median absolute deviation) is marked, never silently removed. Throughput varying more than 10 % between passes is flagged too.

Changing order

The order of implementations changes every pass (a shifted Williams design), so each one runs at a different position and after a different implementation each time. What one leaves behind, such as heat or cache contents, does not always hit the same successor.

Derived metrics such as the throughput index are computed from the medians of one run, so they carry no interval of their own. Rankings count implementations with equal values as tied.

Quality controls

Check Effect
Functional checks before measuring: values, shapes, content types, persisted updates, totals, rejected invalid input Implementation is excluded from the run
Database statements per request (pg_stat_statements) deviate more than 3 % from the scenario specification Result flagged
Errors during a measurement (socket errors, timeouts, non-2xx/3xx) Result flagged with count and rate; failed requests never count as throughput
More than 1 % failed requests, a throughput of zero or a missing result in one of the workloads behind an index No index for that implementation; its scenario results stay visible
The measurement of an implementation fails in a pass Flagged as an error; that pass is missing from the number of passes (n) behind its values
Load generator CPU above 90 % Throughput flagged as a lower bound
Hypervisor steal time above the limit (2 % by default) Measurement repeated (twice by default), then flagged
Fixed-rate scenario reaches less than 97 % of the target rate Result flagged
A result bundle's summary does not match its per-pass data, or the bundle is filed under the wrong machine or run id Rejected by CI; published numbers are always recomputed from the per-pass values

Flags appear next to the affected value in the explorer, on implementation pages and on the run page.

Machines and provenance

Absolute throughput depends heavily on the CPU, so the primary comparison is always within one run. Across machines, relative indices and CPU time per request are more meaningful. Every run stores the machine fingerprint (CPU model, cores, SMT, governor, turbo, memory, kernel, OS, virtualization, Docker version) and the CPU layout. See all machines.

Reference

Measured by the maintainers themselves, who list the run id in the repository. The machine alone does not decide it: every machine of the same model has the same id. The default view prefers complete reference runs.

Community

Submitted by anyone as a pull request containing a result bundle. CI verifies the bundle; maintainers look for implausible values. Submit a run.

Methodology versions

Every run records the methodology it was measured with. Older versions stay visible, clearly labelled.

Preliminary methodology (v0)

Preliminary

Pre-release shell suite: throughput, p50/p99 latency, errors, start-up and memory only; floating image tags; peak memory recorded once per run; 3 s warm-up and 12 s measurement per scenario.

1 published run with this version.

Methodology 1

Current

Engine release: pinned versions and image digests, CPU per request, database work verification, PSS per scenario, steal-time retries, energy, CPU clock, persisted-update and content-type checks, deterministic database content, 5 s warm-up and 15 s measurement.

1 published run with this version.

Limitations

  • Start-up time includes docker start. The overhead is identical for every implementation but means the numbers are not process start times.
  • CPU package energy covers the whole package, including the database and the load generator; compare it within one run only.
  • Memory (PSS) is sampled once per second, so very short spikes can be missed. The cgroup's anonymous-memory peak complements it.
  • The load generator shares the machine. Its CPU use is measured and flagged when it could limit throughput.
  • Implementations are written to be idiomatic, but configuration choices such as pool sizes and worker counts matter. They are documented on each implementation page; better configurations are welcome as pull requests.
  • Categories matter: a bare server without a framework does less work than a full-stack framework. Compare like with like.
  • Workloads are synthetic. They are shaped after common production endpoints but cannot represent every application.