Methodology
How every number on this site is produced, summarized and checked. The full reference lives in the repository; this page combines it with the parameters recorded in the published data.
Goals
- Production-shaped conditions. Services run under the resource limits of a small production pod, against a real database, with realistic request mixes, not only at saturation.
- Same work for everyone. Every implementation passes the same functional checks, and the work it does per request is measured, not assumed.
- Transparent and reproducible. Every number comes with the machine, the exact versions, the configuration and the per-pass raw values.
- Honest uncertainty. Medians with confidence intervals, and quality flags instead of silent filtering.
Environment and resource limits
Each implementation runs alone in its own container with fixed limits. Supporting services (the environment, such as a database) and the load generator run in separate containers on separate cores. All containers share one network namespace and talk over loopback, so no NAT or bridge sits in the request path.
Backend services
Every implementation serves the same API (JSON, single and multiple queries, updates, a list endpoint and a JSON POST) against the same PostgreSQL 18 database. The service gets 2 CPU cores and 1 GiB of memory, like a small production pod; the database and the load generator run on separate cores. Results cover throughput, latency at fixed load, CPU per request, memory, cold start and warm-up.
- CPU limit
- 2 cores
- Memory limit
- 1 GiB
- port
- 8080
- Scenarios
- 12
- Metrics
- 42
CPU layout
The layout is computed from the machine's topology at the start of every run and stored with the results:
- The service gets one logical CPU per physical core for as many cores as its CPU limit. The SMT siblings of those cores stay idle, so the service never shares a core's execution units with other work.
- The remaining physical cores are split between the environment and the load generator; the load generator gets the larger half because it must never be the bottleneck.
- On machines too small for this, roles share CPUs and the run carries a warning in its quality flags.
- CPU 0: Service under test
- CPU 1: Service under test
- CPU 2: Environment (database)
- CPU 3: Environment (database)
- CPU 4: Load generator
- CPU 5: Load generator
- CPU 6: Kept idle
- CPU 7: Kept idle
- CPU 8: Environment (database)
- CPU 9: Environment (database)
- CPU 10: Load generator
- CPU 11: Load generator
- Service under test 0, 1
- Environment (database) 2, 8, 3, 9
- Load generator 4, 10, 5, 11
- Kept idle 6, 7
SMT is on. The service gets one logical CPU per physical core; the SMT siblings of those cores stay idle.
Scenarios
Scenarios are declared as files in the repository; the list below comes straight from the dataset. Closed-loop scenarios keep a fixed number of connections busy to find the maximum throughput; fixed-rate scenarios send the same request rate to every implementation and measure latency with coordinated-omission correction.
| Scenario | Kind | What it does |
|---|---|---|
| Cold start startup | Start-up | Starts a fresh container several times and measures the time until GET /json first succeeds. Also records memory at idle and the image size. GET /json |
| Cold load cold | Cold load | GET /db at full load for 10 seconds right after start, without any warm-up. Shows how much a runtime needs to warm up (JIT, caches, pools) after scaling out. GET /db128 connections10s measured1 SELECT per requestcompared with Single query |
| JSON json | Closed loop | GET /json returns a small JSON object. Pure framework and serialization overhead, no I/O. GET /json |
| Single query db | Closed loop | GET /db reads one random row by primary key and returns it as JSON. GET /db1 SELECT per request |
| 20 queries queries | Closed loop | GET /queries runs 20 sequential primary-key reads on one connection and returns them as a JSON array. Measures driver and async efficiency. GET /queries20 SELECT per request |
| Updates updates | Closed loop | GET /updates reads 5 random rows, changes each value and writes it back with 5 UPDATE statements. GET /updates5 SELECT + 5 UPDATE per request |
| List list | Closed loop | GET /list reads a range of 100 rows, adds a computed field to each and serializes the list, like a typical paginated REST endpoint. GET /list1 SELECT per request |
| JSON POST orders | Closed loop | POST /orders sends a 2.5 KB order with 50 line items. The service parses, validates and totals it and returns a summary. No database. POST /ordersbody orders.json |
| Single query, 16 connections db-c16 | Closed loop | GET /db with only 16 concurrent connections: behaviour at moderate concurrency, where latency rather than queueing dominates. GET /db16 connections1 SELECT per request |
| Single query, 512 connections db-c512 | Closed loop | GET /db with 512 concurrent connections: many idle keep-alive connections and deep queues on 2 cores. GET /db512 connections1 SELECT per request |
| Fixed load (5,000 req/s) rate | Fixed rate | GET /db at a fixed 5,000 requests per second for every implementation (open loop, latency corrected for coordinated omission). Shows latency at normal production load instead of at saturation. GET /db5,000 req/s fixed64 connections20s measured1 SELECT per request |
| Soak (5 minutes) soak | Soak | GET /updates at a fixed 2,000 requests per second for 5 minutes in 30-second slices. Reports p99 drift and memory growth. Optional: runs only when selected with --scenario. GET /updates2,000 req/s fixed64 connections10 × 30s5 SELECT + 5 UPDATE per requestoptional |
Metrics
The metric registry defines every value an implementation can be measured on: label, unit, which direction is better and how prominently the site shows it. primary metrics are shown first, secondary ones on request and diagnostic ones mainly explain measurement quality.
Summary 3 metrics
- Throughput index
-
higher is better
Geometric mean of throughput over the six workload scenarios, each relative to the fastest implementation in that scenario of the same run. 100 means fastest in all six; the leader of a run scores below 100 when it does not lead every workload.
Computed: Geometric mean of “Throughput” over JSON, Single query, 20 queries, Updates, List, JSON POST, each relative to the best implementation in that scenario (best = 100). Only an implementation that is best in every one of them reaches 100, and only implementations with a usable result in every one of them get an index.
throughput_index - CPU efficiency index
-
higher is better
Geometric mean of CPU time per request over the six workload scenarios, each relative to the most efficient implementation in that scenario of the same run. 100 means most efficient in all six.
Computed: Geometric mean of “CPU per request” over JSON, Single query, 20 queries, Updates, List, JSON POST, each relative to the best implementation in that scenario (best = 100). Only an implementation that is best in every one of them reaches 100, and only implementations with a usable result in every one of them get an index.
efficiency_index - Peak memory
-
lower is better
Highest PSS in any scenario of the run.
Computed: Highest median “Memory under load (PSS)” in any scenario.
peak_memory_mib
Throughput 3 metrics
- Throughput
-
higher is better
Successful (2xx/3xx) responses per second in the measured window. Failed requests are counted as errors, not as throughput.
rps - Errors
-
lower is better
Socket errors, timeouts and non-2xx/3xx responses. Any error is flagged next to the result.
errors - Error rate
-
lower is better
error_rate
Latency 9 metrics
- Latency p50
-
lower is better
latency_p50_ms - Latency p75
-
lower is better
latency_p75_ms - Latency p90
-
lower is better
latency_p90_ms - Latency p99
-
lower is better
latency_p99_ms - Latency p99.9
-
lower is better
latency_p99_9_ms - Latency p99.99
-
lower is better
latency_p99_99_ms - Latency mean
-
lower is better
latency_mean_ms - Latency std. dev.
-
lower is better
latency_stdev_ms - Latency max
-
lower is better
latency_max_ms
CPU 4 metrics
- CPU per request
-
lower is better
Service CPU time per request, read exactly from the container's cgroup. Less hardware-dependent than throughput.
cpu_us_per_req - CPU used
-
context only (neither higher nor lower is better)
Average number of cores the service kept busy (limit: 2).
cpu_cores - CPU throttling
-
lower is better
Time the service was throttled by its CPU quota during the window.
throttled_ms
Memory 4 metrics
- Memory under load (PSS)
-
lower is better
Highest proportional set size of all service processes, sampled every second.
mem_pss_peak_mib - Anonymous memory peak
-
lower is better
Heap, stacks and other anonymous memory of the container cgroup (no file cache).
mem_anon_peak_mib - Memory at idle
-
lower is better
PSS five seconds after start, before any load.
idle_pss_mib - Anonymous memory at idle
-
lower is better
idle_anon_mib
Database 3 metrics
- SELECTs per request
-
context only (neither higher nor lower is better)
Measured with pg_stat_statements. Must match the scenario's specification, otherwise the result is flagged.
db_select_per_req - UPDATEs per request
-
context only (neither higher nor lower is better)
db_update_per_req - Database CPU per request
-
lower is better
PostgreSQL CPU time per request. Shows how efficiently an implementation talks to the database.
db_cpu_us_per_req
Start-up 4 metrics
- Cold start
-
lower is better
Median time from container start to the first successful HTTP response.
startup_ms - Cold start (fastest)
-
lower is better
startup_min_ms - Cold start (slowest)
-
lower is better
startup_max_ms - Cold vs. warm throughput
-
higher is better
Throughput in the first seconds after start divided by warm throughput on the same endpoint. 1.0 = no warm-up penalty.
warm_ratio
Footprint 1 metric
- Image size
-
lower is better
Size of the container image as the local Docker image store reports it. Depending on the Docker version this is the unpacked or the compressed size, so compare it within one run only. It depends a lot on the chosen base image.
image_mb
Stability 2 metrics
- p99 drift
-
lower is better
Change of p99 latency between the first and last minutes of a long run at fixed load.
p99_drift_pct - Memory growth
-
lower is better
Slope of PSS over a long run at fixed load. Persistent growth hints at a leak.
mem_growth_mib_per_min
Energy 2 metrics
- CPU package energy
-
context only (neither higher nor lower is better)
Energy of the whole CPU package (RAPL) during the measurement window. Includes every process on the machine. It is a total for the window, not per request: an implementation that does less work in the window uses less, so it says nothing about efficiency on its own. Use energy per request for that.
package_energy_j - Package energy per request
-
lower is better
CPU package energy (RAPL) divided by requests. Includes the database and load generator, so compare only within one run.
energy_mj_per_req
Network 1 metric
- Bytes per response
-
context only (neither higher nor lower is better)
Average response size including headers, as seen by the load generator.
bytes_per_resp
Disk I/O 1 metric
- Disk writes
-
lower is better
io_write_kib
Measurement quality 5 metrics
- Hypervisor steal
-
lower is better
CPU time taken by the hypervisor from this machine during the measurement. Above 2 % the measurement is repeated.
steal_pct - Host load
-
lower is better
1-minute load average of the machine right before the measurement.
host_load - CPU clock
-
context only (neither higher nor lower is better)
Average current frequency of the CPUs used by the service during the measurement.
cpu_mhz - Database CPU
-
context only (neither higher nor lower is better)
Busy share of the database's CPUs. Above 0.9 the database limits the throughput, so fast implementations end up close together.
db_cpu_util - Load generator CPU
-
lower is better
Busy share of the load generator's CPUs. Above 0.9 the throughput is a lower bound.
loadgen_cpu_util
Statistics
Median of passes
A run repeats every measurement in several passes (five by default). Each metric is summarized by its median, which a single disturbed pass cannot move far.
95 % bootstrap interval
The uncertainty of the median is a percentile bootstrap interval with 2,000 resamples and a fixed seed, so summaries are reproducible. With five passes it is the range from the lowest to the highest pass, which covers the true median with 93.75 % probability. With fewer than five passes there is no interval, and the site says so.
Outliers are flagged
A pass whose value has a robust z-score above 3.5 (based on the median absolute deviation) is marked, never silently removed. Throughput varying more than 10 % between passes is flagged too.
Changing order
The order of implementations changes every pass (a shifted Williams design), so each one runs at a different position and after a different implementation each time. What one leaves behind, such as heat or cache contents, does not always hit the same successor.
Derived metrics such as the throughput index are computed from the medians of one run, so they carry no interval of their own. Rankings count implementations with equal values as tied.
Quality controls
| Check | Effect |
|---|---|
| Functional checks before measuring: values, shapes, content types, persisted updates, totals, rejected invalid input | Implementation is excluded from the run |
| Database statements per request (pg_stat_statements) deviate more than 3 % from the scenario specification | Result flagged |
| Errors during a measurement (socket errors, timeouts, non-2xx/3xx) | Result flagged with count and rate; failed requests never count as throughput |
| More than 1 % failed requests, a throughput of zero or a missing result in one of the workloads behind an index | No index for that implementation; its scenario results stay visible |
| The measurement of an implementation fails in a pass | Flagged as an error; that pass is missing from the number of passes (n) behind its values |
| Load generator CPU above 90 % | Throughput flagged as a lower bound |
| Hypervisor steal time above the limit (2 % by default) | Measurement repeated (twice by default), then flagged |
| Fixed-rate scenario reaches less than 97 % of the target rate | Result flagged |
| A result bundle's summary does not match its per-pass data, or the bundle is filed under the wrong machine or run id | Rejected by CI; published numbers are always recomputed from the per-pass values |
Flags appear next to the affected value in the explorer, on implementation pages and on the run page.
Machines and provenance
Absolute throughput depends heavily on the CPU, so the primary comparison is always within one run. Across machines, relative indices and CPU time per request are more meaningful. Every run stores the machine fingerprint (CPU model, cores, SMT, governor, turbo, memory, kernel, OS, virtualization, Docker version) and the CPU layout. See all machines.
Reference
Measured by the maintainers themselves, who list the run id in the repository. The machine alone does not decide it: every machine of the same model has the same id. The default view prefers complete reference runs.
Community
Submitted by anyone as a pull request containing a result bundle. CI verifies the bundle; maintainers look for implausible values. Submit a run.
Methodology versions
Every run records the methodology it was measured with. Older versions stay visible, clearly labelled.
Preliminary methodology (v0)
PreliminaryPre-release shell suite: throughput, p50/p99 latency, errors, start-up and memory only; floating image tags; peak memory recorded once per run; 3 s warm-up and 12 s measurement per scenario.
1 published run with this version.
Methodology 1
CurrentEngine release: pinned versions and image digests, CPU per request, database work verification, PSS per scenario, steal-time retries, energy, CPU clock, persisted-update and content-type checks, deterministic database content, 5 s warm-up and 15 s measurement.
1 published run with this version.
Limitations
-
Start-up time includes
docker start. The overhead is identical for every implementation but means the numbers are not process start times. - CPU package energy covers the whole package, including the database and the load generator; compare it within one run only.
- Memory (PSS) is sampled once per second, so very short spikes can be missed. The cgroup's anonymous-memory peak complements it.
- The load generator shares the machine. Its CPU use is measured and flagged when it could limit throughput.
- Implementations are written to be idiomatic, but configuration choices such as pool sizes and worker counts matter. They are documented on each implementation page; better configurations are welcome as pull requests.
- Categories matter: a bare server without a framework does less work than a full-stack framework. Compare like with like.
- Workloads are synthetic. They are shaped after common production endpoints but cannot represent every application.