Contribute

Benchmania is run by its contributors. Every implementation, benchmark domain and published measurement arrives as a pull request, and anyone can open one. Results from more machines make every comparison more useful.

Measure on your hardware

Submit a measurement

  1. Use Linux with rootful Docker and cgroup v2, at least 4 physical cores and Python 3.11 or newer. Close other workloads; on a laptop use mains power.
  2. doctor checks the machine and check builds every implementation and runs the functional checks. Then run the whole suite from a clean checkout: it takes several hours, so start it in tmux or with nohup.
  3. Open a pull request that adds the bundle written to results/backend/<machine-id>/<run-id>.json.
Read the full guide
Terminal
git clone https://github.com/Cripacx/benchmania && cd benchmania
./benchmania doctor
./benchmania check backend
./benchmania run backend --label "My workstation"
./benchmania validate

What the pull request contains

Only the bundle file: no changes to code, manifests, results/reference-runs.toml or the generated website data. The bundle records the commit that was measured and contains no hostnames, user names or IP addresses. Quick smoke-test runs are never published.

How it is reviewed

CI re-verifies that the summary matches the per-pass data and that every scenario and metric is known. Maintainers then look for implausible values compared with similar hardware, many quality flags and options that differ from a full run.

Reference and community

Merged runs are labelled community. Reference runs are the ones maintainers measured themselves and listed in results/reference-runs.toml. Both are shown with their machine.

2 reference runs published. No community runs yet. Add the first.

Add to the suite

Add an implementation

A new language, framework or variant. Implementations are a directory with an impl.toml, a Dockerfile and the source.

Read the guide
Terminal
./benchmania scaffold backend my-framework --from bun-serve
./benchmania validate
./benchmania check backend --impl my-framework

Add a scenario

A new workload is one TOML file. Its kind must be one the domain harness understands; database work per request is declared and verified.

Read the guide
Terminal
# domains/backend/scenarios/db-c64.toml
id = "db-c64"
title = "Single query, 64 connections"
kind = "http"
order = 95
path = "/db"
connections = 64
work = { select = 1 }

Add a domain

A whole new area such as databases or compute. Domains are discovered automatically, and this website creates their pages as soon as they have results.

Read the guide
Terminal
cp -r domains/_example domains/databases
./benchmania check databases
./benchmania run databases --quick --passes 2

Rules for implementations

Idiomatic and production-like

Use the framework the way its documentation recommends for production. No response caching, precomputed responses, disabled validation or skipped database work.

Same work, pinned versions

Pass the functional checks. Use exact image tags, exact dependency versions and committed lockfiles, and declare the versions in impl.toml.

Honest category and configuration

Declare bare, micro, framework or full-stack truthfully, size workers for the CPU limit and document it in the notes.

Other ways to help

  • Report unfairness or a measurement error. Open a methodology issue with evidence; methodology problems are discussed in public. Issues
  • Share an idea or ask a question. Use Discussions; issues are for bugs and concrete proposals.
  • Improve an existing implementation. Open a pull request that touches only its directory and mention the maintainers listed in its impl.toml.
  • Improve the engine or this website. Open an issue first for larger changes. Changes to the engine, the harnesses and the list of reference runs are reviewed by the maintainers. The website lives in site/ and is generated from the published dataset.
  • Understand the design. Read the architecture and the methodology.