perfkit

Benchmark harness for Fonoster projects: DigitalOcean runs, one report.json schema, one branded report page.

What perfkit is

perfkit benchmarks Fonoster projects (Routr first, then Roomr and Fonoster) the same way and publishes the results in one format.

A unit under test and its load generators. Each run creates dedicated-CPU DigitalOcean droplets in the same region and private VPC: the unit under test, and one to eight load generators (infra.loadgen.count) that drive it with SIPp. The images are pinned by digest when the run starts, and every droplet is deleted on every exit path.
A Kubernetes cluster when a benchmark asks for one. A benchmark with uut.helm installs a Helm chart on a DOKS cluster created for the run in the same VPC (kind with MetalLB on --local), and the page labels each benchmark with its target. The cluster is deleted with its load balancers and volumes on every exit path.
One report format. Every run writes a report.json (schema: perfkit/report/v1), and each project keeps a runs.json history index (schema: perfkit/runs/v1). The formats are the same for every project, so runs compare across releases and projects.
One report page. perfkit renders report.json into a branded static HTML page, published under /benchmarks on the project's GitHub Pages.

A project keeps only its benchmark definitions, in perf/perfkit.yaml (apiVersion: perfkit/v1). See docs/perfkit-yaml.md for every field and docs/report-v1.md for report.json and runs.json. The JSON Schema files are in schemas/.

How it works

perfkit run creates the machines and starts an agent on the first load generator. The agent runs every benchmark on its own, publishes live.json while it runs, and keeps the raw files. perfkit collect fetches them, writes report.json and deletes everything.

flowchart LR
  subgraph CI["GitHub Actions or a laptop"]
    RUN["perfkit run<br/>reads perfkit.yaml"]
    COL["perfkit collect<br/>analyze, report.json"]
  end

  subgraph DO["DigitalOcean, one private VPC per run"]
    direction TB
    LG1["Load generator 1<br/>agent, sampler,<br/>SIPp callers and callees"]
    LGN["Load generators 2 to 8<br/>sampler,<br/>SIPp callers and callees"]
    UUT["Unit under test<br/>one server (Docker Compose)<br/>or a DOKS cluster (Helm)"]
    LG1 -- "SSH, run-only key" --> LGN
    LG1 -- SIP --> UUT
    LGN -- SIP --> UUT
  end

  RUN -- "1 create droplets, firewall,<br/>cluster; start the agent" --> DO
  LG1 -- "2 live.json" --> LIVE["Live page"]
  DO -- "3 raw files" --> COL
  COL -- "4 delete everything" --> DO
  COL --> SITE["Report page<br/>perfkit render"]

Every benchmark spreads load.instances SIPp callers and as many callees round robin over the load generators. The callees register their share of the users, then answer the calls Routr routes to them; no load generator talks to another during the load.

flowchart TD
  A["Next benchmark"] --> B["Deploy the unit under test<br/>compose up, or helm install and seed"]
  B --> C{"Every callee<br/>registered its share?"}
  C -- no --> X["Abort the benchmark"]
  C -- yes --> K{kind}
  K -- capacity --> D["Offer a rate for one step<br/>(30 s), then add a step"]
  D --> F{"p99, failed calls,<br/>any load generator saturated?"}
  F -- ok --> D
  F -- "a limit broke twice" --> G["Limit: the last good step"]
  F -- saturated --> H["Lower bound: the load generator<br/>ran out before the unit under test"]
  K -- soak --> S["Hold a share of the measured limit<br/>for hours; memory, failed calls, restarts"]

Prerequisites

Node 22 or later, and npm 11 to run perfkit from a git tag (see below).
ssh, scp and ssh-keygen (OpenSSH) on the machine that runs perfkit run or perfkit collect. perfkit run --local needs Docker instead, with host networking (on Docker Desktop, turn on host networking in its settings), and kind on the PATH for a benchmark with uut.helm.

For perfkit run and perfkit sweep, a DigitalOcean API token in DIGITALOCEAN_TOKEN. Create it with custom scopes, and grant these, under DigitalOcean's scope names:

Resource	Scopes
Droplet	`droplet:create`, `droplet:read`, `droplet:delete`
VPC	`vpc:create`, `vpc:read`, `vpc:delete`
Firewall	`firewall:create`, `firewall:read`, `firewall:update`, `firewall:delete`
SSH key	`ssh_key:create`, `ssh_key:read`, `ssh_key:delete`
Tag	`tag:create`, `tag:read`, `tag:delete`
Kubernetes, for `uut.helm`	`kubernetes:create`, `kubernetes:read`, `kubernetes:delete`, `kubernetes:access_cluster`, and `load_balancer:delete`, `block_storage:delete` for the cluster's own load balancers and volumes

DigitalOcean adds the read scopes these depend on when you pick them. perfkit run --dry-run needs no token.

Warning

A run costs money. Each perfkit run creates two dedicated-CPU droplets, at the sizes in infra.uut.size and infra.loadgen.size of perfkit.yaml, and they bill by the hour while they exist: about 15 to 100 minutes for a capacity benchmark, and about 7 hours for a soak. perfkit deletes them at the end of every run, whatever the exit; perfkit sweep is the fallback for a run that was killed before it could.

How projects pin it

perfkit is not published to npm. Releases are git tags, vX.Y.Z, and a project runs a tagged release straight from GitHub:

npx -y github:fonoster/perfkit#vX.Y.Z validate perf/perfkit.yaml

npm clones the tag, builds it (the prepare script) and runs the perfkit command. This needs npm 11 or later: npm 10.9, the npm that Node 22 ships with, fails on a git package that has a prepare script (GitFetcher requires an Arborist constructor to pack a tarball). With npm 10, run perfkit through npm 11:

npx -y npm@11 exec --yes --package=github:fonoster/perfkit#vX.Y.Z -- perfkit validate perf/perfkit.yaml

Pin the same tag everywhere a project calls perfkit, and read the release notes before moving to a new one: each release says what changed for perfkit.yaml, the report schema and the workflow inputs.

perfkit validate

perfkit validate <file>

Checks one perfkit.yaml, report.json or runs.json file. It tells the formats apart by the apiVersion field (perfkit/v1) or the schema field (perfkit/report/v1, perfkit/runs/v1). For a perfkit.yaml, it also checks that every file it names exists, relative to the directory of the perfkit.yaml.

$ npx perfkit validate perf/perfkit.yaml
perf/perfkit.yaml: valid perfkit/v1 config, 2 benchmarks
$ echo $?
0
$ npx perfkit validate report.json
report.json: benchmarks[1].verdict: expected one of passed, failed, inconclusive, aborted, not-run; got "ok"
report.json: run.commit: required
$ echo $?
2
$ npx perfkit validate old.json
old.json: unknown schema "perfkit/report/v9"; this perfkit reads perfkit/v1, perfkit/report/v1, perfkit/runs/v1
$ echo $?
2

A valid file prints one line on stdout. Every error is one line on stderr, <file>: <json path>: <message>; a problem with the file itself has no path: <file>: cannot read: <reason>, <file>: cannot parse: <reason> or the unknown-schema line.

Exit code	Meaning
0	The file is valid.
1	Usage error, such as a missing file argument.
2	The file is invalid, unreadable, malformed or of an unknown schema.

perfkit validate --help lists the accepted formats, and perfkit --version prints the perfkit version.

perfkit run

perfkit run --config <perfkit.yaml> --target-version <tag> [--commit <sha>] [--benchmarks <id,id>]
            [--out <dir>] [--detach] [--dry-run] [--local] [--live-port <port>]

Creates the run's infrastructure on DigitalOcean, runs the benchmarks on it, and deletes it at the end. In infra.region of perfkit.yaml, it creates, in this order:

an SSH key (see PERFKIT_SSH_KEY below);
a new VPC, perfkit-<run id>, whose range DigitalOcean assigns, so parallel runs never collide;
two ubuntu-22-04-x64 droplets in it, perfkit-<run id>-uut and perfkit-<run id>-loadgen, tagged perfkit and perfkit-run-<run id>;
when a selected benchmark has uut.helm, a DOKS cluster, perfkit-<run id in lowercase>, of infra.cluster.nodes × infra.cluster.size in the VPC, with the same tags;
a firewall, perfkit-<run id>: from the VPC: all ports (tcp and udp) and icmp, so the load generator reaches the server's API and the server reaches the callees; SSH (22) from this machine's public IP (from https://api.ipify.org) and from the VPC, where the agent on the load generator reaches the server under test; with a cluster, all tcp and udp ports from its node droplets, so its pods reach the callees.

Both droplets get Docker Engine with the compose plugin, 25 MiB socket buffers (net.core.rmem_max, wmem_max, rmem_default, wmem_default) and 1048576 open files, through cloud-init, which holds no secrets. perfkit waits until both droplets are active and cloud-init status --wait has finished over SSH, for at most 10 minutes, and, at the same time, for the cluster to be running, for at most 20 minutes. The cluster's kubeconfig, valid for infra.maxLifetime, goes to the load generator only (/opt/perfkit/keys/kubeconfig, mode 600), never into run.json, a log or the report. Every image runs by the digest its tag had when the run started.

Flag	Default	Meaning
`--config <perfkit.yaml>`	required	The project's benchmark definitions.
`--target-version <tag>`	required	The image tag under test, for every image in `perfkit.yaml`. `--version` prints perfkit's own version.
`--commit <sha>`	`git rev-parse HEAD` in the directory of the config file	The commit under test; part of the run id. Outside a git checkout it is required.
`--benchmarks <id,id>`	every benchmark	Run only these benchmarks.
`--out <dir>`	`.perfkit`	Where the run writes its files.
`--detach`	off	Return once the agent on the load generator has started; `perfkit collect` finishes the run.
`--dry-run`	off	Print the plan and create nothing. Needs no token.
`--local`	off	Run on this machine's Docker instead of DigitalOcean. Needs no token.
`--live-port <port>`	`live.port` of perfkit.yaml, else `8787`	Where live.json is served during the run (see Live results).

Environment	Meaning
`DIGITALOCEAN_TOKEN`	Required, except with `--dry-run`. Never printed, never written to a file, `report.json` or a droplet.
`PERFKIT_NAME_PREFIX`	Optional; containers and compose projects are named `<prefix>-<run id>-<what>`, `perfkit` by default.
`NGROK_AUTHTOKEN`	Optional. Opens an ngrok tunnel to the run's live.json (see Live results). On a DigitalOcean run it goes into the agent container's environment on the load generator only; never into run.json, a log or the report.
`NGROK_DOMAIN`	Optional static ngrok domain when perfkit.yaml has no `live.domain`.
`PERFKIT_SSH_KEY`	Optional ed25519 private key, in OpenSSH format, without a passphrase. When set, the run looks its public key up in the account by fingerprint and reuses it, or registers it as `perfkit-shared`. That key is never deleted, by teardown or by sweep, so parallel runs share it. Without it, the run creates `perfkit-<run id>` for this run only and deletes it at teardown.

--dry-run resolves the digests on Docker Hub and prints the plan:

$ npx perfkit run --config perf/perfkit.yaml --target-version 2.31.6 --dry-run
plan for Routr 2.31.6 (dry run, nothing created)
  region     nyc3, new VPC (range assigned by DigitalOcean)
  uut        c-8   ubuntu-22-04-x64
  loadgen    c-16  ubuntu-22-04-x64
  firewall   from the VPC: all ports (tcp/udp) and icmp; SSH (22) from 203.0.113.7 and the VPC only
  tags       perfkit, perfkit-run-20261103T141210Z-bd3254a
  images     fonoster/routr-edgeport:2.31.6@sha256:8bca…1866
             fonoster/routr-echo:2.31.6@sha256:14a4…a43f
             fonoster/routr-one:2.31.6@sha256:8433…1801
  benchmarks edgeport-echo (capacity, up to 100 min at 50 to 5,000 messages/s)
             one-soak (soak, 6 h 30 min)
  deleted    at the end, or by sweep after 9h
$ echo $?
0

A capacity benchmark runs at most (max - start) / step + 1 steps of every each; a soak runs its warmup plus its duration.

Teardown. perfkit deletes everything the run created, in reverse order (firewall, droplets and the cluster with its load balancers and volumes, VPC, SSH key), on every exit path: success, a failed benchmark, an error, a create that failed part-way, and Ctrl-C, SIGINT or SIGTERM. DigitalOcean deletes only an empty VPC, so perfkit waits up to 5 minutes for the VPC's members to go before it deletes it. Each delete is tried 3 times. The one exception is --detach: once the agent on the droplets has started, the run keeps its droplets, and perfkit collect or perfkit sweep deletes them. Without --detach, the same holds while run follows the run: it deletes everything at the end, as perfkit collect does.

What runs where. Once the droplets are ready, perfkit copies the directory of perfkit.yaml to /opt/perfkit/project on both, and its own package (dist/, package.json and the production node_modules) to /opt/perfkit/agent on the load generator. It then starts the agent there, perfkit agent, as a detached node:22-alpine container with the Docker socket mounted, so the run goes on if this machine disconnects. The agent runs the capacity benchmarks one after another, then the soaks: those of different targets (the server and the cluster) at the same time, those of one target one after another. For a server benchmark: docker compose up of the benchmark's compose file on the UUT, a wait until every service is running (and healthy, with a healthcheck) for at most 5 minutes, the setup script, the samplers, the OpenTelemetry Collector for telemetry, and SIPp. A capacity ramp is checked step by step and SIPp stops at the first step that breaks a rule or doesn't count. Then docker compose down, and the next benchmark. A cluster benchmark installs its chart instead (see docs/perfkit-yaml.md) and ends with helm uninstall. docs/project-setup.md is what a project's compose files, setup script and SIPp scenarios get.

Without --detach, run then follows the run like perfkit collect and prints its progress:

$ DIGITALOCEAN_TOKEN=… npx -y github:fonoster/perfkit#v0.1.0 run --config perf/perfkit.yaml --target-version 2.31.5
plan for Routr 2.31.5 (run 20261103T141210Z-bd3254a)
…
[edgeport-echo] step    50 messages/s  p99 4.1 ms  failed 0.00%  ok
[edgeport-echo] step 1,300 messages/s  p99 62.0 ms  failed 0.05%  over the p99 limit; stopping
[edgeport-echo] passed: limit 1,250 messages/s
[one-soak] passed: memory one +1.8 MB/h, edgeport +0.3 MB/h, postgres +0.1 MB/h; failed calls 0.01%; restarts 0
deleted 2 droplets, firewall, VPC, SSH key
report .perfkit/20261103T141210Z-bd3254a/report.json  (2 of 2 benchmarks passed)

The run's files are in <out>/<run id>/: report.json and raw/<benchmark id>/, the raw results the verdicts come from. A benchmark left out with --benchmarks is in the report as not-run.

The agent's key. The agent reaches the UUT over SSH with a key pair made for this run only: its public key goes into the UUT's authorized_keys and its private key onto the load generator, and both go with the droplets. Neither DIGITALOCEAN_TOKEN nor PERFKIT_SSH_KEY is ever copied to a droplet. Without PERFKIT_SSH_KEY, the run's own login key is kept in ~/.perfkit/runs/<run id>/ so that perfkit collect on this machine can log in.

--local. The same run on this machine's Docker: no droplets, firewall, VPC, SSH keys or token. The UUT's compose project and the agent, SIPp, sampler and collector containers all run locally with host networking; PERFKIT_UUT_IP and PERFKIT_LOADGEN_IP are 127.0.0.1 and PERFKIT_VPC_CIDR is 127.0.0.0/8. report.json says provider: local, with the vCPUs and memory of docker info. The run lives in ~/.perfkit/runs/<run id>/ until it is collected. Use it to check a project's benchmarks before a real run; its numbers are not comparable with one, and on an arm64 machine SIPp (amd64 only) runs emulated.

$ npx -y github:fonoster/perfkit#v0.1.0 run --local --config examples/minimal/perfkit.yaml --target-version 3.7.3
plan for Minimal 3.7.3 (local run 20261103T210141Z-bff8fc4)
  hosts      this machine's Docker, for the server under test and the load generator
  images     pumpagubben/sipp:3.7.3@sha256:ceef…5dd1
  benchmarks sipp-uas (capacity, up to 4 min at 50 to 200 messages/s)
[sipp-uas] step    50 messages/s  p99 1.0 ms  failed 0.00%  ok
…
[sipp-uas] passed: limit at least 200 messages/s
removed the run's containers
report .perfkit/20261103T210141Z-bff8fc4/report.json  (1 of 1 benchmarks passed)

Exit code	Meaning
0	Ok, every benchmark passed; with `--detach`, the agent started.
1	Report written, at least one benchmark not passed.
2	Bad config, flags or a missing token, or a compose service without `network_mode: host`; nothing was created.
3	Infrastructure error, or the UUT was not up within 5 minutes; everything this run created is deleted, no report.
130	Cancelled; everything deleted. Benchmarks not finished are `aborted` (`the run was cancelled`), and the report is written if one finished.

Each error is one line on stderr:

perfkit: DIGITALOCEAN_TOKEN is not set; nothing was created
perfkit: image fonoster/routr-echo:2.31.9 not found on Docker Hub; nothing was created
perfkit: --commit is required outside a git checkout
perfkit: one-soak/compose.yaml: service one must use network_mode: host (Docker's UDP port publishing distorts the results); nothing was created
perfkit: could not create droplet perfkit-20261103T141210Z-bd3254a-uut in nyc3: <DigitalOcean message>. Everything this run created was deleted.
perfkit: could not delete droplet 512034567 (perfkit-20261103T141210Z-bd3254a-uut) after 3 attempts: <message>. Run: perfkit sweep --run 20261103T141210Z-bd3254a

perfkit collect

perfkit collect <run-id> [--wait] [--out <dir>] [--timeout <duration>]

Follows a run started with --detach, from this machine or another: it finds the droplets by the tag perfkit-run-<run id>, adds this machine's public IP to the run's firewall, prints the agent's progress lines, and downloads each benchmark's raw results into <out>/<run id>/raw/ as soon as the benchmark is done. Once the whole run is done it writes <out>/<run id>/report.json, deletes everything and exits 0 or 1. For a --local run it reads ~/.perfkit/runs/<run id>/ instead, and has nothing to delete but containers.

Flag	Default	Meaning
`--wait`	off	Only wait for the run to end: never write the report, never delete.
`--out <dir>`	`.perfkit`	Where the run's files go.
`--timeout <duration>`	`5h50m`	Stop waiting after this, like `5h50m`, `90m` or `30s`: exit 4, nothing deleted.

It logs in with PERFKIT_SSH_KEY, or with the run's own key in ~/.perfkit/runs/<run id>/ on the machine that started the run. With neither: perfkit: no SSH key for run <run id>; set PERFKIT_SSH_KEY (exit 2). It needs DIGITALOCEAN_TOKEN, except for a --local run.

$ perfkit run --config perf/perfkit.yaml --target-version 2.31.5 --detach
run 20261103T141210Z-bd3254a started; collect it with: perfkit collect 20261103T141210Z-bd3254a
$ perfkit collect 20261103T141210Z-bd3254a --wait --timeout 5h50m
[one-soak] running, 3h12m of 6h30m
$ echo $?
4

The load generator stops responding. When 3 reads in a row, 30 s apart, fail, every benchmark without results is aborted (the load generator stopped responding), the report is written and everything is deleted (exit 1). With no results at all there is no report (exit 3).
The UUT stops responding. The agent checks the UUT every 30 s; after 3 failed checks in a row the benchmark is aborted (the server under test stopped responding) with the series collected so far, and the agent goes on to the next one.
Cancelled. SIGINT or SIGTERM stops the agent, marks every unfinished benchmark aborted (the run was cancelled), writes the report if one benchmark finished, deletes everything and exits 130. During --wait it only exits 130.
The agent fails. Before it exits 3, collect downloads the raw results of every benchmark that did not finish, as they are, into <out>/<run id>/raw/<id>/ (without a done marker) and prints [<id>] downloaded the partial raw results to <dir>, so hours of a soak survive the teardown. It does the same for a cancelled run before it deletes everything.

Exit code	Meaning
0	With `--wait`, the run is done; otherwise every benchmark passed.
1	Report written, at least one benchmark not passed.
2	Bad flags, a missing token, or no SSH key for the run.
3	The agent failed (for example the UUT was not up within 5 minutes), or no benchmark produced results; everything deleted unless `--wait`.
4	The timeout passed while the run goes on; nothing was deleted.
130	Cancelled.

For GitHub, docs/github-workflow.md has the reusable workflow that splits a run across jobs.

perfkit summary

perfkit summary --report <report.json>

Prints a report as GitHub Markdown, for a workflow's run summary: the run id, the target version and commit, the overall verdict, and per benchmark its verdict (with its reason when it is inconclusive, aborted or not run). A capacity benchmark adds its capacity limit, where the ramp stopped and a table of the steps (offered and placed rate, p99, failed %, ok or why it stopped); a soak adds its rules against their limits (memory growth per process, failed calls, restarts) and p99 in the first and the last hour. The reusable workflow appends it to $GITHUB_STEP_SUMMARY:

perfkit summary --report .perfkit/<run id>/report.json >> "$GITHUB_STEP_SUMMARY"

It exits 0 when it printed the summary, 1 for a usage error, and 2 when the report is invalid, unreadable or of another schema.

perfkit sweep

perfkit sweep [--config <perfkit.yaml>] [--run <run-id>] [--older-than <duration>] [--dry-run]

The backstop for a run that could not tear down, for example a killed CI job. Run it on a schedule. It deletes droplets tagged perfkit and named perfkit-<run id>-uut, -loadgen or -worker-<n>, and Kubernetes clusters tagged perfkit and named perfkit-<run id in lowercase> (with their load balancers and volumes), that are older than --older-than, and then the firewall, VPC and SSH key named perfkit-<run id> of those runs. It touches nothing else: not an untagged droplet, and never the perfkit-shared key.

Flag	Default	Meaning
`--config <perfkit.yaml>`	none	Read the default of `--older-than` from its `infra.maxLifetime`.
`--run <run-id>`	none	Delete this run's resources, at any age.
`--older-than <duration>`	`infra.maxLifetime` of `--config`, else `9h`	Delete runs older than this, like `9h` or `90m`.
`--dry-run`	off	List what would be deleted; delete nothing.

It needs DIGITALOCEAN_TOKEN.

$ npx perfkit sweep --older-than 9h
deleted droplet perfkit-20261103T021210Z-41d9a2c-uut (age 9h04m)
deleted droplet perfkit-20261103T021210Z-41d9a2c-loadgen (age 9h04m)
deleted firewall, VPC and SSH key of run 20261103T021210Z-41d9a2c
$ echo $?
0

With nothing to delete it prints nothing to sweep and exits 0. It exits 2 for bad flags or a missing token, and 3 for a DigitalOcean API error or a delete that failed.

How verdicts are computed

perfkit turns the raw results of a run into each benchmark's verdict with fixed rules, so two runs of the same build agree. Every threshold comes from perfkit.yaml; none is built in. Latency is the time from the request a scenario marks with start_rtd="1" to the 200 OK it marks with rtd="1", per call as SIPp records it (-trace_rtt). Percentiles are nearest-rank: p99 is the value at rank ⌈0.99 × n⌉ of the sorted calls. SIPp's own averages and buckets are never used. A failed call is one SIPp counts as failed, and CPU is in % of one core, so 100 is one full core.

Capacity

SIPp adds load in steps, such as 50 more calls/s every minute. With callees (load.register), the ramp starts only once every callee process has registered its whole share, with the soak's timeout (users / 20 + 60 s) and abort reason. The uac gives up on a message it expects after 10 s, and a call that timed out counts as failed; with a hold, a call's teardown failure is counted when it happens, up to hold + 10 s after the call starts, so every should be at least 4 × (hold + 10 s). For each step:

Its latency counts the calls whose request was sent in the step, even when the answer came in the next one.
Its attempted, failed and retransmitted calls are SIPp's 10-second counters (-trace_stat) that fall in the step. Failed % is failed / attempted. So every must be a multiple of 10 s: perfkit validate rejects a 15s step, whose counters would mix two steps.
Warm-up. With warmup set, the first step still runs its full every, but it is measured only from warmup after it starts: calls sent, and 10-second counters whose period mid-point falls, before that don't count. A cold JIT (EdgePort's JVM and GraalJS) makes the first seconds slow; the later steps are measured whole.

perfkit checks the steps in order:

Is the measurement valid? A step is invalid when the load generator, not the server, was the bottleneck: any SIPp process (a uac instance, or a callee, sipp-register and sipp-uas) used more than 80% of one core, placed less than 95% of the offered rate, placed more than 105% of it (a burst of calls it owed from earlier steps), or the load generator dropped UDP packets. Then the verdict is inconclusive, the reason names the step and the cause, and no limit is published.
Does it pass? A step passes when p99 is below p99LatencyMs and failed % is below failedCallsPct. Both limits are strict: a p99 of exactly 50 ms breaks a 50 ms limit. The first step that breaks either rule ends the ramp.

The limit is the offered rate of the last step that passed, and the rules are shown at that step, so the verdict is passed. When the first step already breaks a rule there is no limit and the verdict is failed. When no step broke a rule, the limit is a lower bound: the server can take more than the ramp offered.

Each step also shows retransmissions, the server's CPU (the sum of its processes) and UDP drops on both machines. They are reported and never change the verdict.

Soak

A soak holds a fixed rate for hours to find leaks. With rate.fromCapacity, the rate is fraction × the limit that capacity benchmark measured earlier in the same run, rounded down (when it was inconclusive because the load generator gave out, the highest step before that, if every one passed, as a lower bound), or fallbackCps when it found none; the progress log and load.rateSource in report.json say which. On the cluster, a restart is a container restart of a sampled pod (its restart count went up) or a pod replaced by another; a sample the Metrics API left out is not a restart.

Warm-up excluded. Nothing measured during warmup counts toward any rule.
Too little data. When the samples after the warm-up cover less than 90% of duration, the verdict is aborted and no rule is checked. A short soak configured as short, such as a 20-minute smoke run, is fine.
No calls, no response times. The verdict is aborted too when the uac started no call after the warm-up (no calls were placed after the warm-up), or when SIPp left no response times file for it, so the p99 can't be measured.
Callees not registered. With a register role, the uac starts only once every callee has registered. When they have not within users / 20 + 60 seconds, no call is placed and the verdict is aborted (only 150 of 200 callees registered within 70 s, so no call was placed). With several callee processes the reason names the one that fell short (callee 2 registered 10 of 67 within 64 s, so no call was placed).
Memory growth. For each process (or Kubernetes pod), the Theil–Sen slope of its memory (RSS) in MB per hour: the median of the slopes between every pair of samples (an evenly spaced subset of 600 on long runs). The window starts after the warm-up plus a settle period (settle, by default 30 minutes, or half of duration when that is shorter), so a heap that climbs while it settles after start-up does not count, and ends when the load ends, so the samples taken while it winds down don't either. A dip or a spike moves the median far less than a least-squares fit; a steady leak still shows. It fails at the threshold or above: 5.0 MB/h fails a 5 MB/h limit. The headline is the steepest process, and the page says which window the slope covers.
Failed calls. Failed % over every call after the warm-up. It fails at the threshold or above.
Restarts. A process whose PID changed, or that was missing from a sample, restarted once. A name that matches several processes, like a database and its connections, restarted when none of the PIDs of its previous sample is left. It fails above the threshold.

The verdict is failed when any rule failed and passed otherwise. Also reported, never part of the verdict: p99 in the first and the last hour after the warm-up, the most open file descriptors per process, the most concurrent calls, the registrations and their failures, and, with telemetry, each service's p50 and p99 self time (a span's time minus its direct children's).

Live results

While a run goes on, the benchmarks page shows it live. Every run, published config or not:

live.json. The agent writes <run root>/live.json (schema: perfkit/live/v1, see docs/report-v1.md) every 15 seconds: which benchmark runs, each capacity step as the agent judged it, what held so far, a soak's calls and raw memory per process, and when the run should end. Nothing in it is a verdict.
Served read-only. GET /live.json on 127.0.0.1:<live.port> (8787 by default), never cached, with CORS for the origin of project.siteUrl and for http://localhost:*; 503 before the first snapshot, 404 for anything else. On a DigitalOcean run the agent serves it on the load generator; with --local the run (or collect) process serves it on this machine.
ngrok. With NGROK_AUTHTOKEN set, the server is tunnelled through the static domain in live.domain (or NGROK_DOMAIN) with the @ngrok/ngrok package, and the public URL is logged. Without it there is no tunnel. The tunnel and the server close when the run ends.
The page. The live URL (project.live.url, else live.url, else https://<live.domain>/live.json) goes into report.json and into the site's data/site.json, so the latest page (the site root) knows where to look, even before the first run is published (perfkit render --config, below). It reads data/site.json first, then the latest report. It polls every 15 seconds; while a run is in progress it shows it live, marked preliminary. When the run ends it says the report is being published and shows it as soon as data/runs.json lists the run. With no run in progress the page is the latest report, unchanged. Run pages never go live. ?live=<url> on the page overrides the URL.

Watch a local run live

From a perfkit checkout (npm ci && npm run build); the minimal example takes about 4 minutes:

# 1. A site to watch: render any report into a directory (an example works).
node dist/cli.js render --report examples/reports/passed.json \
  --history examples/history/runs-14.json --out /tmp/perfkit-site
# 2. Serve it (leave it running).
node dist/cli.js serve --site /tmp/perfkit-site --port 8080
# 3. In another terminal, a short local run; it serves live.json on http://127.0.0.1:8787.
node dist/cli.js run --local --config examples/minimal/perfkit.yaml --target-version 3.7.3
# 4. Open the page with the local live URL.
open "http://localhost:8080/?live=http://localhost:8787/live.json"
# 5. When the run ends, publish its report; the page switches to it.
node dist/cli.js render --report .perfkit/<run id>/report.json \
  --history /tmp/perfkit-site/data/runs.json --out /tmp/perfkit-site

perfkit serve

perfkit serve --site <dir> [--port 8080] [--host 127.0.0.1]

A read-only static server for a directory perfkit render wrote, as GitHub Pages serves it, with nothing cached: to preview a site, or to watch a --local run live (above). It prints serving <dir> at http://127.0.0.1:8080/ (Ctrl-C stops) and runs until Ctrl-C. It exits 0 when stopped, 1 for a usage error and 2 when it can't listen (the port is taken).

perfkit render

perfkit render --report <report.json> [--history <data/runs.json|https URL>] --out <dir>
perfkit render --config <perfkit.yaml> [--history <data/runs.json|https URL>] --out <dir>

Renders one run's report.json into the project's benchmarks site, a static site the caller publishes (for example under /benchmarks on GitHub Pages). It validates the report first, with the same error lines as perfkit validate.

With --config instead of --report it adds no run: it writes the site from the project's perfkit.yaml (logo and favicon inlined) and the history as it is. Without --history that is a site with no runs yet: the site root shows the project's title and lead and "No published runs yet. When a run is in progress it appears here live.", and goes live as soon as a run starts. Use it to publish (or clear) a project's site before its first published run:

$ npx perfkit render --config perf/perfkit.yaml --out out/benchmarks
wrote out/benchmarks/app/0.1.13/app.js (perfkit 0.1.13)
wrote out/benchmarks/data/runs.json (0 runs)
wrote out/benchmarks/data/site.json (live: https://perfkit.ngrok.app/live.json)
wrote out/benchmarks/index.html (no runs yet)

Flag	Meaning
`--report <file>`	The run's `report.json`. This or `--config` is required, not both.
`--config <perfkit.yaml>`	Instead of `--report`: write the site from the project's perfkit.yaml with no run added.
`--history <file\|url>`	The published `data/runs.json`, as a file or an `https://` URL. A URL that answers 404 is treated as no history. Without it, this is the first run: the page has no deltas and perfkit prints `perfkit: no history: first run, deltas omitted`.
`--out <dir>`	The directory to write the site into. Required.

$ npx perfkit render --report .perfkit/20261103T141210Z-bd3254a/report.json \
    --history https://fonoster.github.io/routr/benchmarks/data/runs.json --out out/benchmarks
wrote out/benchmarks/app/0.2.0/app.js (perfkit 0.2.0)
wrote out/benchmarks/data/runs/20261103T141210Z-bd3254a/report.json
wrote out/benchmarks/runs/20261103T141210Z-bd3254a/report.json
wrote out/benchmarks/data/runs.json (15 runs)
wrote out/benchmarks/data/site.json (live: https://perfkit.ngrok.app/live.json)
wrote out/benchmarks/index.html (latest: 20261103T141210Z-bd3254a)
wrote out/benchmarks/runs/20261103T141210Z-bd3254a/index.html
wrote out/benchmarks/runs/20261027T093000Z-36bf8f8/index.html
...

The site is data-driven: the pages are drawn in the browser, when they are viewed, from JSON files on the same site. The output layout:

data/runs.json: the history, newest first, with this run inserted at its startedAt position. A run already listed (same id) is replaced, so rendering twice is safe. Point --history at the published copy of this file (a runs.json of the old layout, at the site root, also works).
data/site.json (schema: perfkit/site/v1, JSON Schema in schemas/site-v1.schema.json): the project, as report.json carries it: name, repo, site URL, accent, logo and favicon SVGs, title, lead and live URL. Every render rewrites it, from the report or from --config; the app draws the no-runs page from it and reads the live URL from it first.
data/runs/<id>/report.json: this run's report, as read; the app draws the page from it. A copy also goes to runs/<id>/report.json, where the page's report.json link points.
app/<version>/app.js: this perfkit's browser app, the same renderer perfkit has always used, bundled. It validates the data with the same schemas as perfkit validate.
index.html and runs/<id>/index.html, one per run in the history: small shells with the run's title, description and Open Graph tags (for link previews) that load the app. The app reads the run from the URL: the site root is the latest run, runs/<id>/ is that run. Where a report is missing from data/runs/<id>/, it reads runs/<id>/report.json, so a site of the old layout keeps working.

Older reports are never read or rewritten: publishing a run adds its data files and replaces the shells and the history. Each shell pins this perfkit's app, so publishing with a newer perfkit redraws every run with the newer renderer; older app/<version>/ folders can stay. Publish --out without deleting what is already on the site (for example with keep_files on peaceiris/actions-gh-pages). What each section shows, and how the deltas pick the run they compare with, is in docs/report-page.md.

Exit code	Meaning
0	The site was written.
1	Usage error, such as a missing `--out`, or neither or both of `--report` and `--config`.
2	The report, the config or the history is invalid, unreadable or of another schema.
3	The history URL could not be fetched (any error but a 404).

Development

Node 22 (.nvmrc).

npm ci
npm run typecheck
npm run lint
npm test
npm run schemas   # rewrite schemas/*.schema.json after changing src/schemas; commit the result

Unit tests never reach the network (a mocha root hook calls nock.disableNetConnect()) and never see DIGITALOCEAN_TOKEN.

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
docs		docs
examples		examples
schemas		schemas
scripts		scripts
src		src
test		test
.gitignore		.gitignore
.mocharc.json		.mocharc.json
.nvmrc		.nvmrc
.prettierignore		.prettierignore
.prettierrc.json		.prettierrc.json
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
eslint.config.js		eslint.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json

Navigation Menu

Repository navigation

About

Repository files navigation

perfkit

What perfkit is

How it works

Prerequisites

How projects pin it

perfkit validate

perfkit run

perfkit collect

perfkit summary

perfkit sweep

How verdicts are computed

Capacity

Soak

Live results

Watch a local run live

perfkit serve

perfkit render

Development

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

perfkit

What perfkit is

How it works

Prerequisites

How projects pin it

perfkit validate

perfkit run

perfkit collect

perfkit summary

perfkit sweep

How verdicts are computed

Capacity

Soak

Live results

Watch a local run live

perfkit serve

perfkit render

Development

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages