Benchmarking JIT Runtimes Without Lying to Yourself

A benchmark is an experiment, not a stopwatch wrapped around a loop. That distinction matters for a just-in-time compiled runtime because the program being measured can change while it runs. Methods may begin in an interpreter or baseline compiler, become optimized after gathering profiles, and return to less specialized code when an assumption fails. Meanwhile, the runtime allocates memory, collects garbage, starts helper threads, and may discover that the benchmark’s result is unnecessary.

The thesis of trustworthy JIT benchmarking is therefore simple: measure a declared runtime state under a controlled workload, and preserve enough evidence for someone else to reproduce or challenge the conclusion. One number from one warmed process cannot establish that an implementation is faster. A useful result separates startup from steady state, proves that intended work occurred, samples independent processes, reports variation, and states exactly which environment produced it.

This article focuses on measurement discipline. It treats tiering and deoptimization as benchmark variables rather than attempting to teach compiler construction, and it uses a JavaScript example only because the hazards are easy to demonstrate. The same reasoning applies to managed runtimes built around adaptive optimization.

Define the Claim and the Workload First #

Start by writing a falsifiable claim. “The new parser is faster” is not one. “For valid messages between 1 KiB and 64 KiB, the candidate reduces steady-state CPU time per parsed byte without increasing allocations or changing validation results” is much closer. It identifies the population, phase, resource, normalization unit, and correctness boundary.

The workload must represent the claim. Input size alone is rarely sufficient. A parser benchmark may depend on nesting depth, character encoding, valid-to-invalid ratio, key repetition, and whether inputs arrive as strings or byte buffers. A collection benchmark may depend on cardinality, key distribution, mutation rate, and hit ratio. If production has several meaningful modes, benchmark them as separate cases instead of blending them into one average that resembles none of them.

Decide which phase matters:

These are different products. Discarding warm-up can be appropriate for a long-lived service and dishonest for a command that exits after one operation. Conversely, including process startup in every iteration says little about the throughput of a resident worker. Report both phases when users experience both.

Record a semantic oracle before recording time. Candidate and baseline must produce equivalent output for every fixture. A checksum, result count, or full comparison outside the timed region can make work observable while keeping validation cost out of the measurement. Without an oracle, an optimization that skips malformed records or computes a different answer can appear to win.

Observe Warm-Up, Tiering, and Deoptimization #

“Run it a few times first” is not a warm-up policy. Adaptive runtimes optimize according to execution counts, type feedback, call-site behavior, and code size. Different cases can reach optimized code at different iterations, and a later input can invalidate an optimization. A fixed discarded count may cut through the middle of that transition.

Inspect the time series. Store every iteration duration and plot or print it in order. A typical series may contain an initially slow region, one or more drops as tiers change, compilation spikes, and occasional pauses. Do not infer the cause from shape alone; use runtime diagnostics where available to correlate compilation, optimization, deoptimization, and garbage collection with the observations. Diagnostic flags are runtime-version-specific and can perturb timing, so use them in an explanatory run, then measure normally after understanding the phases.

There are three defensible policies:

1. Measure cold behavior explicitly from new processes and retain every operation.
2. Warm until a predeclared stability rule is met, then measure a bounded steady-state window.
3. Measure the complete lifecycle and report the distribution over the expected job length.

A stability rule might require several consecutive windows whose medians differ by less than a chosen tolerance and whose allocation behavior is no longer trending. That tolerance is an experimental parameter, not a universal constant. Put a maximum warm-up limit on it. A case that never stabilizes is a result to investigate, not a reason to warm forever.

Deoptimization deserves deliberate coverage. Suppose a property-access benchmark warms only on objects with one layout, while production also sends objects created by another path. The benchmark measures a friendly specialization, not the deployed workload. Include representative input classes in their realistic proportions, plus separate adversarial cases that reveal cliffs. Preserve ordering when order affects feedback, or randomize from a recorded seed when order should not matter.

Prove the Benchmarked Work Still Exists #

Compilers are allowed to remove work whose result cannot affect observable behavior. Constant inputs can invite constant folding. An unused return value can permit dead-code elimination. Repeating an operation on an unchanging object can enable loop-invariant motion or caching. Some managed runtimes are conservative about these transformations, but relying on that accident makes a harness fragile across versions.

Make inputs unavailable as compile-time constants. Load deterministic fixtures at runtime, generate them from a seed outside the timed region, or pass them through the harness in a way the optimizer cannot trivially replace. Consume results through a checksum accumulated after each invocation. Print or validate that checksum after timing, and make the process fail if baseline and candidate differ.

Be equally suspicious of setup inside the timed operation. If one candidate converts a string to bytes before parsing and the other receives bytes directly, either include conversion for both because the real API starts with strings, or exclude it for both because the claim concerns parsing bytes. Setup is not inherently noise; it is either part of the user-visible operation or a controlled fixture boundary.

Avoid enormous inner-loop counts used only to make the stopwatch move. They can distort branch feedback, cache residency, allocation pressure, and safepoint behavior. Prefer a harness timer with sufficient resolution and automatic batching calibrated to a bounded duration. Report time per logical operation or per byte, but retain batch duration and operation count so normalization can be audited.

Black-box or escape utilities offered by benchmark frameworks can prevent certain eliminations, but they are not magic. Read their guarantees for the specific runtime and version. A value that “escapes” may still leave constant inputs or unrealistic data locality. Disassemble or inspect optimized code for especially consequential microbenchmarks, then pair that evidence with a higher-level benchmark that exercises the real call path.

Build an Isolated Process-Level Harness #

Consider two ways to count ASCII delimiter bytes. The example below is a harness sketch, not a claim that either implementation wins. It separates fixture construction, correctness, warm-up, and sampling, and it emits machine-readable observations rather than a selected summary.

import { performance } from 'node:perf_hooks';

type Candidate = (input: Uint8Array) => number;

const branchy: Candidate = (input) => {
  let count = 0;
  for (const byte of input) {
    if (byte === 44 || byte === 59 || byte === 124) count += 1;
  }
  return count;
};

const table = new Uint8Array(256);
table[44] = 1;
table[59] = 1;
table[124] = 1;

const lookup: Candidate = (input) => {
  let count = 0;
  for (const byte of input) count += table[byte]!;
  return count;
};

function makeFixture(seed: number, length: number): Uint8Array {
  let state = seed >>> 0;
  const input = new Uint8Array(length);
  for (let index = 0; index < input.length; index += 1) {
    state = (Math.imul(state, 1664525) + 1013904223) >>> 0;
    input[index] = 32 + (state % 95);
  }
  return input;
}

function run(name: string, candidate: Candidate): void {
  const fixtures = Array.from({ length: 64 }, (_, index) =>
    makeFixture(10_000 + index, 4_096 + index * 17),
  );
  const expected = fixtures.map(branchy);
  const actual = fixtures.map(candidate);
  if (actual.some((value, index) => value !== expected[index])) {
    throw new Error(`${name} changed the result`);
  }

  for (let round = 0; round < 200; round += 1) {
    candidate(fixtures[round % fixtures.length]!);
  }

  let checksum = 0;
  const samples: number[] = [];
  for (let round = 0; round < 400; round += 1) {
    const input = fixtures[round % fixtures.length]!;
    const started = performance.now();
    checksum += candidate(input);
    samples.push(performance.now() - started);
  }

  process.stdout.write(`${JSON.stringify({ name, checksum, samples })}\n`);
}

const candidates = { branchy, lookup } as const;
const name = process.argv[2] as keyof typeof candidates;
const candidate = candidates[name];
if (!candidate) throw new Error('choose branchy or lookup');
run(name, candidate);

One process should run one candidate and then exit. An outer controller launches many processes, alternates or randomizes candidate order, records the seed and environment, and treats the process as the independent experimental unit. This limits cross-candidate contamination from profiles, generated code, heap history, and global caches. A separate cold-start experiment would omit warm-up and time process launch from the controller.

The numbers in the sketch are harness settings, not recommended universal values. Tune them so timer overhead is small relative to each batch, total duration is manageable, and the operation count resembles the phase being studied. If a benchmark framework already handles calibration, forks, result consumption, and runtime diagnostics correctly, use it instead of maintaining a homegrown substitute.

Analyze Independent Samples, Not the Best Run #

Iterations inside one process are correlated. They share generated code, heap state, CPU assignment, thermal conditions, and background runtime activity. Ten thousand loop timings are not ten thousand independent trials. For comparisons, aggregate within each process and analyze a collection of fresh process runs.

Use robust summaries such as median and selected percentiles, and show the raw process-level distribution. Means remain useful when total resource consumption matters, but they are sensitive to rare pauses. Report uncertainty with a confidence interval obtained by a method suitable for the data, such as a bootstrap over independent process summaries. If candidate and baseline runs are paired by machine and round, analyze their paired differences or ratios rather than pretending they were unrelated.

The basic ratio is easy to communicate:

A ratio above one favors the candidate, but the estimate needs context. Include the interval, sample count, and full distribution. Do not declare victory because a significance test crossed a threshold while the effect is operationally irrelevant. Conversely, do not reject a useful large effect merely because a noisy laptop run lacks power. Predeclare a minimum effect worth shipping and collect enough independent rounds to distinguish it from normal variation.

Multiple cases create another trap. If twenty fixtures are tested and only the largest apparent win is reported, selection bias manufactures a story. Publish all declared cases, including regressions. Weighting them into one score is acceptable only when weights correspond to a documented workload mix; retain per-case results so a severe minority regression remains visible.

Outliers should not be deleted because they look inconvenient. Investigate whether they correspond to GC, deoptimization, OS preemption, compilation, or a harness fault. Exclude a run only under a rule written before examining candidate labels, and report the exclusion. Tail events may be exactly what a latency-sensitive application needs to understand.

Control and Record the Environment #

A reproducible record names the source revision, build mode, runtime distribution and exact version, runtime flags, operating system, architecture, CPU model, available cores, memory limit, power policy, and container or virtual-machine limits. It also records benchmark parameters, fixture hashes, random seeds, process count, warm-up rule, sampling rule, and analysis code version.

Reduce avoidable interference. Use release builds. Keep baseline and candidate on the same host when possible. Disable unrelated services, avoid concurrent CI jobs, and ensure the machine is not thermally throttling. Pinning a process to a core can reduce migration noise, but it can also make the setup less representative; state whether affinity was used. Fixed CPU frequency improves controlled microbenchmarks, while normal frequency scaling may better represent deployed hardware. The right choice follows the claim.

Containers provide packaging, not identical hardware. A runner may share cores with unknown neighbors or receive a changing quota. Cloud instances can vary within a named family. Dedicated benchmark workers and repeated paired rounds help, but do not erase these facts. Continuous integration is useful for catching large regressions with generous budgets; it is usually a poor place to support fine percentage claims unless the runners are intentionally controlled.

Archive raw observations rather than only a dashboard screenshot. Future runtime upgrades can then be compared with the same fixtures and analysis. A compact manifest plus JSON or CSV samples is often enough to reproduce the result and expose accidental changes in flags or hardware.

Recognize Failure Modes and Scope Limits #

Microbenchmarks isolate mechanisms, which is both their strength and their limit. A faster property access or delimiter loop does not imply a faster request. The complete operation may be dominated by allocation, decoding, I/O, locks, or serialization. After a microbenchmark explains a mechanism, verify it in a component benchmark and then in the representative service path.

Common ways to obtain a confident but wrong answer include:

• Benchmarking only steady state for short-lived programs, or only cold state for resident services.
• Warming with uniform objects and measuring a production path with mixed shapes or types.
• Running candidates sequentially in one process so the second inherits runtime state.
• Timing fixture generation for one candidate but not the other.
• Letting the result go unused or using constant data that permits work to disappear.
• Comparing different runtime versions, compiler flags, CPU quotas, or debug settings.
• Reporting the minimum iteration, which mostly selects the luckiest scheduling interval.
• Treating every pause as JIT activity without correlating GC and operating-system events.
• Tuning the harness until a preferred implementation wins, then presenting the final configuration as predetermined.

Runtime upgrades are not noise to hide. Optimization heuristics, collectors, standard libraries, and generated code can change. If the product supports several versions, benchmark each supported version or narrow the claim to one. An implementation that depends on an undocumented optimization may be fast today and brittle tomorrow; readability and algorithmic complexity remain relevant tradeoffs.

Verify the Result Before Acting on It #

Review a benchmark as if it were a production change. First, run correctness fixtures and property tests outside timing. Then inspect the ordered warm-up trace and correlate suspicious transitions with diagnostic runs. Confirm through checksums, allocation counts, or generated-code inspection that the intended work remains. Repeat each candidate in fresh processes with randomized order, and preserve every process summary.

Attempt to falsify the conclusion by varying representative dimensions: input distribution, size, call-site diversity, runtime version, memory pressure, and job length. A genuine mechanism should have an explainable response. A result that reverses unpredictably under tiny harness changes is not ready to guide design.

Finally, take the candidate to the level users experience. Run a component or request benchmark with realistic concurrency and data, watch latency distributions, throughput, errors, CPU, allocation, and GC, and canary the change if its blast radius warrants it. Keep a rollback path and a regression case tied to the original workload.

Trustworthy JIT benchmarking does not require pretending the runtime is static. It requires making adaptation part of the experiment. State the phase, preserve the work, isolate processes, analyze independent samples, publish variation, and record the environment. The result may be less dramatic than the fastest number from a loop, but it is far more likely to survive contact with another machine, another runtime release, and the software that actually matters.