Raxol. Scoreboard
(Raxol v2.8.0)
View Source
Deterministic performance counters for CI, one JSON record per run.
Each scenario drives a playground demo through the frame stages the
rendering engine runs, in-process with no TTY and no engine GenServer: the
theme the engine would resolve (Pipeline.theme/1), view/1, layout
preparation, Raxol.UI.Layout.Engine.apply_layout/3,
Raxol.UI.Renderer.render_to_cells/2, the border-beam pass
(Pipeline.apply_border_beam/2), and the terminal frame
(Backends.terminal_frame/2) diffed against the previous one. The stage
functions are the engine's own, not copies, and frames/2 is held to the
engine's exact output by test/raxol/scoreboard_test.exs, which drives a real
Raxol.Core.Runtime.Rendering.Engine through the same keys. Plugin
transforms are not applied: they come from the dispatcher's plugin manager
at runtime, and the demos run with none loaded, as an app does by default.
It counts three things, and one ratio:
- frame bytes: the size of the ANSI frame payload the engine writes.
Exact. A terminal that supports synchronized output also receives the
16-byte
\e[?2026h/\e[?2026lwrapper per frame, which is not counted. Every frame is first replayed throughRaxol.Terminal.Emulatorand must reproduce the screen it was rendered from, character and style (replay_mismatch/1), so the byte count can only fall by sending less for the same screen. - reductions: the BEAM's unit of work, counted for the measuring
process only (
Process.info(self(), :reductions)). Unlike wall-clock time it does not move with runner load, which is why pull requests can be compared against master on shared CI runners. It is not perfectly repeatable: runs on one machine agree within a few hundredths of a percent, but two CI runs of the same commit have differed by up to 0.2%, so the reduction metrics carry a 0.5% noise tolerance inpriv/scoreboard/metrics.json. Reductions also move with the Elixir/OTP version, which is why the record carries the full toolchain. - screen memory: the screen buffer the engine holds after the table
script, with shared subterms counted once, per cell. Exact. This is the
render path's
Raxol.Terminal.Cell; the buffer gate's memory rows (merged in with--include) measureRaxol.Terminal.Buffer. - work ratio: a keystroke's reductions over the first frame's. A renderer that only redoes what a keystroke changed sits far below 1; the table demo is at 0.98, so a keystroke there costs nearly a full redraw. A ratio also falls when the first frame gets slower, so read it next to the reductions it divides.
Every scenario runs once to load code, then three times in fresh processes;
the reported reductions are the median. Metric ids, labels, units and the
direction of "better" live in priv/scoreboard/metrics.json, which the CI
comment and release-note renderer (.github/scripts/scoreboard.mjs) read
too. Every metric this module emits must be defined there.
Run via MIX_ENV=test mix raxol.scoreboard.
Summary
Functions
Rewrite priv/scoreboard/screens.refs from the current screens. Run from
the project root after an intended visual change, and commit the file.
Compare screen_fingerprints/0 with priv/scoreboard/screens.refs.
Returns :ok, or {:error, problems} listing every frame whose screen
changed, has no reference, or has a reference but no longer exists.
Run every scenario and return %{metric_id => value}.
The metric definitions from priv/scoreboard/metrics.json.
Every frame app draws at 120x40, as {payload, screen}: its first frame (a
keyframe), then one frame per key in keys, each diffed against the one
before. payload is the ANSI the rendering engine writes for the same model
sequence (test/raxol/scoreboard_test.exs holds the two to the same bytes);
screen is the Raxol.Terminal.ScreenBuffer it was rendered from. The
scoreboard's byte metrics are the payload sizes.
Build a record: the collected metrics merged with extra (for example the
buffer gate's memory rows), stamped with the commit and toolchain.
Replays frames (as frames/2 returns them) in order through one
Raxol.Terminal.Emulator and checks, after each frame, that every cell's
character and style equal the screen the frame was rendered from.
A fingerprint of every scenario frame's screen, as {"table:3", sha256}:
the scenario name and frame index (0 is the first frame), and a hash of
every cell's character and full style, serialized stably across Elixir
releases and architectures (the same rules as Raxol.RATE).
Types
Functions
@spec bless_screens() :: :ok
Rewrite priv/scoreboard/screens.refs from the current screens. Run from
the project root after an intended visual change, and commit the file.
@spec check_screens() :: :ok | {:error, [String.t()]}
Compare screen_fingerprints/0 with priv/scoreboard/screens.refs.
Returns :ok, or {:error, problems} listing every frame whose screen
changed, has no reference, or has a reference but no longer exists.
@spec collect() :: metrics()
Run every scenario and return %{metric_id => value}.
Before anything is measured, two checks keep the byte metrics honest, and either failing raises:
- every scenario's frames are replayed through
Raxol.Terminal.Emulatorand must reproduce the screen they were rendered from (replay_mismatch/1), so bytes cannot fall by sending a broken frame; - every frame's screen must match its fingerprint in
priv/scoreboard/screens.refs(check_screens/0), so bytes cannot fall by drawing less. An intended visual change re-blesses the fingerprints (mix raxol.scoreboard --bless) in the same pull request, where review sees it.
@spec definitions() :: [map()]
The metric definitions from priv/scoreboard/metrics.json.
@spec frames(module(), [String.t()]) :: [{binary(), Raxol.Terminal.ScreenBuffer.t()}]
Every frame app draws at 120x40, as {payload, screen}: its first frame (a
keyframe), then one frame per key in keys, each diffed against the one
before. payload is the ANSI the rendering engine writes for the same model
sequence (test/raxol/scoreboard_test.exs holds the two to the same bytes);
screen is the Raxol.Terminal.ScreenBuffer it was rendered from. The
scoreboard's byte metrics are the payload sizes.
Build a record: the collected metrics merged with extra (for example the
buffer gate's memory rows), stamped with the commit and toolchain.
Raises if any metric id is not defined in priv/scoreboard/metrics.json.
@spec replay_mismatch([{binary(), Raxol.Terminal.ScreenBuffer.t()}]) :: map() | nil
Replays frames (as frames/2 returns them) in order through one
Raxol.Terminal.Emulator and checks, after each frame, that every cell's
character and style equal the screen the frame was rendered from.
Returns nil when every frame reproduces its screen, otherwise the first
frame that does not: %{frame: index, cells: mismatched_cell_count, first: {x, y, expected_char, replayed_char}}.
A fingerprint of every scenario frame's screen, as {"table:3", sha256}:
the scenario name and frame index (0 is the first frame), and a hash of
every cell's character and full style, serialized stably across Elixir
releases and architectures (the same rules as Raxol.RATE).