dsh-livebench-panel

Running a LiveBench evaluation used to mean opening a terminal, stitching together a long parameter string, watching the log, and then manually parsing JSONL files to compute scores. dsh-livebench-panel folds all of that into a single tab on the right side of the dsh web trajectory view: pick your model, choose a category, hit Start, and the local LiveBench process takes over.

The panel offers a full parameter experience — every model from every provider is grouped and listed, reasoning-effort levels are encoded into the entry name so runs at different intensities stay independently comparable, and question-set release, six categories, specific tasks, question index range, and max-tokens are all dropdown selections. Up to nine models can run concurrently, logs refresh every 2.5 seconds, and the results table computes per-model-per-task averages directly from the ground-truth judgment file, with sorting, drag-to-reorder rows, and one-click deletion — no need to wait for LiveBench to publish its own leaderboard. A built-in Baseline probe set goes further: it uses a reference model's observed pass/fail records as a yardstick, so a handful of questions the reference model could not solve can quickly bracket where a new model sits.

Built for developers who already work inside the dsh web workflow for conversation, coding, or tool-use tasks, and for evaluators who need to frequently benchmark several models across LiveBench categories. As long as your machine has a LiveBench checkout and a ready evaluation venv, installing the plugin and opening the trajectory view is all it takes — no more memorising CLI flags.