Introduction

When iterating on skills, prompts, or RAG configurations within DeepSeek Harness (DSH), the common practice is to manually test a few dialogues after each change, relying on subjective impressions to judge “seems better.” This subjective comparison is insufficient for supporting release decisions, especially when only the knowledge artifact is changed while keeping the same model and sample set. It also fails to capture gaps in real-world online tasks for regression test cases.

This article introduces the community plugin lizhiyao/oh-my-knowledge (referred to as OMK). Designed for knowledge maintainers of prompts, RAG, skills, agents, and workflows, OMK uses a doctor → eval → observe closed loop to transform questions like “Is it testable?”, “Is it better?”, and “What gaps were exposed online?” into reviewable evidence. It natively supports Codex, Claude Code, and DeepSeek Harness.

What Is This

OMK (oh-my-knowledge) is maintained by lizhiyao under the MIT license. The current npm version is v0.54.0, requiring Node.js >= 22. The project’s positioning is: Observe. Measure. Know. — ensuring every knowledge change in AI applications is backed by evidence.

The core idea is to control variables: using the same model and evaluation samples while only changing the knowledge artifact version. It then provides a one-line conclusion, confidence intervals, failing samples, and cost to indicate whether it’s ready for release. DSH users can install OMK as a native bundle into existing profiles, reusing the current session’s provider, credentials, and sandbox for controlled evaluations, and open persistent DSH task trajectories in Studio.

Core Features

OMK maps common decisions to specific commands. The comparison table in the README serves as a capability index:

Decision Command Evidence Obtained
Is the artifact structure sufficient for evaluation? omk doctor Structure, dependency, security, and testability checks
Is v2 better than v1? omk eval One-line conclusion, confidence intervals, failing samples, cost
Why did it pass or fail? omk studio Report view with scores, diagnostics, and examples
Should this version be accepted? omk promote / omk evolve Evidence-gated acceptance or generation of better candidates
What happened in a real task? omk observe / Studio Task Trajectory Requests, visible Knowledge, tool calls, results, replies, and user corrections
What gaps did online usage expose? omk observe / omk sample --from-traces Draft production gaps for review, convertible into evaluation samples

The main loop is as follows:

Modify prompt / RAG / skill / agent artifact
→ omk doctor
→ omk eval (same model, same samples)
→ Read report / Studio evidence
→ promote or evolve
→ observe real usage, sample --from-traces to draft regression samples

In DSH, the plugin integrates via the host executor: eval.yaml can omit the top-level executor, and the actual executor is always the current DSH host; each sample creates a new DSH agent/session, reusing the profile’s provider, credentials, tools, and sandbox. /omk observe relies on ctx.sessionPersistence to list recent terminal root sessions in read-only mode and generate Studio Task Trajectory URLs.

Installation and Activation

If DSH is already a local harness, it’s preferable to install OMK into existing profiles rather than letting OMK launch another runtime. The official documentation provides the following commands:

dsh plugin --profile web add oh-my-knowledge
dsh --profile web

After entering DSH, you can use slash commands to operate (configuration paths are resolved relative to the current session cwd):

/omk eval eval.yaml
/omk observe
/omk observe <session-id>

The built-in web profile provides ctx.commands and command adapters; headless, ACP, and JSON-RPC surfaces do not currently consume these commands. For local source builds, you can first run npm run build, then link using the absolute path: dsh plugin --profile web add /absolute/path/to/oh-my-knowledge.

Global CLI Installation

When not using the DSH host, you can install globally via npm:

npm i -g oh-my-knowledge

You need to preconfigure at least one authenticated model runtime (Codex CLI, Claude Code, or API executor). Full documentation is available at oh-my-knowledge.pages.dev.

Typical Usage

Five-Minute Demo (CLI)

omk init scaffolds two skill variants and three sample use cases, allowing you to run a comparative evaluation directly:

npm i -g oh-my-knowledge
omk init demo && cd demo
omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
omk eval --control code-review-v1 --treatment code-review-v2

--dry-run is used to preview calls and costs; a formal omk eval generates an HTML report with a one-line conclusion within about five minutes. Initially, with only 3 use cases, the conclusion is often UNDERPOWERED (insufficient data), which is normal. The README suggests accumulating around 20+ use cases before trusting the ship/no-ship judgment.

When setting Codex as the default executor in a regular terminal, you can add the following to your shell profile:

export OMK_EXECUTOR=codex
# Optional: export OMK_MODEL="your-codex-model"

Performing Eval Within DSH

In a DSH session with the plugin loaded, edit the eval.yaml under the project, then execute:

/omk eval eval.yaml

In host mode, the top-level executor in eval.yaml is omitted; the model under test defaults to inheriting the current session unless explicitly specified in the configuration. Evaluation reports are written to the project’s .omk/reports. Sample.mocks is not currently supported in DSH host mode.

Viewing a Single Task Trajectory

To see what happened behind a single Codex or DSH session without first running observe ingest, you can use:

omk studio

Studio opens a local session overview at http://127.0.0.1:7799 by default. Under DSH, using /omk observe <session-id> yields a Task Trajectory URL pointing to the actual listening address, with four columns (Conversation, Actions, Results, Knowledge) showing requests, tool calls, returns, and visible context. Trajectories only reconstruct observable facts from logs, not inferring hidden reasoning.

Installing Agent Skill in Coding Agents

omk install omk-agent-skill

In Claude Code, you can invoke /omk eval, /omk evolve, /omk sample; in Codex, you need to let the agent execute the omk CLI directly, e.g., omk evolve skills/my-skill.md.

Applicable Scenarios and Notes

Who is it for: Authors and maintainers who need to make release decisions for skill, prompt, RAG, or agent artifacts; teams that need to perform controlled A/B testing within DSH profiles and feed online traces back into evaluation sets.

Not ideal for: End users who only want to passively use a skill without caring about version evidence or regression samples.

Usage Notes:

  1. OMK runs with the current DSH process permissions. Before installation, read the GitHub source code and MIT license to confirm it meets your security and compliance requirements.
  2. The DSH plugin directory SkillHub is a community-maintained independent site, with no official affiliation to DeepSeek / High-Flyer.
  3. When comparing results from different executors (e.g., codex vs. claude), OMK will warn about runtime fingerprints; artifact versions should be compared under a fixed executor.
  4. The CLI may prompt for new versions after the first run (at most once every 20 hours); setting OMK_SKIP_UPDATE_CHECK=1 disables this permanently.

Conclusion

OMK brings the question “Can we release this updated knowledge artifact?” back from subjective impressions to reviewable evidence: use doctor and eval before release, and observe and sample after release to close the loop. DSH users can reuse the host runtime within existing profiles by simply running dsh plugin --profile web add oh-my-knowledge.