Oracle AI/ML Design & Evaluation Expert

Problem

LLM features often get stuck between prompts, RAG retrieval, agent tool use, safety guardrails, and cost/latency budgets. Oracle is an AI/ML design and evaluation skill that turns AI requirements into testable, implementable, and monitorable specs. It does not write production code directly; it focuses on system prompts, RAG, LLM-as-judge, OWASP LLM Top 10, and token budgets.

How It Works

Oracle follows ASSESS → DESIGN → EVALUATE → SPECIFY:
- Assess the current prompt versioning, retrieval quality, guardrails, evaluation sets, and cost/latency baselines, then flag anti-patterns and blocking gates.
- Design prompt patterns, RAG chunking/hybrid search, agent/tool contracts, model routing, and graceful degradation paths.
- Evaluate with golden test sets, Recall@5, Faithfulness, regression gates, and LLM-as-judge calibration to avoid single-judge bias.
- Hand off implementation contracts to Builder, test strategy to Radar, security requirements to Sentinel, RAG ingestion to Stream, and monitoring SLOs to Beacon.

It treats prompts as versioned code, prioritizes retrieval quality over model size, and designs safety as layered guardrails. Every proposal should include token budgets, p95 latency, and release gates.

Boundaries

Use it for AI feature design, prompt/RAG/agent architecture, LLM safety, evaluation frameworks, and cost/latency optimization. Route to Builder, Gateway, Sentinel/Probe, Radar, or Nexus when the main task is implementation, general API contracts, penetration testing, test automation, or multi-agent orchestration. Avoid shipping unevaluated prompts, using unvalidated LLM output in critical decisions, and hard-coding model names, and consider RAG poisoning, position bias, length bias, and cache cost tradeoffs.