Knowledge Base Atomic Slicing Engine

Core Challenge: The 'Document Digestion' Problem in RAG Construction

Building an effective RAG knowledge base begins with breaking down lengthy source documents into precisely retrievable chunks. Directly uploading whole documents leads to:

• Semantic Fragmentation: Naive splitting by character or line count brutally severs code blocks, tables, or logical arguments, destroying knowledge integrity.
• Retrieval Noise: Chunks that are too large contain excessive irrelevant context, reducing precision. Chunks that are too small lose context and fail to answer questions.
• Lack of Metadata: Traditional slicing often lacks structured metadata like categories, keywords, and QA pairs, severely undermining vector search and semantic matching.
• Maintenance Overload: Numerous loose slice files without unified management make version tracking, deduplication, and error fixing a nightmare.

How This Skill Works: From Long Documents to Atomic Slices

The knowledge-engineering skill is positioned as the ETL engine for RAG pipelines. Its core is the concept of atomic slicing, where each output file is a self-contained, retrieval-ready knowledge unit. It addresses the above problems through these key capabilities and steps:

1. Structured Pre-analysis & Mandatory Planning: Before any splitting, a mandatory pre-computation Chain-of-Thought (CoT) is executed. This scans the document structure (H1-H3 headings, code blocks, tables) and simulates Token budgets. This ensures cuts are made at logical boundaries, not through mechanical truncation.
2. Intelligent Routing & Directory Isolation: It automatically creates an isolated folder per source file (e.g., source_name/api/) and routes slices into sub-directories based on content attributes (e.g., config vs. guide), physically preventing disorganization.
3. Metadata-Driven Slice Generation: Each .md slice is forced to include a YAML frontmatter. Fields like embedding_hint (semantic summary), hybrid_keywords, and qa_pairs (sample QA pairs) across 14 required fields are reasoned by the Agent. This metadata directly serves vector matching and intent recognition during retrieval.
4. Multi-layered Self-check & Audit Protocol: Post-generation, each slice undergoes immediate validation (validate_slice.py) for YAML completeness, semantic referents (prohibiting vague terms like 'the above'), and factual fidelity. This is followed by batch auditing (batch_audit.py) to ensure continuous indexing, no dead links, and no cross-slice semantic overlap.
5. Incremental Anti-Stall Process: For ultra-long documents, an Init → Step → Poll protocol is employed. It generates and validates only a small batch of slices at a time (max 5 per batch), preventing Token overflow and enabling checkpoint resumption for reliable processing.

Scope and Key Constraints

This skill is not a universal document converter; its boundaries are explicit:

• Input Formats: Supports text-centric formats like Markdown, Text, YAML, JSON, and PDF. It does not support processing images, scanned documents, databases, or dynamic web pages directly.
• Core Output: The artifacts are atomic slice files optimized for vector databases (e.g., Milvus, Pinecone) and retrievers. Its validation (evaluate_retrieval.py) depends on sentence-transformers to simulate real retrieval metrics (R@1, MRR, etc.).
• Execution Mandates: Strict adherence to core red lines like "absolute directory isolation", "semantic integrity over length limits", and the "zero-pronoun principle" is enforced. Any alteration of factual data (numerical values, parameters, error codes) is prohibited. Final delivery must pass both batch_audit.py and retrieval evaluation, outputting a chain of evidence.

In summary, this is a professional slice controller engineered for building high-quality, maintainable RAG knowledge bases, emphasizing process rigor, structured outputs, and proven retrieval effectiveness.