ChromaDB Document Vectorizer

What problem it addresses

Local document retrieval often gets stuck on precision and persistence: keyword search misses paraphrases, pure vector search can miss exact terms, and in-memory vectors disappear after restart. This skill turns Word, Markdown, PDF, and TXT files into a searchable vector store for semantic lookup rather than conversational memory.

How it works

It follows a practical pipeline:
- Extraction and chunking: parses .docx, .md, .pdf, and .txt, then splits text using sentence boundaries such as Chinese periods. Default chunks are around 200 characters to preserve meaning.
- Embedding: uses a dependency-free MD5-hash simulation by default to generate 384-dimensional vectors; it can also switch to paraphrase-multilingual-MiniLM-L12-v2 for multilingual semantic representations.
- Search: search_vectors(query_text, top_k) retrieves results with cosine similarity and optional thresholds. v3 adds an LRU query cache and BM25 + vector hybrid retrieval, with an alpha parameter to balance keyword and semantic signals.
- Management: vectorize_file, get_collection_stats, and clear_collection let you add documents, inspect the collection, and reset it. Data is persisted under chroma_data and restored on startup.

Boundaries and notes

This is best for document knowledge retrieval, not for managing conversation state. Real semantic embeddings require sentence-transformers, and file formats need their corresponding parsers. For large files, batch ingestion is advisable; for large corpora, tune chunk_size to balance recall and precision.