Introduction¶

In local model inference scenarios, reaching the context window limit can cause requests to be rejected, interrupting ongoing sessions. Traditional compression approaches often struggle to work reliably or fail to meet the need for retrieving historical details.

Context Guardian aims to solve this problem. It provides two operating modes: one is an OpenAI-compatible proxy that sits in front of local models (such as Ollama and vLLM); the other is a native DeepSeek Harness (DSH) engine. Both use deterministic compression strategies to ensure that compressing the context does not crash the session, while preserving raw data intact on disk and supporting retrieval of archived details.

What Is It¶

Context Guardian is a plugin maintained by LuminariSoftwares. Its core function is to address session interruptions caused by context overflow in local models.

The project is released under the MIT license. It implements the proxy service in Python and provides a native engine integrated with DeepSeek Harness, supporting deterministic compression and archival management of context.

Core Features¶

Context Guardian includes the following core capabilities:

1. OpenAI-compatible proxy: Runs as a proxy service and supports local backends such as Ollama, LM Studio, vLLM, and LiteLLM.
2. Native DeepSeek Harness engine: Integrates as a DSH plugin and provides context compression.
3. Deterministic compression: Performs compression when the context reaches a threshold.
4. Fails Open: Even if compression fails, it keeps the session available and does not crash the session.
5. Disk persistence: During compression, the raw data is preserved intact on disk.
6. Historical detail retrieval: Supports retrieving archived context details by specific keywords or terms.

Installation and Enablement¶

Install the Proxy¶

First, install and start the proxy service using pip. The proxy listens on port 8786 by default.

pip install context-guardian && context-guardian

Configure DeepSeek Harness¶

Context Guardian provides a native DSH engine. Users need to add the corresponding engine configuration to the agent presets in DSH to enable deterministic compression.

Typical Usage¶

Configure the Upstream Model¶

When using proxy mode, specify the upstream local model endpoint via an environment variable.

GUARDIAN_UPSTREAM_URL=http://localhost:11434/v1

Configure the Client¶

If your client (such as Claude Code) needs to point to the proxy, set the following environment variable:

ANTHROPIC_BASE_URL=http://localhost:8786

Recall Archived Context¶

During an active session, use the command to retrieve previously archived raw context details.

context-guardian-recall "<term>"

Use Cases and Notes¶

Use cases:
* Scenarios where local models have limited context windows and long-term sessions are required.
* Scenarios where context must be compressed while preserving key historical information for on-demand access.

Notes:
* Compatibility: LM Studio, vLLM, and LiteLLM are expected to work, but have not been fully tested yet.
* Environment variable configuration: GUARDIAN_NUM_CTX must be set to the context window size actually supported by the model.
* API support: When the proxy is configured and pointed at the Anthropic Messages API, the proxy will apply compression to that endpoint.

Closing¶

Context Guardian provides a stable, deterministic context management solution for local model inference. Whether used as a proxy or a DSH engine, it effectively prevents sessions from being interrupted by context overflow while retaining complete historical data for later retrieval. You can view more plugin information on the directory page or get the source code on GitHub.