dsh-agent-rate-limit

When multiple agents concurrently call an LLM API, the easiest thing to overlook is quota management. Without unified token tracking and rate limiting, requests quietly push against or beyond TPM/RPM limits, triggering HTTP 429 responses or transient server overloads (such as Nvidia Service temporarily overloaded) and breaking agent workflows. dsh-agent-rate-limit is a DSH client plugin designed to eliminate exactly this pain point.

The plugin intercepts the DSH LLM streaming pipeline and enqueues every call into a 60-second FIFO sliding window. Tokens are recorded from the API usage data block, including cacheReadTokens and cacheWriteTokens, so the window total matches the billed amount. Before each new request is dispatched, the plugin checks remaining headroom and adaptively computes how long to wait, ensuring the window drains below the limit. In multi-agent concurrency, while one agent is waiting the others keep generating tokens; the plugin dynamically extends the wait based on the current overage ratio to avoid repeated collisions with the cap.

On the error-recovery side, the plugin automatically detects transient failures by matching patterns in statusCode, code, and message (rate limit, too many requests, quota, 429, service temporarily overloaded, PI_AI_ERROR) and retries with incremental backoff of 2s, 4s, 8s, 16s, 30s, only surfacing the error to the user after exceeding the configurable retry limit. It is ideal for developers orchestrating multiple concurrent agents against TPM/RPM-billed LLM services such as Alibaba Cloud Bailian, and for anyone who wants to silence 429 noise and let DSH workflows run to completion.