LLM Benchmarking and Self-Optimization

Problem

Teams comparing LLM API performance often face unstable metrics, missing first-token latency, token throughput, percentile stats, and no durable stress-testing workflow. Manual scripts also tend to miss dimensions such as temperature, max_tokens, concurrency, and multilingual behavior, making results hard to reproduce and archive.

How It Works

This skill turns LLM API testing into a repeatable workflow:
- Benchmarking: run quick or full modes to collect TTFT, TPS, ITL, P50/P90/P99, and success rate.
- Stress testing: specify duration and concurrency, with checkpointing, exponential backoff, and crash recovery.
- Reporting and comparison: generate HTML/PDF reports, compare models side by side, and infer trends from historical runs.
- Self-optimization: use optimize suggest and auto-tune to recommend or grid-search parameters such as temperature and max_tokens, with scoring targets like tps, ttft, and balance.

Scope and Caveats

The API endpoint should be OpenAI-compatible, such as /v1/chat/completions. Auto-tuning sends real requests, so it can affect quotas and runtime; stress tests should be run in a controlled environment. Local Ollama can be tested through the compatible endpoint, usually with the key set to ollama.