Observability and Instrumentation
Paste the following prompt into your AI chat to install this skill:
Please install @user_3c6cb52e/observability-and-instrumentation-sh-k1xmdy according to https://skillhub.cn/install/skillhub.md.
About this skill
Problem
The hardest part of a production incident is often not the stack trace, but the inability to answer “how far did the request get, why was it slow, and what did the dependency return?” If telemetry is added after launch, the first failure turns into archaeology. This skill treats observability as part of feature development: define 2–4 on-call questions first, then map each question to structured logs, metrics, or traces. It avoids console.log, high-cardinality labels, and cause-based alerting as substitutes for observability.
How It Works
- Logging: Requires JSON events, stable field names, and a
requestIdcorrelation ID, and forbids secrets, tokens, and full PII. - Metrics: Follows
REDandUSE, emphasizing histograms andp95/p99instead of averages; labels must come from bounded sets. - Tracing: Prefers OpenTelemetry for HTTP, gRPC, and common DB clients, with context propagated across async boundaries.
- Alerting: Keeps only actionable symptom-based alerts, each with a runbook, threshold, duration, and either
pageorticketseverity.
Boundaries
Use it when adding services, endpoints, background jobs, external integrations, or reviewing PRs with retries, queues, and cross-service calls. It does not replace live debugging, performance profiling, or launch-day rollback checklists; those skills depend on the telemetry produced here.
Use Cases
- Before adding a payment callback endpoint, define on-call questions, then configure structured logs, RED metrics, and request IDs.
- When reviewing a PR with retries and queues, check for new telemetry, correlation IDs, bounded labels, and symptom-based alerts.
- After deploying a background job, verify metric series, cross-service spans, and fire a test alert with a working runbook.
- After an incident where slow requests were hard to trace, add RED/USE histograms with p95/p99 and move user IDs from labels to logs.
Best For
- Backend engineers responsible for production services who need queryable request IDs, error rates, and latency percentiles at launch.
- SREs diagnosing cross-service incidents who need logs, metrics, and traces correlated to one request to find slow hops.
- Technical leads reviewing payments, queues, and retries who require telemetry, bounded labels, and no cause-based alerts.
- Ops engineers building alert runbooks who need actionable pages with thresholds, durations, and escalation paths.
Related Skills
An engineer-focused HTTP request smuggling handbook covering CL.TE, TE.CL, TE.TE obfuscation, HTTP/2 downgrade, and client-side desync detection.
Analyzes network captures from Wireshark, tcpdump, Fiddler, and Charles, then pinpoints TCP, HTTP, DNS, and TLS issues with filter expressions and remediation steps.
Lightweight Python Linux HIDS exposing SSH brute-force, web attack, and webshell alerts via MCP with ban controls.
Detects AI-generated images from Stable Diffusion, Midjourney, and GPT-4o via Tencent Cloud image moderation and returns risk scores.