AI Agent Hub
Back to skills
🔒

Observability and Instrumentation

IT Ops & Security Updated 2026.08.30

Paste the following prompt into your AI chat to install this skill:

Please install @user_3c6cb52e/observability-and-instrumentation-sh-k1xmdy according to https://skillhub.cn/install/skillhub.md.

About this skill

Problem

The hardest part of a production incident is often not the stack trace, but the inability to answer “how far did the request get, why was it slow, and what did the dependency return?” If telemetry is added after launch, the first failure turns into archaeology. This skill treats observability as part of feature development: define 2–4 on-call questions first, then map each question to structured logs, metrics, or traces. It avoids console.log, high-cardinality labels, and cause-based alerting as substitutes for observability.

How It Works

  • Logging: Requires JSON events, stable field names, and a requestId correlation ID, and forbids secrets, tokens, and full PII.
  • Metrics: Follows RED and USE, emphasizing histograms and p95/p99 instead of averages; labels must come from bounded sets.
  • Tracing: Prefers OpenTelemetry for HTTP, gRPC, and common DB clients, with context propagated across async boundaries.
  • Alerting: Keeps only actionable symptom-based alerts, each with a runbook, threshold, duration, and either page or ticket severity.

Boundaries

Use it when adding services, endpoints, background jobs, external integrations, or reviewing PRs with retries, queues, and cross-service calls. It does not replace live debugging, performance profiling, or launch-day rollback checklists; those skills depend on the telemetry produced here.

Use Cases

  • Before adding a payment callback endpoint, define on-call questions, then configure structured logs, RED metrics, and request IDs.
  • When reviewing a PR with retries and queues, check for new telemetry, correlation IDs, bounded labels, and symptom-based alerts.
  • After deploying a background job, verify metric series, cross-service spans, and fire a test alert with a working runbook.
  • After an incident where slow requests were hard to trace, add RED/USE histograms with p95/p99 and move user IDs from labels to logs.

Best For

  • Backend engineers responsible for production services who need queryable request IDs, error rates, and latency percentiles at launch.
  • SREs diagnosing cross-service incidents who need logs, metrics, and traces correlated to one request to find slow hops.
  • Technical leads reviewing payments, queues, and retries who require telemetry, bounded labels, and no cause-based alerts.
  • Ops engineers building alert runbooks who need actionable pages with thresholds, durations, and escalation paths.