About this skill

Problem

When a Linux system shows high CPU load, process stalls, memory anomalies, I/O latency, or KVM virtualization jitter, generic tools like top or vmstat often expose symptoms rather than the underlying event path, affected context, or problematic thread/CPU.

How It Works

The skill guides perf-prof-based analysis with a structured workflow:

Bound the problem first: user space vs kernel space, guest vs host, CPU vs process vs I/O vs scheduling.
Choose analyzers such as profile for sampling, multi-trace for latency, kmemprof for allocation, rundelay for scheduling delay, and kvm-exit for VM-exit analysis.
Use -h, references/, and event docs to verify fields, filters, and behavior of options like key, batch, and watermark.
Interpret output progressively: distribution overview → threshold filtering → abnormal stacks and context.

Limits

Best for users familiar with Linux kernel events, tracepoints, perf events, or BPF. It complements application tracing rather than replacing it. Dynamic probes, expressions, and latency correlation require understanding event context and field semantics.

Screenshots

Use Cases

When P99 latency spikes, use multi-trace to correlate event paths and isolate slow syscalls or scheduling delays.
When processes stall intermittently, use task-state and rundelay to analyze R/S/D time and scheduling delay distribution.
When VM load is abnormal, use kvm-exit and kvmmmu to inspect VM-Exit latency and MMU page-table behavior.
When memory leaks are suspected, use kmemprof and kmemleak to track allocation/free byte counts and stacks.

Best For

SREs troubleshooting Linux latency who need to correlate event paths, slow calls, and scheduling delays.
Platform engineers maintaining containers or KVM who need to isolate VM-Exit, MMU, and scheduling anomalies.
Systems engineers handling kernel memory issues who want stacks and byte counts to find allocation/free hotspots or leaks.
Ops engineers analyzing CPU or I/O anomalies who need to separate user space, kernel space, and guest/host causes.