Week 22 - Tracing and Performance Mastery: ftrace, perf, BPF

22.1 Conceptual Core

• The Linux observability triad: ftrace (function tracer; in-kernel only), perf (sampling profiler + tracepoint subscriber), eBPF (programmable, low-overhead).
• Each has its niche. For "what is the kernel doing?" → ftrace function_graph. For "where is CPU time spent?" → perf record + flamegraph. For "summarize a behavior with low overhead" → eBPF.

22.2 Mechanical Detail

• ftrace via /sys/kernel/tracing/. Set current_tracer, filter with set_ftrace_filter, dump trace. Modern frontend: trace-cmd.
• perf:
• `perf stat - counter snapshot.
• perf record -g + `perf report - sampling profiler with call graphs.
• perf script to feed into FlameGraph.pl for flamegraphs.
• `perf trace - strace-equivalent with low overhead.
• `perf top - live profiler.
• bpftrace for one-liners; libbpf C for production tools.

22.3 Lab-"End-to-End Profiling"

1. Take a service running on a host. Capture: perf record -F 99 -ag -- sleep 30.
2. Generate a flamegraph.
3. Identify the top three CPU consumers; for each, propose a hypothesis and a fix.
4. Compare with the same workload profiled by parca or pyroscope if available.

22.4 Hardening Drill

• perf requires kernel.perf_event_paranoid ≤ 2 for unprivileged use. Decide your policy: tighter (=3, perf disabled for non-root) or looser (=1, allow unprivileged users to profile their own processes).

22.5 Performance Tuning Slice

• Run runqlat, cpudist, offcputime (BPF) on a busy host. Build a one-page "what's wrong with this host?" diagnostic flow.

Comments