perf counts, perf samples — pick the right one

perf has two modes and they answer different questions. Decide which one you are asking before you collect anything.

Count when the question is how much: cycles, instructions, cache misses, context switches, page faults over a window long enough to mean something.

perf stat -p "$(pidof example)" sleep 30

Hardware counters are scarce, so when you request more events than the CPU has slots, perf multiplexes them and scales the numbers. The run reports the fraction of time the counters were actually running; below roughly 90% on a short window, treat the result as indicative only. Long window, few events, and the question never arises.

Sample when the question is where the time went:

perf record -F 99 -a -g --call-graph dwarf -o /tmp/prof.data -- sleep 30
perf report --stdio --sort comm,dso,symbol --percent-limit 0.5
perf report --stdio -g graph

-F 99 is the conventional fixed sample rate; the default for cycles is adaptive and throttles itself, which quietly distorts a thirty-second run. --call-graph decides whether your stacks are usable at all: fp (the default) needs frame pointers, so a binary built with -fno-omit-frame-pointer yields truncated stacks; dwarf unwinds through .eh_frame at a real CPU cost and needs nothing from the compiler, which makes it the sane default on x86-64; lbr uses the branch recorder, and is x86-only and approximate.

A browser is optional. perf report --stdio --percent-limit 1 finds the top consumers on its own, and perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg (Brendan Gregg’s FlameGraph) turns the same file into a flame graph when you want the shape rather than the ranking. perf top -p PID is the live version of the same idea.

Two things stop this working, and both have obvious fixes:

• Permissions. cat /proc/sys/kernel/perf_event_paranoid — distributions set it tightly on purpose. Without hardware access a software event still profiles userspace: perf record -e cpu-clock -a -g --all-user. Where the value blocks even that, you need root or a sysctl change, which is the host owner’s decision rather than yours to make in a shell.
• Symbols. Frames shown as [unknown] or as raw addresses mean the debuginfo for that build is missing. Install the matching debuginfo packages, set DEBUGINFOD_URLS so build-ids resolve over the network, or rebuild with -g.

Thirty seconds of samples is noise; aim for thousands of samples. To hand the profile to someone else, perf archive bundles it with the binaries and debuginfo they need to open it.