Joshua Rosato~/linux
cd ../notes

TIL ·

Core dumps are free until you need one

The kernel refuses to write core dumps by default (RLIMIT_CORE starts at 0), which is why production crashes usually leave nothing behind. systemd raises that limit for everything it starts and registers itself as the kernel’s kernel.core_pattern handler, so on a systemd host dumps already work — they just land somewhere nobody looks:

bash
sysctl kernel.core_pattern          # |/usr/lib/systemd/systemd-coredump
sysctl kernel.core_ulimit
systemctl show example.service -p LimitCORE

systemd-coredump logs the signal, command line, cgroup and a backtrace to the journal, then stores a compressed core under /var/lib/systemd/coredump/, aged out by /usr/lib/tmpfiles.d/systemd.conf. coredumpctl is the interface to all of it:

bash
coredumpctl list -1
coredumpctl info 1234              # or match by command name or executable path
coredumpctl debug 1234             # opens gdb on the core, with the right paths
coredumpctl -o core.1234 dump 1234 # the raw core file

Matching works on PID, COMM or EXE, plus --since. An unprivileged user only sees their own crashes, because the metadata lives in the journal — a real reason for a reporting path that runs privileged.

Two things decide whether a dump is useful when you finally want it. Symbols: a backtrace of bare addresses means the debuginfo for that exact build is not on the host, so install the matching -dbg/debuginfo package, point DEBUGINFOD_URLS at a debuginfod server, or check which build you are dealing with via eu-unstrip -n core. Size: a multi-gigabyte JVM core is a disk-full event, so /etc/systemd/coredump.conf is where you bound it — ProcessSizeMax=, ExternalSizeMax=, MaxUse=, KeepFree= — or set Storage=none to keep the log line and discard the file.

For a process that is alive but wedged, do not make it crash to get its memory:

bash
gcore -o /tmp/example 1234
gdb -p 1234 -batch -ex 'thread apply all bt' -ex 'gcore /tmp/core'

Managed runtimes have a better answer still: -XX:+HeapDumpOnOutOfMemoryError for the JVM, GOTRACEBACK=crash to turn a Go panic into a SIGABRT and a core, faulthandler.enable to print a Python traceback instead of a bare segfault. Containers need the plumbing — bind the host’s /run/systemd/coredump into the container, or run with --ulimit core=-1 and a host-side handler — because without one a containerised crash vanishes at the container boundary.

esc