Linux Troubleshooting Cheat Sheet

Fast system snapshot

Capture context before changing anything.

date -Is
hostname
whoami
uptime
cat /etc/os-release | head
systemctl --failed
free -h
df -hT
df -i
ps -eo pid,ppid,user,stat,%cpu,%mem,etime,cmd --sort=-%cpu | head
ip -br addr
ip route
ss -s
journalctl -p warning -n 30 --no-pager

Read the signals correctly

High %us

Application/user-space work is consuming CPU.

High %sy

Kernel/system work is elevated. Investigate syscall-heavy work, networking, drivers, or I/O paths.

High %wa

Supports an I/O hypothesis, but does not prove storage saturation. Correlate it with device latency, queues, blocked tasks, and I/O pressure.

High load with low CPU

Check blocked tasks, I/O, NFS, storage, locks, and D-state processes.

High %st in a VM

The hypervisor may not be giving the guest the CPU time it expects.

Process clues

ps -eo pid,ppid,user,stat,wchan:24,%cpu,%mem,etime,cmd --sort=-%cpu | head -20

ps -eo stat,pid,ppid,cmd | awk '$1 ~ /^D/'

pstree -ap <PID>

cat /proc/<PID>/status

cat /proc/<PID>/limits

lsof -p <PID>

Advanced escalation

strace -f -tt -T -p <PID>

Memory

• free -h

  Focus on

  Available, not just Free.

• cat /proc/meminfo

  Focus on

  MemAvailable, SwapFree, Dirty, Slab.

• vmstat 1

  Focus on

  Paging and sustained pressure.

• swapon --show

  Focus on

  Unexpected swap use or no safety margin.

• journalctl -k | grep -Ei "oom|out of memory|killed process"

  Focus on

  OOM victim process and timeline.

• cat /proc/pressure/memory

  Focus on

  Sustained some/full memory pressure.

Host healthy, workload unhealthy?

A process can hit a container or cgroup resource limit even when the host still has plenty of CPU or memory.

Useful cgroup v2 evidence

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events
cat /sys/fs/cgroup/cpu.stat

Exact cgroup paths depend on the workload and service manager.

• Memory limits
• OOM events
• CPU throttling
• Resource pressure isolated to the workload

Disk and filesystem

• df -hT

  Investigate

  Filesystem capacity, type, and mount point.

• df -i

  Investigate

  Inode exhaustion even when blocks remain.

• du -xhd1 /path | sort -h

  Investigate

  Largest directories on one filesystem.

• find /path -xdev -type f -size +1G -ls

  Investigate

  Large files without crossing mounts.

• lsof +L1

  Investigate

  Deleted files still held open by a process.

• lsblk -f

  Investigate

  Block devices, filesystems, labels, and mount relationships.

• findmnt

  Investigate

  Actual mount sources, targets, and options.

• iostat -xz 1

  Investigate

  Device utilization, latency, and queue behavior.

• cat /proc/pressure/io

  Investigate

  Workload time stalled on I/O.

• journalctl -k -b

  Investigate

  Filesystem, block-device, and kernel errors.

systemd: work from state to evidence

• systemctl --failed

  What it tells you

  Failed units across the system.

  Watch for

  Correlated failures or an obvious starting point.

• systemctl status <unit>

  What it tells you

  Current unit state and recent log lines.

  Watch for

  Exit status, restart loop, or dependency failure.

• journalctl -u <unit> -b

  What it tells you

  Unit logs from the current boot.

  Watch for

  The first error and the timeline around it.

• journalctl -u <unit> --since "15 min ago"

  What it tells you

  Recent unit-specific events.

  Watch for

  What changed immediately before failure.

• systemctl cat <unit>

  What it tells you

  The effective unit definition and drop-ins.

  Watch for

  Overrides, paths, users, and environment files.

• systemctl show <unit>

  What it tells you

  Detailed runtime properties.

  Watch for

  Limits, result codes, dependencies, and cgroup data.

• ss -lntup

  What it tells you

  Listening TCP/UDP sockets and owning processes.

  Watch for

  Wrong port, wrong bind address, or no listener.

Build a timeline, not a pile of logs

journalctl -b -p warning --no-pager
journalctl -k -b --no-pager
dmesg -T | tail -100
journalctl --since "10 minutes ago" --until "now"

dmesg may require elevated privileges.

Boot and startup

systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain
journalctl -b -1
last -x | head

When "permission denied" is not obvious

id
groups
namei -l /path/to/file
ls -ld /path /path/to/file
stat /path/to/file
getfacl /path/to/file
getenforce
ausearch -m AVC -ts recent
aa-status
journalctl -k --grep=apparmor
cat /proc/<PID>/limits
ls /proc/<PID>/fd | wc -l

• getfacl may not be installed.
• getenforce and ausearch apply to SELinux systems.
• aa-status applies to AppArmor systems and may require root.

When IP works but hostname fails

getent hosts app.example.com
cat /etc/resolv.conf
resolvectl status
dig app.example.com
getent ahosts app.example.com

Check the resolver the application actually uses.

Move from simple checks to packet evidence

curl -v --connect-timeout 5 https://host/path

nc -vz host 443

tracepath host

openssl s_client -connect host:443 -servername host </dev/null

openssl s_client -connect host:443 -servername host </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dates

tcpdump -nn -i any host <IP> and port 443

• Netcat may not be installed.
• tcpdump requires appropriate privilege.

Do not forget time

Clock drift can break TLS, authentication, tokens, and distributed systems.

date -Is
timedatectl status

When the usual tools are missing

Linux gives you raw evidence under /proc and /sys.

• /proc/loadavg

  Evidence

  Load information

• /proc/stat

  Evidence

  CPU counters

• /proc/meminfo

  Evidence

  Memory counters

• /proc/<PID>/status

  Evidence

  Process state and memory

• /proc/<PID>/fd/

  Evidence

  Open file descriptors

• /proc/pressure/{cpu,memory,io}

  Evidence

  Pressure stall information

• /sys/fs/cgroup/

  Evidence

  Workload resource limits and cgroup state

• /sys/class/net/

  Evidence

  Network interface device statistics