Linux Performance Tools

Also Known As

TL;DR

Explanation

CPU: top / htop (real-time process stats), mpstat (per-CPU stats), perf top (CPU profiling by function). Memory: free -h (overview), vmstat -s (detailed), smem (per-process). Disk IO: iostat -x (saturation and utilisation), iotop (per-process IO). Network: nethogs (per-process bandwidth), iftop (per-connection traffic), ss -s (socket summary). Tracing: strace -p PID (system calls made by a process — what is it doing?), ltrace (library calls). For PHP: strace on a stuck PHP-FPM worker shows exactly which system call it's blocked on.

Common Misconception

Why It Matters

Common Mistakes

• Only looking at CPU usage — IO wait shows as CPU idle but means processes are blocked on disk.
• Not knowing strace -p for live process inspection — invaluable for hung process diagnosis.
• Running strace in production on a busy process — overhead can be 10-100x; use carefully.
• Confusing %iowait (CPU waiting for IO) with disk utilisation — %iowait doesn't directly show disk saturation.

Code Examples

# Performance analysis by guesswork:
# 'The server is slow'
# Restart PHP-FPM (sometimes works, doesn't fix root cause)
# Check top: CPU 30% — nothing obvious
# Give up, ticket to sysadmin
# Hours wasted

# Systematic performance investigation:
# 1. CPU and load:
htop           # Real-time, with color
mpstat 1 5     # Per-CPU every second for 5 seconds

# 2. Memory:
free -h        # Quick overview
vmstat 1 5     # Including swap activity

# 3. Disk IO:
iostat -x 1 5  # Is any disk at 100% util?
iotop          # Which process is using IO?

# 4. What is a stuck PHP-FPM worker doing?
strace -p $(pgrep php-fpm | head -1) -e trace=network,file -T
# Shows: connect() to DB taking 3.5 seconds — DB is the bottleneck!

References

• ↗ https://brendangregg.com/linuxperf.html

Tags

Related Terms