nmon – Linux Performance Monitor – A Complete “Zero‑to‑Hero” Manual

1. Introduction & History

1.1 What is nmon?

Performance problems are rarely polite. They don’t knock, they don’t announce themselves, and they almost never happen while you’re watching. Instead, they appear at 3 AM, under peak load, during a deployment—or worse, during a demo.

This is exactly the kind of problem nmon was built to solve.

nmon (short for Nigel's performance Monitor for Linux) is a single‑binary, interactive, command‑line performance monitoring tool for Linux (and originally AIX). It shows CPU, memory, disks, network, processes, file systems, kernel metrics, and more in real time, while also being capable of recording all data to a compact binary (*.nmon) file for offline analysis.

Key attributes:

1.2 Origin & Evolution

nmon was originally developed by Nigel Griffiths at IBM for performance analysis on AIX systems. Over time, it was ported to Linux and became popular among:

• System administrators
• Performance engineers
• Capacity planners
• DevOps & SRE teams

History:

Today, nmon runs on:

• Linux (most distributions)
• AIX
• Power systems
• Containers and virtual machines

Despite its age, nmon remains relevant because performance fundamentals never change.

1.3 Why Use nmon Over Other Tools?

If you’ve been working with Linux for a while, you already know the usual suspects. You’ve probably typed top or htop more times than you can count. You may have used vmstat and iostat to dig a little deeper, or relied on Prometheus and Grafana dashboards to keep an eye on your systems.

So the natural question is: why would you need nmon at all?

The answer is simple—nmon is built for the moments when other tools start to fall short.

Most real-time tools are great while you’re watching the screen. But performance problems rarely wait for you to be ready. They happen overnight, during peak load, or right after a deployment—when no one is logged in running top.

This is where nmon really shines.

nmon is designed to quietly collect detailed performance data for long periods of time, with very little impact on the system itself. Even when a server is already under heavy load, nmon keeps recording without making things worse.

It’s especially powerful for post-mortem analysis. When something goes wrong, you don’t have to guess what happened—you can open a single .nmon file and replay the system’s behavior over time: CPU, memory, disks, network, and top processes, all in one place.

In short, nmon is the tool you use when:

• You need reliable performance data over hours or days
• The system is already stressed and can’t afford heavy monitoring
• You want to understand what happened, not just what is happening now
• You prefer one compact file instead of scattered logs and metrics

nmon may not look fancy, but when things go wrong, it tells the story clearly—and that’s exactly what you need.

2. Getting Started

2.1 Supported Platforms

Note: nmon works on both System‑V init and systemd. For containers, you must mount /proc and /sys (default in most images).

2.2 Installation

Getting nmon installed is refreshingly simple.

2.2.1 Using Package Managers

That’s it. No agents. No services. No configuration wizard.

All packages ship the standard binary compiled without GPU support. If you need GPU, see §2.2.3.

2.2.2 Downloading Pre‑Built Binaries

If you’re on a restricted system or older OS, you can grab a static binary and just drop it in /usr/local/bin.

# Example for x86_64 Linux (v19.2)
wget -O nmon https://github.com/nmon/nmon/releases/download/v19.2/nmon_x86_64_linux
chmod +x nmon
sudo mv nmon /usr/local/bin/

2.2.3 Building from Source (optional GPU, eBPF, static)

Building from source is mostly useful if:

• You want static builds
• You’re experimenting with eBPF
• You’re on exotic hardware

Most users can skip this safely.

# Prereqs
sudo apt-get install -y build-essential libncurses5-dev git

# Clone repo
git clone https://github.com/nmon/nmon.git
cd nmon

# Build (CPU‑only)
make clean && make

# Build with NVIDIA GPU support (requires CUDA headers)
make clean && make GPU=1

# Install
sudo cp nmon /usr/local/bin/

Static linking is possible (make STATIC=1). For ARM64, use the aarch64 source tree (nmon_aarch64_linux).

2.2.4 Running nmon Inside Docker

Yes, you can run nmon in containers—but remember:

• Containers see their own namespaces
• For host metrics, you need access to /proc and /sys

This is more advanced and usually unnecessary unless debugging containerized workloads.

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y nmon && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/usr/bin/nmon"]

Run container with host proc & sys mounted (required for accurate stats):

docker run --rm -it \
  --pid=host \
  -v /proc:/proc:ro \
  -v /sys:/sys:ro \
  my-nmon-image -f -s 30 -c 1440

2.3 First Run – “Hello‑World”

Just type:

nmon

nmon is waiting for you to tell it what you want to see. You will see a screen like:

   NMON - version 19.2    2025-10-13 14:20:01

   C: CPU    M: Memory    D: Disk    N: Network   T: Top Processes
   L: LPAR   V: Virtual   P: Power   G: GPU

   Press: ? for help, q to quit, s to start/stop snapshot

Press c to toggle the CPU view, m for memory, d for disks, etc. The UI updates every second by default.

Tip: Press t to view the top‑10 processes sorted by CPU% (same as top).

3. Understanding the UI

3.1 Real‑time Screen Layout

nmon is keyboard-driven.
Each screen shows one aspect of the system. Some essential keys::

Hot‑Key Summary is in Appendix A.

3.2 Keyboard Shortcuts & Hot‑Keys

4. Collecting Data

Watching live stats is nice, but nmon can work in interactive mode (as above) or in batch (recording) mode, where it writes a compact ASCII CSV‑like file (*.nmon).

4.1 Recording Mode (-f)

nmon -f -s <interval> -c <count>

Example: Record 1‑minute data every 5 seconds for 12 hours (864 snapshots):

nmon -f -s 5 -c 8640 -t -Z

The file will be myhost_20251013_1500.nmon.gz.

4.2 Choosing Interval & Duration

Rule of thumb: Keep interval × count ≤ 2 hours for a single file (to avoid huge files); split longer runs using cron or systemd timers.

4.3 Output File Naming Conventions

4.4 Running nmon as a Service (systemd)

Create /etc/systemd/system/nmon-collector.service:

[Unit]
Description=Nmon data collector
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/nmon -f -s 60 -c 1440 -t -o /var/log/nmon
StandardOutput=append:/var/log/nmon/nmon.log
StandardError=append:/var/log/nmon/nmon.err
Restart=on-failure
User=root
Group=root

[Install]
WantedBy=multi-user.target

Enable & start:

sudo systemctl daemon-reload
sudo systemctl enable --now nmon-collector.service

Tip: Use systemd.timer for periodic runs, e.g., every 6 hours.

[Unit]
Description=Run nmon collector every 6h

[Timer]
OnCalendar=*:0/6
Persistent=true

[Service]
Type=oneshot
ExecStart=/usr/bin/nmon -f -s 30 -c 720 -t -Z -o /var/log/nmon

5. Converting nmon Files to CSV

The nmon2csv script (bundled) parses the .nmon file and creates one CSV per metric.

nmon2csv myhost_20251013_1500.nmon

Resulting files:

Automated conversion is performed automatically if you install the nmon2csv package (available on most distros).

4.5 Using Python for Quick Plotting

import pandas as pd
import matplotlib.pyplot as plt

df_cpu = pd.read_csv('myhost_20251013_1500_CPU.csv')
df_cpu.plot(x='Date', y='%usr')
plt.title('CPU User% Over Time')
plt.show()

5. Analyzing nmon Data

Below are common analysis patterns.

5.1 Detecting CPU Bottlenecks

grep -E '^CPU' myhost_20251013_1500.nmon | \
awk '{print $1,$2,$3,$4}' | \
gnuplot -p -e "set title 'CPU Utilisation'; plot '-' using 1:4 with lines"

Or use the generated CSV:

csvcut -c "Date,CPU1%usr,CPU1%sys" myhost_20251013_1500_CPU.csv | \
csvstat

5.2 Disk Latency & Utilisation

nmon2csv myhost_20251013_1500.nmon
csvcut -c Date,Device,KBread/s,KBwritten/s,await,%util \
  myhost_20251013_1500_DISK.csv | \
csvlook

Look for %util > 80% and await > 20 ms – indicates saturation.

5.3 Network Throughput

csvcut -c Date,Interface,RxKB/s,TxKB/s myhost_20251013_1500_NET.csv | \
csvstat

Note: For high‑speed NICs (10 GbE+), use -s 1 to capture bursts.

5.4 Top‑Processes Trend

csvcut -c Date,PID,%CPU,%MEM,Command myhost_20251013_1500_TOP.csv | \
awk '$3 > 20 {print $0}' | less

Shows processes that consistently used >20 % CPU.

5.5 Capacity Planning Example (CPU & Memory)

# Compute average CPU utilisation per day
awk '/^CPU/ {print $2,$3}' myhost_20251013_1500.nmon | \
awk '{cpu+=$3; count++} END {print "Avg CPU%:", cpu/count}'

Plot memory usage over the period:

csvplot -c Date,MemUsed% myhost_20251013_1500_MEM.csv

6. Automating nmon with Cron / systemd

6.1 Cron Example (Hourly Capture)

0 * * * * /usr/bin/nmon -f -s 30 -c 120 -t -Z -o /var/log/nmon

This runs every hour, captures data for 1 hour (30 sec interval, 120 snapshots).

6.2 systemd Timer Example (Every 6 hours)

Create /etc/systemd/system/nmon-collector.service (as in §4.4) and /etc/systemd/system/nmon-collector.timer:

[Unit]
Description=Run nmon collector every 6h

[Timer]
OnCalendar=*:0/6
Persistent=true

[Install]
WantedBy=timers.target

Enable timer:

sudo systemctl enable --now nmon-collector.timer

7. Common Use Cases & Best Practices

7.1 Performance Impact

*Measured on an idle system with nmon v19.2.

Best Practice: Run nmon -s 30 -c 1200 -f for routine monitoring; keep -t on only when needed.

7.2 Integrating with Monitoring Platforms

8. Advanced Topics

8.1 Using nmon with eBPF (experimental)

nmon -e -s 5 -c 720 -t

Caution: eBPF mode is experimental and may need root privileges.

8.2 Parsing nmon Output in Real‑Time (Shell)

/usr/bin/nmon -f -s 10 -c 60 -t | \
awk '/^CPU/ {print strftime("%Y-%m-%d %H:%M:%S"), $2, $3, $4, $5}'

Outputs a live stream of CPU percentages with timestamps.

8.3 Custom Scripts for Alerts

#!/bin/bash
FILE=/var/log/nmon/$(date +%Y%m%d%H).nmon
/usr/bin/nmon -f -s 30 -c 120 -t -o $FILE
CPUAVG=$(csvcut -c Date,%usr $FILE_CPU.csv | \
        awk '{sum+=$2; cnt++} END {print sum/cnt}')
if (( $(echo "$CPUAVG > 85" | bc -l) )); then
   echo "High CPU average: $CPUAVG%" | mail -s "CPU Alert" admin@example.com
fi

Set executable and schedule via cron.

8.4 Using nmon on AIX

AIX ships nmon (AIX version) with slightly different syntax:

nmon -f -s 30 -c 120 -t

All options are similar; -f writes a file with the same format, convertible with nmon2csv (available for AIX).

9. Troubleshooting

9.1 Recreating Missing Snapshots

If a run stopped early, you can append additional data:

/usr/bin/nmon -f -s 30 -c 10 -t -o /var/log/nmon

Then concatenate .nmon files (they are plain text) before converting.

10. Summary

• nmon is a powerful, low‑overhead tool for capturing system performance data on AIX, Linux, and (experimental) eBPF‑enabled kernels.
• Use -s (interval) and -c (count) to control data granularity and duration.
• -t enables top‑process logging; -V for virtualization; -g for GPU metrics.
• Convert .nmon files with nmon2csv to obtain per‑metric CSVs for analysis.
• Integrate with monitoring stacks (Grafana/InfluxDB, Prometheus, Splunk, ELK) using provided exporters or custom parsers.
• Automate collection via cron or systemd timers, keeping resource impact minimal.
• Follow best‑practice patterns for troubleshooting, capacity planning, and performance tuning.

Quick Reference Cheat‑Sheet

# Live UI, 1‑sec interval, 5‑min capture, include top processes
nmon -s 1 -c 300 -t

# Background collection, 30‑sec interval, 2‑hour retention, gzip
nmon -f -s 30 -c 240 -t -Z -o /var/log/nmon

# Convert to CSV (creates multiple files)
nmon2csv myhost_20251101_0000.nmon

# Plot CPU utilisation (Python)
python - <<'PY'
import pandas as pd, matplotlib.pyplot as plt
df = pd.read_csv('myhost_20251101_0000_CPU.csv')
df.plot(x='Date', y='%usr')
plt.show()
PY