Skip to content

NVMe Timeout Issue – Analysis & Fix

✅ Issue Summary

  • Kernel logs show: 
    nvme nvme2: I/O tag ... timeout, reset controller
nvme nvme2: D3 entry latency set to 8 seconds
  • Indicates NVMe controller resets due to power state latency.
  • Drive health (SMART):
  • percentage_used: 9% (healthy)
  • media_errors: 0
  • unsafe_shutdowns: 60 (possible power instability)
  • Performance tests (fio) show disk is healthy.

🔍 Root Cause

  • Aggressive NVMe power saving (APST) allows up to 100 ms latency (default_ps_max_latency_us=100000).
  • Combined with previous unsafe shutdowns, this can cause I/O timeouts.

✅ Fix: Disable NVMe Power Saving

Temporary (runtime):

echo 0 | sudo tee /sys/module/nvme_core/parameters/default_ps_max_latency_us

Permanent (RHEL/CentOS/Rocky):

  1. Edit GRUB: 
    sudo nano /etc/default/grub
    Add: 
    GRUB_CMDLINE_LINUX="nvme_core.default_ps_max_latency_us=0"
  2. Update GRUB: 
    sudo grub2-mkconfig -o /boot/grub2/grub.cfg
    (For UEFI: /boot/efi/EFI/redhat/grub.cfg)
  3. Reboot: 
    sudo reboot
  4. Verify: 
    cat /sys/module/nvme_core/parameters/default_ps_max_latency_us
# Should output: 0

✅ Monitoring Commands

  • Check NVMe errors: 
    dmesg | grep nvme
  • Check APST status: 
    dmesg | grep -i apst
  • Check SMART health: 
    nvme smart-log /dev/nvme2