Skip to content

nvme/Pi 5: DRAM-less SSD with HMMIN > 32 MiB runs with HMB disabled (32 MiB DT default, 26e6d69) → intermittent rootfs wedge #7445

@by

Description

@by

Describe the bug

On a Raspberry Pi 5 booting from a DRAM-less NVMe SSD (Corsair MP600 GS, Phison PS5021-E21), the device-tree default nvme.max_host_mem_size_mb=32, set in bcm2712-rpi.dtsi by commit 26e6d69, is below the minimum HMB the drive reports (HMMIN = 64 MiB).

Because the requested minimum exceeds the cap, nvme_setup_host_mem() takes the min > max path, warns, and disables HMB entirely (it does not fall back to a smaller buffer). The DRAM-less drive then runs with no host memory buffer:

nvme nvme0: min host memory (64 MiB) above limit (32 MiB).

Under sustained operation the board then intermittently wedges after ~10–12 h: the NVMe/rootfs write path stalls while the kernel network stack stays up, so the machine answers TCP but can't service anything that needs to write to disk, recoverable only by a forced reboot.

Overriding the cap with e.g., nvme.max_host_mem_size_mb=128 restores the full buffer and stops the wedges:

nvme nvme0: allocated 64 MiB host memory buffer (2 segments).

The 32 MiB compromise helps drives with HMMIN ≤ 32 MiB, but for drives reporting HMMIN > 32 MiB it does the opposite of what was intended: instead of a smaller-but-nonzero buffer, HMB is disabled outright, the worst outcome for exactly the DRAM-less class the default was meant to help.

Steps to reproduce the behaviour

Use a Pi 5 with a DRAM-less NVMe SSD whose HMMIN exceeds 32 MiB. Confirm:
sudo nvme id-ctrl /dev/nvme0 | grep -E 'hmpre|hmmin' → here both report 16384 (= 64 MiB).
Boot with the stock DT default (nvme.max_host_mem_size_mb=32, inherited from bcm2712-rpi.dtsi; no override in cmdline.txt).
sudo dmesg | grep -i 'host mem' shows min host memory (64 MiB) above limit (32 MiB) and no allocated … host memory buffer line, HMB is disabled.
Run under sustained I/O. After ~10–12 hours the rootfs write path stalls: new SSH sessions are closed before the banner, journald stops logging, but the network stack still answers (closed ports RST instantly); vcgencmd get_throttled = 0x0 (not power). Only a forced reboot recovers it.
Append nvme.max_host_mem_size_mb=128 to cmdline.txt and reboot. dmesg now shows allocated 64 MiB host memory buffer, and the wedge no longer occurs.

Device (s)

Raspberry Pi 5

System

Raspberry Pi 5 Model B Rev 1.0 (BCM2712). Storage: Corsair MP600 GS (Phison PS5021-E21, DRAM-less) on a Pimoroni NVMe Base (passive single-M.2 carrier, M.2 2230/2242/2260/2280; no ID EEPROM, SSD direct-attached under the BCM2712 bridge), PCIe Gen 2 ×1.

Raspberry Pi reference 2021-10-30
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, c12b1df4ed6416fb0df33ba1731c5b13c1bdbdf8, stage2
2026/05/26 16:01:25 
Copyright (c) 2012 Broadcom
version 086b83e3 (release) (embedded)
Linux chronos 7.1.0-v8-16k-NTP+ #14 SMP PREEMPT_RT Mon Jun 15 22:29:02 CEST 2026 aarch64 GNU/Linux

Logs

Drive HMB parameters (nvme id-ctrl /dev/nvme0, 4 KiB units):

hmpre : 16384   # 64 MiB preferred
hmmin : 16384   # 64 MiB minimum  -> exceeds the 32 MiB DT cap

PCIe topology — SSD direct-attached (no HAT switch); lspci independently tags the controller DRAM-less:

0001:00:00.0 PCI bridge: Broadcom BCM2712 PCIe Bridge [14e4:2712] (rev 21)
0001:01:00.0 Non-Volatile memory controller: Phison PS5021-E21 PCIe4 NVMe Controller (DRAM-less) [1987:5021] (rev 01)

Before (stock DT default =32), HMB disabled:

nvme nvme0: min host memory (64 MiB) above limit (32 MiB).

After (nvme.max_host_mem_size_mb=128 in cmdline.txt), HMB allocated:

[ 0.358199] nvme nvme0: allocated 64 MiB host memory buffer (2 segments).

Power check (rules out undervoltage):

throttled=0x0

Additional context

History:
#6504 originally restricted HMB to 0 (allocation came from CMA; superficial testing showed no impact).
26e6d69 bumped the default to 32 MiB as a compromise for small DRAM-less drives, enabled by 6686634 ("nvme-pci: manually allocate Host Memory Buffer segments on arm64").

Workaround:
Append to cmdline.txt (last occurrence wins; survives kernel/DT updates, which keep shipping =32):

nvme.max_host_mem_size_mb=128

Stable for 18 hours; already past the ~10–12 h window in which the unpatched config used to wedge.

Suggested fixes (in order of preference):

  1. Don't disable HMB when the device minimum exceeds the soft cap, grant the device its reported HMMIN (up to a sane hard ceiling) rather than dropping to 0 MiB. The current min > max → no-HMB result is the most harmful outcome for the affected drives.
  2. Raise the BCM2712 default (e.g., 64 MiB) so common DRAM-less drives with HMMIN = 64 MiB work out of the box; Pi 5 RAM budgets make this cheap.
  3. At minimum, document the failure mode and the override; the symptom (network-alive but wedged, logs vanish) is hard to diagnose and the triggering message is lost when the box hangs.

Notes:
Because the failure also kills journald, the triggering kernel message is lost on reboot unless captured out-of-band (netconsole/serial).
The command line carries determinism parameters for a GNSS/NTP timing build (isolcpus, irqaffinity, pcie_aspm=off, nvme_core.default_ps_max_latency_us=0); none touch HMB.
Reproducible on a stock kernel: the DT default and nvme-pci HMB path are unmodified upstream, so any HMMIN > 32 MiB drive should hit it.

Happy to test patches or provide further data (netconsole capture of a wedge, longer-term stability).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions