[Bug 299954] mana(4): accelerated networking lost on Azure Dv7 (AMD) - missed MSI-X interrupts stall event queues

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=299954

            Bug ID: 299954
           Summary: mana(4): accelerated networking lost on Azure Dv7
                    (AMD) - missed MSI-X interrupts stall event queues
           Product: Base System
           Version: CURRENT
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: aftahi334@gmail.com

Created attachment 276342
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=276342&action=edit
mana: Recover event queues after a missed MSI-X interrupt

Reproduced on FreeBSD 15.0-RELEASE (releng/15.0-n280995-7aedc8de6446)
on Azure Standard_D2ads_v7. The mana(4) driver code is identical in
15.1-RELEASE, and the bug is still present in main. The attached patches
are against main and were validated on 15.0-RELEASE. Please consider
merging to stable/15 as well.

Environment
-----------
- Azure Standard_D2ads_v7 (2 vCPU, AMD EPYC 9V45), Accelerated Networking
  enabled (the MANA VF is presented even when it is disabled in Azure).
- Image freebsd:freebsd-15_0:15_0-release-amd64-gen2-ufs:15.0.0, GENERIC,
  stock if_mana.ko.
- MANA PCI 1414:00ba, 3 MSI-X vectors (HWC + 2 queues).

Symptoms
--------
All of these are the same underlying problem:
1. First boot of a new VM: mana0 attaches and hn0 switches to the VF,
   then all traffic stops. The Azure agent cannot reach the WireServer,
   so VM provisioning times out (OSProvisioningTimedOut).
2. Later boots: under load one MANA queue stops permanently (SSH and
   other flows hashed to it hang). vmstat -i shows that vector's count
   frozen while the other queue keeps working.
3. Some boots: "mana0: Failed to establish HWC: 60" or
   "mana0: test_eq timed out on queue 0", leaving only synthetic hn0.

How to reproduce
----------------
Boot the VM and download a large file in a loop over the accelerated
path, e.g.:
  while :; do fetch -o /dev/null \
    https://download.freebsd.org/releases/amd64/15.0-RELEASE/src.txz; done
With the stock driver a queue stalled within 10-60 seconds in every run.

Analysis
--------
A MANA EQ raises an interrupt only for events written while it is armed,
and it stays disarmed until the driver processes the event and rings the
EQ doorbell. Using a release-matched diagnostic build, at every stall:
- The stalled EQ held a valid, unprocessed EQE, and both CQs on that EQ
  held new CQEs; every CQ had been re-armed after each poll.
- The vmstat -i count for the vector equalled the handler entry count and
  stopped: the guest never received the MSI-X message.
- The MSI-X table entry was unmasked and the PBA was clear.
- Re-ringing the EQ arm doorbell alone did not produce an interrupt;
  running the EQ handler once restored traffic immediately.
- Serializing doorbell writes, flushing the arm doorbell and re-checking
  the EQ, deferring CQ callbacks until after re-arming, Linux-style CQ
  doorbell limits, and issuing HVCALL_RETARGET_INTERRUPT (as Linux
  pci-hyperv does) did not prevent stalls.
- nvme(4) on the same VMs prints "System interrupt issues?" on every boot,
  so other vPCI devices also miss interrupts; nvme recovers by polling.

Interrupts were lost about once per 100k-200k MANA interrupts. The
platform cause is not known, but mana(4) turns one lost interrupt into a
permanent outage of the queue (or the HWC during attach).

Separately, the port_stats sysctl handlers dereference apc->rxqs and
apc->tx_qp, which only exist while the port is up. Reading
"sysctl dev.mana.0" while mana0 is down panics (page fault, fault virtual
address 0x70, in sysctl_root_handler_locked -> if_mana).

Fix (attached, two commits against main)
----------------------------------------
1. mana: Fix port statistics sysctls while the port is down
   Take the port lock and skip queues that do not exist.
2. mana: Recover event queues after a missed MSI-X interrupt
   A per-device callout, started before the HWC is created, runs the EQ
   handler when an EQ has a pending event but its vector handled no
   interrupt since the previous check. Interval: hw.mana.eq_watchdog_ms
   (default 10 ms, 0 disables). Recoveries are counted in
   dev.mana.N.missed_interrupts and logged at most once a minute. A
   per-vector mutex serializes the interrupt handler, the watchdog and
   handler registration/teardown.

Testing (Standard_D2ads_v7, 15.0-RELEASE, same changes applied to 15.0)
-----------------------------------------------------------------------
- Sustained download load: stock stalled a queue within 1 minute in every
  run; patched ran 30 minutes, 154/154 downloads succeeded, 154 missed
  interrupts recovered, no stall.
- "sysctl dev.mana.0" with mana0 down: stock panics; patched returns zeros.
- Cold start (deallocate/start): HWC established, VF active, WireServer
  reachable; a further 5-minute load ran without stalls.
- Both commits build with -Werror and pass tools/build/checkstyle9.pl.

Attachments
-----------
0001-mana-Fix-port-statistics-sysctls-while-the-port-is-d.patch
0002-mana-Recover-event-queues-after-a-missed-MSI-X-inter.patch

-- 
You are receiving this mail because:
You are the assignee for the bug.