[Bug 299221] nfscl: 45s NFSv4.1 outage when CREATE_SESSION returns NFS4ERR_STALE_CLIENTID within one nfsc_renew

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=299221

            Bug ID: 299221
           Summary: nfscl: 45s NFSv4.1 outage when CREATE_SESSION returns
                    NFS4ERR_STALE_CLIENTID within one nfsc_renew
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: olivier@freebsd.org

Created attachment 275500
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=275500&action=edit
patch

After NFS4ERR_BADSESSION, recovery first tries CREATE_SESSION with the
extant ClientID. If that returns NFS4ERR_STALE_CLIENTID and a full recovery
already ran within the last nfsc_renew (lease/2, ~45s), nfsrpc_setclient()
returns NFSERR_IO instead of falling back to EXCHANGE_ID (the retok path,
recover_done_time gate in nfscl_renewthread()). The mount then does no I/O
until the window expires. Under load, 5ab48fb9f7ac re-arms recovery on every
BADSESSION, so this also becomes a CREATE_SESSION storm (+78 CREATE_SESSION,   
    +2 EXCHANGE_ID in one minute observed).

RFC 8881 says a failed CREATE_SESSION with the extant ClientID is followed by
EXCHANGE_ID. STALE_CLIENTID is definitive; waiting does not help.

Trigger: Amazon EFS drops session and ClientID whenever it resets a client
TCP connection (seen 30 times in one hour on one mount target). Linux 6.19
on the same mount target recovers in 8 ms (pcap: SEQUENCE 10052,
DESTROY_SESSION 10052, CREATE_SESSION 10022, EXCHANGE_ID, CREATE_SESSION 0).

DTrace (fbt on nfsrpc_setclient/createsession/exchangeid; retok=1 is the
CreateSession-only path):

  18:30:41 setclient start /mnt/efs-soak retok=0
  18:30:41 createsession ret=10022
  18:30:41 exchangeid ret=0
  18:30:41 createsession ret=0                <- recovered
  18:30:59 setclient start /mnt/efs-soak retok=1   <- 2nd reset, 18s later
  18:30:59 createsession ret=10022
  18:30:59 setclient ret=5 retok_out=1        <- EIO, no EXCHANGE_ID
  18:31:41 setclient start /mnt/efs-soak retok=0   <- 42s later
  18:31:41 createsession ret=10022
  18:31:41 exchangeid ret=0
  18:31:41 createsession ret=0

4 of 17 recoveries in 2.5h took this path: outages of 42, 46, 46 and 45s.

Fix (attached): in nfsrpc_setclient(), on NFSERR_STALECLIENTID from the
extant-ClientID CREATE_SESSION, clear *retokp and continue to EXCHANGE_ID,
so nfscl_recover() also reclaims opens and delegations. Other errors keep
the current behaviour. Trade-off: a server that keeps dropping the ClientID
now gets a full recovery per BADSESSION instead of one per nfsc_renew; the
RECVRINPROG/RECOVER flags still serialize recoveries.

-- 
You are receiving this mail because:
You are the assignee for the bug.