ZFS recv oddity

I have noticed that an incremental "zfs recv" to a backup pool consistently
stalls for an extended period (roughly 5 minutes) at the start before it
starts copying data.  During this time, there's basically no IO on the pool
(<1 IOPS) and basically no CPU activity (~0.5% increase vs the background).
I can understand the "zfs recv" needing to do some book-keeping when it
starts but I would expect that to involve non-trivial CPU and/or IO.  Once
the "zfs recv" starts actually writing data, everything transfers as
expected.  Does anyone have any suggestions as to what might be going on?

Overall backup approach:
  zfs import -N -d $DEVICE $POOL_ID bupool
  zfs send -RLeI @$OLD pool@$NEW | <<network and memory buffer>> | zfs recv -Fuv bupool

The pool has just under 23k snapshots and just under 100 filesystems.

Reporting from buffer tool shows that it quickly fills to its 1GB limit and
writes about 1.8KiB to the "zfs recv" then sits for 5 minutes before the
"zfs recv" starts reporting "receiving incremental stream...".

I see similar behaviour whether the receiving side is a one or two HDD pool
on an arm64 system or a 6-HDD pool on a amd64 system.

-- 
Peter Jeremy