ZFS: OoM with very large kmem_cache_create("abd_chunk", ...) USED count

From: Dave Baukus <daveb_at_spectralogic.com>
Date: Fri, 08 Apr 2022 19:02:49 UTC
On FreeBSD_stable13 I'm seeing multiple OoM crash dumps with very large adb_chunk
used count as reported by vmstat -z -M <crash dump> -N <kern>:

For example on a system with 128G memory:
vmstat -z -M <crash dump> -N <kern>m reports:

ITEM                   SIZE  LIMIT     USED     FREE      REQ     FAILSLEEP XDOMAIN
...
abd_chunk:             4096,      0,22967817,    5657,7737623005,   0,   0,3345348454
...

Meanwhile ARC stats are reasonable:
arc_stats.arcstat_size:  68955842224
arc_stats.arcstat_c:     68107622992
arc_stats.arcstat_c_max: 68643903488

But, adbstat_struct_size is impossibly large:

  abdstat_struct_size = {
    name = "struct_size", '\000' <repeats 243 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 207162941552,
      l = 207162941552,
      ul = 207162941552,
    }
  },
  abdstat_scatter_cnt = {
    name = "scatter_cnt", '\000' <repeats 243 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 76786,
      l = 76786,
    }
  },
  abdstat_scatter_data_size = {
    name = "scatter_data_size", '\000' <repeats 237 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 65734410752,
      l = 65734410752,
    }
  },
  abdstat_scatter_chunk_waste = {
    name = "scatter_chunk_waste", '\000' <repeats 235 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 482816,
      l = 482816,
    }
  },
  abdstat_linear_cnt = {
    name = "linear_cnt", '\000' <repeats 244 times>,
    data_type = 4 '\004',
    value = {
      l = 9822,
      ul = 9822,
    }
  },
  abdstat_linear_data_size = {
    name = "linear_data_size", '\000' <repeats 238 times>,
    data_type = 4 '\004',
    value = {
      ui64 = 34973696,
      l = 34973696,
    }
  }

=========

anyone seen anything similar ?
--

Dave Baukus