`sysctl vm.pmap.kernel_maps' spins on 12.2-RELEASE-p3 w/ nvdimm.ko

[Alan Somers]

On Wed, Feb 24, 2021 at 4:49 PM Konstantin Belousov <kostikbel at gmail.com>
wrote:

> On Wed, Feb 24, 2021 at 03:37:12PM -0800, Ravi Pokala wrote:
> > Hi folks,
> >
> > A colleague and I both independently observed `sysctl -a' appear to hang
> on nodes running FreeBSD 12.2-RELEASE-p3; it didn't emit any output, and ^C
> didn't kill it. We could still establish a new terminal session to the
> node, via SSH or serial console, and we were able to see that it was
> actually spinning, not hung, and was consuming an entire CPU.
> >
> > We eventually determined that it was specifically `sysctl
> vm.pmap.kernel_maps' which was spinning, and subsequently that it only
> spinned if nvdimm.ko was loaded. It was not necessary to access the device
> node associated with the NVDIMM; merely having the module loaded was
> sufficient.
> >
> > I know nvdimm(4) isn't terribly widely used, but hopefully someone who
> uses it can at least confirm my findings on this. Help in debugging would
> be even more appreciated.
> >
>
> How large your nvdimms are?  Their' SPAs are mapped into KVA fully and this
> could be quite large.  It could be busy dumping page tables.
>
> Try to skip large map in pmap.c:sysctl_kmaps() (just increment i over it).
>

Speaking of vm.pmap.kernel_maps, that thing is huge.  It easily dwarfs all
other sysctls combined, and tends to grow with time.  Would it be possible
to hide it from sysctl -a's output?  I think there are other sysctls like
that, that are treated as opaque binary values.  I once had to fix a bug in
py37-salt that caused a common operation to take _6_hours_ as opposed to <
1 minute because of a huge vm.pmap.kernel_maps value, coupled with some
O(n^2) string processing.