ZFS server has gone crazy slow

[Chris Ross]

I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror (of two GPT disks) as it’s disk.  It’s many years old, and has only been misbehaving like this for a day or so.  I’m trying to figure out what’s wrong.

I confirmed that internet connectivity isn’t the problem, and a reboot didn’t fix it.  (The reboot took 10-15 minutes to finish going multi-user, starting daemons, due to the underlying problem described below.)

Truss’ing a very basic command (date), I can see that close() and exit() calls are taking 1-2 seconds.  All of the files being opened are on ZFS, but I don’t know if that’s for sure related.  Similarly, using shell builtin “echo foo” always is immediate, but “/bin/echo” sometimes works quickly, but sometimes the close() on /var/run/ld-elf.so.hints takes 3-5 seconds.

I _think_ this is a filesystem problem.  It’s very hard to diagnose because logging in, and doing anything, takes many seconds per command.  zpool status shows my mirror as online, so I’m not sure where I should check.

I’d appreciate any help!  Thanks much…

           - Chris