[Bug 287783] GEOM_ELI: Crypto request failed (ENOMEM).

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=287783

mail@rubenvos.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mail@rubenvos.com

--- Comment #1 from mail@rubenvos.com ---
Hi Mason,

My NAS's main purpose is serving ZFS datasets over NFS as well. It also pulls
backups from other devices on my home-network.

The raidz2 pool consists of 8 magnetic SATA drives backed by a ZIL (consisting
of 2 m2 nvme ssds, which are mirrored). The SATA drives are passed through
GELI.

Recently, I've started encountering errors that look a lot like yours:

[root@nas:/usr/home/fux]# cat /var/log/messages | grep 'GEOM_ELI: Crypto
request failed' | wc -l
     451
[root@nas:/usr/home/fux]# 

[root@nas:/usr/home/fux]# cat /var/log/messages | grep 'GEOM_ELI: Crypto
request failed' | tail -n 8
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFE8UAK_zfs_data.eli[WRITE(offset=6099482865664, length=331776)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHKL2K_zfs_dataGEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHM9LP_zfs_data.eli[WRITE(offset=6099482918912, length=282624)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFE8UAK_zfs_data.eli[WRITE(offset=6099482746880, length=430080)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHYX1K_zfs_data.eli[WRITE(offset=6099482865664, length=331776)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHM9LP_zfs_data.eli[WRITE(offset=6105742626816, length=315392)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFE8UAK_zfs_dataGEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHM9LP_zfs_data.eli[WRITE(offset=6099482746880, length=430080)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHKL2K_zfs_dataGEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHYX1K_zfs_data.eli[WRITE(offset=6099482918912, length=282624)]
Sep 18 08:52:23 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHM9LP_zfs_data.eli[WRITE(offset=6099482873856, length=327680)]

[root@nas:/usr/home/fux]# cat /var/log/messages | grep 'GEOM_ELI: Crypto
request failed' | head -n 8
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHYX1K_zfs_data.eli[WRITE(offset=7381933195264, length=1044480)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHKL2K_zfs_data.eli[WRITE(offset=7381933199360, length=1048576)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHYX1K_zfs_data.eli[WRITE(offset=7381933195264, length=1044480)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHA2MP_zfs_data.eli[WRITE(offset=7381933195264, length=1040384)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHBNKK_zfs_data.eli[WRITE(offset=7381933195264, length=1040384)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHKL2K_zfs_data.eli[WRITE(offset=7381933199360, length=1048576)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHYX1K_zfs_data.eli[WRITE(offset=7381933195264, length=1044480)]
Aug 22 08:52:33 nas kernel: GEOM_ELI: Crypto request failed (ENOMEM).
gpt/SDFHBNKK_zfs_data.eli[WRITE(offset=7381933195264, length=1040384)]
[root@nas:/usr/home/fux]# 

Unfortunately, I've been unable to pinpoint them to 1 hardware device..

After some log parsing though, I found out that my issue appears to be related
to 1 particular rsync backup job, which pulls data from a remote Debian machine
running KVM guests. I forgot to exclude a raw image file of a VM running on it
(backing it up this way doesnt necessarily produce a consistent VM backup..).
The VM-volume backend file that is rsynced is 25G big, and probably changes
somewhat during transfer (although the VM does not perform very IO intensive
operations).

I've limited the rsync-job's  bandwith after identifying the correlation
between  the cronjob and the error message, I'm hoping to see fewer of these
error-occurances (less IO, less chance of error, right?).

These errors DO look scary, I don't remember seeing them occur before upgrading
to FreeBSD 14.3 from 14.2 but I don't exactly know when they began occuring :/

The system has proven itself to be fairly robust (has been running reliably for
a couple of years now) and I'd very much like to find out how to prevent
situations that are a danger to the integrity of my zpools from occuring in the
future.

Have you been able to diagnose your error-messages any further?

-- 
You are receiving this mail because:
You are the assignee for the bug.