cannot delete some files in stub /usr

From: Eugene M. Zheganin <eugene_at_zhegan.in>
Date: Thu, 11 Nov 2021 13:44:20 UTC
Hello.

I have a server with 12.2-R, it was installed on a remote hosting witch 
patched bsdinstall which didn't create the separate /usr zfs dataset. 
Recently I've created one, mounted it ontop of the /, exported the / 
with nfs, mounted it on the same server and tried to delete the stub /usr.

Everything is fine except... yeah, I cannot delete some files:

# mount
zroot/ROOT/default on / (zfs, NFS exported, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
zroot/www on /var/www (zfs, local, noatime, nfsv4acls)
zroot/home on /home (zfs, local, noatime, nfsv4acls)
zroot/mysql on /var/db/mysql (zfs, local, noatime, nfsv4acls)
zroot/usr on /usr (zfs, local, noatime, nfsv4acls)
zroot/ports on /usr/ports (zfs, local, noatime, nfsv4acls)
zroot/ports/distfiles on /usr/ports/distfiles (zfs, local, noatime, 
nfsv4acls)
127.0.0.1:/ on /mnt/usr (nfs)

# pwd
/mnt/usr/usr/lib32

# ls -l
total 1210
-r--r--r--  1 root  wheel  1778272 11 нояб. 12:31 libc.so.7
-r--r--r--  1 root  wheel    56492 11 нояб. 12:31 libcrypt.so.5
-r--r--r--  1 root  wheel    19592 11 нояб. 12:31 librt.so.1
-r--r--r--  1 root  wheel   121324 11 нояб. 12:31 libthr.so.3

# rm libc.so.7
override r--r--r-- root/wheel for libc.so.7? y
rm: libc.so.7: Input/output error

There's more in stub /usr/bin. This is what nfsv3 gives me. nfsv2 says 
"operation not permitted".

I'm aware about chflags, currently there are none:

# ls -ol
total 1210
-r--r--r--  1 root  wheel  - 1778272 11 нояб. 12:31 libc.so.7
-r--r--r--  1 root  wheel  -   56492 11 нояб. 12:31 libcrypt.so.5
-r--r--r--  1 root  wheel  -   19592 11 нояб. 12:31 librt.so.1
-r--r--r--  1 root  wheel  -  121324 11 нояб. 12:31 libthr.so.3


Filesystem seems to be fine, I'm able to create any files in these 
cursed directories. :P zfs shows no errors, didn't make the whole scrub, 
but I suppose it won't find any issues.

Ideas ? The server is fully operatioinal, and rebootable with no other 
issues than these remains (reboot doesn't clear them). There's one more 
in old /usr/lib and some in old /usr/bin. Also tried to mount it with 
lockd/statd just in case, but this diodn't help either.

Thanks.

Eugene.