Re: PEFS and advisory locking on ZFS

On Tue, Apr 5, 2022 at 7:19 AM Daniel O'Connor <darius@dons.net.au> wrote:
>
> Hi,
> I maintain the PEFS port/project (https://github.com/freebsd-pefs/pefs/) which is an encrypted file system which transparently runs on top of other file systems.
>
> I've updated it to work OK however someone has discovered that if it's running on top of ZFS then locking doesn't work, >1 process can  own a lock (as tested with lockf)
>
> It FreeBSD 13.1-RC1 (tested with releng/13.1-n250053-6fe29001573 GENERIC arm64) - when testing on -current (14.0-CURRENT #1 main-3468cd95c) it does work.
>
> I tried implemented VOP_ADVLOCK but it didn't help (not really surprising but still).
>
> The test is pretty simple, if /testtank is ZFS, then:
>
> # Create crypto FS
> sudo mkdir -p /testtank/test/pefs
> echo test123 >keyfile
> sudo pefs addchain -fZj keyfile /testtank/test/pefs
>
> # Mount it and add the key
> sudo pefs mount /testtank/test/pefs /testtank/test/pefs
> sudo pefs addkey -cj keyhole /testtank/test/pefs
>
> # Test locking
> sudo lockf -k -t 0 /testtank/test/pefs/lock sleep 5 &
> sudo lockf -k -t 0 /testtank/test/pefs/lock echo foo
>
> When it's working the second lockf will print:
> lockf: /testtank/test/pefs/lock: already locked
>
> ZFS itself is fine, the lock test passes if PEFS isn't mounted, and on the same version PEFS on UFS works fine also.
>
> I plan on bisecting it but if anyone has a suggestion I'm all ears.
>
> Thanks.
>
> --
> Daniel O'Connor
> "The nice thing about standards is that there
> are so many of them to choose from."
>  -- Andrew Tanenbaum

Does it use flock or fcntl with F_GETLK/F_SETLK?  Or worse, does it
mix the two?  Is fusefs involved?  And does it work on top of UFS?