Re: Filesystem extended attributes and Capsicum

From: Shawn Webb <shawn.webb_at_hardenedbsd.org>
Date: Sat, 23 Mar 2024 16:00:42 UTC
On Sat, Mar 23, 2024 at 02:54:22PM +0000, Shawn Webb wrote:
> On Sat, Mar 23, 2024 at 02:25:26PM +0000, Shawn Webb wrote:
> > On Sat, Mar 23, 2024 at 07:58:37AM -0600, alan somers wrote:
> > > On Fri, Mar 22, 2024 at 9:52 PM Shawn Webb <shawn.webb@hardenedbsd.org> wrote:
> > > >
> > > > On Fri, Mar 22, 2024 at 08:07:17PM -0600, Alan Somers wrote:
> > > > > On Fri, Mar 22, 2024 at 6:56 PM Shawn Webb <shawn.webb@hardenedbsd.org> wrote:
> > > > > >
> > > > > > On Fri, Mar 22, 2024 at 06:20:48PM -0600, Alan Somers wrote:
> > > > > > > On Fri, Mar 22, 2024 at 5:38 PM Shawn Webb <shawn.webb@hardenedbsd.org> wrote:
> > > > > > > >
> > > > > > > > Hey all,
> > > > > > > >
> > > > > > > > I'm writing an application in which I hope to enable Capsicum. I'm
> > > > > > > > experiencing an issue whereby extattr_get_fd fails with a file
> > > > > > > > descriptor that has all the extended attribute capabilities enabled
> > > > > > > > (CAP_EXTATTR_DELETE, CAP_EXTATTR_GET, CAP_EXTATTR_LIST, and
> > > > > > > > CAP_EXTATTR_SET).
> > > > > > > >
> > > > > > > > Looking at the kernel source (sys/kern/vfs_extattr.c) tells me that
> > > > > > > > kern_extattr_get_fd only requires CAP_EXTATTR_GET.
> > > > > > > >
> > > > > > > > So I'm a bit puzzled as to why my call to extattr_get_fd(2) is
> > > > > > > > failing. Am I doing something wrong or are filesystem extended
> > > > > > > > attributes not supported in a Capabilities-enabled process?
> > > > > > > >
> > > > > > > > Here's how I'm creating the file descriptor (before calling
> > > > > > > > cap_enter(2)):
> > > > > > > >
> > > > > > > > ==== BEGIN CODE ====
> > > > > > > > static int
> > > > > > > > open_file(const char *path)
> > > > > > > > {
> > > > > > > >         cap_rights_t rights;
> > > > > > > >         int fd;
> > > > > > > >
> > > > > > > >         fd = open(path, O_PATH | O_CLOEXEC);
> > > > > > > >         if (fd == -1) {
> > > > > > > >                 return (-1);
> > > > > > > >         }
> > > > > > > >
> > > > > > > >         memset(&rights, 0, sizeof(rights));
> > > > > > > >         cap_rights_init(&rights, CAP_EXTATTR_DELETE, CAP_EXTATTR_GET,
> > > > > > > >             CAP_EXTATTR_LIST, CAP_EXTATTR_SET);
> > > > > > > >         cap_rights_limit(fd, &rights);
> > > > > > > >
> > > > > > > >         return (fd);
> > > > > > > > }
> > > > > > > > ==== END CODE ====
> > > > > > > >
> > > > > > > > Eventually, after calling cap_enter(2), the following code is called:
> > > > > > > >
> > > > > > > > ==== BEGIN CODE ====
> > > > > > > > #define ATTRNAME_ENABLED "hbsd.pax.aslr"
> > > > > > > >         sz = extattr_get_fd(fd, ctx->hc_namespace, ATTRNAME_ENABLED, NULL, 0);
> > > > > > > >         if (sz <= 0) {
> > > > > > > >                 if (errno == ENOATTR) {
> > > > > > > >                         /*
> > > > > > > >                         * This is okay, it just means that nothing has been set.
> > > > > > > >                         * No error condition here.
> > > > > > > >                         */
> > > > > > > >                         return (RES_SUCCESS);
> > > > > > > >                 }
> > > > > > > >                 return (RES_FAIL);
> > > > > > > >         }
> > > > > > > > ==== END CODE ====
> > > > > > > >
> > > > > > > > For reference, the program's code is here:
> > > > > > > > https://git.hardenedbsd.org/shawn.webb/hbsdctrl/-/tree/main?ref_type=heads
> > > > > > > >
> > > > > > > > The library code, which is what's responsible for calling the
> > > > > > > > filesystem extended attribute related syscalls is here:
> > > > > > > >
> > > > > > > > https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/tree/hardened/current/hbsdcontrol-v2/lib/libhbsdcontrol?ref_type=heads
> > > > > > > >
> > > > > > > > From the rights(4) manual page, I'm instructed all I need are to apply
> > > > > > > > those capabilities to that file descriptor:
> > > > > > > >
> > > > > > > > ==== BEGIN PASTE ====
> > > > > > > >      CAP_EXTATTR_DELETE   Permit extattr_delete_fd(2).
> > > > > > > >
> > > > > > > >      CAP_EXTATTR_GET      Permit extattr_get_fd(2).
> > > > > > > >
> > > > > > > >      CAP_EXTATTR_LIST     Permit extattr_list_fd(2).
> > > > > > > >
> > > > > > > >      CAP_EXTATTR_SET      Permit extattr_set_fd(2).
> > > > > > > > ==== END PASTE ====
> > > > > > > >
> > > > > > > > So I'm a bit unsure if I'm doing something wrong.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > --
> > > > > > > > Shawn Webb
> > > > > > > > Cofounder / Security Engineer
> > > > > > > > HardenedBSD
> > > > > > > >
> > > > > > > > Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
> > > > > > > > https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
> > > > > > >
> > > > > > > What error code does it fail with?  If it's ENOTCAPABLE, then I
> > > > > > > suggest using dtrace to find the reason why it fails.  Do something
> > > > > > > like this:
> > > > > > >
> > > > > > > dtrace -i 'fbt:kernel::return /arg1 == 93 && pid == $target/
> > > > > > > {trace(".");}' -c ./my_application
> > > > > > >
> > > > > > > That will print the name of every non-inlined kernel function that
> > > > > > > returns ENOTCAPABLE during your process.  But it will also print the
> > > > > > > names of any other kernel functions that return an integer value of
> > > > > > > 93.  From there, guess which function is the real source of the error.
> > > > > > > Then you can do
> > > > > >
> > > > > > DTrace is unavailable on this particular system.
> > > > > >
> > > > > > It does indeed fail with ENOTCAPABLE. I have the kern.trap_enotcap sysctl
> > > > > > set to 1 so that I can know at exactly what point we're failing, and
> > > > > > it's indeed at extattr_get_fd.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Shawn Webb
> > > > > > Cofounder / Security Engineer
> > > > > > HardenedBSD
> > > > > >
> > > > > > Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
> > > > > > https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc
> > > > >
> > > > > Without dtrace, you've got your work cut out for you.  I suggest
> > > > > simply adding all capabilities, verifying that extattr_get_fd works,
> > > > > and then removing capabilities until it fails.  Or, run your program
> > > > > on vanilla FreeBSD with dtrace.
> > > >
> > > > HardenedBSD doesn't have any modifications that would affect Capsicum
> > > > in this manner. Regardless, I reproduced the problem successfully on
> > > > FreeBSD 14.0-RELEASE without any code changes. I tried running your
> > > > DTrace script on FreeBSD 14.0-RELEASE and got this output:
> > > >
> > > > ==== BEGIN OUTPUT ====
> > > > $ sudo dtrace -i 'fbt:kernel::return /arg1 == 93 && pid == $target/ {trace(".");}' -c "obj/hbsdctrl pax list /bin/ls"
> > > > dtrace: description 'fbt:kernel::return ' matched 31396 probes
> > > > aslr:   sysdef
> > > > mprotect:       sysdef
> > > > pageexec:       sysdef
> > > > segvguard:      sysdef
> > > > dtrace: pid 29270 has exited
> > > > CPU     ID                    FUNCTION:NAME
> > > >   3  47780        foffset_unlock_uio:return   .
> > > >   3  50605              foffset_lock:return   .
> > > >   3  47778          foffset_lock_uio:return   .
> > > > ==== END OUTPUT ====
> > > >
> > > > But I'm still unsure what I'm missing, if anything.
> > > 
> > > That's red herring.  Those functions return void, but dtrace doesn't
> > > know it.  So the "93" is just register garbage.  I also notice that
> > > kern_extattr_get_fd isn't listed.  Are you sure that your program is
> > > really failing with ENOTCAPABLE?  You can also try running it with
> > > ktrace.  kdump will show you exactly what capabilities you limited the
> > > file descriptor to.  That can help you verify if you applied the
> > > limits correctly.
> > 
> > I'm pretty sure it's failing with ENOTCAPABLE, since the
> > kern.trap_enotcap sysctl logic is being hit at the point of the
> > extattr_get_fd syscall. I'll see what I can do
> > with ktrace.
> 
> ktrace output captured here:
> https://hardenedbsd.org/~shawn/2024-03-23_kdump-01.txt
> 
> The relevant portion is pasted here:
> 
> ==== BEGIN ktrace OUTPUT ====
>  41878 hbsdctrl CALL  openat(AT_FDCWD,0x6894a36d4f86,0x500000<O_RDONLY|O_CLOEXEC|O_PATH>)
>  41878 hbsdctrl NAMI  "/scratch/tmp/ls"
>  41878 hbsdctrl RET   openat 3
>  41878 hbsdctrl CALL  cap_rights_limit(0x3,0x6894a36d4bc0)
>  41878 hbsdctrl STRU  cap_rights_t CAP_SEEK,CAP_FSTATAT,CAP_EXTATTR_DELETE,CAP_EXTATTR_GET,CAP_EXTATTR_LIST,CAP_EXTATTR_SET,CAP_ACL_GET
>  41878 hbsdctrl RET   cap_rights_limit 0
>  41878 hbsdctrl CALL  cap_enter
>  41878 hbsdctrl RET   cap_enter 0
>  41878 hbsdctrl CALL  extattr_get_fd(0x3,0x2,0x33ff63afbaf,0,0)
>  41878 hbsdctrl NAMI  "freebsd:system:hbsd.pax.segvguard"
>  41878 hbsdctrl CAP   restricted VFS lookup
>  41878 hbsdctrl RET   extattr_get_fd -1 errno 94 Not permitted in capability mode
>  41878 hbsdctrl PSIG  SIGTRAP SIG_DFL code=TRAP_CAP
> ==== END ktrace OUTPUT ====
> 
> As you can see, I'm adding a bunch more capabilities to the file
> descriptor. The point of failure is still extattr_get_fd returning
> ENOTCAPABLE.
> 
> I'm beginning to suspect that while extattr_get_fd is documented to be
> allowed with a file descriptor with the CAP_EXTATTR_GET capability,
> this functionality is broken in FreeBSD.
> 
> The thing that really confuses me is that the extattr_get_fd syscall
> entry in syscalls.master has the CAPENABLED flag set:
> 
> ==== BEGIN syscalls.master ====
> 372     AUE_EXTATTR_GET_FD      STD|CAPENABLED {
>                 ssize_t extattr_get_fd(
>                     int fd,
>                     int attrnamespace,
>                     _In_z_ const char *attrname,
>                     _Out_writes_bytes_(nbytes) void *data,
>                     size_t nbytes
>                 );
>         }
> ==== END syscalls.master ====
> 
> I think it's now time for me to file a bug report in Bugzilla.

Bug report submitted:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277908

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc