Re: libc/libsys split coming soon

On 2/3/24, David Chisnall <theraven@freebsd.org> wrote:
> On 3 Feb 2024, at 09:15, Mateusz Guzik <mjguzik@gmail.com> wrote:
>>
>> Binary startup is very slow, for example execve of a hello world
>> binary in a Linux-based chroot on FreeBSD is faster by a factor of 2
>> compared to a native one. As such perf-wise this looks like a step in
>> the wrong direction.
>
> Have you profiled this?  Is the Linux version using BIND_NOW (which comes
> with a load of problems, but it often the default for Linux systems and
> reduces the number of slow-path entries into rtld)?  Do they trigger the
> same number of CoW faults?  Is there a path in rtld that’s slower than the
> equivalent ld-linux.so path?
>

I only profiled FreeBSD, it was 4 years ago. I have neither time nor
interest in working on this.

Relevant excerpts from profiling an fexecve loop:

Sampling what syscalls was being executed when in kernel mode
(or trap):

syscalls:
           pread              108
           fstat              162
       issetugid              250
     sigprocmask              303
            read              310
        mprotect              341
            open              380
           close             1547
            mmap             2787
            trap             5421
[snip]
In userspace most of the time is spent here:
                      ld-elf.so.1`memset              406
              ld-elf.so.1`matched_symbol              431
                      ld-elf.so.1`strcmp             1078
               ld-elf.so.1`reloc_non_plt             1102
                 ld-elf.so.1`symlook_obj             1102
                 ld-elf.so.1`find_symdef             1439

find_symdef iterates a linked list, which I presume induces strcmp calls
due to unwanted entries.
[snip]

Full profile
 user:
          libc.so.7`__je_extent_heap_new               71
          libc.so.7`__vdso_clock_gettime               73
                        libc.so.7`memset               75
                       ld-elf.so.1`_rtld               83
                      ld-elf.so.1`getenv               85
        libc.so.7`__je_malloc_mutex_boot              132
                   ld-elf.so.1`reloc_plt              148
                ld-elf.so.1`__crt_malloc              163
             ld-elf.so.1`symlook_default              166
             ld-elf.so.1`digest_dynamic1              184
        libc.so.7`__je_malloc_mutex_init              205
              ld-elf.so.1`symlook_global              281
                      ld-elf.so.1`memset              406
              ld-elf.so.1`matched_symbol              431
                      ld-elf.so.1`strcmp             1078
               ld-elf.so.1`reloc_non_plt             1102
                 ld-elf.so.1`symlook_obj             1102
                 ld-elf.so.1`find_symdef             1439
 kernel:
             kernel`vm_reserv_alloc_page               89
                    kernel`amd64_syscall               95
                     kernel`0xffffffff80              102
       kernel`vm_page_alloc_domain_after              114
             kernel`vm_object_deallocate              117
                kernel`vm_map_pmap_enter              122
                kernel`pmap_enter_object              140
                   kernel`uma_zalloc_arg              148
              kernel`vm_map_lookup_entry              148
         kernel`pmap_try_insert_pv_entry              156
                   kernel`vm_fault_dirty              168
                         kernel`pagecopy              177
                         kernel`vm_fault              260
                     kernel`get_pv_entry              265
                    kernel`pagezero_erms              367
          kernel`pmap_enter_quick_locked              380
                       kernel`pmap_enter              432
                     kernel`0xffffffff80             1126
                     kernel`0xffffffff80             2017
                             kernel`trap             2097
                        syscalls:
                                   pread              108
                                   fstat              162
                               issetugid              250
                             sigprocmask              303
                                    read              310
                                mprotect              341
                                    open              380
                                   close             1547
                                    mmap             2787
                                    trap             5421

Counting fexecve:
dtrace -n 'fbt::sys_fexecve:entry { @[count] = stack(); } tick-30s { exit(0); }'

dtrace script, can be run as: dtrace -w -x aggsize=128M -s script.d
assumes binary name is a.out

syscall::fexecve:entry
{
        self->inexec = 1;
}

syscall::fexecve:return
{
        self->inexec = 0;
}

fbt::trap:entry
{
        self->trap = 1;
}

fbt::trap:return
{
        self->trap = 0;
}

profile:::profile-4999
/execname == "a.out" && arg1 && self->inexec == 0/
{

        @user[usym(arg1)] = count();
}

profile:::profile-4999
/execname == "a.out" && arg0 && self->inexec == 0 && self->trap == 0/
{

        @kernel[sym(arg0)] = count();
        @kernel_syscall[stringof(`syscallnames[curthread->td_sa.code])]
= count();
}

profile:::profile-4999
/execname == "a.out" && arg0 && self->inexec == 0 && self->trap == 1/
{

        @kernel[sym(arg0)] = count();
        @kernel_syscall["trap"] = count();
}

tick-5s
{
        system("clear");

        trunc(@user, 20);
        trunc(@kernel, 20);
        printa("%40A %@16d\n", @user);
        printa("%40a %@16d\n", @kernel);
        clear(@user);
        clear(@kernel);

        trunc(@kernel_syscall, 10);
        printf("\t\t\tsyscalls:\n");
        printa("%40s %@16d\n", @kernel_syscall);
        clear(@kernel_syscall);
}

fexecve-loop.c:
#include <sys/types.h>
#include <sys/time.h>

#include <err.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int
main(int argc, char **argv)
{
        char buf[8];
        char *cargv[3];
        int fd;

        switch (argc) {
        case 1:
                fd = open(argv[0], O_EXEC);
                if (fd == -1)
                        err(1, "open");
                cargv[0] = argv[0];
                cargv[1] = buf;
                sprintf(cargv[1], "%d", fd);
                cargv[2] = NULL;
                break;
        case 2:
                fd = atoi(argv[1]);
                cargv[0] = argv[0];
                cargv[1] = argv[1];
                cargv[2] = NULL;
                break;
        default:
                printf("invalid argc %d\n", argc);
                exit(1);
        }

        fexecve(fd, cargv, NULL);
        err(1, "fexecve");


-- 
Mateusz Guzik <mjguzik gmail.com>