git: af949c590bd8 - main - Disable stack gap for ntpd during build.
Cy Schubert
Cy.Schubert at cschubert.com
Fri May 21 17:53:57 UTC 2021
In message <CAPv3WKe4O--Jne20ozpMfLe3XvyPZXawUx+LgvOF8bsDEVsa7g at mail.gmail.c
om>
, Marcin Wojtas writes:
> Hi Cy,
>
> pt., 21 maj 2021 o 16:46 Cy Schubert <Cy.Schubert at cschubert.com> napisał(a):
> >
> > In message <02078965-24BE-4F23-92D5-5E8E54A0C3E7 at freebsd.org>, Jessica
> > Clarke w
> > rites:
> > > > On 21 May 2021, at 15:11, Marcin Wojtas <mw at semihalf.com> wrote:
> > > >
> > > > Hi Jess
> > > >
> > > > pt., 21 maj 2021 o 15:39 Jessica Clarke <jrtc27 at freebsd.org> napisał(a
> ):
> > > >>
> > > >> On 21 May 2021, at 14:34, Marcin Wojtas <mw at FreeBSD.org> wrote:
> > > >>>
> > > >>> The branch main has been updated by mw:
> > > >>>
> > > >>> URL: https://cgit.FreeBSD.org/src/commit/?id=af949c590bd8a00a5973b587
> 5d7e
> > > 0fa6832ea64a
> > > >>>
> > > >>> commit af949c590bd8a00a5973b5875d7e0fa6832ea64a
> > > >>> Author: Marcin Wojtas <mw at FreeBSD.org>
> > > >>> AuthorDate: 2021-05-21 09:29:22 +0000
> > > >>> Commit: Marcin Wojtas <mw at FreeBSD.org>
> > > >>> CommitDate: 2021-05-21 13:33:06 +0000
> > > >>>
> > > >>> Disable stack gap for ntpd during build.
> > > >>>
> > > >>> When starting, ntpd calls setrlimit(2) to limit maximum size of its
> > > >>> stack. The stack limit chosen by ntpd is 200K, so when stack gap
> > > >>> is enabled, the stack gap is larger than this limit, which results
> > > >>> in ntpd crashing.
> > > >>
> > > >> Isn’t the bug that the unusable gap counts as usage?
> > > >>
> > > >> Jess
> > > >>
> > > >
> > > > An alternative solution was submitted
> > > > (https://reviews.freebsd.org/D29832), so that to extend the limit for
> > > > ntpd, but eventually it was recommended to simple disable the stack
> > > > gap for it until it's fixed upstream (see the last comment in the
> > > > linked revision).
> > >
> > > That’s my point, there is nothing to “fix†upstream. NTPD uses less
> tha
> > > n 200K
> > > of stack, thus it is perfectly reasonable for it to set its limit to that
> . Th
> > > e
> > > fact that FreeBSD decides to count an arbitrary, non-deterministic amount
> of
> > > additional unusable virtual address space towards that limit is not its f
> ault
> > > ,
> > > but a bug in FreeBSD that needs to be fixed as it’s entirely unreasonab
> le f
> > > or
> > > applications to have to account for that.
> >
> > This latest problem is not stack gap. It is PIE.
> >
>
> I have to disagree.
We are talking cross purposes. Your examples later on in your email prove
my point.
> ntpd does not start because of stack gap, not PIE, even though it may
> seem like PIE causes this. This is due to the fact that stack gap is
> disabled if PIE is disabled. Because of that value of sysctl
> kern.elf64.aslr.stack_gap does not matter when kern.elf64.aslr.pie_enable
> is set to 0. When pie_enabled is set to 1 and stack gap is enabled, then
> ntpd fails to start, but when pie_enabled is set to 1 and stack_gap
> is set to 0, then ntpd starts without any issue. We verified this on
> FreeBSD-CURRENT snapshot from 2021-05-20.
I verified the PIE problem on a -CURRENT as of my comments in the review.
Enabling stack gap and disabling PIE resolved the issue. The reason for
stack gap is not a problem is that ntpd disables stack gap at line 441 of
ntpd.c.
Furthermore enabling stack gap and disabling PIE circumvents the problem. I
tested this myself and left that note in the review.
Enable stack gap and disable PIE: It works. But look at line 441 of ntpd.c
to see stack gap disabled before ntpd forks itself.
>
> The fact that this is a stack gap issue can be verified using following
> procedure:
> 1. Install FreeBSD-CURRENT snapshot from 2021-05-20 using default
> configuration.
> 2. On a newly installed system start ntpd. With default configuration
> it should start successfully.
> 3. Set sysctl kern.elf64.aslr.pie_enable=1 and start ntpd. This time ntpd
> should fail. An entry indicating that ntpd was killed because of signal
> 11 should be visible in /var/log/messages.
> 4. Set sysctl kern.elf64.aslr.stack_gap=0 and start ntpd once again. This
> time ntpd should start even though pie_enable is set to 1.
>
> Exact log from the boot it was tested:
> root at freebsd-ntpd-test:~ # sysctl -a | grep aslr
> kern.elf32.aslr.stack_gap: 3
> kern.elf32.aslr.honor_sbrk: 1
> kern.elf32.aslr.pie_enable: 0
> kern.elf32.aslr.enable: 0
> kern.elf64.aslr.stack_gap: 3
> kern.elf64.aslr.honor_sbrk: 1
> kern.elf64.aslr.pie_enable: 0
> kern.elf64.aslr.enable: 0
> vm.aslr_restarts: 0
> root at freebsd-ntpd-test:~ # ntpd
> root at freebsd-ntpd-test:~ # ps aux | grep ntpd
> root 826 0.0 0.2 22060 6960 - Ss 17:38 0:00.01 ntpd
> root 828 0.0 0.1 12976 2416 0 S+ 17:38 0:00.00 grep ntpd
> root at freebsd-ntpd-test:~ # killall ntpd
> root at freebsd-ntpd-test:~ # ps aux | grep ntpd
> root 831 0.0 0.1 12976 2416 0 S+ 17:38 0:00.00 grep ntpd
> root at freebsd-ntpd-test:~ # sysctl kern.elf64.aslr.pie_enable=1
> kern.elf64.aslr.pie_enable: 0 -> 1
This causes the problem.
> root at freebsd-ntpd-test:~ # ntpd
> root at freebsd-ntpd-test:~ # ps aux | grep ntpd
> root 836 0.0 0.1 14128 2452 0 S+ 17:39 0:00.00 grep ntpd
> root at freebsd-ntpd-test:~ # cat /var/log/messages | tail
> May 21 17:38:25 freebsd-ntpd-test ntpd[826]: ntpd exiting on signal 15
> (Terminated)
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: ntpd 4.2.8p15-a (1): Starting
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: Command line: ntpd
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]:
> ----------------------------------------------------
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: ntp-4 is maintained by
> Network Time Foundation,
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: Inc. (NTF), a non-profit
> 501(c)(3) public-benefit
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: corporation. Support and
> training for ntp-4 are
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]: available at
> https://www.nwtime.org/support
> May 21 17:39:14 freebsd-ntpd-test ntpd[833]:
> ----------------------------------------------------
> May 21 17:39:14 freebsd-ntpd-test kernel: pid 834 (ntpd), jid 0, uid
> 0: exited on signal 11 (core dumped)
> root at freebsd-ntpd-test:~ # sysctl kern.elf64.aslr.stack_gap=0
> kern.elf64.aslr.stack_gap: 3 -> 0
> root at freebsd-ntpd-test:~ # sysctl -a | grep aslr
> kern.elf32.aslr.stack_gap: 3
> kern.elf32.aslr.honor_sbrk: 1
> kern.elf32.aslr.pie_enable: 0
> kern.elf32.aslr.enable: 0
> kern.elf64.aslr.stack_gap: 0
> kern.elf64.aslr.honor_sbrk: 1
> kern.elf64.aslr.pie_enable: 1
This is the problem.
> kern.elf64.aslr.enable: 0
> vm.aslr_restarts: 1
> root at freebsd-ntpd-test:~ # ntpd
> root at freebsd-ntpd-test:~ # ps aux | grep ntpd
> root 845 0.0 0.2 22060 6924 - Ss 17:40 0:00.01 ntpd
> root 847 0.0 0.1 12976 2440 0 S+ 17:40 0:00.00 grep ntpd
> root at freebsd-ntpd-test:~ # cat /var/log/messages | tail
> May 21 17:39:14 freebsd-ntpd-test kernel: pid 834 (ntpd), jid 0, uid
> 0: exited on signal 11 (core dumped)
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: ntpd 4.2.8p15-a (1): Starting
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: Command line: ntpd
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]:
> ----------------------------------------------------
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: ntp-4 is maintained by
> Network Time Foundation,
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: Inc. (NTF), a non-profit
> 501(c)(3) public-benefit
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: corporation. Support and
> training for ntp-4 are
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]: available at
> https://www.nwtime.org/support
> May 21 17:40:52 freebsd-ntpd-test ntpd[844]:
> ----------------------------------------------------
> May 21 17:40:52 freebsd-ntpd-test ntpd[845]: leapsecond file
> ('/var/db/ntpd.leap-seconds.list'): stat failed: No such file or
> directory
> root at freebsd-ntpd-test:~ # killall ntpd
>
> Best regards,
> Marcin
Running on my firewall, which has had this same ASLR configuration for
about a year.
cwfw# sysctl kern.elf64.aslr
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 1
cwfw# ps auxww | grep ntpd
ntpd 1499 0.0 0.1 22044 5776 - Ss 09:30 0:00.28
/usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f
/var/db/ntp/ntpd.drift -g
root 3032 0.0 0.0 13044 2384 0 S+ 10:49 0:00.00 grep ntpd
cwfw# uptime
10:49AM up 1:20, 1 user, load averages: 1.06, 1.02, 0.97
cwfw# uname -a
FreeBSD cwfw 14.0-CURRENT FreeBSD 14.0-CURRENT #151
komquats-n246804-af949c590bd8-dirty: Fri May 21 07:09:32 PDT 2021
root at cwsys:/export/obj/opt/src/git-src/amd64.amd64/sys/PROD2 amd64
cwfw#
My laptop:
slippy# sysctl kern.elf64.aslr
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 1
slippy# ps auxww | grep ntpd
ntpd 2100 0.0 0.1 22036 8600 - Ss 09:35 0:00.27
/usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f
/var/db/ntp/ntpd.drift -g
root 4632 0.0 0.0 13040 2724 1 S+ 10:51 0:00.00 grep ntpd
slippy# uptime
10:51AM up 1:17, 0 users, load averages: 0.11, 0.16, 0.16
slippy# uname -a
FreeBSD slippy 14.0-CURRENT FreeBSD 14.0-CURRENT #155
komquats-n246804-af949c590bd8-dirty: Fri May 21 07:07:22 PDT 2021
root at cwsys:/export/obj/opt/src/git-src/amd64.amd64/sys/BREAK amd64
slippy#
One of my poudriere machines:
cwsys# sysctl kern.elf64.aslr
kern.elf64.aslr.stack_gap: 3
kern.elf64.aslr.honor_sbrk: 1
kern.elf64.aslr.pie_enable: 0
kern.elf64.aslr.enable: 1
cwsys# ps auxww | grep ntpd
ntpd 4039 0.0 0.1 22040 7340 - Ss 09:34 0:00.46
/usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f
/var/db/ntp/ntpd.drift -g
root 6385 0.0 0.0 13044 2712 2 S+ 10:52 0:00.01 grep ntpd
cwsys# uptime
10:52AM up 1:19, 2 users, load averages: 0.26, 0.25, 0.24
cwsys# uname -a
FreeBSD cwsys 14.0-CURRENT FreeBSD 14.0-CURRENT #155
komquats-n246804-af949c590bd8-dirty: Fri May 21 07:07:22 PDT 2021
root at cwsys:/export/obj/opt/src/git-src/amd64.amd64/sys/BREAK amd64
cwsys#
Three examples of stack gap enabled and PIE disabled. When I enable PIE,
ntpd fails.
--
Cheers,
Cy Schubert <Cy.Schubert at cschubert.com>
FreeBSD UNIX: <cy at FreeBSD.org> Web: https://FreeBSD.org
NTP: <cy at nwtime.org> Web: https://nwtime.org
The need of the many outweighs the greed of the few.
More information about the dev-commits-src-all
mailing list