From nobody Sun Sep 10 15:53:03 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RkDs32d8Jz4smkS for ; Sun, 10 Sep 2023 15:53:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RkDs31Hh8z3GXr for ; Sun, 10 Sep 2023 15:53:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-98377c5d53eso450097666b.0 for ; Sun, 10 Sep 2023 08:53:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1694361188; x=1694965988; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ZfJK3wAogQjmDvglFZJ70SBIOy2l57NwLuX6SK5uiPk=; b=NYWC19CbSP4BElBMIO/Pf4jhfLUjHcNNYs53DQ27zUTShVISh03dmvkJxuLtiCzJ8g qMZStacaFDwfgNI4Zys7+es7wk2LkYw8dfPhURhAhVeU+XGzcJIkHrOuw9foY3F+6WQJ ltTTQe8Hs243rDjgzPfEjM9i2L5Twu1HZ9wOzi9+ToPWVXpnFHHUdIX9m8w6Lurk+2iX B/LVjT3+u4Y0ZNxW2apP9T3l6LvGkw1uNO+/ReW+qec8+vWcrdf63uqSEgTkIhS258Gd g46GI+2QXS7hT+YXz7TcYYSY9j81zYWre+EvUr5725+Qb8Bhv3dThkTYcw12TvpWctiq Or1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694361188; x=1694965988; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZfJK3wAogQjmDvglFZJ70SBIOy2l57NwLuX6SK5uiPk=; b=Lj654T8xHJ8uc7SfIW1xlipYm/nnQAoHq9HYPqVa4r8tZRsjTRLBztp52coZUNEVI+ 9YsHrXSrixXjEWmcFGHcsG0N5R2EilkJo7Hw+EeU/nkqvJNllnFJhd9C6SkLBgjZyDXL YZakZxW6tt+bqfadaheTJaWp9S4lqDFdf0U7hU7BbqAaQo9eBDBjfyStLivAoS8Nl69F JuKGMnEG51RHYyCyL0wIuwSkvXqejT02xDIINnaBjKJFAG9kp+8LK4zNDXhoje0s0qXT 69bOxxBf8pVqPhaQEIAuPTRmfrMarlEGEeRYGABZGwk1cpromyIBBb3bwrBGxjmNuYZb kouA== X-Gm-Message-State: AOJu0Yw4YLNr0iheFFZTlLmBAaDq0W0zRC0qs5O7CT9rnNFh9Ry7hWo7 V66WRTAuqm4onplMNCXXPP39cA4HxqfUQ+NgfXRISxO2RGsd9O3GtFQ= X-Google-Smtp-Source: AGHT+IEYOIIdxgOuKY5Lh7zQ/NNz13tSjNqlH6pCCaqsYN9JoVXUXo9qGOuj5TVeWFSlSjM8HOxsEV4ZB4jNgAp+aZ8= X-Received: by 2002:a17:906:56:b0:9a5:7ade:b5e1 with SMTP id 22-20020a170906005600b009a57adeb5e1mr6369355ejg.70.1694361188414; Sun, 10 Sep 2023 08:53:08 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <30a59ac0-28fd-f6ed-505c-9ef6d8e84fc3@tsoft.com> <86edj7qnia.fsf@ltc.des.no> In-Reply-To: From: Warner Losh Date: Sun, 10 Sep 2023 09:53:03 -0600 Message-ID: Subject: Re: sed in CURRENT fails in textproc/jq To: Robert Clausecker Cc: FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000c1cb950605033405" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4RkDs31Hh8z3GXr --000000000000c1cb950605033405 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Sep 10, 2023, 7:36 AM Robert Clausecker wrote: > Hi Warner, > > I have pushed a fix. It should hopefully address those failing tests. > The same issue should also affect memcmp(), but unlike for memchr(), it i= s > illegal to pass a length to memcmp() that extends past the actual end of > the buffer as memcmp() is permitted to examine the whole buffer regardles= s > of where the first mismatch is. > > I am considering a change to improve the behaviour of memcmp() on such > errorneous inputs. There are two options: (a) I could change memcmp() th= e > same way I fixed memchr() and have implausible buffer lengths behave as i= f > the buffer goes to the end of the address space or (b) I could change > memcmp() to crash loudly if it detects such a case. I could also > (c) leave memcmp() as is. Which of these three choices is preferable? > What does the standard say? I'm highly skeptical that these corner cases ar= e UB behavior. I'd like actual support for this statement, rather than your conjecture that it's illegal. Even if you can come up with that, preserving the old behavior is my first choice. Especially since many of these functions aren't well defined by a standard, but are extensions. As for memchr, https://pubs.opengroup.org/onlinepubs/009696799/functions/memchr.html has no such permission to examine 'the entire buffer at once' nor any restirction as to the length extending beyond the address space. I'm skeptical of your reading that it allows one to examine all of [b, b + len), so please explain where the standard supports reading past the first occurance. Also, all the tests that started failing with your commit, available here, https://ci.freebsd.org/job/FreeBSD-main-amd64-test/24136/ should be fixed at a bare minimum. Yes, there's a lot of other tests that are failing: those should be fixed or disabled also. Have these tests been fixed? They are showing up as failed still in the latest run https://ci.freebsd.org/job/FreeBSD-main-amd64-test/24141/ shows them still failing by my reading. Also, we're several weeks in, and we're still fixing basic, fundamental functions that show breakage. This indicates to me that proper care hasn't been taken to replace the critical functions in the tree. It also indicates to me that a stronger level of rigor is needed given that the problems are trickling in every few days to weeks. This suggests that the unusual step of backing all this work out of stable/14 and releng/14.0 would be indicated until we can go through this process to the point that the new bugs have stopped (which given the decay rate seen so far suggests we won't be done before 14.0 is released). i know this is an enormous pain, but I think the track record so far supports backing these out of the release we're about to do. They just arrived too late for the normal 'cooking' process of FreeBSD to sufficiently cook them in time for 14.0. Warner Yours, > Robert Clausecker > > Am Sun, Sep 10, 2023 at 05:51:43AM -0600 schrieb Warner Losh: > > On Sat, Sep 9, 2023, 10:51 PM Robert Clausecker wrote: > > > > > Greetings, > > > > > > I apologise for the inconvenience. The issue seems to boil down to > > > various places calling > > > > > > memchr(buf, c, SIZE_MAX); > > > > > > which causes an overflow when my newly written memchr() computes buf = + > > > len to find the end of the buffer. A patch to alleviate this issue c= an > > > be found here: > > > > > > > > > > http://fuz.su/~fuz/freebsd/0001-lib-libc-amd64-string-memchr.S-fix-behavi= our-with-ov.patch > > > > > > Please check if it does the trick for you. If yes, I'll go ahead and > > > push it tomorrow-ish. > > > > > > > > > There are half a dozen or do kyua tests that are likely failing because > of > > this or other reasons related to strings. When you push this fix you'l= l > > get the list. They are hidden among about 80 or so networking tests tha= t > > fail. I plan on disabling those tests soon If no one fixes them. > > > > Warner > > > > > > > > Yours, > > > Robert Clausecker > > > > > > Am Sat, Sep 09, 2023 at 07:12:29PM +0200 schrieb Dag-Erling Sm=C3=B8r= grav: > > > > Antoine Brodin writes: > > > > > Yuri writes: > > > > > > Either something has changed in sed(1) in CURRENT, or sed just > fails > > > > > > during the configure stage of textproc/jq: > > > > > > > > > > > > sed: No error: 0 > > > > > > checking for sys/cygwin.h... eval: ${+...}: Bad substitution > > > > > This seems to be a recent issue (less than 5 days). > > > > > Hundreds of configure scripts now fail to run on 15-current due t= o > > > > > this sed failure: [...] > > > > > > > > Try adding ARCHLEVEL=3Dscalar to CONFIGURE_ENV on one of these. If > that > > > > helps, yell at fuz@ :) > > > > > > > > DES > > > > -- > > > > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org > > > > > > > > > > -- > > > () ascii ribbon campaign - for an 8-bit clean world > > > /\ - against html email - against proprietary attachments > > > > > > > > -- > () ascii ribbon campaign - for an 8-bit clean world > /\ - against html email - against proprietary attachments > --000000000000c1cb950605033405 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Sep 10, 2023, 7:36 AM Robert = Clausecker <fuz@fuz.su> wrote:
Hi Warner,

I have pushed a fix.=C2=A0 It should hopefully address those failing tests.=
The same issue should also affect memcmp(), but unlike for memchr(), it is<= br> illegal to pass a length to memcmp() that extends past the actual end of the buffer as memcmp() is permitted to examine the whole buffer regardless<= br> of where the first mismatch is.

I am considering a change to improve the behaviour of memcmp() on such
errorneous inputs.=C2=A0 There are two options: (a) I could change memcmp()= the
same way I fixed memchr() and have implausible buffer lengths behave as if<= br> the buffer goes to the end of the address space or (b) I could change
memcmp() to crash loudly if it detects such a case.=C2=A0 I could also
(c) leave memcmp() as is.=C2=A0 Which of these three choices is preferable?=



has no such per= mission to examine 'the entire buffer at once' nor any restirction<= /div>
as to the length extending beyond the address space. I'm skep= tical of your reading
that it allows one to examine all of [b, b = + len), so please explain where the standard
supports reading pas= t the first occurance.

Also, all the tests tha= t started failing with your commit, available here,


s= hould be fixed at a bare minimum. Yes, there's a lot of other tests
that are failing: those should be fixed or disabled also. Have these=
tests been fixed? They are showing up as failed still in the lat= est run


shows them still failing by my read= ing.

Also, we're several weeks in, and we&= #39;re still fixing basic, fundamental
functions that show breaka= ge. This indicates to me that proper care
hasn't been taken t= o replace the critical functions in the tree. It also
indicates t= o me that a stronger level of rigor is needed given that the
prob= lems are trickling in every few days to weeks. This suggests that
the unusual step of backing all this work out of stable/14 and releng/14.0=
would be indicated until we can go through this process to the p= oint
that the new bugs have stopped (which given the decay rate s= een so
far suggests we won't be done before 14.0 is released)= . i know this is
an enormous pain, but I think the track record s= o far supports backing
these out of the release we're about t= o do. They just arrived too late
for the normal 'cooking'= process of FreeBSD to sufficiently cook them
in time for 14.0.

Warner

Yours,
Robert Clausecker

Am Sun, Sep 10, 2023 at 05:51:43AM -0600 schrieb Warner Losh:
> On Sat, Sep 9, 2023, 10:51 PM Robert Clausecker <fuz@fuz.su> wrote:<= br> >
> > Greetings,
> >
> > I apologise for the inconvenience.=C2=A0 The issue seems to boil = down to
> > various places calling
> >
> >=C2=A0 =C2=A0 =C2=A0memchr(buf, c, SIZE_MAX);
> >
> > which causes an overflow when my newly written memchr() computes = buf +
> > len to find the end of the buffer.=C2=A0 A patch to alleviate thi= s issue can
> > be found here:
> >
> >
> > http://fuz.su/~fuz/freebsd/0001-lib-libc-amd64-string-memchr.S-= fix-behaviour-with-ov.patch
> >
> > Please check if it does the trick for you.=C2=A0 If yes, I'll= go ahead and
> > push it tomorrow-ish.
> >
>
>
> There are half a dozen or do kyua tests that are likely failing becaus= e of
> this or other reasons related to strings.=C2=A0 When you push this fix= you'll
> get the list. They are hidden among about 80 or so networking tests th= at
> fail. I plan on disabling those tests soon If no one fixes them.
>
> Warner
>
> >
> > Yours,
> > Robert Clausecker
> >
> > Am Sat, Sep 09, 2023 at 07:12:29PM +0200 schrieb Dag-Erling Sm=C3= =B8rgrav:
> > > Antoine Brodin <antoine@freebsd.org> writes:
> > > > Yuri <yuri@freebsd.org> writes:
> > > > > Either something has changed in sed(1) in CURRENT,= or sed just fails
> > > > > during the configure stage of textproc/jq:
> > > > >
> > > > > sed: No error: 0
> > > > > checking for sys/cygwin.h... eval: ${+...}: Bad su= bstitution
> > > > This seems to be a recent issue (less than 5 days).
> > > > Hundreds of configure scripts now fail to run on 15-cur= rent due to
> > > > this sed failure: [...]
> > >
> > > Try adding ARCHLEVEL=3Dscalar to CONFIGURE_ENV on one of the= se.=C2=A0 If that
> > > helps, yell at fuz@ :)
> > >
> > > DES
> > > --
> > > Dag-Erling Sm=C3=B8rgrav - des@FreeBSD.org
> > >
> >
> > --
> > ()=C2=A0 ascii ribbon campaign - for an 8-bit clean world
> > /\=C2=A0 - against html email=C2=A0 - against proprietary attachm= ents
> >
> >

--
()=C2=A0 ascii ribbon campaign - for an 8-bit clean world
/\=C2=A0 - against html email=C2=A0 - against proprietary attachments
--000000000000c1cb950605033405--