From nobody Sat Mar 25 18:23:04 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkSBK5hz0z41XKC for ; Sat, 25 Mar 2023 18:23:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-25.consmr.mail.gq1.yahoo.com (sonic303-25.consmr.mail.gq1.yahoo.com [98.137.64.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkSBJ4VHtz3GZ6 for ; Sat, 25 Mar 2023 18:23:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=D+6zf49O; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679768598; bh=A8T2cNzKjrsZDGbrEGIYM7bIWpUg0nzqkgILXEUJhXA=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=D+6zf49OomzPJHKKg4UVxApKyMJ2lZEQRSusDAa+6BiQDGbMzOXZHCxrvD18GyZaNSri02Phj4D8+ZYxU2W1Qrk2r00g22hUINKFGoIp3zXqFIaqV4qqkERtrtWdFpKV25Nbx75dnKYtG1Lwn4VTUYNGB+ljkL96uLv4zUrOMeA2G94eQkGtuer58kqFilLxk/2SPAP03Ed1a3Hd177QM8zGslV2B9aDh3VrHO+/r0L6PhVbFY5W77pbgcSs3uuKHXHisGjwZ929O0Txw7h05Zj1fIeNzvLT4HRn1nyUglulkOi/sOG72cJtFb+N7RrOrHyFINFk9CK0cLtbNAjJ5Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679768598; bh=f5NJerh2kbt178W7xkr2xE6JR6DKZ1oTjiuRF+KdG5g=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=kq74UdPZC1F7HbYanvxKSLEnZ1J/OFPv8HXVGbfE5Z7A1M7CWjZ9K5bWQkT75YAIWvzTRrkIXByWF17qB8MM0EHIaFsXwZGqVNUPFVy3+IJXaZEbHZpQmPuzVB/c1wKmFc1xmoaQutm38LvrKyDeI2BgDoYUdOW0NZFflko2eLoY5A6Hpx4TL/4unho1KvQKm4AnKRBZWUbrv22buUvLAcUMW8I6cisv7aJC8tq3gjzBT9Vsm3bWHjvlgBQIoMNfVrKu7m21tcARJLyp70Pw2SxpoFCW+2mLu6H4DS09EEGeS7md9mZBMtwhpaT6w8iLMMGW0KjR3IGJZWgK8jaGJg== X-YMail-OSG: HA5zqMwVM1kA68Pe9RS_1wymrDWc0eW0pHhX4RHXLM4X1K0zScwS9NK1u33g.Ir dR3Bnd0ilxYJ.k6Ko4.l5I0o_HF5IEJdgAebjYNsi0CyTnkqFzlmOhHGbnchhk9sYwmYbF.C8ewc AHcrgZjnHijLNi_TWzu0pwOAnMB4juy_mt8aNuEL9GQVERJ_fVBTOC.Fp2MIGQGg4OXpE7svZ_Yc bOoelHyIt3.f_XoCsJ3Y7_RI_3inWDDG0tzYFoOPDijrnod0c7EmgcObWWhkt9adw3t0yms1JDOD dWMIYXFld3NcHwgBsiOStujWuCNhrDj6jK4ThSH6eYXPWlY3nv4bFG6.YSRd_WuIKr7mfVP4AOfG stj8L8ki8VJNnBaB4cCoj2.ZQl16rhW3g.aS4hjr3wUtuX11W6SKWdPEcSGXsnkFPHTHFsnRtoYc MpFawOXLRAm8agv8fGWMKIt2REk1FXf4iZqeqNgRBKnXsRJ_tm99Eq6I.co3bcmBTzFUQPlRhzMK BoIXgEOUGUdXqDEhMuOEjtngJnDLqVA1UCKOITPqRrdwII.Yln9n97UnITqRU21nqtpBU.dCQQlt s2IknreWIMKlVL8kC6Y6v60D.uVXlaRupSYrsUqolpxbWXUhjj81lx9ljlJw7mzhZ5YhoaCAvrwv Tbn.s5D83htwyynELRX99dyv0wiKeTP1fc4oCBFoJvaVCv3CCET7mKrI2FbIDFXDdiWNPG7LxEc8 HluDJFnM7VX8RM7HrkDJuJiTcxxAQmR.azPJyQLuxQqq908_hwHfwadBMfECqn88yuuNQBwrvJeP ESKQq17sueTJkSvj2kQHM0.dkGHcJNSL.mHc2wGaIu_WcgAj7nYbe4_yq3zFHPzYnvXj0FoUuOAJ 91ZZ5BzlrQqRtWDFsq0nlrcOUMckmVh_1HNZRXIFEVDZ6cRJUhrx41YrhWP..8EGLJcGOnRhnZA8 BcQQOvo.0NuXqrXwQpno4dXpa5FGCWj20NNMJ4WAnkPEqpjFMl1dHLp6TK.9_WOlldNCjjlFPLRU n9AoK6K3MrzmsvDCtGgRQTbZrUu.82LV.qoVgtvYi8wTAAgyGtHwi9ex6XB05rggmJdIcAkF0BcR oVQrN2Fy4yKImHQ9xv5btOMUzbBaaW7oAUtErNhjrcohGtP2DdoDTK.6EbbcugBegwefdhNc5Mdo 5xQW9a8SFUNn3nfTQ87GI4Qkv_ZKhPiARP6HuDuBnLthbsk.oYQRUGtvsdsoqjbR1fMQX3aw4JFz uzFL_o2p7fwuW.x5_gt8kOi9TA3kBn4aYu5TAgxyh7jaf.EzGYQT3I6_NJ1uRkK7eYG7ZLqDnMC6 ypDI63C6BOl19gFeYoiMobSNCva2CEJLlyXDPZ2WMA3POAgwVbiHsmuEewRUs7C1FNhL4mxwNGEu 0ToZPnrEcFgtlmkKZIxyVjBTzyJyX2caiiSF2P4Ck1gMHlCJdFkH22auRDo3XbFeDoUhU09XvTW2 JpsqEpZhEDxZulI0kRpUQqBaz.26vCG1tNNWVzX_uxFQUJYO6XF.HKi1uHMdsiZyJEfP_nAR.B3V S6QmgPfelsuF82lTvXxhnoOnWMb6XoevKrh8WsqImDJnQKJeGDeav_t0n5w_RbhLX0VbBsOZm9Rd UWmfdMNA552zU4QIKVKGWokB3mZoYANNas89f_3A8krrW_fYaxX4V0qYvQ.vZmzw6T66fJW4RaZv AlFcNb5rvAHOvWJ2Cb0mOyVUodxFinm3CkUI6nQNeR9fF7hC1Ytqot6.nNtd6ouVF1Sywv1XOVr2 PCAqQ2Q7gdmUFd1KgjGbGDYpn7sqw5MMl37KXpK3f_qu0IJNn2nHpyHQrS9KPXcGHqBnubfV_RIe uMwGANEq114js4sXp4OQl89r4b_3R6ks4on00_jn38HS8UDJNVDKSjC2xZzJHKD53tgcnzNdPlhW pptGxqUMpMOceRlN6_E7wrjh1Uzgr8VyvSyGWu_MuNWWWnTxylGdNj8rEF8bzeuxvyrSp5XXso9N KN2i9D6fuvW7XL0WzL._twx6zLy8cI005cVWz2tcRvi8UF4zDerTGRof3wcZGZyD_JurvjumynLH vz1vwaHPPP__AE3lFseXT91j2NzrytoAWZUxLH7MhyuEVgGRTPHe28JnYtI5qr7pynIMF8LyNpja YaTEAGlgJYXPnXz8UGKmF2i4k8.dDNwJS8VAB7twB7RkHkrELr1p.6UUadp.ATTZnJwZVOu2iSXI 1qk.mpMHAbAkOzKIYS7qUxvdYnE8WzEWR7fnooEL8rnvtFiyjM2r0xxDly3ps2yELKCN8oLRhxC1 JAQ-- X-Sonic-MF: X-Sonic-ID: 9d834da9-91bd-4f56-bbdb-2a4069b030c4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Mar 2023 18:23:18 +0000 Received: by hermes--production-gq1-6cf7749bc8-q7lrl (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d2f9f0dd8de28b81e8a773019631e00d; Sat, 25 Mar 2023 18:23:15 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Date: Sat, 25 Mar 2023 11:23:04 -0700 References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> To: Peter , FreeBSD Hackers In-Reply-To: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> Message-Id: <76DAACBB-C865-4779-A340-D66C35D610B4@yahoo.com> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.25 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.75)[-0.752]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.206:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PkSBJ4VHtz3GZ6 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 25, 2023, at 11:14, Mark Millard wrote: > Peter wrote on > Date: Sat, 25 Mar 2023 15:47:42 UTC : >=20 >> Quoting George Mitchell : >>=20 >>>> = https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thres= h.85 >>>>=20 >>> Thank you! -- George >>=20 >> You're welcome. Can I get a success/failure report? >>=20 >>=20 >> --------------------------------------------------------------------- >>>> On 3/22/23, Steve Kargl wrote: >>>>>=20 >>>>> I reported the issue with ULE some 15 to 20 years ago. >>=20 >> Can I get the PR number, please? >>=20 >>=20 >> --------------------------------------------------------------------- >> Test usecase: >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>=20 >> Create two compute tasks competing for the same -otherwise unused- = core,=20 >> one without, one with syscalls:=20 >>=20 >> # cpuset -l 13 sh -c "while true; do :; done" &=20 >> # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null=20 >>=20 >> Within a few seconds the two task are balanced, running at nearly the=20= >> same PRI and using each 50% of the core:=20 >>=20 >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND=20 >> 5166 root 1 88 0 13M 3264K RUN 13 9:23 51.65% sh=20 >> 10675 root 1 87 0 13M 3740K CPU13 13 1:30 48.57% gzip=20 >>=20 >> This changes when the tar reaches /usr/include with it's many small=20= >> files. Now smaller blocks are delivered to gzip, it does more=20 >> syscalls, and things get ugly:=20 >>=20 >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND=20 >> 5166 root 1 94 0 13M 3264K RUN 13 18:07 95.10% sh=20 >> 19028 root 1 81 0 13M 3740K CPU13 13 1:23 4.87% gzip=20 >=20 > Why did PID 10675 change to 19028? >=20 >> This does not happen because tar would be slow in moving data to=20 >> gzip: tar reads from SSD, or more likely from ARC, and this is=20 >> always faster than gzip-9. The imbalance is made by the scheduler. >=20 >=20 > When I tried that tar line, I get lots of output to stderr: >=20 > # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null > tar: Removing leading '/' from member names > a . > a root > a wrkdirs > a bin > a usr > . . . >=20 > Was that an intentional part of the test? >=20 > To avoid this I used: >=20 > # tar cvf - / 2>/dev/null | cpuset -l 13 gzip -9 2>&1 > /dev/null >=20 > At which point I get the likes of: >=20 > 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 = 3.95% gzip -9 > 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 = 0.27% tar cvf - / (bsdtar) > 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 = 95.93% sh -c while true; do :; done >=20 > up front. >=20 > For reference, I also see the likes of the following from > "gstat -spod" (it is a root on ZFS context with PCIe Optane media): >=20 > dT: 1.063s w: 1.000s > L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name > . . . > 0 68 68 14 937 0.0 0 0 0 0.0 = 0 0 0 0.0 0 0.0 0.1| nvd2 > . . . >=20 >=20 I left it running and I'm now seeing: 17129 root 1 107 0 14192Ki 3628Ki CPU13 13 3:01 = 48.10% gzip -9 17128 root 1 21 0 58300Ki 15428Ki pipdwt 20 0:04 = 2.02% tar cvf - / (bsdtar) 17097 root 1 115 0 13364Ki 3060Ki RUN 13 16:30 = 51.77% sh -c while true; do :; done Also examples of the likes of: dT: 1.063s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name . . . 0 1213 1213 5 6456 0.0 0 0 0 0.0 = 0 0 0 0.0 0 0.0 1.2| nvd2 . . . FYI: ThreadRipper 1950X context. Looks like what I'll see is very dependent on when I look at what it is doing: the details involved matter. =3D=3D=3D Mark Millard marklmi at yahoo.com