From nobody Fri Feb 10 16:23:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PCzZL2dJNz3pFCL for ; Fri, 10 Feb 2023 16:23:54 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PCzZK2SgFz4Y4B for ; Fri, 10 Feb 2023 16:23:53 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=LZJdbhd+; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::f33 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org; dmarc=none Received: by mail-qv1-xf33.google.com with SMTP id i12so3858877qvs.2 for ; Fri, 10 Feb 2023 08:23:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=0IR3mSl+iVqnlcjdq7/lFp8WbcOF2ahpK206R/HH+hc=; b=LZJdbhd+dobdMvM3Cez8OQc65AEwrgT221fnzdd/udTR9bF7+UbpdcSZiWIGx4Dx3l dVk1+hYy6/sME4AOuP9HXj0INt32AtSbKUNTEnwejvbfSZNvj0m5LhofMba7ePo5tA7V zarsJjE4+zbtK/KfnRMWNoCW8l7hlswQKb1yjk/zd+u+z9jNCTdLo2BK72dkb7sK+xdO WHV+8qWNydXeIYvNpWBK0QGwmZgfZg3jyFyBBRkDl27hB9xFWa9jwWTnodQoNqZx7qHp 0zZpiHUx/dxitnqGYdKbiZJGScoOlQ1uC5WiN2C0A/Lhn3+VM/5PBZjdsf9B92JjnmNF 3ogg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=0IR3mSl+iVqnlcjdq7/lFp8WbcOF2ahpK206R/HH+hc=; b=WsSnF8H0aWO4d6XmI4nDuh/gf+D/zLbEaOqeN8UUGVUdt8He601NXrGMdZppNFNT+a XEUdtyOmh3hmbSIqaGIZqL3KxX3usShPz82/Rruv/Cnp9G/AjIf6u7JjFJQYKhqsyQ4m EgKucG297lyVlQIBrYcZQu9uN9cycKBLLVxPArk0/gnZ2LrZK4XOaDRlUerMpg8GhVyL tFaALBc8TbOFhrqmekrMXxLKyNU4pcUSryrlx9x7mx/g3AXohsTl6HS/pHRCNgNalUEM tGuNA5OaikIUZc+LH3SFqTDULLMgvsljP9+6TqSfNI92govU99aicFc6V88BFj84f4lX c/7Q== X-Gm-Message-State: AO0yUKUQE5Q/wam7Ao0AIshB3MhG4byMAZXETZeJc4UUSaP366r4naiF Yzcsp/AKb3ldadvu/Wa9IiyNHA== X-Google-Smtp-Source: AK7set9B5r2YI47y2zg8kjd1T4zrWsbPpJWEHGk7P20A/59VxjvkjdfsuIEKnTtwrY2ZqefGuEdplg== X-Received: by 2002:ad4:5bcc:0:b0:56b:f6a0:2333 with SMTP id t12-20020ad45bcc000000b0056bf6a02333mr25462176qvt.28.1676046232011; Fri, 10 Feb 2023 08:23:52 -0800 (PST) Received: from mutt-hbsd (pool-100-16-219-215.bltmmd.fios.verizon.net. [100.16.219.215]) by smtp.gmail.com with ESMTPSA id k189-20020a3788c6000000b0071dc769d5e7sm3772193qkd.56.2023.02.10.08.23.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Feb 2023 08:23:51 -0800 (PST) Date: Fri, 10 Feb 2023 11:23:50 -0500 From: Shawn Webb To: David Chisnall Cc: freebsd-hackers Subject: Re: CFT: snmalloc as libc malloc Message-ID: <20230210162350.gg5g7dihp3zef3ov@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="7ww2svwck3f2m2k5" Content-Disposition: inline In-Reply-To: <2f3dcda0-5135-290a-2dff-683b2e9fe271@FreeBSD.org> X-Spamd-Result: default: False [-5.10 / 15.00]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_DKIM_ALLOW(-0.20)[hardenedbsd.org:s=google]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::f33:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[hardenedbsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4PCzZK2SgFz4Y4B X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N --7ww2svwck3f2m2k5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 09, 2023 at 12:08:49PM +0000, David Chisnall wrote: > Hi, >=20 > For the few yearsI've been running locally with snmalloc as the malloc in > libc. Eventually I'd like to propose this for upstreaming but it needs s= ome > wider testing first. >=20 > For those unfamiliar with snmalloc (https://github.com/microsoft/snmalloc= ), > it is an allocator (or, rather, a toolkit for building allocators) from my > team at Microsoft Research designed for both performance and security. A > few highlights: >=20 > - Snmalloc uses a message-passing design, which makes allocating on one > thread and freeing on another cheap. > - Very fast allocation performance > - Randomisation of relative locations of allocations > - Most metadata is stored out-of-band > - In-band metadata uses some lighweight encryption to protect against > corruption. > - Support for CHERI. >=20 > In the (limited!) testing that I've done, it outperforms jemalloc and > results in a smaller libc binary. >=20 > I've also previously managed to use it in the kernel, though that code > hasn't been tested in a while (last used with FreeBSD 11): >=20 > https://github.com/microsoft/snmalloc/blob/main/src/snmalloc/pal/pal_free= bsd_kernel.h >=20 > It is also used in the Verona process sandboxing work, which makes it easy > to isolate a library in a capsicum Sandbox: >=20 > https://github.com/microsoft/verona/tree/master/experiments/process_sandb= ox >=20 > We test on FreeBSD in CI upstream and the code is actively maintained. > We have implemented compatibility wrappers for all of the jemalloc > non-standard APIs that FreeBSD's libc exposes. >=20 > In particular, snmalloc is designed to make it very cheap to find the sta= rt > and end of an allocation, given a heap pointer. This means that we can > insert bounds checks in critical libc functions to prevent heap overflow. > This is done in the branch for memcpy, which some investigation of a corp= us > of security vulnerabilities showed was the root cause of about 10% of > arbitrary-code-execution vulnerabilities. >=20 > The bounds checks are controlled via an environment variable > LIBC_BOUNDS_CHECKS. Setting this to 0 disables checks, to 1 checks on > destination arguments, and to 2 checks sources and destinations. An ifunc > resolver selects the correct memcpy implementation at load time. >=20 > I did have a version that checked a bunch of other libc functions (e.g. > sprintf, puts) but it was quite hacky (and the way the ifunc resolves was > implemented broke tcl). >=20 > The current branch puts two things behind the MALLOC_PRODUCTION toggle: >=20 > - The additional security checks that detect corruption of malloc state. > - Pretty-printing errors. >=20 > We are currently separating the former into separate knobs upstream, some > subset should probably be turned on by default in production. The latter > has less of a performance impact than it had and will probably be on for = all > configurations at some point once we've refactored slightly to ensure the > compiler can tail call the failure function (which moves it entirely off = the > fast path). With this enabled, you get errors that look like this: >=20 > Fatal Error! > memcpy with source out of bounds of heap allocation: > range [0x14823c02440, 0x14823c0246a) > allocation [0x14823c02440, 0x14823c02450) > range goes beyond allocation by 0x1a bytes >=20 > Abort trap (core dumped) >=20 > Without it, you just get an illegal instruction trap. >=20 > There are a few limitations in the current branch: >=20 > - The memcpy integration is broken on non-amd64 platforms (patches welco= me > from people who can test these!). > - Only memcpy (not, for example, memmove) has bounds checks. > - The memcpy in rtld is naive, which may impact performance. > - MALLOC_PRODUCTION conflates too many things >=20 > The branch is here: >=20 > https://github.com/davidchisnall/freebsd-src/tree/snmalloc2 >=20 > It adds snmalloc as a submodule in contrib. FreeBSD is allergic to > submodules, so upstreaming will need to replace this with something more > complicated. You should be able to cherry-pick the top commit on any > vaguely-recent -CURRENT. So I took a little bit of a different approach, which should provide the same end result as your submodule approach. Note that I'm doing this in HardenedBSD 14-CURRENT (the hardened/current/master branch). 1. git cherry-pick -xs a5c83c69817d03943b8be982dd815c7e263d1a83 2. git rm -f .gitmodules contrib/snmalloc 3. git commit 4. git subtree add -P contrib/snmalloc \ git@github.com:microsoft/snmalloc.git main I believe this should leave me with a tree that populates contrib/snmalloc and pulls in your non-contrib/ changes, leading me to end up in the same end state as your submodule approach. I am seeing some build errors. I've uploaded a WITHOUT_CLEAN=3Dyes log to: https://hardenedbsd.org/~shawn/2023-02-10_snmalloc-01.log.txt Note that this is with llvm 15.0.7 that just landed in FreeBSD main. Any non-XKCD[138]-conforming pointers would be appreciated. ;-) Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --7ww2svwck3f2m2k5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmPmb48ACgkQ/y5nonf4 4foIoQ//Y90bPyFfbRmK8vPOs0cZ4EKWuui5YZvyvv1BxM7yU9VFWeVwF/C1HIP1 0MKkxRyxE3s68K/xMBj2VCx6OgBShofVlzIkz9RWiza39CliU5Qx0ZcSZbDRT5ng UXTCut/VTIbaNJtpZMYswl6tdMs4U1yroi8r9UsytmGkV9PREl51YDFTPnopVmSC m79/INU4IOdz0uee9LDZfx95G1fU/sgjjNL9dQtTHuDSrq5iOynILlfr1bcEoR1O IZAOp+P36AeqodHMDh+E7G/vO04qhR7MugHkNpUP/WQEIa0dLtffo58BWnvzCymd h9I8wtTl5RGyy0gkgnrQWdOEEz8GV6hIMDtr5/LrL/YHQUPC2xoF5cyvIh/4fmLr oxKY/GfCDu49wy5sHZ14iBM5v8zGESZK1nx19XX/vctpwV3xQkH6FnOqGFzX9Rkw vLR5AtGOSf8Wsk+4Doki3UjG0Ax+KzCWhQGkKEW2hNWHEz9k1oq0gBykURSedqEA UPqabFF/bc7TP3d6lec8OwFmaCjCKZBmLZxvG4K3tDYb+wqtYZ6EeNDJ29gamFmW lwxdHfOuPE/NrWx1f/Ie9a6xkTTM4o10397pOBqeFmEWzvAkz7k928ib0INxh1qP CAcfH7NaDFCAI2Vfeq2NckdRXdYwOqFFDmlDOi4sYQUdam/8LMU= =8Zb7 -----END PGP SIGNATURE----- --7ww2svwck3f2m2k5--