From nobody Wed Apr 06 15:28:59 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 743CF1A836BF for ; Wed, 6 Apr 2022 15:29:32 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KYT2g4FWpz3sfT for ; Wed, 6 Apr 2022 15:29:31 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-io1-xd2a.google.com with SMTP id g21so3287442iom.13 for ; Wed, 06 Apr 2022 08:29:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FEu/bhJlTBVkdfIkobaSalDTOudXuaDPJ+G5DbpS7Nc=; b=TCi+BNcRYZCvE6gBIkIXlbEjEoDOE1Jsm3Uxpk6dAlZBEED9tquJLsFTvOsCW3g28y 4D+KqRey2tCH+IB+YV1iI8mZtzi76LrfWTRX0MMgYpQkaBs61aBDnBP15m+bHJPlY2c+ 9RuJn9YZup92QkoG+vU/V4vEc3RgcrKBUAuIYkkv9FV67MhX/yYJl4VpxWoBXaXRrpPv MckOa9JsF8650l5D/zWV6bmJ7JeXW+OxEmbRjKNQl8nMEOjbcS0XrPSG2iVlQSHEQoSD boOKGiqybm7QKa70VJlBUOZWFMUqIv7FOFHcmlEEvNsrZKxmFUF0J/oquV0zgo0CnZra qQog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FEu/bhJlTBVkdfIkobaSalDTOudXuaDPJ+G5DbpS7Nc=; b=tP5oN9ejA3CkAlU2XAsyOOtz+HP2uB6UwjaTVh5dKl5XwsgAshMvykKl7afN6WIDsq zuZQO4Jh9/mS1snK/Wed8YZrYjkiL8jzI4oMalxhBDBJOqxzCmrjv1pkQvM+ytQNtnBz fpBI1J50PrxPa6+WAJQ9JImPwUmQMaUPdgSjnYSSjGcyrjyRrSeOGTagU3vyzm+p4fjJ yEqUJVd7ht6d3GQC6VfJtEgj85+Y4gtxsNneTe6dRNelEe+dTMj1dqNd9FS4IVM41LyM s7/MxVOUb35pJ/tM0vaoF0rp5G7XYGELlOgkRtuzKWcL3f/PcIHZYCUcJfDwiybCpy+c 3GgA== X-Gm-Message-State: AOAM5334ETYHEKH7ViYdiNwgCL5JpuXPJ4mJwT/b+etXWJAiRcLc0VgW ujIPHf6n5e2LYcyHKIKtQaYeXocAES+fi16MMcgO8g== X-Google-Smtp-Source: ABdhPJwFSZvOxKQP2zIegmKDfRHd011uM1c4IAdjGpyccSfJefn8Tl4T6YXW2C4dlghB9r9Pj89XpXkZ3avuhKXZwBQ= X-Received: by 2002:a05:6638:dd4:b0:323:d48f:83b5 with SMTP id m20-20020a0566380dd400b00323d48f83b5mr4599284jaj.174.1649258965332; Wed, 06 Apr 2022 08:29:25 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> In-Reply-To: From: Steven Hartland Date: Wed, 6 Apr 2022 16:28:59 +0100 Message-ID: Subject: Re: Desperate with 870 QVO and ZFS To: John F Carr Cc: "egoitz@ramattack.net" , "freebsd-fs@freebsd.org" Content-Type: multipart/alternative; boundary="000000000000c5950205dbfe06ab" X-Rspamd-Queue-Id: 4KYT2g4FWpz3sfT X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20210112.gappssmtp.com header.s=20210112 header.b=TCi+BNcR; dmarc=pass (policy=none) header.from=multiplay.co.uk; spf=pass (mx1.freebsd.org: domain of steven@multiplay.co.uk designates 2607:f8b0:4864:20::d2a as permitted sender) smtp.mailfrom=steven@multiplay.co.uk X-Spamd-Result: default: False [-3.70 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20210112.gappssmtp.com:s=20210112]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20210112.gappssmtp.com:+]; DMARC_POLICY_ALLOW(-0.50)[multiplay.co.uk,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::d2a:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-fs]; FORGED_SENDER(0.30)[killing@multiplay.co.uk,steven@multiplay.co.uk]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[killing@multiplay.co.uk,steven@multiplay.co.uk] X-ThisMailContainsUnwantedMimeParts: N --000000000000c5950205dbfe06ab Content-Type: text/plain; charset="UTF-8" What does gstat -pd report? On Wed, 6 Apr 2022 at 15:59, John F Carr wrote: > On Apr 6, 2022, at 07:15 , egoitz@ramattack.net wrote: > > > > Good morning, > > > > I write this post with the expectation that perhaps someone could help > me > > > > I am running some mail servers with FreeBSD and ZFS. They use 870 QVO > (not EVO or other Samsung SSD disks) disks as storage. They can easily have > from 1500 to 2000 concurrent connections. The machines have 128GB of ram > and the CPU is almost absolutely idle. The disk IO is normally at 30 or 40% > percent at most. > > > > The problem I'm facing is that they could be running just fine and > suddenly at some peak hour, the IO goes to 60 or 70% and the machine > becomes extremely slow. ZFS is all by default, except the sync parameter > which is set disabled. Apart from that the ARC is limited to 64GB. But even > this is extremely odd. The used ARC is near 20GB. I have seen, that meta > cache in arc is very near to the limit that FreeBSD automatically sets > depending on the size of the ARC you set. It seems that almost all ARC is > used by meta cache. I have seen this effect in all my mail servers with > this hardware and software config. > > > > My system with > > nvd0: NVMe namespace > > has a problem with high write volume. If I build llvm with debugging > symbols, which writes about 70 GB, the filesystem nearly grinds to a halt. > I have to use a spinning disk to get decent performance on this workload. > There is some old talk on the mailing lists about certain drives handling > TRIM commands badly. See comment by Ted Ts'o here: > https://forums.freebsd.org/threads/ssd-trim-maintenance.56951/ > > Unfortunately the documentation for adjusting trim settings is out of date. > > > --000000000000c5950205dbfe06ab Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
What does gstat -pd report= ?

On Wed, 6 Apr 2022 at 15:59, John F Carr <jfc@mit.edu> wrote:
On Apr 6, 2022, at 07:15 , egoitz@ramattack.net wrote:
>
> Good morning,
>
> I write this post with the expectation that perhaps someone could help= me <d8974688.gif>
>
> I am running some mail servers with FreeBSD and ZFS. They use 870 QVO = (not EVO or other Samsung SSD disks) disks as storage. They can easily have= from 1500 to 2000 concurrent connections. The machines have 128GB of ram a= nd the CPU is almost absolutely idle. The disk IO is normally at 30 or 40% = percent at most.
>
> The problem I'm facing is that they could be running just fine and= suddenly at some peak hour, the IO goes to 60 or 70% and the machine becom= es extremely slow. ZFS is all by default, except the sync parameter which i= s set disabled. Apart from that the ARC is limited to 64GB. But even this i= s extremely odd. The used ARC is near 20GB. I have seen, that meta cache in= arc is very near to the limit that FreeBSD automatically sets depending on= the size of the ARC you set. It seems that almost all ARC is used by meta = cache. I have seen this effect in all my mail servers with this hardware an= d software config.
>

My system with

nvd0: <Samsung SSD 970 EVO 1TB> NVMe namespace

has a problem with high write volume.=C2=A0 If I build llvm with debugging = symbols, which writes about 70 GB, the filesystem nearly grinds to a halt.= =C2=A0 I have to use a spinning disk to get decent performance on this work= load.=C2=A0 There is some old talk on the mailing lists about certain drive= s handling TRIM commands badly.=C2=A0 See comment by Ted Ts'o here: https://forums.freebsd.org/threads/ssd-tr= im-maintenance.56951/

Unfortunately the documentation for adjusting trim settings is out of date.=


--000000000000c5950205dbfe06ab--