From nobody Thu Sep 08 23:30:31 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MNwNM5Gjkz4c2n8 for ; Thu, 8 Sep 2022 23:30:43 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MNwNM03wVz40WZ for ; Thu, 8 Sep 2022 23:30:43 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-qt1-f175.google.com with SMTP id x5so144743qtv.9 for ; Thu, 08 Sep 2022 16:30:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date; bh=WxyGsnRQxKB4JRIEyoxu+Ei/vnzKondUwOKXVwAyZv0=; b=DFer4IVz5ze86RSnNiTc9jTsQCxtFdgeYfgC28uCsd1OnjuH0hLdBvhOIkSmApLXYB BanmgDtNHVcdSwTizLPo052XqRRYTTsqmMQ0EO8DOtseIdW4ywSS/rGB7VQYJpJg8XnL N6csiYPhjTCG6mY1Gc1UlAO5EsVHHpZYbSaIhnqeGtE4kExJS8CTao0dpiLo71KNtFFH d9qvrGxa05qcsJmzBlxIMkhT1GJ29ScOQiynQS5bkR66kLwRYVJnhHw1oxzNNQSzj0jD P6KTmv4cvYx7u7vQtAjXnQrMswtIIcyJX+/Eri2pB2a9HGACe7A69jKb2StSjTTaSIsY kJSw== X-Gm-Message-State: ACgBeo3QLwO0N/DhK50dyQfEfZKYpnHZnWcdnPi1uVPPbT6BPRxU+AQ2 wCUlDJVzv2PY3il13nyoaTrJaovl56bBkzGZlJc= X-Received: by 2002:ac8:59c3:0:b0:343:6528:db29 with SMTP id f3-20020ac859c3000000b003436528db29mt10317716qtf.575.1662679842072; Thu, 08 Sep 2022 16:30:42 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Thu, 8 Sep 2022 17:30:31 -0600 Message-ID: Subject: Re: sysctl is too slow Cc: Mateusz Guzik , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4MNwNM03wVz40WZ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.160.175 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-0.92 / 15.00]; MISSING_TO(2.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-0.93)[-0.926]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.175:from]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.160.175:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FREEFALL_USER(0.00)[asomers]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] X-ThisMailContainsUnwantedMimeParts: N I finally got around to this. Here's the review: https://reviews.freebsd.org/D36500 On Mon, Aug 23, 2021 at 6:54 AM Alan Somers wrote: > > Ideally, but it's not very high priority, since it's merely a performance issue in a monitoring tool. > > On Mon, Aug 23, 2021 at 6:05 AM Mateusz Guzik wrote: >> >> So is this something you plan on fixing? >> >> On 8/17/21, Alan Somers wrote: >> > Actually, I did get a flamegraph, and only 0.77% of samples were in ZFS. >> > >> > On Mon, Aug 16, 2021 at 7:19 PM Mateusz Guzik wrote: >> > >> >> On 8/16/21, Alan Somers wrote: >> >> > Yes, I see what you're talking about now. There are a bunch of linked >> >> > lists in sysctl_find_oid etc. Good point. >> >> > -Alan >> >> > >> >> >> >> You still want to get a flamegraph, chances are most of the problem is in >> >> zfs. >> >> >> >> > On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik >> >> > wrote: >> >> > >> >> >> Last time I checked lookup of a sysctl was very bad with linear scans >> >> all >> >> >> over. >> >> >> >> >> >> Short of complete revamp of the entire thing I would start with >> >> >> replacing the scans with a RB tree at each level. As is if you indeed >> >> >> have 5000 datasets, you are doing increasingly longer walks. >> >> >> >> >> >> On 8/16/21, Alan Somers wrote: >> >> >> > ztop feels very sluggish on a server with 5000 ZFS datasets. Dtrace >> >> >> shows >> >> >> > that almost all of its time is spent in sys_sysctl. ktrace shows >> >> >> > that >> >> >> both >> >> >> > ztop and sysctl(8) call sys_sysctl a total of five times for each >> >> >> > sysctl >> >> >> > they care about: >> >> >> > >> >> >> > 1) To get the next oid >> >> >> > 2) To get the sysctl's name >> >> >> > 3) To get the oidfmt >> >> >> > 4) To get the size of the value >> >> >> > 5) To get the value itself. >> >> >> > >> >> >> > Each of these steps takes about equal time, and together all five >> >> >> > take >> >> >> > about 100us. If the time per call is mostly syscall overhead, then >> >> the >> >> >> > process could be sped up by 80% by combining all of these things >> >> >> > into >> >> a >> >> >> > single syscall: return the next oid, its name, its format, the size >> >> >> > of >> >> >> its >> >> >> > value, and optimistically the value itself, assuming the user passed >> >> >> > a >> >> >> > sufficiently large buffer. >> >> >> > >> >> >> > Am I missing something? Is there any other reason why sysctl is so >> >> >> > slow? >> >> >> > Or should I forget about it, and try to export ZFS's dataset stats >> >> >> through >> >> >> > devstat instead? >> >> >> > -Alan >> >> >> > >> >> >> >> >> >> >> >> >> -- >> >> >> Mateusz Guzik >> >> >> >> >> > >> >> >> >> >> >> -- >> >> Mateusz Guzik >> >> >> > >> >> >> -- >> Mateusz Guzik