From nobody Fri Apr 19 22:23:51 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VLq1d23RLz5HsB6 for ; Fri, 19 Apr 2024 22:24:05 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vk1-f174.google.com (mail-vk1-f174.google.com [209.85.221.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VLq1c36d9z4Hfq for ; Fri, 19 Apr 2024 22:24:04 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.221.174 as permitted sender) smtp.mailfrom=asomers@gmail.com Received: by mail-vk1-f174.google.com with SMTP id 71dfb90a1353d-4daa91c0344so770979e0c.3 for ; Fri, 19 Apr 2024 15:24:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713565443; x=1714170243; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=x71Exf+DYrTrHrLNSKVuEDO8b23MEqFpBlcYsekN/JU=; b=HozcKKEHZJAmmL3DvFhoghPytlOSf5UDDVLNOrcPq85aLKjny1aeFJaKTMVI+wby+Z OxT0lOXAdKcpyw+B7YAd05c3V7JA4ZS9F3qvwEEAfJoFI/GcLzTToAfWvyBf3W3r1MjP 7OBgbHvORyCqa0S9cU3XJHBX2TGN+anAqQypHUlnVJh+/70vAyF2bhWLA/DJ2poIQsnK SowBdptrUBVD29LQW/xjRtdF4WIGIVnVWd9oc/z0IuS82AuIgDfD7LO/RTKJtJC0PDeE EkGt0N6CZj/Bl08vLWIvwWEIhGkXNq7FaRNXh6tstG3DlWWErQLcGjnxoDc1t68bLV0M psiQ== X-Gm-Message-State: AOJu0YzpHzcOXmpAHnIIaN1jclJAlW+2tnoZj1CHq1LxyKSDj8nG9NZA 53mfaphD/Qpw0V8lrn/F0MwSgefjkFXMeL8AYfjLFG7vCdZX2fM3RnXDnCgHnkVERRUo9VDr71K +MfULaXjyQWYa88Lxiq/jCVm+gBmvm2QP X-Google-Smtp-Source: AGHT+IET23J+45MdLa1MLoxkwB+liWUHCMH3/fQw+Wwrn+DRTl6+LQ3G5luxvXqAhuyttND1B2uSKifXb3bdLHK5esc= X-Received: by 2002:a05:6122:3d05:b0:4d3:43f8:8541 with SMTP id ga5-20020a0561223d0500b004d343f88541mr3784652vkb.1.1713565442665; Fri, 19 Apr 2024 15:24:02 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 From: Alan Somers Date: Fri, 19 Apr 2024 16:23:51 -0600 Message-ID: Subject: Stressing malloc(9) To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: - X-Spamd-Result: default: False [-1.92 / 15.00]; NEURAL_HAM_LONG(-0.97)[-0.970]; NEURAL_HAM_MEDIUM(-0.53)[-0.531]; NEURAL_HAM_SHORT(-0.52)[-0.516]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[asomers]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_ONE(0.00)[1]; R_DKIM_NA(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.221.174:from]; TO_DOM_EQ_FROM_DOM(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_IN_DNSWL_NONE(0.00)[209.85.221.174:from] X-Rspamd-Queue-Id: 4VLq1c36d9z4Hfq TLDR; How can I create a workload that causes malloc(9)'s performance to plummet? Background: I recently witnessed a performance problem on a production server. Overall throughput dropped by over 30x. dtrace showed that 60% of the CPU time was dominated by lock_delay as called by three functions: printf (via ctl_worker_thread), g_eli_alloc_data, and g_eli_write_done. One thing those three have in common is that they all use malloc(9). Fixing the problem was as simple as telling CTL to stop printing so many warnings, by tuning kern.cam.ctl.time_io_secs=100000. But even with CTL quieted, dtrace still reports ~6% of the CPU cycles in lock_delay via g_eli_alloc_data. So I believe that malloc is limiting geli's performance. I would like to try replacing it with uma(9). But on a non-production server, none of my benchmark workloads causes g_eli_alloc_data to break a sweat. I can't get its CPU consumption to rise higher than 0.5%. And that's using the smallest sector size and block size that I can. So my question is: does anybody have a program that can really stress malloc(9)? I'd like to run it in parallel with my geli benchmarks to see how much it interferes. -Alan