From nobody Tue Aug 23 21:27:55 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MC2Q85h98z4ZL75 for ; Tue, 23 Aug 2022 21:28:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MC2Q80Df0z3fFc for ; Tue, 23 Aug 2022 21:28:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt1-x829.google.com with SMTP id e28so11413571qts.1 for ; Tue, 23 Aug 2022 14:28:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :from:to:cc; bh=BrRb89lx5McEEEOBPd7g0E0BJm62Bu8rxZxhkZP5pLs=; b=hwnv5GNRgZOHpwZMMfTEjoaJsJTRTvhJba4okoJIUCn/MDpT9evJ7UcOsKgK6OP40V yB8N8ppY9oGUpvP5/y0WMcKV6MMMcUwDU/MR3rlVUdO/zYyWRIAj45dB6B55I6JvlgyP qSpJLZjOQ2BoewdTkdCb4PAsJREozn6Z66fYKD2aXyUjn/n35txz8BUt8KFI6ztu6/mD Gejeg9iDojfyQoE4utGYj6yeqiinBjsa98lO2rhUi2TO4lwN26FmuU9QZAIm4SW2K7jS p8nwU8+pmZbiA9jWJki8jEI5VPsnxLAsw5mWS8iDH2e95/8D806jcTGVctWY3J7HNon8 vrDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc; bh=BrRb89lx5McEEEOBPd7g0E0BJm62Bu8rxZxhkZP5pLs=; b=xAsuRzUe9eZfVPpS7Lmxol8ThSZcYg1YJAQ3nVwRA7UBsI0dGykNJfwCmHkXcfy3IF Do21JHOVmoSgONtHsfSzFbMEQCf5UdWrhIXMyi26GP8S/IHMfodANw6TQa9luBDycDvM JGaOEpAwKZpANmp3hi0z2z6nKDCARmrMz9zLVqpPmsEeJVMbPHtp6LwQ3tOnrCDrt1GL 7HEqhxuNq4/+zhpP+MV3DLMhMorkc3lveK4v9hV5XJ5Am7DOqvuxn0/bsx9PjJ9BO43w ixDwj6B/bZiDG4lifiMIEWxISKYGIiYe0IWRmJ8LhMRMkpQugRGyirbCYAkj9EPF9EwW b60w== X-Gm-Message-State: ACgBeo0M7ywvho9NyOAdWztmbcTgO1P8PCQiMcBvWp/efSpj+A95WSVU oUV8dZzRysMV38HD5Sfkj/vHS5+iBB8= X-Google-Smtp-Source: AA6agR75abFDDwg8c32RWZjKcvnZutvPHJcSxTMki4DN/04Vnbh9w5VOCDlbJSP/v6oYdl1MDY6e4g== X-Received: by 2002:a05:622a:4c:b0:343:5d26:562f with SMTP id y12-20020a05622a004c00b003435d26562fmr21303768qtw.613.1661290079301; Tue, 23 Aug 2022 14:27:59 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id y10-20020a05620a44ca00b006bb41ac3b6bsm16205969qkp.113.2022.08.23.14.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Aug 2022 14:27:57 -0700 (PDT) Date: Tue, 23 Aug 2022 17:27:55 -0400 From: Mark Johnston To: "Cavallo, Joseph" Cc: "freebsd-hackers@freebsd.org" Subject: Re: Quick question about geom/g_mirror.c g_mirror_sync_stop() function Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 4MC2Q80Df0z3fFc X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=hwnv5GNR; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::829 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-2.69 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_LONG(-1.00)[-0.997]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::829:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; TO_DN_EQ_ADDR_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Mon, Aug 22, 2022 at 10:30:15PM +0000, Cavallo, Joseph wrote: > Hi Mark, > > Thank you for your response while traveling. > > In g_mirror_sync_stop() it quickly calls “free(disk->d_sync.ds_bios, M_MIRROR);”. But at that point can’t these bios still be in progress in the g_mirror_disk.d_sync.ds_consumer or in the g_mirror_disk.ds_consumer? They can, yes. Note though that disk->d_sync.ds_bios is an array of pointers to BIOs, not an array of BIOs, so freeing ds_bios doesn't impact pending synchronization I/O. g_mirror_sync_request() will notice in the BIO_WRITE path that the sync GEOM has gone away and will free the BIO at that point, or earlier if an I/O error occurred. Alternately, if the mirror itself is being destroyed, g_mirror_destroy_provider() will free any BIOs queued for the worker thread (see the special handling of synchronization BIOs there). > In the g_mirror_ctl_remove() path it calls “g_mirror_event_send(disk, G_MIRROR_DISK_STATE_DESTROY, G_MIRROR_EVENT_DONTWAIT);” That leads to “g_mirror_destroy_disk()” which quickly calls “g_mirror_sync_stop()”. I can’t find where it quiesces those bios so that it can free them right then and there. Later it does the g_mirror_kill_consumer() but that would just jump out when it got to the g_mirror_is_busy() test unless it was all quiesced (I think). > > I’ve run FreeBSD 13.1 with 2 64GB SCSI disks mirror (data volume). It looks like it defaults to 2 sync bios. I did the remove of the disk getting synched in the middle of a sync, and of course it works fine 100% of the time. It does not crash. But I’m trying to understand how that is. How does it avoid the pitfall that I think I see? > > Many thanks in advance. Take your time. Enjoy your travels. > > --Joe > > From: Mark Johnston > Sent: Saturday, August 20, 2022 2:20 PM > To: Cavallo, Joseph > Subject: [EXTERNAL] Re: Quick question about mirror.c > > > [EXTERNAL SENDER: This email originated from outside of Stratus Technologies. Do not click links or open attachments unless you recognize the sender and know the content is safe.] > > ________________________________ > Hi Joe, > > I am happy to try to answer questions about that code, but I'm traveling and might take a few days to reply. Please include freebsd-hackers@freebsd.org in the email, in case someone else is able to answer. > > Thanks, > -Mark > > On Fri., Aug. 19, 2022, 12:08 Cavallo, Joseph, > wrote: > Hi Mark, > > I reside in MA, USA. > > I'm using the FreeBSD Geom/mirror project that you're a contributor to. > > I have a quick question about the g_mirror_sync_stop() routine. Would you be able to help? > > Thx, Joe