From nobody Sat Feb 26 12:29:12 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 899DD19ECB85 for ; Sat, 26 Feb 2022 12:29:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K5Qtc2lWcz4jTy for ; Sat, 26 Feb 2022 12:29:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3BB461396B for ; Sat, 26 Feb 2022 12:29:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 21QCTCjm030683 for ; Sat, 26 Feb 2022 12:29:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 21QCTCIc030682 for bugs@FreeBSD.org; Sat, 26 Feb 2022 12:29:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 262189] ZFS volume not showing up in /dev/zvol when 1 CPU Date: Sat, 26 Feb 2022 12:29:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: zedupsys@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1645878552; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=APq+qopQCH0gJPPUEY7fMCl3/d/e7BGw0ppINmkj+bU=; b=V8pC0/IoXEcc3z2EZR2RK06MvEhGAodGbOCEx8sIXo0ix3gX/nL2DhQ3UUH61my9wTuP7v c20XIjzCqVzmLVuFgTUj0zZKPM7qYrJ5CGFc0wXkFqwM6LyhfHMUbvtiTKPnnzYJm5kG8Z efTHJeg8SP+5J9KYSIPLQbsFP7q6j9axU3FyS1RbkJPCR/n75tHQuQdpMoLky3BM2Vizv5 8Cafzys60eAhFml1A7GozyZ4AKOY1mqwFuEhlEzIpMZXFDGZUqI9Di1+FkLLNeGnIIXWBL lhFo5onwdxsl8d20UI2EMYr8sE+bxwbnt7EJGoZ6Tim4OAo4sDFQQ12lvIr0og== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1645878552; a=rsa-sha256; cv=none; b=UdXJXrpq8rhNIADlZeFksA4cdXIBleIILuHWqWyq/P13efwkrn58hwzIcLhbF2xTEz+ZHx eIMmXGwkX/KW3h6tNTURdBXAUJePEuaXIeCLuP7YjQAieo9K4ZUsFE8zHcnOXH9W0f4tFB uDjZuZ6+1eUZnqFYH6mNS8NNLerCbfrb6/IlotTbI8PBQrMl+IKLuq7IRIP6rAht/Du/Q7 AhNqje1x6f0t+0eDMJ+c85b8B9C+5MqLavymSmLazvWewqOYf3pYKl7RDkUX9GKloSetV7 5j1qTceCTTnS459s89roJoxvxYqd3LjEGBV8lzs5odERX8w4ut+sNmIPzLzNPQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D262189 --- Comment #6 from Janis --- Thank's for the useful information, i did not know the internals in this regard. Now i understand what the bug #250297 report is speaking about. That bug seemed might be relevant to my case. I tried to reproduce it with zfs create/destroy shell script loop and could not hit kernel panic as stated t= here in comments. I did not get that he speaks about how volmode=3Ddev is created internally, so his use-case seemed a bit bizarre. About asynchronous nature for "zfs create" command. At first i thought this= is the case, but it does not seem to be. There are two problems as i see: 1) if this was just that "zfs create" returns too early, it would be a mini-bug, since it is expected that command returns when things are done. I guess, i wouldn't even report it. 2) with sleep between "zfs create" and "dd" on multi-core systems it "solve= s" some of dd problems, but not all of the missing ZVOL cases; there are still ZVOLs that never show up in /dev/zvol but can be seen in "zfs list". On sin= gle CPU case, it solves nothing at all and ZVOL never appears, only after reboot/export-import. To illustrate 1 CPU case, i run script: #!/bin/sh name_pool=3Dzroot/stress echo `date` ls /dev/zvol seq 1 10 | while read i; do zfs create -o volmode=3Ddev -V 1G $name_pool/data$i done sleep 300 echo `date` ls /dev/zvol Output is: Sat Feb 26 12:21:08 EET 2022 ls: /dev/zvol: No such file or directory Sat Feb 26 12:26:11 EET 2022 ls: /dev/zvol: No such file or directory even now after a while # date Sat Feb 26 12:35:03 EET 2022 # ls /dev/zvol ls: /dev/zvol: No such file or directory I do not know how long is considered asynchronous, but it seems too long, s= o i assume that ZVOL will never show up. With 1 CPU machine in file "cat /var/run/devd.pipe" i get lines like so for each create command: !system=3DDEVFS subsystem=3DCDEV type=3DCREATE cdev=3Dzvol/zroot/stress/dat= a20 !system=3DGEOM subsystem=3DDEV type=3DCREATE cdev=3Dzvol/zroot/stress/data20 !system=3DDEVFS subsystem=3DCDEV type=3DDESTROY cdev=3Dzvol/zroot/stress/da= ta20 !system=3DGEOM subsystem=3DDEV type=3DDESTROY cdev=3Dzvol/zroot/stress/data= 20 This seems wrong, since both "last" events are DESTROY. With 4 CPUs it is harder to reproduce, so i ran with 2 CPUs enabled in BIOS. Physical hardware, 16G RAM. So i ran script as follows: #!/bin/sh name_pool=3Dzroot/stress zfs create -o mountpoint=3Dnone $name_pool seq 1 1000 | while read i; do zfs create -o volmode=3Ddev -V 1G $name_pool/data$i done Testing result: # zfs list | grep stress | wc -l 1001 # ls /dev/zvol/zroot/stress/ | wc -l 638 Output clearly shows that ZVOLs are missing (even if ignoring output header= and ZVOL parent, diff is too big) I created files and will attach them (though maybe it is enough with my pointers): zfs list -H -o name > /service/log/zfs_list_001.log ls ls /dev/zvol/zroot/stress/ > ls_dev_zvol_stress__001.log cat /var/run/devd.pipe| grep -v "!system=3DZFS" > /service/log/grepped_devd.pipe_no_dd_seq_1000__001.log With diff and sorting we see that, i.e. there is no ZVOL for: -data526 -data527 -data528 -data529 # ls /dev/zvol/zroot/stress/data526 ls: /dev/zvol/zroot/stress/data526: No such file or directory # zfs get -H volmode zroot/stress/data526 zroot/stress/data526 volmode dev local For non-existing case we see this in devd.pipe file: # cat /service/log/grepped_devd.pipe_no_dd_seq_1000__001.log | grep data526 !system=3DDEVFS subsystem=3DCDEV type=3DCREATE cdev=3Dzvol/zroot/stress/dat= a526 !system=3DGEOM subsystem=3DDEV type=3DCREATE cdev=3Dzvol/zroot/stress/data5= 26 !system=3DDEVFS subsystem=3DCDEV type=3DDESTROY cdev=3Dzvol/zroot/stress/da= ta526 !system=3DGEOM subsystem=3DDEV type=3DDESTROY cdev=3Dzvol/zroot/stress/data= 526 It's the same fingerprint as in 1 CPU case, double destroy events are last ones. Whereas for existing ZVOLs there is: # cat /service/log/grepped_devd.pipe_no_dd_seq_1000__001.log | grep data525 !system=3DDEVFS subsystem=3DCDEV type=3DCREATE cdev=3Dzvol/zroot/stress/dat= a525 !system=3DGEOM subsystem=3DDEV type=3DCREATE cdev=3Dzvol/zroot/stress/data5= 25 !system=3DDEVFS subsystem=3DCDEV type=3DDESTROY cdev=3Dzvol/zroot/stress/da= ta525 !system=3DGEOM subsystem=3DDEV type=3DDESTROY cdev=3Dzvol/zroot/stress/data= 525 !system=3DDEVFS subsystem=3DCDEV type=3DCREATE cdev=3Dzvol/zroot/stress/dat= a525 --=20 You are receiving this mail because: You are the assignee for the bug.=