From nobody Fri Apr 15 09:06:12 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6E1282DE559 for ; Fri, 15 Apr 2022 09:06:20 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kfr6M3gPrz4Qvm for ; Fri, 15 Apr 2022 09:06:19 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lj1-x22d.google.com with SMTP id f5so1331175ljp.8 for ; Fri, 15 Apr 2022 02:06:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=reply-to:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=hx6EpiLqpJbRPslduuxGcR5A4SgCuJ/HlI2LtQpHq4Q=; b=Qa9u3QPinEHSPRgMeXA5/cxeWD3hpv2jm0k0NKAfmyAB+AT+js2+v5Kc7x8QUEFBuA VahCf7TP1pVps66O8ptAHsSz4616bgwyzXe7B0KVh7YylEo+o6o1X09xgVHKe7EiRFau A7VJIZT3Mjna5P/szSMlwo7rMgzSRbAITEiqNlVJhd11cyVQgNX+ctjaSDL3x8H/kIy6 qpEid5CqkD/Ipfp7FywXBOmJmXwW1ln5/xGM7M/RKkBjUAT0ufeAagVCkBuyFrwp2l55 SMRaEutZRZqnfO8AN1BrLfsHMo09bEBH2b3Cvh/52EJ/2AyjDGLIY4cQgTGd1IyIQxk5 K6Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=hx6EpiLqpJbRPslduuxGcR5A4SgCuJ/HlI2LtQpHq4Q=; b=2+fOshbjC2Y4DVaPjhU0Tc92ic1/yflF4bRvQxgQjQ2kmHBC4JG+fY5jPGvCG3jJc9 XgZv9R8roSN4Pgts+Ne7hZSoZDpzcnhBmEvBOKncKieaizhCcAOxytxBwofxWrUydJoZ 9mvDhFtdCo+VBQhwFCcECcqdspt1oWiRetGryYQvyU7lamUjiFR579MkReHNvgL5IdEL 5NO1ROeb4bLguuaRLqsua1amGthZxDuCU1z6xaKde0rOS7QUpQisyMoyUZipqaKJOEHY uJVxnbrV2hNRlCQ6e4lD9+bVXmssTX+1RNg/aNcc7rpEnvs8DHlF9XqFSNFhGzEfQznt FpMQ== X-Gm-Message-State: AOAM531lk6bvc/Dasbb6jPk4a6ENwCYmFxH/pyjMjcOBWGtJrQGmAy9j Uztv+YfnKlw2oICgCG1cF5M= X-Google-Smtp-Source: ABdhPJzK+cP9JBWjB3SaQNc7QH1zGx1okPLxem2iQiXhvr2Sx8GFaUe92cdOVmlCt+g9Xizi2DCgQA== X-Received: by 2002:a2e:8748:0:b0:24d:a7a0:1ae3 with SMTP id q8-20020a2e8748000000b0024da7a01ae3mr747776ljj.8.1650013578045; Fri, 15 Apr 2022 02:06:18 -0700 (PDT) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id f16-20020ac24990000000b0046bb8e2ca13sm267957lfl.14.2022.04.15.02.06.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 15 Apr 2022 02:06:17 -0700 (PDT) Reply-To: zedupsys@gmail.com Subject: Re: ZFS + FreeBSD XEN dom0 panic To: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org References: <48b74c39-abb3-0a3e-91a8-b5ab1e1223ce@gmail.com> <22643831-70d3-5a3e-f973-fb80957e80dc@gmail.com> <209c9b7c-4b4b-7fe3-6e73-d2a0dc651c19@gmail.com> <1286cb59-867e-e7d0-2bd3-45c33feae66a@gmail.com> From: Ze Dupsys Message-ID: Date: Fri, 15 Apr 2022 12:06:12 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: lv Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Kfr6M3gPrz4Qvm X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=Qa9u3QPi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::22d as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-1.76 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.77)[-0.766]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-0.99)[-0.991]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[0.999]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22d:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 2022.04.14. 10:39, Roger Pau Monné wrote: > .. > Thanks. I will groom those patches in order to prepare them for > commit. Regardless of whether there are other issues still lurking I > think those changes are worth committing now. Hi, So the tests still are running with 3 patches, seems a lot better than without, soon i think i will stop them, since they have proved a point. uptime 11:22AM up 1 day, 19:18, 8 users, load averages: 2.17, 2.04, 2.12 There has been some problem though, because at this stage xl list has (null) VM. xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1023 4 r----- 211752.0 (null) 346 0 1 --ps-d 61.7 xen-vm2nonic-zvol-5 557 1024 1 r----- 47.2 xen-vm1nonic-zvol 558 1024 1 -b---- 42.0 I have no idea why it went in that state. I've been collecting vmstat -m since start and after filtering out, i think i got maybe useful hints. Pictures in url: https://file.fm/u/k67uhj436#/ In general we can see, that xbbd with 3 patches seems not to leak memory. I did not see any component whose memory usage was just growing. Images show values of InUse of vmstat -m, y is bytes, x unix timestamp, but i don't know if maybe i should have looked into MemUse instead. We see that solaris takes quiet a chunk, which is expected for ZFS i guess. But the interesting thing is a spike in newblk, looking deeper it can be seen that this spike happens at the same time when jsegdep spike is, and then there is at smaller scale, but still spike at the same time for jseg. At the same time when there is spike for newblk there is a little bit dip for solaris. If there are some specific ones i should pay more attention, i could look at them. I would like to speculate that "..pmap_growkernel.." panic happens at those times when there is spike high enough and that for this current case it was just a lucky coincidence that system had more mem and did not panic. Or maybe this is where this (null) VM appeared. Unfortunateley i did not log output of xl list with timestamps. Currently xenstore-ls -fp, does not contain any row with 346, so i suppose that disks have been freed, i don't see any suspicious sysctl variables either, so i do not know what state this (null) VM is in and why it is not cleaning up. Is there some useful command in this case to collect more info about (null) VM? Thanks.