From nobody Tue Mar 21 15:37:02 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgwhR2zNJz40RlS for ; Tue, 21 Mar 2023 15:37:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgwhQ3R1wz3hW3; Tue, 21 Mar 2023 15:37:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 32LFb21D036784 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 21 Mar 2023 17:37:05 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 32LFb21D036784 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 32LFb2g5036783; Tue, 21 Mar 2023 17:37:02 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 21 Mar 2023 17:37:02 +0200 From: Konstantin Belousov To: Ken Merry Cc: hackers@freebsd.org Subject: Re: Getting v_wire_count from a kernel core dump? Message-ID: References: <66742036-C8DF-4A13-9D4A-CDA71217E574@freebsd.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <66742036-C8DF-4A13-9D4A-CDA71217E574@freebsd.org> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4PgwhQ3R1wz3hW3 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Tue, Mar 21, 2023 at 11:11:30AM -0400, Ken Merry wrote: > I have kernel core dumps from several machines out in the field (customer sites) that were out of memory panics, and I’m trying to figure out, from the kernel core dumps, whether we’re dealing with a potential page leak. > > For context, these machines are running stable/13 from April 2021, but they do have the fix for this bug: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256507 > > Which is this commit in stable/13: > > https://cgit.freebsd.org/src/commit/?id=6094749a1a5dafb8daf98deab23fc968070bc695 > > On a running system, I can get a rough idea whether there is a page leak by looking at the VM system page counters: > > # sysctl vm.stats |grep count > vm.stats.vm.v_cache_count: 0 > vm.stats.vm.v_user_wire_count: 0 > vm.stats.vm.v_laundry_count: 991626 > vm.stats.vm.v_inactive_count: 39733216 > vm.stats.vm.v_active_count: 11821309 > vm.stats.vm.v_wire_count: 11154113 > vm.stats.vm.v_free_count: 1599981 > vm.stats.vm.v_page_count: 65347213 > > So the first 5 numbers add up to 65300245 in this case, for a difference of 46968. > > Am I off base here as far as the various counts adding up to the page count? (e.g. is the wire count just an additional attribute of a page and not another separate state like active, inactive or laundry?) > > Looking at the kernel core dump for one of the systems I see: > > kgdb) print vm_cnt > $1 = {v_swtch = 0xfffffe022158f2f8, v_trap = 0xfffffe022158f2f0, > v_syscall = 0xfffffe022158f2e8, v_intr = 0xfffffe022158f2e0, > v_soft = 0xfffffe022158f2d8, v_vm_faults = 0xfffffe022158f2d0, > v_io_faults = 0xfffffe022158f2c8, v_cow_faults = 0xfffffe022158f2c0, > v_cow_optim = 0xfffffe022158f2b8, v_zfod = 0xfffffe022158f2b0, > v_ozfod = 0xfffffe022158f2a8, v_swapin = 0xfffffe022158f2a0, > v_swapout = 0xfffffe022158f298, v_swappgsin = 0xfffffe022158f290, > v_swappgsout = 0xfffffe022158f288, v_vnodein = 0xfffffe022158f280, > v_vnodeout = 0xfffffe022158f278, v_vnodepgsin = 0xfffffe022158f270, > v_vnodepgsout = 0xfffffe022158f268, v_intrans = 0xfffffe022158f260, > v_reactivated = 0xfffffe022158f258, v_pdwakeups = 0xfffffe022158f250, > v_pdpages = 0xfffffe022158f248, v_pdshortfalls = 0xfffffe022158f240, > v_dfree = 0xfffffe022158f238, v_pfree = 0xfffffe022158f230, > v_tfree = 0xfffffe022158f228, v_forks = 0xfffffe022158f220, > v_vforks = 0xfffffe022158f218, v_rforks = 0xfffffe022158f210, > v_kthreads = 0xfffffe022158f208, v_forkpages = 0xfffffe022158f200, > v_vforkpages = 0xfffffe022158f1f8, v_rforkpages = 0xfffffe022158f1f0, > v_kthreadpages = 0xfffffe022158f1e8, v_wire_count = 0xfffffe022158f1e0, > v_page_size = 4096, v_page_count = 65342843, v_free_reserved = 85343, > v_free_target = 1392195, v_free_min = 412056, v_inactive_target = 2088292, > v_pageout_free_min = 136, v_interrupt_free_min = 8, v_free_severe = 248698} > (kgdb) print vm_ndomains > $2 = 4 > (kgdb) print vm_dom[0].vmd_pagequeues[0].pq_cnt > $3 = 6298704 > (kgdb) print vm_dom[0].vmd_pagequeues[1].pq_cnt > $4 = 3423939 > (kgdb) print vm_dom[0].vmd_pagequeues[2].pq_cnt > $5 = 629834 > (kgdb) print vm_dom[0].vmd_pagequeues[3].pq_cnt > $6 = 0 > (kgdb) print vm_dom[1].vmd_pagequeues[0].pq_cnt > $7 = 2301793 > (kgdb) print vm_dom[1].vmd_pagequeues[1].pq_cnt > $8 = 7130193 > (kgdb) print vm_dom[1].vmd_pagequeues[2].pq_cnt > $9 = 701495 > (kgdb) print vm_dom[1].vmd_pagequeues[3].pq_cnt > $10 = 0 > (kgdb) print vm_dom[2].vmd_pagequeues[0].pq_cnt > $11 = 464429 > (kgdb) print vm_dom[2].vmd_pagequeues[1].pq_cnt > $12 = 9123532 > (kgdb) print vm_dom[2].vmd_pagequeues[2].pq_cnt > $13 = 1037423 > (kgdb) print vm_dom[2].vmd_pagequeues[3].pq_cnt > $14 = 0 > (kgdb) print vm_dom[3].vmd_pagequeues[0].pq_cnt > $15 = 5444946 > (kgdb) print vm_dom[3].vmd_pagequeues[1].pq_cnt > $16 = 4466782 > (kgdb) print vm_dom[3].vmd_pagequeues[2].pq_cnt > $17 = 785195 > (kgdb) print vm_dom[3].vmd_pagequeues[3].pq_cnt > $18 = 0 > (kgdb) > > > Adding up the page queue counts: > > 6298704 > 3423939 > 629834 > ++p > 10352477 > 2301793 > 7130193 > 701495 > ++p > 10133481 > +p > 20485958 > 464429 > 9123532 > 1037423 > ++p > 10625384 > +p > 31111342 > 5444946 > 4466782 > 785195 > ++p > 10696923 > +p > 41808265 > > So, about 23M pages short of v_page_count. > > v_wire_count is a per-CPU counter, and on a running system it gets added up. But trying to access it in the kernel core dump yields: > > (kgdb) print vm_cnt.v_wire_count > $2 = (counter_u64_t) 0xfffffe022158f1e0 > (kgdb) print *$2 > Cannot access memory at address 0xfffffe022158f1e0 > > Anyone have any ideas whether I can figure out whether there is a page leak from the core dump? > Did you looked at UMA/malloc stats? It could be genuine VM leaking pages, but more often it is some kernel subsystem leaking its own allocations. For later, try both vmstat -z and vmstat -m on the kernel.debug + vmcore. Often the leakage is immediately obvious.