[Bug 277389] Reproduceable low memory freeze on 14.0-RELEASE-p5

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277389

            Bug ID: 277389
           Summary: Reproduceable low memory freeze on 14.0-RELEASE-p5
           Product: Base System
           Version: 14.0-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: pascal.guitierrez@gmail.com

I can reliably reproduce a freeze using iozone on 14.0-RELEASE-p5 using a
simple ZFS pool backed onto SATA HDDs as such:

NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP   
HEALTH  ALTROOT
tank        1.81T   512G  1.31T        -         -    36%    27%  1.00x   
ONLINE  -
  mirror-0  1.81T   512G  1.31T        -         -    36%  27.6%      -   
ONLINE
    ada0    1.82T      -      -        -         -      -      -      -   
ONLINE
    ada1    1.82T      -      -        -         -      -      -      -   
ONLINE


To reproduce I create a ZFS filesystem with compression=off, then run iozone -i
0,1 -l 512 -r 4k -s 1g

The write tests execute fine (writing out 512 1g files), however when it runs
the read tests (1=read/re-read) the system freezes due to an apparent low
memory condition after only a few minutes.

I've tried increasing the following sysctls from their defaults:
vm.v_free_target
vm.v_free_min
vm.v_free_severe
vm.v_free_reserved
vfs.zfs.arc.sys_free
vfs.zfs.arc.free_target

and reducing the following:
vfs.zfs.arc_max

But no effect, free memory reliably drops below 100M and does not recover and
soon after the system will freeze and require a hard reset.


This machine has 16GB of RAM with no swap (have also tried with swap on zvol
but same effect), dmesg is below:


---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-RELEASE-p5 #0: Tue Feb 13 23:37:36 UTC 2024
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
amd64
FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git
llvmorg-16.0.6-0-g7cbf1a259152)
VT(efifb): resolution 800x600
CPU: Intel(R) Atom(TM) CPU C3758 @ 2.20GHz (2200.21-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506f1  Family=0x6  Model=0x5f  Stepping=1
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended
Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>
  Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x14000c69<RDCL_NO,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16552595456 (15785 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 2 3 5 1 4 6 7
random: entropy device external interface
kbd0 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x7f0c4000-0x7f0c401e
smbios0: Version: 3.0, BCD Revision: 3.0
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <SUPERM SMCI--MB>
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <processor> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> mem 0xdf440000-0xdf45ffff at device 9.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xdf420000-0xdf43ffff at device 16.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> mem 0xdf400000-0xdf41ffff at device 17.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.0 on pci4
pci5: <ACPI PCI bus> on pcib5
vgapci0: <VGA-compatible display> port 0xd000-0xd07f mem
0xde000000-0xdeffffff,0xdf000000-0xdf01ffff at device 0.0 on pci5
ahci0: <Intel Denverton AHCI SATA controller> port
0xe090-0xe097,0xe080-0xe083,0xe040-0xe05f mem
0xdf476000-0xdf477fff,0xdf47d000-0xdf47d0ff,0xdf47c000-0xdf47c7ff at device
19.0 on pci0
ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich6: <AHCI channel> at channel 6 on ahci0
ahcich7: <AHCI channel> at channel 7 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
ahci1: <Intel Denverton AHCI SATA controller> port
0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem
0xdf474000-0xdf475fff,0xdf47b000-0xdf47b0ff,0xdf47a000-0xdf47a7ff at device
20.0 on pci0
ahci1: AHCI v1.31 with 4 6Gbps ports, Port Multiplier not supported
ahcich11: <AHCI channel> at channel 3 on ahci1
ahcich12: <AHCI channel> at channel 4 on ahci1
ahcich13: <AHCI channel> at channel 5 on ahci1
ahcich14: <AHCI channel> at channel 6 on ahci1
ahciem1: <AHCI enclosure management bridge> on ahci1
xhci0: <Intel Denverton USB 3.0 controller> mem 0xdf460000-0xdf46ffff at device
21.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pcib6: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci6: <ACPI PCI bus> on pcib6
ix0: <Intel(R) X553 (1GbE)> mem 0xddc00000-0xdddfffff,0xdde04000-0xdde07fff at
device 0.0 on pci6
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 8 RX queues 8 TX queues
ix0: Using MSI-X interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: 3c:ec:ef:e9:aa:30
ix0: eTrack 0x80000877
ix0: netmap queues/slots: TX 8/2048, RX 8/2048
ix1: <Intel(R) X553 (1GbE)> mem 0xdda00000-0xddbfffff,0xdde00000-0xdde03fff at
device 0.1 on pci6
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 8 RX queues 8 TX queues
ix1: Using MSI-X interrupts with 9 vectors
ix1: allocated for 8 queues
ix1: allocated for 8 rx queues
ix1: Ethernet address: 3c:ec:ef:e9:aa:31
ix1: eTrack 0x80000877
ix1: netmap queues/slots: TX 8/2048, RX 8/2048
pcib7: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci7: <ACPI PCI bus> on pcib7
ix2: <Intel(R) X553 (1GbE)> mem 0xdd600000-0xdd7fffff,0xdd804000-0xdd807fff at
device 0.0 on pci7
ix2: Using 2048 TX descriptors and 2048 RX descriptors
ix2: Using 8 RX queues 8 TX queues
ix2: Using MSI-X interrupts with 9 vectors
ix2: allocated for 8 queues
ix2: allocated for 8 rx queues
ix2: Ethernet address: 3c:ec:ef:e9:aa:32
ix2: eTrack 0x8000087c
ix2: netmap queues/slots: TX 8/2048, RX 8/2048
ix3: <Intel(R) X553 (1GbE)> mem 0xdd400000-0xdd5fffff,0xdd800000-0xdd803fff at
device 0.1 on pci7
ix3: Using 2048 TX descriptors and 2048 RX descriptors
ix3: Using 8 RX queues 8 TX queues
ix3: Using MSI-X interrupts with 9 vectors
ix3: allocated for 8 queues
ix3: allocated for 8 rx queues
ix3: Ethernet address: 3c:ec:ef:e9:aa:33
ix3: eTrack 0x8000087c
ix3: netmap queues/slots: TX 8/2048, RX 8/2048
pci0: <simple comms> at device 24.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
pci0: <serial bus> at device 31.5 (no driver attached)
apei0: <ACPI Platform Error Interface> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ns8250: UART FCR is broken
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounter "TSC-low" frequency 1099999978 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST2000NM001A-2J2101 SN03> ACS-4 ATA SATA 3.x device
ada0: Serial Number WS10R331
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST2000NM001A-2J2101 SN03> ACS-4 ATA SATA 3.x device
ada1: Serial Number WS10R3FB
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors)
ses0 at ahciem0 bus 0 scbus8 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses1 at ahciem1 bus 0 scbus13 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses0: ada0,pass0 in 'Slot 00', SATA Slot: scbus0 target 0
ses0: ada1,pass1 in 'Slot 01', SATA Slot: scbus1 target 0
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device Generic
USB Storage (0x05e3:0x0736)
Root mount waiting for: usbus0
usb_msc_auto_quirk: UQ_MSC_NO_TEST_UNIT_READY set for USB mass storage device
Generic USB Storage (0x05e3:0x0736)
usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device
Generic USB Storage (0x05e3:0x0736)
usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device
Generic USB Storage (0x05e3:0x0736)
ugen0.2: <Generic USB Storage> at usbus0
umass0 on uhub0
umass0: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc101
umass0:14:0: Attached to scbus14
da0 at umass-sim0 bus 0 scbus14 target 0 lun 0
da0: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device
da0: Serial Number 000000000272
da0: 40.000MB/s transfers
da0: 3968MB (8126464 512 byte sectors)
da0: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device Generic
USB Storage (0x05e3:0x0736)
Root mount waiting for: usbus0
usb_msc_auto_quirk: UQ_MSC_NO_TEST_UNIT_READY set for USB mass storage device
Generic USB Storage (0x05e3:0x0736)
usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device
Generic USB Storage (0x05e3:0x0736)
usb_msc_auto_quirk: UQ_MSC_NO_SYNC_CACHE set for USB mass storage device
Generic USB Storage (0x05e3:0x0736)
ugen0.3: <Generic USB Storage> at usbus0
umass1 on uhub0
umass1: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 2> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0xc101
umass1:15:1: Attached to scbus15
da1 at umass-sim1 bus 1 scbus15 target 0 lun 0
da1: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device
da1: Serial Number 000000000272
da1: 40.000MB/s transfers
da1: 3968MB (8126464 512 byte sectors)
da1: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
ugen0.4: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 3> on usbus0
uhub1: MTT enabled
Root mount waiting for: usbus0
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.5: <vendor 0x0557 product 0x7000> at usbus0
uhub2 on uhub1
uhub2: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 4> on
usbus0
uhub2: 4 ports with 3 removable, self powered
ugen0.6: <vendor 0x0557 product 0x2419> at usbus0
ukbd0 on uhub2
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 5> on
usbus0
kbd1 at ukbd0
Root mount waiting for: usbus0
ugen0.7: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub3 on uhub1
uhub3: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 6> on usbus0
uhub3: MTT enabled
uhub3: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.8: <Logitech Logitech USB Keyboard> at usbus0
ukbd1 on uhub3
ukbd1: <Logitech Logitech USB Keyboard, class 0/0, rev 1.10/28.00, addr 7> on
usbus0
kbd2 at ukbd1
Dual Console: Video Primary, Serial Secondary
ichsmb0: <Intel Denverton SMBus controller> port 0xe000-0xe01f mem
0xdf478000-0xdf4780ff at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
lo0: link state changed to UP
ums0 on uhub2
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 5> on
usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ix0: link state changed to UP

-- 
You are receiving this mail because:
You are the assignee for the bug.