CPU hog on -current (pfSense 23.05)

Hi,

since cca 7 days I’ve started to observe a CPU hog of one CPU core on APU2 box running pfSense 23.05 (if that matters).

mjg@ suggested to run a dtrace oneliner that showed:

~~~~~~
                        kernel`pmap_copy              33
                    kernel`amd64_syscall              33
                  kernel`vm_radix_insert              35
                kernel`vm_map_pmap_enter              37
         kernel`vm_radix_lookup_unlocked              38
                       kernel`memcpy_std              38
             kernel`vm_object_deallocate              39
          kernel`pmap_enter_quick_locked              41
         kernel`em_update_stats_counters              43
               kernel`copyout_nosmap_std              43
           kernel`ck_epoch_poll_deferred              44
                   kernel`sbuf_put_bytes              46
           kernel`vm_page_pqbatch_submit              48
                  kernel`pmap_remove_pte              51
                  kernel`pmap_pvh_remove              53
          kernel`vm_pqbatch_process_page              54
               kernel`cpu_search_highest              56
                     kernel`get_pv_entry              57
         kernel`pmap_try_insert_pv_entry              59
              kernel`vm_map_lookup_entry              65
                  kernel`epoch_call_task              92
                       kernel`pmap_enter             101
                         kernel`vm_fault             110
                         kernel`pagecopy             110
                     kernel`0xffffffff81             133
                       kernel`lock_delay             145
                kernel`pmap_remove_pages             203
                     kernel`_thread_lock             415
                     kernel`pagezero_std             490
                        kernel`assert_rw             532
                      kernel`acpi_cpu_c1             600
                     kernel`callout_lock             641
                       kernel`kern_yield            1206
               kernel`_callout_stop_safe            2010
                   kernel`spinlock_enter            2032
                   kernel`tcp_timer_stop            2703
                                     0x0            5927
                    kernel`spinlock_exit           40964
                         kernel`cpu_idle           61943
                     kernel`sched_idletd           76722
~~~~~~~

The symptom is that the kernel thread "kernel{if_io_tqg_1}” consumes 100% of the CPU core.

Can this be debugged any further?

I also suspect the hardware problem (I have one spare box where I’ll put my XML config
and test whether the problem will persist).

I’d be very thankful for any pointers.

Thanks!
otis


—
Juraj Lutter
otis@FreeBSD.org