high CPU usage in FreeBSD for a PCIe card driver

[Steevan Rodrigues]

Thank you for the quickly reply.     The same driver gives pretty good
performance with low CPU usage in kernel mode.
I see high CPU usage only for user mode application.

Here is the Dtrace hotkernel output details provided in the earlier
attachment.

CPU usage details on a  12 core 24 thread server PC.

part of the perl hotkernel  output

kernel`free                                              1314   0.2%
kernel`_sx_slock                                         1415   0.3%
kernel`dev_relthread                                     1435   0.3%
kernel`devvn_refthread                                   1842   0.3%
kernel`_mtx_lock_spin_cookie                             2059   0.4%
kernel`copyin                                            2164   0.4%
kernel`amd64_syscall                                     2235   0.4%
kernel`__mtx_lock_spin_flags                             3566   0.6%
kernel`bzero                                             4889   0.9%
kernel`__mtx_lock_sleep                                  9323   1.7%
kernel`acpi_cpu_idle                                    11166   2.0%
kernel`pmap_extract                                     17491   3.1%
kernel`spinlock_exit                                    64850  11.5%
kernel`cpu_idle                                        134617  23.9%
kernel`lock_delay                                      263330  46.8%

-------------------------------------------------

CPU usage details on a  2 core 4 thread Intel i3 based Desktop PC.

part of the perl hotkernel  output


kernel`0xffffffff80                                       558   0.6%
kernel`copyout                                            592   0.6%
0xffffffff822368a3                                        639   0.7%
kernel`amd64_syscall                                      683   0.7%
kernel`__mtx_unlock_flags                                 728   0.8%
kernel`copyin                                            1311   1.4%
kernel`bzero                                             1434   1.5%
kernel`pmap_extract                                      1615   1.7%
kernel`lock_delay                                        2329   2.4%
0xffffffff822b0b5e                                       2528   2.6%
kernel`spinlock_exit                                    20950  21.8%
kernel`cpu_idle                                         48124  50.2%

------------------------------------------------------

Thanks
Steevan

On Wed, Jun 27, 2018 at 7:41 PM, Warner Losh <imp at bsdimp.com> wrote:

> No. It's not from mtx_lock. It's likely from lock contention. Since the
> list ate the attachment, I can't check for sure, but high CPU usage in
> locking primitives is a classic sign of poor lock design leading to
> contention.
>
> Warner
>
> On Wed, Jun 27, 2018 at 6:28 AM, Steevan Rodrigues <
> steevanxperia at gmail.com> wrote:
>
>> Hi
>>
>> I  come from Linux background and new to FreeBSD.
>>
>> I am investigating a high CPU usage issue on FreeBSD for PCIe driver of a
>> card . The card can process data at a rate of 30 to 50 Gbps . However, I
>> see very high CPU usage even when I run a single threaded user side
>> application which uses this card .
>>
>> I ran hotkernel script from Dtrace toolkit.   A portion of the output is
>> given in the attached file  dtraceDiff.txt. Here I see that lock_delay
>> seems to be the reason for high CPU usage. Looks like it comes from mutex
>> locks
>>
>> This driver uses  mutex locks ( mtx_lock).
>>
>> The CPU usage is very high  on a multicore (Xeon CPU) server node than a
>> dual core (Intel i3) Desktop .
>>
>> It shows 30 to 40 % CPU usage on a Intel i3 ( 2 core 4 threads) and 60 to
>> 70% CPU usage  on a Xeon  CPU with 12 core (24 threads) for same output
>> data rate .
>>
>> I wonder whether usage of mtx_lock is the cause of high CPU usage ? Any
>> pointers or suggestions welcome.
>>
>> Using  FreeBSD 11.1 Release
>>
>> Thanks
>>
>> Steev
>>
>> _______________________________________________
>> freebsd-hackers at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org
>> "
>>
>>
>