MUTEX_PROFILING option has been removed ...
Robert Watson
rwatson at FreeBSD.org
Sat Nov 11 00:54:01 PST 2006
On Fri, 10 Nov 2006, Kip Macy wrote:
> and replaced by LOCK_PROFILING.
>
> - When LOCK_PROFILING is compiled in and enabled the kernel will now
> profile hold times for all locks (spin mutex, blocking mutex, rwlock, sx
> lock, and lockmgr).
> - We now track the wait-to-acquire time, which I believe to be a more
> useful metric of contention than hold time or number of times contested.
> - The overhead of having LOCK_PROFILING compiled in but not enabled has
> been reduced by moving large chunks of code out of line - on the T1 the
> measured overhead is < 1%.
> - There is no longer a single mutex for serializing updates to the
> profiling hash - reducing the locking contention of measuring lock
> contention.
This sounds like really great work -- thanks for working on this!
Robert N M Watson
Computer Laboratory
University of Cambridge
>
> Thanks to DES for the MUTEX_PROFILING implementation and Kris Kennaway for
> many of the optimizations that made their way into this patch.
>
> Please report to me any issues caused by this change. I give some examples
> of its immediate utility below:
>
>
> I'm running a buildworld that isn't using all the system threads. I sorted
> on the third column (maximum total wait) - the first is due to the idle
> threads constantly trying to get work. The third and fourth are from make
> using select. Looking at kern_select - one sees that it is clearly fairly
> single-threaded. Oddly enough, makes Job.c already has support for kqueue,
> but it isn't the default. I defined USE_KQUEUE and select went away as a
> point of contention during builds. We see here that the page queue mutex is
> a major point of contention.
>
> max total wait_total count avg wait_avg
> cnt_hold cnt_lock name
> 24 3566322 1311264691 1358360 2 965 6800925
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/kern_idle.c:121
> (sched lock)
>
> 29 1218447 414601055 172116 7 2408 533196
> 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/subr_sleepqueue.c:529
> (sched lock)
> 2 3013 413907132 8359 0 49516 14603
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/sys_generic.c:812
> (sched lock)
>
> 1027 242236 413829518 14365 16 28808 4462
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/sys_generic.c:776
> (sellck)
>
> 1894753 787273038 55823553 726605 1083 76 0
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/vfs_default.c:263
> (nfs)
>
> 253 104799 8583672 204689 0 41 39672
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_fault.c:844
> (vm page queue mutex)
> 153 264890 3935024 227674 1 17 46885
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_fault.c:902
> (vm page queue mutex)
>
> 316 3238931 2650089 227674 14 11 113827
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/sun4v/sun4v/pmap.c:956
> (vm page queue mutex)
> 35 101146 1916077 82252 1 23 16275
> 0 /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_fault.c:342
> (vm page queue mutex)
>
> 4 106600 1665429 285490 0 5 475928
> 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/subr_sleepqueue.c:318
> (sched lock)
>
>
>
> Here we do a make -j32 of the kernel, so all cpu threads are in use (thus no
> issues with the idle threads). The turnstile lock contention is likely a
> result of all the cpu threads contending for the page queue mutex. This
> could probably be improved by adaptively spinning if the current holder of
> the mutex is running. Many of page queue mutex acquisitions are merely to
> protect setting flags in an individual page. In the case of a 32 cpu system
> having a lock per vm_page would probably be the way to go - however, this
> would penalize systems with 4 and fewer cpus. Perhaps alc should look into
> varying the granularity of locking as a function of the number of cpus.
> max total wait_total count avg wait_avg cnt_hold
> cnt_lock name
> 5 7266196 206805560 7522452 0 27
> 48266619 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/subr_turnstile.c:487
> (turnstile chain)
> 457 528521 180592127 550284 0 328
> 1469872 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_fault.c:844 (vm page queue
> mutex)
> 15057461 1679582934 117520488 87978 19090 1335
> 0 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/ufs/ffs/ffs_vnops.c:366 (ufs)
> 214 1076256 112489341 559032 1 201
> 1520471 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_fault.c:902 (vm page queue
> mutex)
> 424 8250360 105249196 559031 14 188
> 1767340 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/sun4v/sun4v/pmap.c:956 (vm page
> queue mutex)
> 72563452121 218316084315 94216669 452713 482239 208
> 0 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/vfs_default.c:263 (nfs)
> 23 1349030 14049785 280685 4 50
> 923679 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/kern/kern_idle.c:121 (sched lock)
> 73 214117 11078161 63944 3 173
> 2505 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/nfsclient/nfs_socket.c:1235
> (Giant)
> 42 92768 10431233 40012 2 260
> 122966 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_fault.c:342 (vm page queue
> mutex)
> 6429 2155581 6645086 18297 117 363
> 105550 0
> /usr/flatstor/shared/freebsd/kmacy/src/sys/vm/vm_object.c:651 (vm page queue
> mutex)
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
>
More information about the freebsd-current
mailing list