b_freelist TAILQ/SLIST

[Konstantin Belousov]

On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote:
> Hi.
> 
> While doing some profiles of GEOM/CAM IOPS scalability, on some test 
> patterns I've noticed serious congestion with spinning on global 
> pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is 
> already very simple, I've tried to optimize probably the only thing 
> possible there: switch bswlist from TAILQ to SLIST. As I can see, 
> b_freelist field of struct buf is really used as TAILQ in some other 
> places, so I've just added another SLIST_ENTRY field. And result 
> appeared to be surprising -- I can no longer reproduce the issue at all. 
> May be it was just unlucky synchronization of specific test, but I've 
> seen in on two different systems and rechecked results with/without 
> patch three times.
This is too unbelievable.  Could it be, e.g. some cache line conflicts
which cause the trashing, in fact ? Does it help if you add void *b_pad
before b_freelist instead of adding b_freeslist ?

> 
> The present patch is here:
> http://people.freebsd.org/~mav/buf_slist.patch
> 
> The question is how to do it better? What is the KPI/KBI policy for 
> struct buf? I could replace b_freelist by a union and keep KBI, but 
> partially break KPI. Or I could add another field, probably breaking 
> KBI, but keeping KPI. Or I could do something handmade with no breakage. 
> Or this change is just a bad idea?
The same question about using union for b_freelist/b_freeslist, does the
effect of magically fixing the contention still there if b_freeslist
is on the same offset as the b_freelist ?

There are no K{B,P}I policy for struct buf in HEAD, just change it as
it fits.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20130628/032f05f9/attachment.sig>