HAST performance / dtrace / _umtx_op

[Jilles Tjoelker]

On Sat, Apr 26, 2014 at 11:30:25AM +0100, Karl Pielorz wrote:
> I've been looking at HAST recently - and noticed there's a huge
> performance hit (on 10-STABLE anyway) even if you setup a disk with a
> remote of 'none' - switching from the 'raw' disk to '/dev/hast/disk'
> shows a 50% performance hit [in testing having an active secondary via
> 10Gbit link doesn't seem to show any further performance drop].

> Thinking of digging deeper I setup a single disk HAST system (with no
> remote node) and ran dtrace on the daemon handling that disk. I then
> DD'ed 1Gbyte of data to that disk.

> The output from dtrace shows:

> "
> Elapsed Times for PID 5219,
> 
>          SYSCALL          TIME (ns)
>            pread            4439902
>           pwrite         7617448542
>            ioctl        11370282332
>     sigtimedwait        20015976362
>         _umtx_op        36514993910
> "

> To my untrained eye that looks like it spent more time on locking than
> anything else? - Followed by ioctl's - then, considerably less time on
> pread/pwrite (which must be the bit that's copying the data)

> Presumably the time spent in sigtimedwait would have actually been
> 'waiting' time - i.e. not active CPU usage kind of time?

> Does that seem to be the case? - At least then I'll hopefully have
> some pointer where to look further for why there's such a performance
> hit.

_umtx_op implements blocking not only for mutexes but also for condition
variables, and in the latter case waiting for a long time does not
indicate a problem.

-- 
Jilles Tjoelker