Performance of disk i/o with 3ware

[Manjunath R Gowda]

On 11/6/07, Nico -telmich- Schottelius <
nico-freebsd-performance at schottelius.org> wrote:
>
> Manjunath R Gowda [Tue, Nov 06, 2007 at 09:54:09AM -0800]:
> > On 11/6/07, Nico -telmich- Schottelius <
> > nico-freebsd-performance at schottelius.org> wrote:
> > >
> > > Hello!
> > >
> > > I've the problem that sometimes there are many disk waiting processes
> > > (sysctl -n vm.vmtotal), but systat -vmstat shows da0 and da1 busy with
> > > 0-10%.
> >
> > I woudn't worry about how many are sleeping. But, How long they sleep
> would
> > be interesting data.
>
> And also why they sleep. Having about 700 processes in sleep state plus
> 400 processes in diskwait state makes me wonder somehow.



At a given time only few can run,  remaining once will sleep which is an
expected behavior.

There are also some processes from cron, which are older and also in
> state "I" (as shown by ps).


You have to dig deep and see what those cron jobs do.

> > I guess that the disk i/o is at about 100%, but wondering why I see
> > > those strange values.
> >
> > Why do you think it should be 100%? Are you running any disk I/O intense
> > application?
>
> Because it's the well known bottleneck in that server:
>
> - it has 4 cpus, which are about 30-50% idle
> - it has 4 GiB ram, of which 2GiB is mostly inactive
> - it has 4 10k rpm hds in 2 raid1 disk arrays, one for
> the mailboxes + root and one for the qmail-queue
>
> Normally systat -vmstat shows 80-100% busy state on the disk arrays,
> but currently it's at about 10%, which cannot be right.
>
> I think I'll have a look closer look at the patch we used from CVS to
> patch the
> 3ware twa driver.


Looks like your blaming the twa driver  based on your experience. OS is a
complex piece of software and involves many components, scheduler, vm,
scsi-stack, driver, problem could be any where in this. That's why specific
data would be useful. Something like on 6.2 running x takes y and on
6.2+patch it takes z would be useful.

> > Anyone an idea,
> > >   a) why systat -vmstat shows so small busy values?
> > >   b) how to debug it further?
> >
> > You can try experimenting  with diffrent I/O loads to make sure that
> there
> > is a problem before start debugging it.
>
> There is a big problem, that even results in taking about 30 seconds
> until the '220' messages comes from qmail when connecting via telnet
> from outside to it.



Your adding one more variable, network here. Verify that  when you did
cvsup the network driver  was not updated.

To summarise, what runs on the server:
>
> - qmail + patches => has /var/qmail/queue on it's own disk-array
> - courier imapd
> - vpopmail (pop3 auth)
> - mysql (needed by vpopmail)
> - clamd
> - spamassassin
> - Webmail (horde on apache)
>
> So it's mainly a customer mailserver, with nothing special installed,
> which may have some load, but not in the way it's not explainable.
>
> So I'm still hoping somebody with similar problems reads this mail ;-)
>
> Sincerly
>
> Nico
>
> --
> Think about Free and Open Source Software (FOSS).
> http://nico.schottelius.org/documentations/foss/the-term-foss/
>
> PGP: BFE4 C736 ABE5 406F 8F42  F7CF B8BE F92A 9885 188C
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFHMMFxuL75KpiFGIwRAoKxAKDjh7aN8n/b7B3YPqcdKuqMPv9CGACgsGuf
> lI6wV4jUuP8GnUhd37oW/mQ=
> =4Yn4
> -----END PGP SIGNATURE-----
>
>