(long) Re: writing to 3ware escalade uses up 100% cpu time in
system.
Klaus Robert Suetterlin
krs at robert2.mpe-garching.mpg.de
Thu Jan 15 04:44:20 PST 2004
Hi,
I'll try to provide more information on the system I use and on the
problem I see along the line of three emails I recieved so far.
My system is a P3-933 Compaq Proliant ML350, with 512MB of Ram.
Using software raid (i.e. ccd) on 4 scsi-2 or ultra ata-100 disks
I can get 80MByte/sec easily, without any significant cpu usage.
Unfortunately ultra ata cable length and scsi prices are too limiting
for my storage needs (1GByte in 4 Disks), thus I opted for a serial
ata raid controller.
"Subhro" <subhro at fusemail.com> asked:
> Can you post this?
>
> sysctl -a |grep dma
vfs.nfs.iodmaxidle: 120
vfs.nfs.iodmax: 20
hw.ata.ata_dma: 1
hw.ata.atapi_dma: 0
But I do not see the point here. twe(4) does not list any sysctl
variables for dma, so I'd expect to find none.
``sysctl -a|grep twe'' gives
kern.disks: twed0 ad0
twe commands 255 16K 16K 255 64
hw.twe0.driver_version: 1.50.00.000
Doug White <dwhite at gumbysoft.com> wrote:
> What speed of bus is the 3ware attached to?
Unfortunately my system only has a 33MHz 64Bit PCI Slot. The Adapter
I use supports 66MHz.
> You might run 'systat -vmstat 2' and check the amount of CPU going to
> interrupts. I suspect you're maxing out on the interrupt handler. Also
> look at the # of interrupts to the twe.
According to ``systat -vmstat 2'' my cpu is used 70-90% by System
(the exact number depends on the amount of free memory), 2% by
Interrupt. The number of interrupts in twe is approx. 270. The
twe is serving 570 tps with 128KB/t for a grand total of 70MB/s and
is 75% busy.
I get a total of 500 Interrupts and the cpu is at 8%-30% Idle. Cpu
usage depends on the ammount of free memory. In the beginning (free
memory available) cpu is at approx. 30% Idle, after the first few
GByte of transferred data cpu is at 10% Idle.
> What kind of workload are you expecting to put on? Its very, very rare
> that you would be getting large sequential writes like those you are
> getting with dd.
I expect linear writes of 200 GByte at 10 - 80 MByte / sec.
Scott Long comments:
> The interrupt handler for the driver is actually pretty lean; most real
> work is offloaded to a taskqueue. However, the driver as a whole is not
> locked, so it's going to have contention on the Giant mutex. He might
> well get some better performance just my making the interrupt handler
> be INTR_MPSAFE. Unfortunately, this requires a bit of work and I don't
> have any hardware to test on.
Sorry I had no time and no expertise to investigate this.
> Another cheap hack to help performance would be to remove the
> INTR_ENTROPY flag from the call to bus_setup_intr() in twe_freebsd.c.
> This will affect the quality of your entropy a bit, but it would be an
> interesting test.
I will do this today, if I find out what it means :).
> One issue that might be at play here is that the 3ware hardware has some
> limits on DMA alignment for data buffers. If the buffers don't meet
> those limits, then the driver has to manually allocate and copy
> temporary buffers. The driver doesn't seem to keep any stats on this,
> but it would be quite useful if it did.
According to twe(4) this is only a problem when userland tries to
access the raw media... I guess this is because kernel buffercache
uses aligned buffers only?
> Other tests to try would include doing I/O directly to the device, and
> turning off softupdates, especially if you're doing lots of file
> operations.
Huh? I don't understand how this would help with throughput.
Regards, Robert S.
More information about the freebsd-current
mailing list