different behaviour between 4.x and 5.x (ping response/disk io)

[Jochen Keil]

Hi all.

I got a dual Piii 800 box here with these specifications:
-Supermicro 370DL3 Mainboard
-Adaptec aic7892 Ultra160 SCSI adapter (onboard)
-2x 256MB Micron ECC RAM (tested with memtest)
-2x IBM IC35L036UWDY10-0 (SCSI Drives, only one connected for testing)
-LG GSA-4040B (connected via onboard IDE Controller)
(a few IDE drives which weren't connected für testing)

Full dmesg output is attached to the end of this email.

Now to the description of my problem.
I tested various Operating systems on this box, FreeBSD 5.3, NetBSD 2.0, 
FreeBSD 4.11 and an elder version of the Knoppix (Linux 2.4) CD.

Every installation except FreeBSD 4.11 showed almost the same behaviour 
when doing the following test. First i ran iozone on the SCSI drive. 
While the benchmark was performed i issued a ping to another host in the 
network. The result drove me insane. Wether with the onboard fxp0 either 
the em0 i was able to get a "stable" ping response time which looks like 
that: i get a bunch (~20-30, depends on disk io) of ping response times 
around 0.2ms and then there are some really long response times above 
1ms (the longest took about 90ms).
However FreeBSD 4.x seems to be the only exception to this as it gives 
me over a complete iozone test only three pings with a response time 
around 1ms. The other ping times are mostly around 0.2ms with some 
seldom peaks at 0.5-0.6ms.
Due to this behaviour i figured out that this cannot be the fault of the 
hardware, at least i hope so. A hint from a NetBSD Mailing List pointed 
me to have a look at the interrupts. On disk IO the interrupt rate on 
ahc0 is going up to around 300/s. But this is all the same for FreeBSD 
4.x and 5.x (i used vmstat -i for that). There also seems to be no 
difference between using SMP and a single CPU kernel (Information based 
on FreeBSD 5.3 and NetBSD 2.0 tests as i didn't test this with FreeBSD 
4.11 yet).
Now i am really puzzled because i cannot understand why 4.x behaves 
relatively good compared to 5.x on this specific issue. Is there a good 
explanation or does one have to investigate this further?
The NetBSD Mailing List i'm talking about can be found here: 
http://mail-index.netbsd.org/netbsd-users/2005/01/. (Discussion starts 
on 05.01.2005: High Load / bad response times).

I'm looking forward to your answers and with kind regards,

Jochen Keil


Dmesg from 4.11:
FreeBSD 4.11-RC2 #0: Sun Jan  2 09:47:04 GMT 2005
     root at perseus.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel Pentium III (799.62-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0x683  Stepping = 3
 
Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 536805376 (524224K bytes)
config> di pcic0
config> di psm0
config> di sio1
config> di sio0
config> di ppc0
config> di fdc0
config> di bt0
config> di ata1
config> di aic0
config> di aha0
config> di adv0
config> q
avail memory = 516833280 (504720K bytes)
Preloaded elf kernel "kernel" at 0xc0556000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc055609c.
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 9 entries at 0xc00f5230
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <S3 968 graphics accelerator> at 3.0 irq 11
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xd800-0xd83f mem 
0xfe800000-0xfe8fffff,0xfe9ef000-0xfe9effff irq 9 at device 6.0 on pci0
fxp0: Ethernet address 00:30:48:21:74:63
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 ATA33 controller> port 0xffa0-0xffaf at 
device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
ohci0: <OHCI (generic) USB controller> mem 0xfeaff000-0xfeafffff irq 10 
at device 15.2 on pci0
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: (0x1166) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
ahc0: <Adaptec aic7892 Ultra160 SCSI adapter> port 0xe800-0xe8ff mem 
0xfebff000-0xfebfffff irq 11 at device 3.0 on pci1
aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
orm0: <Option ROM> at iomem 0xc0000-0xc7fff on isa0
pmtimer0 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
acd0: DVD-R <HL-DT-ST DVDRAM GSA-4040B> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
Mounting root from ufs:/dev/da0s1a
da0 at ahc0 bus 0 target 1 lun 0
da0: <IBM IC35L036UWDY10-0 S28C> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged 
Queueing Enabled
da0: 34715MB (71096640 512 byte sectors: 255H 63S/T 4425C)