Odd performance problems after upgrade from 4.11 to 6.0-Stable

[Kevin Oberman]

> Kevin Oberman wrote:
> I recently upgraded my last desktop system from 4.11-Stable to
> 6.0-Stable. I did an update to 5.3 then to RELENG_5,
> RELENG_6_0_0_RELEASE and on to RELENG6_0.
>
> This system has been updated regularly from the days of at least 4.1.
> The hardware is a 1GHz PIII with an ICH2 chipset. 256 MB of memory. 
>
> After the upgrade, the system performs poorly. Specifically, it seems to
> be blocking on I/O. If I have a dump or bsdtar running, a CPU intensive
> job produced minimal CPU utilization and little progress. The system is
> 70% or more idle all of the time. X is particularly sluggish. Many times
> the system will simply appear to be locked up for seconds at a time like
> all of the I/O is synchronous and the entire system is blocking on GIANT
> until the disk write finishes. A level 0 dump of the system drive piped
> through gzip and over the network took about 10 hours. Now it is 17
> hours. My network link only ran at 3.4 Mbps (yes, that's bits, not
> Bytes) over a 100 Mbps connection to the storage server. The rate was
> much higher under 4.11.
>
> My attempts to track down the source of the problem simply failed. I
> have examined my kernel configuration and found nothing out of the
> ordinary. In fact, the the configuration varies from GENERIC only in
> devices (both added and deleted), dropping 486 and 586 CPU code and a
> larger SC_HISTORY. I have nothing in sysctl.conf and only enable DMA on
> my ATAPI devices in loader.conf. I have confirmed that the disks are
> running ATA100.
>
> Any suggestions on where to look? At this point I am baffled.

Well, I have tracked the problem down a bit. It happens when the system
runs out of memory and starts to use the swap file. The glimpse indexing
operation is very memory intensive and under V6 it simply crawls when it
starts paging. On my old 256 MB system, this operation was taking over 6
hours. On my new system (1 GB), it takes about 7 minutes. 

This is not really a surprise. I know how running out of memory slows
things down. What does bother me is why it went from 6 hours in V4 14
hours on V6.

Unfortunately, my old box is going away in two weeks and I doubt that
I'll have much time to look at things before then. All I can say is
that either the wired memory for V6 is up so much that I page and swap a
lot mote or that the operations have slowed down way more than they
should have.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman at es.net			Phone: +1 510 486-8634