System crawls after Upgrade 7.0->7.2

[Tobias Lott]

On Tue, 8 Sep 2009 12:19:45 +0200
Tobias Lott <tlott at gamesnet.de> wrote:

> Hey Everyone,
> 
> I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all OS
> related Stuff is located on an UFS Slice, Application is on a ZFS
> Volume.
> 
> After the Upgrade everything seemed fine, but a User noticed one PHP
> Script which is basically loading a plain Textfile into Mysql times
> out. PHP Timeout was set to 60 secs, that was more then enough just
> one day before the upgrade.
> 
> System-wise its like Mysql can't get get the data fast enough, process
> is max at 10% cpu usage most of the time in sbwait state.
> 
> Haven't changed any sysctl (kern.maxvnodes="200000",
> vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="100M"
> but had those before already) or kernel parameter at that time. But
> according to the Wiki's ZFS Tuning Guide those aren't needed anymore
> so I tried without but no change there either.
> 
> I checked the following (and the current used values each):
> maxproc
> maxfiles
> kern.ipc.somaxconn
> kern.ipc.nmbclusters
> 
> Getting no error messages dmesg, syslog wise.
> 
> Tried moving Mysql to the UFS Slice but same thing happens happens
> there.
> 
> Some very basic testing like dd-ing:
> # dd if=/dev/da0 of=/dev/null bs=1024 count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes transferred in 157.698602 secs (6808823 bytes/sec)
> 
> FreeBSD hostname 7.2-STABLE FreeBSD 7.2-STABLE #3 r196954:
> Tue Sep  8 02:10:22 CEST 2009
> root at hostname:/usr/obj/usr/src/sys/SPIRIT  amd64
> 
> Since I upgraded the Machines zpool to version 13 already I don't
> really wanna go back to 7.0, but it seems the only way out atm.
> 
> Hopefully someone can give me a Hint where maybe I forgot to check.
> 
> 
> Somehow it feels like the last Versions, I'd say starting with 7.x
> doesn't really feel that Rock-Stable as Versions Prior used to
> be. Prolly cause I'm using an Experimental Feature like ZFS, so no
> Offense nor Blame and this is not intended as a Flame or whatsoever!
> So don't get me wrong, I always liked, used, recommended FreeBSD since
> 4.0 and will continue to do so!
> 
> Best Regards
> 

I've tested some more it seems the Bottleneck is the HD, Raid Array is
Optimal checked it.
Running bonnie++ takes like forever, couldn't finish it since its a
Productive Machine, cause it renders the Server useless, every try to
access the server while bonnie is running times out. Weird thing is
bonnie++ is using max 1-3% CPU. But according to gstat its only ~3mb/s
(displayed as 100% busy) during first 3 Tests that is.


savecore: reboot after panic: page fault

But had no luck with the coredumb

# kgdb kernel.debug  /var/crash/vmcore.12
<snip>
Cannot access memory at address 0x0

Or is there any other method I could try?

Cause this is getting nasty unstable

-- 
Tobias Lott