System crawls after Upgrade 7.0->7.2
tlott at gamesnet.de
Wed Sep 9 08:58:02 UTC 2009
On Tue, 8 Sep 2009 12:19:45 +0200
Tobias Lott <tlott at gamesnet.de> wrote:
> Hey Everyone,
> I upgraded a Dual Core Machine to 7.2-Stable (2 Days ago), all OS
> related Stuff is located on an UFS Slice, Application is on a ZFS
> After the Upgrade everything seemed fine, but a User noticed one PHP
> Script which is basically loading a plain Textfile into Mysql times
> out. PHP Timeout was set to 60 secs, that was more then enough just
> one day before the upgrade.
> System-wise its like Mysql can't get get the data fast enough, process
> is max at 10% cpu usage most of the time in sbwait state.
> Haven't changed any sysctl (kern.maxvnodes="200000",
> vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="100M"
> but had those before already) or kernel parameter at that time. But
> according to the Wiki's ZFS Tuning Guide those aren't needed anymore
> so I tried without but no change there either.
> I checked the following (and the current used values each):
> Getting no error messages dmesg, syslog wise.
> Tried moving Mysql to the UFS Slice but same thing happens happens
> Some very basic testing like dd-ing:
> # dd if=/dev/da0 of=/dev/null bs=1024 count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes transferred in 157.698602 secs (6808823 bytes/sec)
> FreeBSD hostname 7.2-STABLE FreeBSD 7.2-STABLE #3 r196954:
> Tue Sep 8 02:10:22 CEST 2009
> root at hostname:/usr/obj/usr/src/sys/SPIRIT amd64
> Since I upgraded the Machines zpool to version 13 already I don't
> really wanna go back to 7.0, but it seems the only way out atm.
> Hopefully someone can give me a Hint where maybe I forgot to check.
> Somehow it feels like the last Versions, I'd say starting with 7.x
> doesn't really feel that Rock-Stable as Versions Prior used to
> be. Prolly cause I'm using an Experimental Feature like ZFS, so no
> Offense nor Blame and this is not intended as a Flame or whatsoever!
> So don't get me wrong, I always liked, used, recommended FreeBSD since
> 4.0 and will continue to do so!
> Best Regards
I've tested some more it seems the Bottleneck is the HD, Raid Array is
Optimal checked it.
Running bonnie++ takes like forever, couldn't finish it since its a
Productive Machine, cause it renders the Server useless, every try to
access the server while bonnie is running times out. Weird thing is
bonnie++ is using max 1-3% CPU. But according to gstat its only ~3mb/s
(displayed as 100% busy) during first 3 Tests that is.
savecore: reboot after panic: page fault
But had no luck with the coredumb
# kgdb kernel.debug /var/crash/vmcore.12
Cannot access memory at address 0x0
Or is there any other method I could try?
Cause this is getting nasty unstable
More information about the freebsd-stable