Stability Issues on 5.4-RELEASE Box

[alex at schnarff.com]

> I'm not running your application mix, but I've never seen random reboots
> unless there were hardware issues.  With 5.X these included having
> hyperthreading turned on, which I know caused problems with my dual XEON
> system.

Hmmm...two answers so far, two people saying "hyperthreading can be an 
issue." I'll definitely have that turned off ASAP.

>> * Issues with files that are not found on startup sometimes, but are 
>> other times. Prime example: the Zope CMS system that's been 
>> installed failed to find libmysqlclient.so after a planned soft 
>> reboot, but found it with no trouble on a subsequent boot a few 
>> minutes later, with no config changes in between.
>>
> Haven't seen that; are there any messages indicating you're having
> filesystem problems?

Thanks for asking; I see some new nasties in /var/log/messages:

Feb 27 09:05:37 www fsck: /dev/ad4s1f: PARTIALLY TRUNCATED INODE I=9397392
Feb 27 09:05:37 www fsck: /dev/ad4s1f: UNEXPECTED SOFT UPDATE 
INCONSISTENCY; RUN fsck MANUALLY.

>> Thus far, besides simply scanning log files, constantly watching 
>> "top" and "ps", etc., I've not been able to do much with the box. As 
>> I said, I upgraded OpenSSL/OpenSSH to current versions, and I 
>> installed pf as the firewall (there was none before I 
>> arrived...don't even get me started on that). This weekend the guy 
>> who was the previous admin will be running a Memtest for me and 
>> disabling hyperthreading (which there's no performance justification 
>> for, and which has caused me stability issues at least on Linux in 
>> the past), since the server is in Oregon and I'm in the DC area. 
>> That's about the extent of what I've been able to do to date, since 
>> this is a production box.
>>
>
> How did you upgrade OpenSSL/SSH - cvsup + buildworld, etc., from the
> ports, or some other way?  I've always did this using cvsup since
> OpenSSL/SSH is now built into the OS

 From source on openssl.org and openssh.org. I'm not yet familiar with 
all of the cool helper things that FreeBSD has such as cvsup, and I was 
conerned about getting those two specifically fixed fast when I first 
started this task (as the version on there was old enough that there 
were known remote exploits).

>> * Does 5.4-RELEASE have any known bugs that might cause stability 
>> issues like the ones I've described here? More importantly, would an 
>> upgrade to 6.2-RELEASE be worthwhile (as is my instinct), in terms 
>> of being generally more stable and/or having better hardware 
>> support? Would such an upgrade be possible/relatively painless to 
>> perform without being physically at a console, as has been the case 
>> with OpenBSD over the years?
>>
>
> I am not running your mix, but I have a 4.11 box, a 5.5-STABLE box, and a
> 6.2-STABLE box, and haven't seen anything like this. the 4.11 box looks a
> lot like yours - Supermicro P4DC6 with dual Xeons, RAID1 with the onboard
> Adaptec controller + the add-in RAID board, RAID5 using vinum with an
> Adaptec (39160?) board. 5.X plus hyperthreading was definitely a problem
> with that system.  I have another similar box which I'm testing with
> 6.2-STABLE and haven't seen any problems yet.

Again, since the two answers I have so far say essentially the same 
thing here, it looks like my concerns about hardware are valid.

>> * Given my dmesg below, do you see any specific problems?
>
> The interrupt storm on uhci1+ is not a good thing.

Any thoughts on how to fix it?

Thanks,
Alex Kirk