cpu timer issues
freebsd at jdc.parodius.com
Tue Sep 28 09:30:54 UTC 2010
On Tue, Sep 28, 2010 at 05:54:15PM +1000, Jurgen Weber wrote:
> Hello List
> We have been having issues with some firewall machines of ours using
> FreeBSD smash01.ish.com.au 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #0:
> Sun Dec 6 23:20:31 EST 2009 sullrich at FreeBSD_7.2_pfSense_1.2.3_snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7
> MotherBoard: http://www.supermicro.com/products/motherboard/Xeon3000/3200/X7SBi-LN4.cfm
> Originally the systems started out by showing a lot of packet loss,
> the system time would fall behind, and the value of "#vmstat -i |
> grep timer" was dropping below 2000. I was lead to believe by the
> guys at pfSense that this is where the value should sit. I would
> also receive errors in messages that looked like " kernel: calcru:
> runtime went backwards from 244314 usec to 236341".
> We tried a variety of things, disabling USB, turning off the Intel
> Speed Step in the BIOS, disabling ACPI, etc, etc. All having little
> to no effect. The only thing that would right it is restarting the
> box but over time it would degrade again. I talked to the SuperMicro
> and they said that this is a FreeBSD issue and pretty much washed
> their hands of it.
> After a couple of months of dealing with this and just rebooting the
> systems reguarly, the symptoms slowly but surely disappeared. eg.
> The kernel messages went away, the system time was not falling
> behind and I was experiencing no packet loss but the "#vmstat -i |
> grep timer" value would continue to decrease over time. Eventually I
> think, when it finally got the 0 the machine restarted (I am only
> guessing here).
> After this restart it worked again for a couple of hours and then it
> restarted again.
> After the second time the system has not missed a beat, it has been
> fine and the "#vmstat -i | grep timer" value remained near the 2000
> mark... We setup some zabbix monitoring to watch it. As mentioned it
> was fine for about a month. Until today. Today the value has dropped
> to 0, but the system has not restarted and over the last couple of
> hours the value has increased to 47.
> This machine is mission critical, we have two in a fail over
> scenario (using pfSense's CARP features) and it seems unfortunate
> that we have an issue with two brand new SuperMicro boxes that
> affect both machines. While at the moment everything seems fine I
> want to ensure that I have no further issues. Does anyone have any
> Lastly I have double check both of the below:
> We disabled EIST.
> # dmesg | grep Timecounter
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Timecounters tick every 1.000 msec
> # sysctl kern.timecounter.hardware
> kern.timecounter.hardware: i8254
> Only have one timer to choose from.
I have a subrevision of this motherboard in use in production, which ran
RELENG_7 and now runs RELENG_8, without any of the problems you
describe. I don't have any experience with the -LN4 submodel though,
although I do have experience with the X7SBA-LN4.
Our hardware in question:
The machine in question consists of 4 disks (1 OS, 3 ZFS raidz1), uses
both NICs (two separate networks) at gigE rates, handles nightly backups
for all other servers, acts as an NFS server, a time source (ntpd) for
other servers on the network, and a serial console head. Oh, it also
has EIST enabled, and runs powerd with some minor (well-known) tunings
in loader.conf for it.
Secondly, here's our sysctl kern.timecounter tree on our system, in
addition to our SMBIOS details (proving the system is what I say it is).
Note that we have multiple timecounter choices, and APCI-fast is chosen.
I would expect problems if i8254 was chosen, but the question is why
this is being chosen on your systems and why alternate timecounter
choices aren't available. You said you tried booting with ACPI
disabled, which might explain why ACPI-fast or ACPI-safe are missing.
$ sysctl kern.timecounter
kern.timecounter.choice: TSC(-100) ACPI-fast(1000) i8254(0) dummy(-1000000)
$ kenv | grep smbios
smbios.bios.vendor="Phoenix Technologies LTD"
Fourthly, here's our BIOS settings (using BIOS 1.30, which is referred
to as "R 1.3a" on Supermicro's site):
Supermicro SuperServer 5015B-MT BIOS Settings
Current BIOS: 1.30
Reset to Factory Defaults, then change:
--> Set to GMT, not local time!
* Serial ATA
--> Native Mode Operation --> Serial ATA
--> SATA AHCI Enable --> Enabled
* Boot Features
--> Quiet Boot --> Disabled
* I/O Device Configuration
--> Serial port B --> Disabled
--> Parallel port --> Disabled
* Console Redirection
--> Com Port Address --> On-board COM A
--> Baud Rate --> 115.2K
--> Console Type --> VT100+
--> Continue C.R. after POST --> On (SEE NOTE #2)
NOTE #2: CR after POST
If the system is running RELENG_7, ***do not*** enable this option. The
bootloader and thus kernel appear to get confused by who controls the
interrupt, and you end up without *any* serial console output period.
RELENG_8 has addressed this problem, and you *should* enable this feature
when using that OS. This will allow you to see LAN option ROM messages
during PXE booting, or boot0 (if you use it; usually we don't).
Since you have two systems with the same problem, I really don't know
what to tell you. What I can tell you is that we've run RELENG_7 and
RELENG_8 on all of the following hardware without any problems:
* Supermicro SuperServer 5015B-MTB
* Supermicro SuperServer 5015M-T+B
* Supermicro X7SBA
* Supermicro X7SBL-LN2
Can you provide any tuning you do in loader.conf or sysctl.conf, as well
as your kernel configuration?
Otherwise, if you continue to have problems of this nature, I would
strongly recommend replacing the hardware. Clock skew of this nature,
at least based on what I've seen at my day/night job, is usually the
sign of a crystal going bad on the motherboard. Yes, I realise you have
two systems which are exhibiting the same behaviour, but for all I know
a manufacturer (not Supermicro) released a batch of bad crystals into
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
More information about the freebsd-stable