cpu timer issues
jurgen at ish.com.au
Tue Sep 28 08:05:02 UTC 2010
We have been having issues with some firewall machines of ours using
FreeBSD smash01.ish.com.au 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #0: Sun
Dec 6 23:20:31 EST 2009
sullrich at FreeBSD_7.2_pfSense_1.2.3_snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7
Originally the systems started out by showing a lot of packet loss, the
system time would fall behind, and the value of "#vmstat -i | grep
timer" was dropping below 2000. I was lead to believe by the guys at
pfSense that this is where the value should sit. I would also receive
errors in messages that looked like " kernel: calcru: runtime went
backwards from 244314 usec to 236341".
We tried a variety of things, disabling USB, turning off the Intel Speed
Step in the BIOS, disabling ACPI, etc, etc. All having little to no
effect. The only thing that would right it is restarting the box but
over time it would degrade again. I talked to the SuperMicro and they
said that this is a FreeBSD issue and pretty much washed their hands of it.
After a couple of months of dealing with this and just rebooting the
systems reguarly, the symptoms slowly but surely disappeared. eg. The
kernel messages went away, the system time was not falling behind and I
was experiencing no packet loss but the "#vmstat -i | grep timer" value
would continue to decrease over time. Eventually I think, when it
finally got the 0 the machine restarted (I am only guessing here).
After this restart it worked again for a couple of hours and then it
After the second time the system has not missed a beat, it has been fine
and the "#vmstat -i | grep timer" value remained near the 2000 mark...
We setup some zabbix monitoring to watch it. As mentioned it was fine
for about a month. Until today. Today the value has dropped to 0, but
the system has not restarted and over the last couple of hours the value
has increased to 47.
This machine is mission critical, we have two in a fail over scenario
(using pfSense's CARP features) and it seems unfortunate that we have an
issue with two brand new SuperMicro boxes that affect both machines.
While at the moment everything seems fine I want to ensure that I have
no further issues. Does anyone have any suggestions?
Lastly I have double check both of the below:
We disabled EIST.
# dmesg | grep Timecounter
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec
# sysctl kern.timecounter.hardware
Only have one timer to choose from.
More information about the freebsd-stable