time issues and ZFS

[John Nielsen]

On Jan 22, 2013, at 2:40 AM, Adrian Chadd <adrian at freebsd.org> wrote:

> On Jan 21, 2013, at 4:33 AM, Daniel Braniss <danny at cs.huji.ac.il> wrote:
> 
>> host: DELL PowerEdge R710, 16GB, 

I administer a Dell PowerEdge R710 and I've been seeing the exact same thing. It's currently running FreeBSD 9.0-STABLE #0 r236355. It has a ZFS pool which sees moderate load most of the time but can be very high at times (when certain scripts run, etc.). I hadn't previously correlated the issue with ZFS load but that is very possible.

I set a cron job to restart ntpd when it dies (because the time difference exceeds the sanity check). The cron job runs "every 20 minutes", but that varies greatly when the system stops counting. The time offset from ntpdate (which the script runs before restarting ntpd) varies a lot, but always in increments of 300 seconds. I've seen everything from 1200 to 23100. (Yes, that's 23 thousand seconds aka 6 hours 25 minutes that the system wasn't keeping time for.)

Sysctl kern.timecounter.hardware defaults to HPET. I experimented with setting it to ACPI-fast but the issue persisted so I put it back.
kern.timecounter.choice: TSC-low(-100) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000)

I first installed the box with an older 9.0-STABLE and this issue was not present. I have been tracking -STABLE on it (albeit irregularly) so I'm not sure when the issue came up.

> Have you run tests with the machdep.idle value changed, and fiddling
> kern.eventtimer.periodic / kern.eventtimer.idletick ?

I would love to resolve this and am able to do some experimenting. I've _usually_ been seeing the issue 2-3 times every 1-2 days, but I did just make some changes:
	disabling ZFS compression and deduplication on all pools
	updated to 9.1-STABLE from yesterday (r245821)

If the issue persists I will try changing some of the sysctls above and follow up with the result. If it goes away, I'll try to remember to report that too.

JN