time issues and ZFS

[Daniel Braniss]

After many trials (and errors), here are some facts:

host: DELL PowerEdge R710, 16GB, 
 mfi0: <Dell PERC H700 Integrated>
 mfid0: 14305280MB (29297213440 sectors) RAID volume 'r5' is optimal
 mfi1: <Dell PERC 6> 
 mfid1: 12393472MB (25381830656 sectors) RAID volume 'Virtual Disk 0' is 
optimal

we have NO problems with FreeBSD-8.3-STABLE, but with 9.1-STABLE, the real-time
clock slows down when doing some zfs stuff like send|receive, typing 'date' 
when less that 1000s went by seems to crorrect the problem,
ntpd kicks in and on track again.

I have a cron job just logging date every 5 minutes, and the loghost sees:

|-- local time on loghost             | time on problematic host
Jan 20 19:56:19 store-02.cs.huji.ac.il Jan 20 19:56:19 danny: Sun Jan 20 
19:56:19 IST 2013	-- ok
Jan 20 20:15:00 store-02.cs.huji.ac.il Jan 20 20:15:00 danny: Sun Jan 20 
20:15:00 IST 2013	-- ok
Jan 20 21:30:00 store-02.cs.huji.ac.il Jan 20 20:21:06 danny: Sun Jan 20 
20:21:06 IST 2013	-- off by 1:09
Jan 20 21:33:53 store-02.cs.huji.ac.il Jan 20 20:25:00 danny: Sun Jan 20 
20:25:00 IST 2013	-- off by 1:08
Jan 20 21:38:54 store-02.cs.huji.ac.il Jan 20 20:30:00 danny: Sun Jan 20 
20:30:00 IST 2013	-- off by 1:09
...
Jan 20 22:03:54 store-02.cs.huji.ac.il Jan 20 20:55:00 danny: Sun Jan 20 
20:55:00 IST 2013	-- diff is now constant
..
Jan 20 22:04:13 store-02.cs.huji.ac.il Jan 20 20:55:19 ntpd[1848]: time 
correction of 4134 seconds exceeds sanity limit (1000); set clock manually to 
the correct UTC time.
...
Jan 20 22:58:53 store-02.cs.huji.ac.il Jan 20 21:50:00 danny: Sun Jan 20 
21:50:00 IST 2013


strangely, when running 8.3, ACPI-fast is chosen:
	kern.timecounter.choice: TSC(-100) HPET(900) ACPI-fast(1000) i8254(0) 
dummy(-1000000)
but with 9.1 TSC-low gets chosen:
	kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) HPET(950) i8254(0) 
dummy(-1000000)

so I did sysctl kern.timecounter.hardware=ACPI-fast, but the same happens - 
unless it can't be changed after boot.

I realy need help here!

thanks,
	danny