Synchronizing TSC

[Konstantin Belousov]

On Wed, Apr 17, 2013 at 09:46:15AM +0300, Alexander Motin wrote:
> On 17.04.2013 03:25, Jim Harris wrote:
> >
> > On Tue, Apr 16, 2013 at 3:04 PM, Alexander Motin <mav at freebsd.org
> > <mailto:mav at freebsd.org>> wrote:
> >
> >     Hi.
> >
> >     Recently I've got 6-core/12-thread system on Sandy Bridge-E Core
> >     i7-3930K CPU and was unpleasantly surprised to see that TSCs are not
> >     synchronized there. While all 11 APs were synchronized, BSP was far
> >     behind them. Since it is single-socket system, I don't know any good
> >     reason for such behavior except some BIOS bug. But I've recalled
> >     that somewhere was some discussions about possible TSC
> >     synchronization. I've implemented patch below that allows to adjust
> >     TSC values of BSPs to AP's one on boot using CPU MSRs, hoping that
> >     they should not diverge after that:
> >     http://people.freebsd.org/~__mav/tsc_adj2.patch
> >     <http://people.freebsd.org/~mav/tsc_adj2.patch>
> >
> >     I don't know very much about all different TSC hardware to predict
> >     when it is safe to enable the functionality, but at least on my
> >     system being enabled via loader tunable it seems working well.
> >
> >     Comments?
> >
> >
> > You may be remembering this thread on r238755 last year:
> >
> > http://lists.freebsd.org/pipermail/svn-src-head/2012-July/038992.html
> >
> > This was a bug fix in the TSC synchronization test code though, not
> > anything for trying to adjust out-of-sync TSCs.
> 
> I remember that thread, but I think I've seen somebody told somewhere 
> that it could be interesting to implement some MI mechanism. Never mind.
> 
> > The Intel SDM (volume 3, section 17.13 of March 2013 revision) says
> > earlier models can only write to lower 32 bits of
> > IA32_TIME_STAMP_COUNTER, but these models also should not have invariant
> > TSC so they would never even get to your new routine.  So your patch
> > seems OK for Intel CPUs, at least as a tunable that is disabled by default.
> 
> Thanks.
> 
> > My only concern would be why TSC on the BSP started out-of-sync on your
> > system.  Theoretically, BIOS could adjust TSCs in SMM to try to hide SMI
> > code execution from the OS, which could then make them out-of-sync
> > again.  Not sure if that's what's happening here, but might be worth a
> > test putting the TSC test code on a periodic timer to see if they ever
> > get out of sync again.
> 
> I did one more interesting observation: on every reboot drift between 
> BSP and APs is growing proportionally to the previous system power-on 
> time. On first boot it is -3878361036 (just above one second), after 
> reboot some minutes later it is -1123454492776 (about 6 minutes), after 
> another reboot it is -1853033521804 (about 10 minutes).
> 
> Unless my adjustment code would be active, I would guess that AP's TSC 
> is running linearly while BSP's for some reason reset to zero on every 
> reboot. But since I am synchronizing them on each boot, the only 
> possibility for it I see is that there is some other timer(s) / 
> counter(s) not affected by MSR writes that ticks linearly and reloading 
> AP's TSC, but for some reason not reloading BSP's.

For me it sounds as the BIOS bug, indeed. Could you verify the content
of IA32_TSC_ADJUST on all cores (I believe it is present on E5) ?
Also, using TSC_ADJUST to correct the skew seems to be preferrable,
according to the Intel docs.

Why do you use cpuid in the assembly sequence ? As I understand, you
ensure that there is a serialization point, but why do you need it ?

The common knowledge is that for CPUs with invariant TSC, the TSC
counter is single-instance and located on uncore. For single-socket
configurations, your patch would be fine. But, for multi-socket
machines, each package has its own counter, and counters might drift.
As result, the initial synchronization would still allow the eventual
de-sync and this is problematic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20130417/5f52ffa2/attachment.sig>