XEN 5.5.0 and clflush

[Kostik Belousov]

On Tue, Sep 22, 2009 at 01:19:18PM -0400, Larry Baird wrote:
> > > I think I will have to disable CLFLUSH support for intel CPUs when
> > > self-snoop
> > > is not reported.
> > >
> > >
> > That's the kinda weird part about this though... It's not triggering an
> > Invalid Instruction, but a GPF. Looking at AMD's description of how CLFLUSH
> > is supposed to work, I don't see why it's faulting with what looks like a
> > valid address.
> > 
> > While this is probably far outside the scope of what their entry-level
> > support techs will understand, I can try raising this as a bug with Citrix
> > under our support contract if you're confident that this is broken on Xen's
> > end.
> I keep wondering about comments having to do with AMD processor.  The
> native processor is an an Intel Xeon.  Is this piece of the puzzle
> important?

Ok, the whole story (mostly to be able to point out an URL with description
until the issue is sorted out).

FreeBSD got ability to specify page caching attributes in the page table
entries, so called Page Attribute Table (PAT). The physical to linear
page mapping may specify the caching attribute, like WB (write-back), UC
(uncacheable), WT (write-through) and so on.

The procedure to set the caching attribute for the page involves
removing old mapping, flushing TLB, creating new mapping, and then
flushing the cache line that could be allocated for the remmapped
physical memory. Flushing the cache may be performed by one of the three
mechanisms:
- do nothing; this is acceptable if CPU has so called Self-Snoop feature.
  For future discussion it is important to note that the feature is declared
  in the feature bitmap returned by CPUID instruction.
- do the series of CLFLUSH instructions over the linear region. This
  method should be used if CPU has CLFLUSH support, again reported
  by CPUID.
- do the WBINVD instruction that flushes the whole CPU cache.

The methods above are ordered from most efficient (do nothing) to very
time-consuming, and FreeBSD falls to the next method if previous is not
supported.

Real Intel CPUs (almost) all have all three methods implemented, and we
do nothing to flush cache since self snoop is there. AMD CPUs, at least
older models, do not have self-snoop, I believe, but do support CLFLUSH.
I got no reports of problems caused by CLFLUSH on the AMD.

If I manually disable self-snoop on Intel CPU, I get a reversed trap
when doing CLFLUSH over the mapping for APIC registers page. I do
not know the cause for this, my unfounded speculation is that there
is some cache line allocated for it, and it is written out on CLFLUSH.
Speculation is flimsy since BIOS should have programmed MTRR so
that APIC is UC mode.

Now, add the XEN to the picture. When whole machine virtualization
(how it is called by XEN ?) runs GENERIC/i386 kernel, it uses so
called VT feature of recent Intel CPUs, that, among other, gives
the hypervisor the ability to intercept CPUID instruction. XEN seems
to modify reported CPU features, removing self-snoop. As result. we
fall back to CLFLUSH method.

XEN seems to translate a fault caused by cache write-back to the APIC
registrers, into #gpf. This is the issue that you get on XEN.

So, I do not know why Intels cause trap on CLFLUSH, this may be either
some stupidness in my clflush code, or an additional factor that I
did not taken care of yet. Patch I sent disables CLFLUSH unconditionally,
that is bad for performance on AMD.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-xen/attachments/20090923/b7a7ef59/attachment.pgp