svn commit: r345359 - in head/sys/cddl/dev/dtrace: amd64 i386

[Bruce Evans]

On Thu, 21 Mar 2019, Bruce Evans wrote:

> On Wed, 20 Mar 2019, Mark Johnston wrote:
>
>> On Wed, Mar 20, 2019 at 08:20:18PM -0700, Rodney W. Grimes wrote:
>>>> Author: markj
>>>> Date: Thu Mar 21 02:52:22 2019
>>>> New Revision: 345359
>>>> URL: https://svnweb.freebsd.org/changeset/base/345359
>>>> 
>>>> Log:
>>>>   Don't attempt to measure TSC skew when running as a VM guest.
>>>> 
>>>>   It simply doesn't work in general since VCPUs may migrate between
>>>>   physical cores.  The approach used to measure skew also doesn't
>>>>   make much sense in a VM.
>
> I think it simply doesn't work.
>
> [... lots]

I don't normally use dtrace, but built a kernel with it to test this, and
got the expected garbage for skew values on SandyBridge and Haswell systems.

First, the build was broken on at least i386:
- i386 NOTES only has KTRACE_HOOKS.  All other dtrace options (devices) are
   commented out.  This defeats the reason for existence of NOTES.  It is
   not to supply documenation, but to test everything except negative options
   using LINT.
- since dtrace was not tested in LINT, of course it doesn't compile
- "device dtrace" doesn't compile, since it depends on the "optional"
   "module" "device fbt" for 1 symbol
- "device fbt" doesn't compile, since kern.pre.mk has an invalid -I path
   for it, and at least gcc considers invalid -I paths as errors.

Fix for the -I path:

XX Index: kern.pre.mk
XX ===================================================================
XX --- kern.pre.mk	(revision 345315)
XX +++ kern.pre.mk	(working copy)
XX @@ -202,9 +202,13 @@
XX  DTRACE_S=	${CC} -c ${DTRACE_ASM_CFLAGS}	${WERROR} ${.IMPSRC}
XX 
XX  # Special flags for managing the compat compiles for DTrace/FBT
XX -FBT_CFLAGS=	-DBUILDING_DTRACE -nostdinc -I$S/cddl/dev/fbt/${MACHINE_CPUARCH} -I$S/cddl/dev/fbt -I$S/cddl/compat/opensolaris -I$S/cddl/contrib/opensolaris/uts/common -I$S ${CDDL_CFLAGS}
XX +FBT_CFLAGS=	-DBUILDING_DTRACE -nostdinc -I$S/cddl/dev/fbt \
XX +		-I$S/cddl/compat/opensolaris \
XX +		-I$S/cddl/contrib/opensolaris/uts/common -I$S ${CDDL_CFLAGS}
XX  .if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386"
XX  FBT_CFLAGS+=	-I$S/cddl/dev/fbt/x86
XX +.else
XX +FBT_CFLAGS+=	-I$S/cddl/dev/fbt/${MACHINE_CPUARCH}
XX  .endif
XX  FBT_C=		${CC} -c ${FBT_CFLAGS}		${WERROR} ${PROF} ${.IMPSRC}
XX

This also fixes some style bugs (a too-long line).  kern.pre.mk has about 12
other too-long lines, with about half of these for cddl.

Perhaps the order of the -I paths matters, but this worked for me.

I once eliminated almost all -I paths in kernel Makefiles.  There are about
100 times as many now as before I removed some :-(.  At least the -I maze
for FBT doesn't affect other subsystems.

Results of calculating skews:

On SandyBridge i5-2560M 2 cores * 2 threads:

               boot1  boot2  boot3  boot4
               -----  -----  -----  -----
tsc_skew[0] =     0      0      0      0
tsc_skew[1] =  0x40   0x58   0x5c   0x8c
tsc_skew[2] =  0x68   0x68   0x7c   0x7c
tsc_skew[3] = 0x118   0x98  0x104  0x104

On Haswell i7-4790K 4 cores * 2 threads:

               boot1  boot2  boot3  boot4
               -----  -----  -----  -----
tsc_skew[0] = 0          0      0      0
tsc_skew[1] = 0x40    0x40   0x40   0x30
tsc_skew[2] = 0x40   -0x40   0x54   0x5c
tsc_skew[3] = 0xf0    0xf4   0x74   0x58
tsc_skew[4] = 0xdc    0x60   0x6c  0x138
tsc_skew[5] = 0x38    0x6c   0x38   0x38
tsc_skew[6] = 0x4c    0x38   0xe0   0xb0
tsc_skew[7] = 0x3c    0x3c  -0x38   0x38

This looks like IPC delays with jitter from sloppy measurements.  Nothing
to do with skews, except it approximates 0.

Bruce