svn commit: r237434 - in head/lib/libc: amd64/sys gen i386/sys include sys

Bruce Evans brde at optusnet.com.au
Sat Jun 23 15:26:25 UTC 2012


On Sat, 23 Jun 2012, Konstantin Belousov wrote:

> On Sat, Jun 23, 2012 at 03:17:57PM +0200, Marius Strobl wrote:
>> On Fri, Jun 22, 2012 at 10:48:17AM +0300, Konstantin Belousov wrote:
>>> On Fri, Jun 22, 2012 at 09:34:56AM +0200, Marius Strobl wrote:
>>>> On Fri, Jun 22, 2012 at 07:13:31AM +0000, Konstantin Belousov wrote:
>>>>> Author: kib
>>>>> Date: Fri Jun 22 07:13:30 2012
>>>>> New Revision: 237434
>>>>> URL: http://svn.freebsd.org/changeset/base/237434
>>>>>
>>>>> Log:
>>>>>   Use struct vdso_timehands data to implement fast gettimeofday(2) and
>>>>>   clock_gettime(2) functions if supported. The speedup seen in
>>>>>   microbenchmarks is in range 4x-7x depending on the hardware.
>>>>>
>>>>>   Only amd64 and i386 architectures are supported. Libc uses rdtsc and
>>>>>   kernel data to calculate current time, if enabled by kernel.
>>>>
>>>> I don't know much about x86 CPUs but is my understanding correct
>>>> that TSCs are not synchronized in any way across CPUs, i.e.
>>>> reading it on different CPUs may result in time going backwards
>>>> etc., which is okay for this application though?
>>>
>>> Generally speaking, tsc state among different CPU after boot is not
>>> synchronized, you are right.
>>>
>>> Kernel has somewhat doubtful test which verifies whether the after-boot
>>> state of tsc looks good. If the test fails, TSC is not enabled by
>>> default as timecounter, and then usermode follows kernel policy and
>>> falls back to slow syscall. So we err on the safe side.
>>> I tested this on Core i7 2xxx, where the test (usually) passes.
>>
>> Okay, so for x86 the TSCs are not used as timecounters by either
>> the kernel or userland in the SMP case if they don't appear to
>> be synchronized, correct?
> Correct as for now.  But this is bug and not a feature. The tscs shall
> be synchronized, or skew tables calculated instead of refusing to use it.
>>>
>>> While you are there. do you have comments about sparc64 TICK counter ?
>>> On SMP, the counter of BSP is used by IPI. Is it unavoidable ?
>>
>> The TICK counters are per-core and not synchronized by the hardware.
>> We synchronize APs with the BSP on bring-up but they drift over time
>> and the initial synchronization might not be perfect in the first
>> place. At least in the past, drifting TICK counters caused all sorts
>> of issues and strange behavior in FreeBSD when used as timecounter
>> in the SMP case. If my understanding of the above is right, as is
>> this still rules them out as timecounters for userland.
>> Linux has some complex code (based on equivalent code origining in
>> their ia64 port) for constantly synchronizing the TICK counters.
>> In order to avoid that complexity and overhead, what I do in
>> FreeBSD in the SMP case is to (ab)use counters (either intended

Attempted synchronization of TSCs is left out for the same reason on x86.
Except some half-baked synchronization for a home made time function in
dtrace (dtrace_gethrtime() on amd64 and i386) crept in.

>> for that purpose or bus cycle counters probably intended for
>> debugging the hardware during development) available in the
>> various host-to-foo bridges so it doesn't matter which CPU they
>> are read by. This works just fine except for pre-PCI-Express
>> based USIIIi machines, where the bus cycle counters are broken.
>> That's where the TICK counter is always read from the BSP
>> using an IPI in the SMP case. The latter is done as sched_bind(9)
>> isn't possible with td_critnest > 1 according to information
>> from jhb@ and mav at .

How can it work fine?  Buses are too slow.  On x86, ACPI-fast takes
700-1900 nsec on machines that I've tested (mostly pre-PCIe ones).
HPET seems to be only slightly faster (maybe 500 nsec).

>> So apart from introducing code to constantly synchronize the
>> TICK counters, using the timecounters on the host busses also
>> seems to be the only viable solution for userland. The latter
>> should be doable but is long-winded as besides duplicating
>> portions of the corresponding device drivers in userland, it
>> probably also means to get some additional infrastructure
>> like being able to memory map registers for devices on the
>> nexus(4) level in place ...

There is little point in optimizations to avoid syscalls for hardware.
On x86, a syscall takes 100-400 nsec extra, so if the hardware takes
500-2000 nsec then reduction the total time by 100-400 nsec is not
very useful.

> Understand. I do plan eventually to map HPET counters page into usermode
> on x86.

This should be left out too.

> Also, as I noted above, some code to synchronize per-package counters
> would be useful for x86, so it might be developed with multi-arch
> usage in mind.

It's only worth synchonizing fast timecounter hardware so that it can be
used in more cases.  It probably needs to be non-bus based to be fast.
That means the TSC on x86.

The new timeout code to support tickless kernels looks like it will give
large pessimizations unless the timecounter is fast.  Instead of using
the tick counter (1 atomic increment on every clock tick) and some
getbinuptime() calls in places like select(), it uses the hardware
timecounter via binuptime() in most places (since without a tick counter
and without clock interrupts updating the timehands periodically, it takes
a hardware timecounter read to determine the time).  So callout_reset()
might start taking thousands of nsec for per call, depending on how slow
the timecounter is.  This fix is probably to use a fuzzy time for long
long timeouts and to discourage use of short timeouts and/or to turn them
into long or fuzzy timeouts so that they are not very useful.

Bruce


More information about the svn-src-head mailing list