[PATCH] hwpmc(4) changes to use 'mp_maxid' instead of 'mp_ncpus'.

[John Baldwin]

On Saturday 15 March 2008 01:43:00 am Joseph Koshy wrote:
> >  FreeBSD has been trying to not be quite as i386-centric as it used to
> > be.  If you look at other code in the kernel that handles per-cpu data
> > such as UMA you will see that it uses mp_maxid and CPU_ABSENT().  There
> > are other places in the kernel that are broken though (such as ndis(4)).
>
> HWPMC is very x86 centric, for obvious reasons.

Considering other CPU archictectures support various performance counters it 
really shouldn't be designed to be x86-centric even if it is currently only 
implemented for x86 CPUs.

> >  > - Will sysctl hw.ncpus represent the count of present CPUs or will it
> >  >   represent the maximum CPU id?
> >
> >  hw.ncpus is always mp_ncpus
> >  kern.smp.cpus is also mp_ncpus
> >  kern.smp.maxcpus is MAX_CPUS.
> >
> >  Userland can just iterate from 0 to kern.smp.maxcpus while handling
> > absent CPUs.  (For example, the kern.cp_time[] sysctl just writes out all
> > 0's for absent CPUs so that is how userland can determine an absent CPU
> > in that case.)
>
> I thought of that.   For PMCTools use, using the proposed 'online_cpus'
> mask would be a better option.   MAX_CPUS is a compile time value and could
> be large, whereas most machines will have far fewer CPUs than that limit.
> Why waste cycles needlessly?

Userland cycles are "cheaper". :)  I think having both is fine and userland 
can choose which to use (maxcpus is probably easier to impl but perhaps less 
efficient).

> Now it appears to me that in the scheme of things described
> above one of mp_maxid and mp_ncpus is superfluous.
>
> Here is the reasoning:
>
> 0) We need a compile time limit for the kernel; this is kern.smp.maxcpus.
>
> 1) A given machine has a maximum number of CPUs that can fit in it.
>    This is usually <<= MAXCPUS.     Let us call this {MACHINE-MAX}.
>    We need to scale kernel data structures based on {MACHINE-MAX}
>    since using {MAXCPUS} is probably wasteful.  We cannot just count the
>    current number of CPUS, as we do today, because more could be
>    hotplugged in later.
>
> 2) At any given instant a subset of CPUs 0..{MACHINE_MAX} will be
>     online. This would be tracked by the kern.smp.online_cpus/all_cpus
>     bitmask.
>
> Therefore we can use either a count (mp_ncpus) or a maximum id
> (mp_maxid) to represent {MACHINE-MAX}, but  either one would do.
>
> However, x86 MD code uses both, with newer code seeming to prefer
> mp_maxid.  So I am puzzled.  There are far more uses of mp_ncpus
> there though.

The mp_ncpus uses are mostly bugs (e.g. ndis).  I think mp_ncpus's primary use 
is for userland so people can do:

	make -j $(sysctl hw.ncpus)

or the like.  That is, if you need a simple count of CPUs in the system, that 
is what mp_ncpus is for.  If you need to address invididual CPUs by ID, then 
mp_ncpus is not appropriate and you need to iterate from 0 to mp_maxid 
suitable to some bitmask (e.g. all_cpus via CPU_ABSENT, or a 
not-yet-implemented onlines_cpus wth CPU_ONLINE/CPU_OFFLINE wrappers).

> jk> Changing HWPMC and its userland before the base kernel itself
> jk> changes does not seem to be the right thing to do.
>
> jb>  While the userland intIerface is somewhat lacking, all of the
> in-kernel jb>  infrastructure has been in place for at least the past 4
> years, and there is
> jb>  no excuse for any in-kernel code not properly handling sparse CPU IDs.
>
> I try keep userland, kernel and documentation associated with PmcTools
> in sync.
>
> Looking around, there appear to be lots of nits that need correction.
> For one,  the kern.smp sysctl hierarchy is undocumented.

Not entirely:

> sysctl -d kern.smp
kern.smp: Kernel SMP
kern.smp.maxcpus: Max number of CPUs that the system was compiled for.
kern.smp.active: Number of Auxillary Processors (APs) that were successfully 
started
kern.smp.disabled: SMP has been disabled from the loader
kern.smp.cpus: Number of CPUs online

(On a UP 6.3 box)

-- 
John Baldwin