Invalid memory stats from vmstat and sysctl vm.vmtotal?

[John Baldwin]

On 12/1/11 1:05 PM, Jason Hellenthal wrote:
>
> On Thu, Dec 01, 2011 at 10:44:58AM -0000, Steven Hartland wrote:
>> ----- Original Message -----
>> From: "Jason Hellenthal"<jhell at DataIX.net>
>>
>>> This goes along with the thoughts I had about 4 months ago tending to some
>>> zfs statistics as well top showing greater than 100% actual CPU usage. This
>>> is a big pet peave of mine. Its like saying you ate 134% of a bannanna when
>>> in all reallity it is impossible. You can never have more than 100% usage of
>>> anything and when seen is a clear notice that some math is considerably
>>> incorrect leading to other such miscalculations to be performed. Things like
>>> the above already have checks in place that ensure no boundries are being
>>> crossed/overflowed or underrun but it surely makes processing results building
>>> future products a bitch. One instance is the calculation of threads for example
>>> firefox can be seen using upto or more 338% of the CPU. Thats impossible its
>>> like saying anyones CPU grew by 400%.
>>
>> I could understand a bit of overflow as stats are snapshots which may not
>> be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue /
>> overflow.
>
> I agree
>
>>
>> With respect to top showing greater than 100% by how much are you talking?
>> Do your realise that each core = 100%? So if you have a quad core your system
>> total will be 400% not 100%?
>>
>
> Yeah I realize that but it still would lead you to believe that if a proccessor has 4 cores on the same die then total for each core could only be 25% usage. And the usage for a proccess only consuming full usage of 1 core is 100%. But you can start firefox on a single uniproccessor and like stated above see large usage percents near 338% or greater which is impossible and leads me to believe were forcing calculation for the entire proccess of threads onto tthread 0. This makes accounting pretty difficult.

A single-package machine with 4 cores on the die is not a uniprocessor 
machine.  It is an SMP machine.  Try booting a kernel without SMP, 
_that_ will give you a UP machine.  The %CPU usage is actually very 
simple if you stop trying to make it so complicated.  Each potential 
concurrent thread of execution is mapped to 100%.  Thus, if you have a 
system with 16 potential threads (either due to 4 quad-core packages, or 
2 quad-core package where each core has 2 threads via HTT, etc.), then 
you have a total CPU usage of 1600%.  Why does this make sense?  Because 
a machine with 16 concurrent threads can (theoretically) do 16x the work 
of an otherwise identical machine with a single thread of execution 
within a given unit of wall time.

Another way to look at it is that in FreeBSD, each thread of execution 
is treated as a "CPU" in top, etc.  Thus, 100% CPU means that a given 
thread is using all of the available cycles on a CPU.  If you have a 
multithreaded app (like Firefox) that is using all of the available 
cycles on 4 CPUs, then that would be 400% CPU (it's using 4 CPUs).  It 
may only be using 25% of the available system-wide CPU cycles, but that 
is not what %CPU measures.

Anyways, pretty much everyone I've ever talked to about this gets it 
right away, so the current arrangement is fine for the majority of 
folks.  It has also been that way since FreeBSD first added SMP over a 
decade ago.  It is also true on other OS's such as OS X, so the current 
arrangement is here to stay.

-- 
John Baldwin