svn commit: r319744 - vendor/illumos/dist/lib/libdtrace/common

[Andriy Gapon]

Author: avg
Date: Fri Jun  9 15:04:10 2017
New Revision: 319744
URL: https://svnweb.freebsd.org/changeset/base/319744

Log:
  8269 dtrace stddev aggregation is normalized incorrectly
  
  illumos/illumos-gate at 79809f9cf402f130667349b2d4007ecd65d63c6f
  https://github.com/illumos/illumos-gate/commit/79809f9cf402f130667349b2d4007ecd65d63c6f
  
  https://www.illumos.org/issues/8269
    It seems that currently normalization of stddev aggregation is done
    incorrectly.
    We divide both the sum of values and the sum of their squares by the
    normalization factor. But we should divide the sum of squares by the
    normalization factor squared to scale the original values properly.
  
  Reviewed by: Bryan Cantrill <bryan at joyent.com>
  Approved by: Robert Mustacchi <rm at joyent.com>
  Author: Andriy Gapon <avg at FreeBSD.org>

Modified:
  vendor/illumos/dist/lib/libdtrace/common/dt_consume.c

Modified: vendor/illumos/dist/lib/libdtrace/common/dt_consume.c
==============================================================================
--- vendor/illumos/dist/lib/libdtrace/common/dt_consume.c	Fri Jun  9 15:03:07 2017	(r319743)
+++ vendor/illumos/dist/lib/libdtrace/common/dt_consume.c	Fri Jun  9 15:04:10 2017	(r319744)
@@ -381,8 +381,10 @@ dt_stddev(uint64_t *data, uint64_t normal)
 	 * The standard approximation for standard deviation is
 	 * sqrt(average(x**2) - average(x)**2), i.e. the square root
 	 * of the average of the squares minus the square of the average.
+	 * When normalizing, we should divide the sum of x**2 by normal**2.
 	 */
 	dt_divide_128(data + 2, normal, avg_of_squares);
+	dt_divide_128(avg_of_squares, normal, avg_of_squares);
 	dt_divide_128(avg_of_squares, data[0], avg_of_squares);
 
 	norm_avg = (int64_t)data[1] / (int64_t)normal / (int64_t)data[0];