Re: Measuring performance impact of tracing a system under load

> On Oct 18, 2021, at 2:32 PM, Mateusz Piotrowski <0mp@FreeBSD.org> wrote:
> 
> Hello everyone,
> 
> I would like to measure the overhead / performance impact that of tracing with DTrace has on a system under load. I'm looking for the right way to do it. Obviously, I should not be using DTrace to measure that in this situation. I could use other stats(7) tools but it does not seem. Measuring the impact indirectly sounds like a good idea (e.g., by looking at the throughput between two machines in a test lab as they transfer bytes from A to B in two scenarios: with active tracing and without). Are there any other options? Do you have any insights to share regarding measuring DTrace overhead?
> 
> I'd be grateful for all suggestions and pointers.
> 

I have done this for several sub-systems when I had to — for work — show the progress of bpftrace against DTrace and formulate plans for sampling strategies based on observed overhead.

Running in one shell:

dd if=/dev/urandom of=/tmp/trashfile bs=1 count=5000000

And another [Linux] shell:

bpftrace -e 'tracepoint:syscalls:sys_enter_read { }’

This resulted in a 23% performance hit on the dd while bpftrace was running. We ran the test 10x and on multiple pieces of hardware, under multiple Operating Systems, and tried our best to control the testing environment parameters.

The hit on FreeBSD was barely noticeable. I think I calculated it as less than 6% at its worst.

By the way, did you know you can run DTrace on CentOS?

cd /etc/yum.repos.d
wget http://public-yum.oracle.com/public-yum-ol7.repo
yum install -y kernel-uek{,-devel} dtrace-utils
# EFI
cp /boot/efi/EFI/centos/grub.cfg{,.`date +%s`.bak}
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
# BIOS
grubby --set-default-index 0
reboot

ASIDE: And it’s actually fun to use and quite powerful. Back to the question at-hand though …

I have done similar things with “nc” in place of “dd”

I also suggest checking out “dpv” in FreeBSD base for rate monitoring with and with-out tracing active to gain insights on overhead.
— 
Devin