Timing issue with Dummynet on high kernel timer interrupt

[Rasool Al-Saadi]

Hello all,

While we were implementing new AQMs (Codel, PIE, FQ-Codel and FQ-PIE) in FreeBSD targeting ipfw+Dummynet, we noticed timing issue with Dummynet when we increase kernel timer interrupt Hz (changing the value of kern.hz in /boot/loader.conf) to 3000 or higher. The issue is that spikes (upward and downward) appear in RTT when we use FreeBSD+ipfw+Dummynet as a router with high Hz value.

We did simple experiment (configuration listed below) with six runs and Hz={1000, 2000, 3000, 3250, 3500, 4000} to demonstrate the issue. I have attached two RTT vs Time graphs for our experiments. You can see in these graphs everything is smooth when Hz ~ <=3000, and the issue starts to appear when Hz ~>=3250.

As part of figuring out why this happen, I found that removing C_DIRECT_EXEC and C_HARDCLOCK flags from callout_reset_sbt() call(in dn_reschedule() function) causes the issue with high kernel interrupt hz (>3000 in my case) to disappear even when kern.hz=10000.

I did another two experiment (kern.hz=10000, with and without the flags mentioned above) and I logged dn_cfg.curr_time from dummynet_task() (after the adjustments code). Then I calculated the difference (delta) between curr_time[i] and curr_time[i-1] of the log file (the logged dn_cfg.curr_time). I found that when kern.hz=10000 and C_DIRECT_EXEC and C_HARDCLOCK flags are set, the delta values vary from 0 to ~600,  while when C_DIRECT_EXEC and C_HARDCLOCK flags are unset, the delta values vary from 1 to ~80. It seems like the high jumps of delta values when the flags are set causes the issue. I have attached a graph that compares delta values for the two experiments.

Does anyone have thoughts on what we can test next to narrow down the root-cause of these unusual timing jumps?

Our configurations:
Router PC: Intel(R) Core(TM)2 Duo @ 2.33GHz processer, 4GiB RAM.
       FreeBSD 10.1 (also I tried FreeBSD11-CURRENT-r289420).       
ipfw+Dummynet: 10mbit traffic shaping , emulated delay 30ms (RTT=60ms), queue size is 100 slices, droptail
Hosts: Linux 3.17
Traffic: One TCP flow, iperf,  TCP NewReno
NICs: Intel PRO/1000 GT
Testbed experiments under the control of TEACUP (http://caia.swin.edu.au/tools/teacup/)

Regards,
Rasool Al-Saadi