packet delay because of blackhole

Anthony Pankov ap00 at
Tue Apr 1 07:03:49 PDT 2008

Hello Rui,

Tuesday, April 01, 2008, 2:41:29 PM, you wrote:

RP> On Fri, Mar 28, 2008 at 08:14:58PM +0300, Anthony Pankov wrote:
>> Just for somebody convince.
>> While analyzing client<->server HTTPS conversation one second delay in
>> packet exchange was discovered (strongly reproducible):
>> Sample:
>> N        time
>> 6       0.002303      SSL     Client Hello
>> 7       0.106710      TCP     443 > 1447 [ACK] Seq=1 Ack=103 Win=65535 Len=0
>> 8       1.045712      TLSv1   Server Hello, Certificate, Server Hello Done
>> Another sample:
>> 10      0.011722      TLSv1   Application Data
>> 11      0.115933      TCP     443 > 1442 [ACK] Seq=839 Ack=519 Win=65466 Len=0
>> 12      1.054037      TLSv1   Application Data
>> The reason for delay is sysctl tcp.blackhole value grater than 0, much to surprise.
>> So, turning tcp.blackhole to 0 eliminate any delay (strongly reproducible).
>> System: FreeBSD 6_2_stable

RP> I'm not sure how performance penalty can induce a cache miss and I
RP> it's very processor specific. So, you're best guess is to profile the
RP> kernel.

RP> Regards,

I'm not fully understand what cache miss do you mean.

I'll try to be more clear.

During client<->server HTTPS conversation there is a packet delay (see "sample" and
"Another sample") about 900 ms. Delay appear one per conversation in
random place (between 7-8 packet in "sample", 11-12 in "another
sample"). Of course, it's not depending from SSL session cache, SSL
negotiation or any other apache/mod_ssl/OpenSSL setting/performance,
otherwise i should write to another maillist.

I have disabled all my sysctl tuning, one by one. No effect has
But when i turn tcp.blackhole to zero, all things became fine. Maximum
delay between packet is 6 ms.

It is strange, so i've reported to all.

I suggest to keep tcp.blackhole=0 and use firewall for protection. If
one would raise tcp.blackhole value, than he should dump packets and
make sure that there is no strange delay between packets.

It most likely FreeBSD net issue.

P.S. "Another sample" is not a sequel of "Sample", it is a dump of
different transaction.

Best regards,
 Anthony                            mailto:ap00 at

More information about the freebsd-performance mailing list