packet delay because of blackhole

[Anthony Pankov]

Hello Rui,

Tuesday, April 01, 2008, 2:41:29 PM, you wrote:

RP> On Fri, Mar 28, 2008 at 08:14:58PM +0300, Anthony Pankov wrote:
>> Just for somebody convince.
>> 
>> While analyzing client<->server HTTPS conversation one second delay in
>> packet exchange was discovered (strongly reproducible):
>> 
>> Sample:
>> N        time
>> 6       0.002303        10.28.4.14      10.28.4.50      SSL     Client Hello
>> 7       0.106710        10.28.4.50      10.28.4.14      TCP     443 > 1447 [ACK] Seq=1 Ack=103 Win=65535 Len=0
>> 8       1.045712        10.28.4.50      10.28.4.14      TLSv1   Server Hello, Certificate, Server Hello Done
>> 
>> Another sample:
>> 10      0.011722        10.28.4.14      10.28.4.50      TLSv1   Application Data
>> 11      0.115933        10.28.4.50      10.28.4.14      TCP     443 > 1442 [ACK] Seq=839 Ack=519 Win=65466 Len=0
>> 12      1.054037        10.28.4.50      10.28.4.14      TLSv1   Application Data
>> 
>> The reason for delay is sysctl tcp.blackhole value grater than 0, much to surprise.
>> 
>> So, turning tcp.blackhole to 0 eliminate any delay (strongly reproducible).
>> 
>> System: FreeBSD 6_2_stable


RP> I'm not sure how performance penalty can induce a cache miss and I
RP> it's very processor specific. So, you're best guess is to profile the
RP> kernel.

RP> Regards,

I'm not fully understand what cache miss do you mean.

I'll try to be more clear.

During client<->server HTTPS conversation there is a packet delay (see "sample" and
"Another sample") about 900 ms. Delay appear one per conversation in
random place (between 7-8 packet in "sample", 11-12 in "another
sample"). Of course, it's not depending from SSL session cache, SSL
negotiation or any other apache/mod_ssl/OpenSSL setting/performance,
otherwise i should write to another maillist.

I have disabled all my sysctl tuning, one by one. No effect has
achieved.
But when i turn tcp.blackhole to zero, all things became fine. Maximum
delay between packet is 6 ms.

It is strange, so i've reported to all.

I suggest to keep tcp.blackhole=0 and use firewall for protection. If
one would raise tcp.blackhole value, than he should dump packets and
make sure that there is no strange delay between packets.

It most likely FreeBSD net issue.

P.S. "Another sample" is not a sequel of "Sample", it is a dump of
different transaction.



-- 
Best regards,
 Anthony                            mailto:ap00 at mail.ru