pf and hnX interfaces

[Kristof Provost]

On 13 Oct 2020, at 10:58, Eugene M. Zheganin wrote:
> I'm running a FreeBSD 12.1 server as a VM under Hyper-V. And although 
> this letter will make an impression of another lame post blaming 
> FreeBSD for all of the issues while the author should blame himselm, 
> I'm atm out of another explanation. The thing is: I'm getting loads of 
> sendmail errors like:
>
>
> ===Cut===
>
> Oct 13 13:49:33 gw1 sm-mta[95760]: 09D8mN2P092173: SYSERR(root): 
> putbody: write error: Permission denied
> Oct 13 13:49:33 gw1 sm-mta[95760]: 09D8mN2P092173: SYSERR(root): 
> timeout writing message to <whatever>.mail.protection.outlook.com.: 
> Permission denied
>
> ===Cut===
>
A “Permission denied” on outbound packets can indeed happen when pf 
decides to block the packet.

> The relay address is just random. The thing is, I can successfully 
> connect to it via telnet. Even send some commands. But when this is 
> done by senamil - and when it's actually sending messages, I get 
> random errors. Firstly I was blaming myself and trying to get the rule 
> that actually blocks something. I ended up having none of the block 
> rules without log clause, and in the same time tcpdump -netti pflog0 
> shows no droppen packets, but sendmail still eventually complains.
>
> If it matters, I have relatively high rps on this interface, about 25 
> Kpps.
>
> I've also found several posting mentionsing that hnX is badly handling 
> the TSO and LRO mode, so I switched it off. No luck however, with 
> vlanhwtag and vlanmtu, which for some reason just cannot be switched 
> off. the if_hn also lacks a man page for some reason, so it's unclear 
> how to tweak it right.
>
While it’s possible that there are issues with TSO/LRO those 
wouldn’t look like this. (As an aside, I am interested in any 
reproducible setups where pf has issues with TSO/LRO. As far as I’ve 
been able to see all such issues have been resolved.)

> And the most mysterious part  - when I switch the pf off, the errors 
> stops to appear. This would clearly mean that pf blocks some packets, 
> but then again, this way the pflog0 would show them up, right (and yes 
> - it's "UP" )?
>
It’s possible for pf to drop packets without triggering log rules. For 
example, if pf decides to drop the packet before it matches any rule 
(e.g. it’s a corrupt packet) it won’t show up in pflog.

> Is there some issue with pf and hn interfaces that I'm unaware about?
>
There’s no interface specific code in pf, so it wouldn’t be specific 
to hn interfaces.

> Are these symptoms of a bug ?
>
Perhaps. It can also be a symptom of resource exhaustion.
Are there any signs of memory allocation failures, or incrementing error 
counters (in netstat or in pfctl)?

Best regards,
Kristof