igb fc (aka flowcontrol) issue...

From: John-Mark Gurney <jmg_at_funkthat.com>
Date: Mon, 02 Aug 2021 01:58:52 UTC
I have a dual port igb card:
igb0: <Intel(R) PRO/1000 82576> port 0x2020-0x203f mem 0xd1020000-0xd103ffff,0xd0c00000-0xd0ffffff,0xd1044000-0xd1047fff irq 17 at device 0.0 on pci3
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: xxxx
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) PRO/1000 82576> port 0x2000-0x201f mem 0xd1000000-0xd101ffff,0xd0800000-0xd0bfffff,0xd1040000-0xd1043fff irq 18 at device 0.1 on pci3
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: xxxx
igb1: netmap queues/slots: TX 4/1024, RX 4/1024

And I have observed an issue with it when adjusting flow control, it
seems to adjust both ports at the same time:
# sysctl dev.igb.0.fc
dev.igb.0.fc: 3
# sysctl dev.igb.1.fc
dev.igb.1.fc: 3
# sysctl dev.igb.0.fc=0
dev.igb.0.fc: 3 -> 0
# sysctl dev.igb.1.fc
dev.igb.1.fc: 0

Is this correct behavior?

Also, the fc sysctl is not documented, I propose the following:
diff --git a/share/man/man4/em.4 b/share/man/man4/em.4
index a1fa22c..bad82e9 100644
--- a/share/man/man4/em.4
+++ b/share/man/man4/em.4
@@ -265,6 +265,26 @@ If
 is non-zero, this tunable limits the maximum delay in which a transmit
 interrupt is generated.
 .El
+.Sh SYSCTL VARIABLES
+The following variable is availabel:
+.Bl -tag -width "dev.em.X.fc"
+.It Va dev.em.X.fc
+This sysctl sets the flow control for the card (this means both ports
+on a dual port card).  The values are as follows:
+.Bl -tag -width "XX" -offset "XXXX" -compact
+.It 0
+Disabled
+.It 1
+Process received pause frames, do not transmit them.
+.It 2
+Send pause frames, but to not process received frames.
+.It 3
+Process and send pause frames.
+.It 4
+No software override, use EEPROM configuration.
+.El
+.El
+Note: That the variable is available for igb as well.
 .Sh FILES
 .Bl -tag -width /dev/led/em*
 .It Pa /dev/led/em*

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."