bhyve, ixgbe and debian 8

Mon Nov 7 14:13:02 UTC 2016

On 11/07/2016 07:41 AM, Rajil Saraswat wrote:
> On 11/07/2016 07:08 AM, Rajil Saraswat wrote:
>> Hello,
>>
>> I have PCI passthrough working now for the Intel X540-AT2 NIC and is
>> seen inside debian 8 vm.
>>
>> # lspci
>>
>> 00:00.0 Host bridge: Network Appliance Corporation Device 1275
>> 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
>> 00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
>> 00:06.0 Ethernet controller: Intel Corporation Ethernet Controller
>> 10-Gigabit X540-AT2 (rev 01)
>> 00:1f.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
>>
>>
>> However, on loading the module the kernel gives the error
>>
>> #modprobe ixgbe
>>
>> [   34.929546] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
>> version 3.19.1-k
>> [   34.929549] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
>> [   34.929706] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
>> [   34.929708] ixgbe 0000:00:06.0: PCI INT A: no GSI
>> [   34.930354] ixgbe 0000:00:06.0: Adapter removed
>> [   34.931710] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
>> [   34.931717] ixgbe: probe of 0000:00:06.0 failed with error -5
>>
>> Any idea what could be the issue?
>>
>> Thanks,
>>
>> Rajil
>>
> I tried with the second onboard NIC and it worked the first time. iperf
> showed 10g speeds. After restarting the VM, i get an error for both
> NICs. This a dual CPU motherboard Supermicro X10DriT with a pair of
> onboard NICs.
>
> [    0.628758] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
> version 3.19.1-k
> [    0.628760] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
> [    0.628899] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
> [    0.628901] ixgbe 0000:00:06.0: PCI INT A: no GSI
> [    0.629519] ixgbe 0000:00:06.0: Adapter removed
> [    0.630143] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
> [    0.630147] ixgbe: probe of 0000:00:06.0 failed with error -5
> [    0.630199] ixgbe 0000:00:06.1: can't derive routing for PCI INT B
> [    0.630200] ixgbe 0000:00:06.1: PCI INT B: no GSI
> [    0.630789] ixgbe 0000:00:06.1: Adapter removed
> [    0.639469] ixgbe 0000:00:06.1: can't derive routing for PCI INT B
> [    0.639473] ixgbe: probe of 0000:00:06.1 failed with error -5
>
After a cold reboot of the server, the devices are again seen in debian.
I guess somehow the devices become unusable when the vm is restarted.

[    1.146946] ixgbe 0000:00:06.0: PCI Express bandwidth of 32GT/s available
[    1.146949] ixgbe 0000:00:06.0: (Speed:5.0GT/s, Width: x8, Encoding
Loss:20%)
[    1.146953] ixgbe 0000:00:06.0: This is not sufficient for optimal
performance of this card.
[    1.146955] ixgbe 0000:00:06.0: For optimal performance, at least
60GT/s of bandwidth is required.
[    1.146956] ixgbe 0000:00:06.0: A slot with more lanes and/or higher
speed is suggested.
[    1.189458] ixgbe 0000:00:06.0: MAC: 3, PHY: 3, PBA No: 030C00-000
[    1.189461] ixgbe 0000:00:06.0: 0c:c4:7a:de:cc:c0
[    1.339231] ixgbe 0000:00:06.0: Intel(R) 10 Gigabit Network Connection
[    1.339364] ixgbe 0000:00:06.1: can't derive routing for PCI INT B
[    1.339366] ixgbe 0000:00:06.1: PCI INT B: no GSI
[    1.348140] input: PS/2 Generic Mouse as
/devices/platform/i8042/serio1/input/input2
[    1.566630] tsc: Refined TSC clocksource calibration: 2198.264 MHz
[    1.793929] ixgbe 0000:00:06.1: irq 47 for MSI/MSI-X
[    1.793950] ixgbe 0000:00:06.1: irq 48 for MSI/MSI-X
[    1.795079] ixgbe 0000:00:06.1: Multiqueue Disabled: Rx Queue count =
1, Tx Queue count = 1
[    1.859258] ixgbe 0000:00:06.1: PCI Express bandwidth of 32GT/s available
[    1.859261] ixgbe 0000:00:06.1: (Speed:5.0GT/s, Width: x8, Encoding
Loss:20%)
[    1.859262] ixgbe 0000:00:06.1: This is not sufficient for optimal
performance of this card.
[    1.859264] ixgbe 0000:00:06.1: For optimal performance, at least
60GT/s of bandwidth is required.
[    1.859265] ixgbe 0000:00:06.1: A slot with more lanes and/or higher
speed is suggested.
[    1.902549] ixgbe 0000:00:06.1: MAC: 3, PHY: 3, PBA No: 030C00-000
[    1.902554] ixgbe 0000:00:06.1: 0c:c4:7a:de:cc:c1
[    2.055631] ixgbe 0000:00:06.1: Intel(R) 10 Gigabit Network Connection

# cat /proc/interrupts
           CPU0      
  0:         45   IO-APIC-edge      timer
  1:          9   IO-APIC-edge      i8042
  4:       3756   IO-APIC-edge      serial
  8:          0   IO-APIC-edge      rtc0
  9:          0   IO-APIC-fasteoi   acpi
 12:        138   IO-APIC-edge      i8042
 40:          0   PCI-MSI-edge      virtio0-config
 41:       5667   PCI-MSI-edge      virtio0-requests
 42:          0   PCI-MSI-edge      virtio1-config
 43:        139   PCI-MSI-edge      virtio1-input.0
 44:          1   PCI-MSI-edge      virtio1-output.0
 45:        120   PCI-MSI-edge      eth1-TxRx-0
 46:          0   PCI-MSI-edge      eth1
 47:       1170   PCI-MSI-edge      eth2-TxRx-0
 48:          3   PCI-MSI-edge      eth2
NMI:          0   Non-maskable interrupts
LOC:       8265   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RTR:          0   APIC ICR read retries
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:          1   Machine check polls
HYP:          0   Hypervisor callback interrupts
ERR:          0
MIS:          0