Server doesn't boot when 3 PCIe slots are populated
Mehmet Erol Sanliturk
m.e.sanliturk at gmail.com
Sun Jan 14 16:18:11 UTC 2018
On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka <list1 at gjunka.com> wrote:
> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote:
>> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka <list1 at gjunka.com
>> <mailto:list1 at gjunka.com>> wrote:
>> I am installing a FreeBSD server based on Supermicro H8SML-iF.
>> There are three PCIe slots to which I installed 2 NVMe drives and
>> one network card Intel I350-T4 (with 4 Ethernet slots).
>> I am observing a strange behavior where the system doesn't boot if
>> all three PCIe slots are populated. It shows this message:
>> nvme0: <Generic NVMe Device> mem 0xfd8fc000-0xfd8fffff irq 24 at
>> device 0.0 on pci1
>> nvme0: controller ready did not become 1 within 30000 ms
>> nvme0: did not complete shutdown within 5 seconds of notification
>> The I see a kernel panic/dump and the system reboots after 15 seconds.
>> If I remove one card, either one of the NVMe drives or the network
>> card, the system boots fine. Also, if in BIOS I set PnP OS to YES
>> then sometimes it boots (but not always). If I set PnP OS to NO,
>> and all three cards are installed, the system never boots.
>> When the system boots OK I can see that the network card is
>> reported as 4 separate devices on one of the PCIe slots. I tried
>> different NVMe drives as well as changing which device is
>> installed to which slot but the result seems to be the same in any
>> What may be the issue? Amount of power drawn by the hardware? Too
>> many devices not supported by the motherboard? Too many interrupts
>> for the FreeBSD kernel to handle?
>> Any help would be greatly appreciated.
>> From my experience from other trade marked main boards , an action may be
>> to check manual of your server board to see whether there are rules about
>> use of these slots : Sometimes differently shaped slots are supplied with
>> same ports : If one slot is occupied , the other slot should be left open ,
>> or rules about not to insert such a kind of device into a slot , for
>> example , graphic cards .
>> Mehmet Erol Sanliturk
> I checked the manual but couldn't find any restrictions regarding PCIe
> ports. It only says how many lanes are available in each slot. Would there
> be any obvious BIOS setting that could cause this issue? I tried after
> resetting BIOS to default settings but maybe something is set incorrectly
> by default?
On the above page , click "OS Compatibility"
On the following page , click "SR5650"
OS Compatibility Chart
On the column ( third )
there listed only
>From this list , it may be said that , this mother board date is old ,
means , it seems that the new OS versions are not tested after currently
tested OS versions .
To check interaction between operating system and your Supermicro H8SML-iF
, select one of the suitable operating system ( Unix class OSes are more
suitable ) for you and tested on this card , and try to install it as you
like your installed components . If it boots successfully , it means that
there is an incompatibility between your FreeBSD and the main board . If no
one of them boots , then you may conclude that , there is a problem in your
BIOS settings are important , because , OS communicates with the main board
through these settings .
In manual ( downloaded from the above page :
Manual Revision 1.0c
Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is
If PnP is selected YES. OS adjusts some device settings . If NO is
selected , BIOS adjusts some device settings . When BIOS adjusted device
settings are not conforming to OS parameters , the result will be "FAIL" .
Therefore , more suitable selection is YES .
Another point is that , there are many more BIOS selectable parameters and
jumpers about PCI slots and others .
There are some BIOS settings for PCI slots :
PCI X4 Slot 6 ( page 4-9 )
PCI x8 Slot 7 ( page 4-10 )
Please review these BIOS settings in your manual and set them with respect
to your requirements .
Mehmet Erol Sanliturk
More information about the freebsd-questions