Server doesn't boot when 3 PCIe slots are populated

Mehmet Erol Sanliturk m.e.sanliturk at gmail.com
Sun Jan 14 16:18:11 UTC 2018


On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka <list1 at gjunka.com> wrote:

>
> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote:
>
>>
>>
>> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka <list1 at gjunka.com
>> <mailto:list1 at gjunka.com>> wrote:
>>
>>     Hello,
>>
>>     I am installing a FreeBSD server based on Supermicro H8SML-iF.
>>     There are three PCIe slots to which I installed 2 NVMe drives and
>>     one network card Intel I350-T4 (with 4 Ethernet slots).
>>
>>     I am observing a strange behavior where the system doesn't boot if
>>     all three PCIe slots are populated. It shows this message:
>>
>>     nvme0: <Generic NVMe Device> mem 0xfd8fc000-0xfd8fffff irq 24 at
>>     device 0.0 on pci1
>>     nvme0: controller ready did not become 1 within 30000 ms
>>     nvme0: did not complete shutdown within 5 seconds of notification
>>
>>     The I see a kernel panic/dump and the system reboots after 15 seconds.
>>
>>     If I remove one card, either one of the NVMe drives or the network
>>     card, the system boots fine. Also, if in BIOS I set PnP OS to YES
>>     then sometimes it boots (but not always). If I set PnP OS to NO,
>>     and all three cards are installed, the system never boots.
>>
>>     When the system boots OK I can see that the network card is
>>     reported as 4 separate devices on one of the PCIe slots. I tried
>>     different NVMe drives as well as changing which device is
>>     installed to which slot but the result seems to be the same in any
>>     case.
>>
>>     What may be the issue? Amount of power drawn by the hardware? Too
>>     many devices not supported by the motherboard? Too many interrupts
>>     for the FreeBSD kernel to handle?
>>
>>     Any help would be greatly appreciated.
>>
>>     GregJ
>>
>>     _______________________________________________
>>
>>
>>
>>
>>
>> From my experience from other trade marked main boards , an action may be
>> to check manual of your server board to see whether there are rules about
>> use of these slots : Sometimes differently shaped slots are supplied with
>> same ports : If one slot is occupied , the other slot should be left open ,
>> or rules about not to insert such a kind of device into a slot , for
>> example , graphic cards .
>>
>>
>> Mehmet Erol Sanliturk
>>
>>
> I checked the manual but couldn't find any restrictions regarding PCIe
> ports. It only says how many lanes are available in each slot. Would there
> be any obvious BIOS setting that could cause this issue? I tried after
> resetting BIOS to default settings but maybe something is set incorrectly
> by default?
>
> GregJ
> _______________________________________________
>
>



http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56x0/H8SML-iF.cfm
H8SML-iF


On the above page , click "OS Compatibility"


On the following page , click "SR5650"

http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm
OS Compatibility Chart


On the column ( third )

H8SML-7F
H8SML-7
H8SML-iF
H8SML-i


there listed only


FreeBSD 8.0
FreeBSD 9.1

>From this list , it may be said that , this mother board date is old ,
means , it seems that the new OS versions are not tested after currently
tested OS versions .


To check interaction between operating system and your Supermicro H8SML-iF
, select one of the suitable operating system ( Unix class OSes are more
suitable  ) for you and tested on this card , and try to install it as you
like your installed components . If it boots successfully , it means that
there is an incompatibility between your FreeBSD and the main board . If no
one of them boots , then you may conclude that , there is a problem in your
settings .


BIOS settings are important , because , OS communicates with the main board
through these settings .


In manual ( downloaded from the above page :
Manual Revision 1.0c
Release Date: March 12, 2014 ) , page 4-9  , "PCI/PnP Configuration" is
defined .
If PnP is selected YES. OS adjusts some device settings  . If NO is
selected , BIOS adjusts some device settings . When BIOS adjusted device
settings are not conforming to OS parameters , the result will be "FAIL" .

Therefore , more suitable selection is YES .


Another point is that , there are many more BIOS selectable parameters and
jumpers about PCI slots and others  .
There are some BIOS settings for PCI slots :

PCI X4 Slot 6 ( page 4-9 )
PCI x8 Slot 7 ( page 4-10 )



Please review these BIOS settings in your manual and set them with respect
to your requirements .



Mehmet Erol Sanliturk


More information about the freebsd-questions mailing list