kern/170713: [cxgb] Driver must be loaded after boot due to timing issues checking for kern.ipc.nmb* values set via /boot/loader.conf

Garrett Cooper yanegomi at gmail.com
Fri Aug 17 20:50:05 UTC 2012


>Number:         170713
>Category:       kern
>Synopsis:       [cxgb] Driver must be loaded after boot due to timing issues checking for kern.ipc.nmb* values set via /boot/loader.conf
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Aug 17 20:50:04 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Garrett Cooper
>Release:        7.3-RELEASE/7.4-RELEASE
>Organization:
EMC Isilon
>Environment:
FreeBSD  7.4-RELEASE-p10 FreeBSD 7.4-RELEASE-p10 #0: Fri Aug 17 07:15:01 UTC 2012     root@:/usr/obj/mnt/freebsd/releng/7.4/sys/ISI-GENERIC  amd64
>Description:
There's a timing issue where the following section of code in cxgb_sge.c always fails on 7.3-RELEASE/7.4-RELEASE when trying to load the driver because the values being checked are set to default values (in particular the jumbo_q_size check fails):

    383 #if __FreeBSD_version >= 700111
    384         if (cxgb_use_16k_clusters)
    385                 jumbo_q_size = min(nmbjumbo16/(3*nqsets), JUMBO_Q_SIZE);
    386         else
    387                 jumbo_q_size = min(nmbjumbo9/(3*nqsets), JUMBO_Q_SIZE);
    388 #else
    389         jumbo_q_size = min(nmbjumbop/(3*nqsets), JUMBO_Q_SIZE);
    390 #endif
    391         while (!powerof2(jumbo_q_size))
    392                 jumbo_q_size--;
    393
    394         if (fl_q_size < (FL_Q_SIZE / 4) || jumbo_q_size < (JUMBO_Q_SIZE / 2))
    395                 device_printf(adap->dev,
    396                     "Insufficient clusters and/or jumbo buffers.\n");

Example (7.3-RELEASE with the patch noted below):

cxgbc0: <Chelsio T320, 2 ports> mem 0xfaf7e000-0xfaf7efff,0xfaf7f000-0xfaf7ffff irq 24 at device 0.0 on pci8
cxgbc0: Insufficient clusters and/or jumbo buffers (4096 < 1024 or 256 < 512).
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Ethernet address: 00:07:43:07:41:1c
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Ethernet address: 00:07:43:07:41:1d
cxgbc0: Firmware Version 7.8.0

Example (7.4-RELEASE):

cxgbc0: <Chelsio T320, 2 ports> mem 0xfaf7e000-0xfaf7efff,0xfaf7f000-0xfaf7ffff irq 24 at device 0.0 on pci8
cxgbc0: Insufficient clusters and/or jumbo buffers.
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Ethernet address: 00:07:43:07:41:1c
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Ethernet address: 00:07:43:07:41:1d
cxgbc0: Firmware Version 7.8.0

I've worked around this issue by loading the driver later in the boot process,
but this chicken and egg problem should be properly resolved with the proper calls being made to kern_ipc.c before sge is attached to the kernel.

Please note that this issue does not exist on 9.0-RELEASE (or at least didn't exist in the limited testing I've done.. the race still might be there but ameliorated); did not verify if this issue exists on 8.x.

More details:

# cat /boot/loader.conf 
if_cxgb_load="YES"
if_em_load="YES"

kern.ipc.nmbclusters="250000"
kern.ipc.nmbjumbo9=262144
kern.ipc.nmbjumbo16=262144
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
kern.ipc.maxsockbuf=2097152
>How-To-Repeat:
1. Set values for kern.ipc to values shown above.
2. Load if_cxgb at boot.
3. Boot the kernel.
>Fix:
Something needs to be backported from 9.x to 7.x in order to fix this chicken and egg problem, or better sequencing needs to be done, so the mbuf tunables are read in before the driver is probed and attached.

>Release-Note:
>Audit-Trail:
>Unformatted:


More information about the freebsd-bugs mailing list