kern/120231: GEOM_CONCAT error adding second drive

[Richard Bejtlich]

>Number:         120231
>Category:       kern
>Synopsis:       GEOM_CONCAT error adding second drive
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Feb 03 02:10:00 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Richard Bejtlich
>Release:        7.0-RC1
>Organization:
TaoSecurity
>Environment:
uname -a
FreeBSD edited.edited.com 7.0-RC1 FreeBSD 7.0-RC1 #0: Mon Dec 24 12:18:24 UTC 2007     root at logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

>Description:
I want to use gconcat to make ad4s1h and ad5s1d appear to be a single /dev/concat/nsm partition.  When ad5s1d occupies the entire ad5s1 slice, then GEOM_CONCAT tries to add ad5s1 instead of ad5s1d.  GEOM_CONCAT doesn't work at that point.  Please see text below for an example.
>How-To-Repeat:
# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad4s1a    989M    129M    781M    14%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad4s1e    989M     22K    910M     0%    /home
/dev/ad4s1h    5.9G    4.0K    5.4G     0%    /nsm1
/dev/ad5s1d     72G    4.0K     66G     0%    /nsm2
/dev/ad4s1f    989M     12K    910M     0%    /tmp
/dev/ad4s1d    4.8G    755M    3.7G    17%    /usr
/dev/ad4s1g     20G    232K     18G     0%    /var

# mount
/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad4s1e on /home (ufs, local, soft-updates)
/dev/ad4s1h on /nsm1 (ufs, local, soft-updates)
/dev/ad5s1d on /nsm2 (ufs, local, soft-updates)
/dev/ad4s1f on /tmp (ufs, local, soft-updates)
/dev/ad4s1d on /usr (ufs, local, soft-updates

# umount /nsm1
# umount /nsm2
# gconcat label -v nsm ad4s1h ad5s1d
Metadata value stored on ad4s1h.
Metadata value stored on ad5s1d.
Done.
# dmesg | tail
ad4: 38146MB <Seagate ST340014AS 8.05> at ata2-master SATA150
ad5: 76293MB <SAMSUNG HD080HJ/P ZH100-34> at ata2-slave SATA150
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad4s1a
GEOM_CONCAT: Device nsm created (id=3517115025).
GEOM_CONCAT: Disk ad4s1h attached to nsm.
GEOM_CONCAT: Disk ad5s1 attached to nsm.
GEOM_CONCAT: Device nsm activated.
GEOM_CONCAT: Cannot add disk ad5s1d to nsm (error=17).

>Fix:
As a workaround, I can create two partitions on ad5s1 instead of one.

# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad4s1a    989M    129M    781M    14%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad4s1e    989M     22K    910M     0%    /home
/dev/ad4s1h    5.9G    4.0K    5.4G     0%    /nsm1
/dev/ad5s1e     71G    4.0K     65G     0%    /nsm2
/dev/ad4s1f    989M     12K    910M     0%    /tmp
/dev/ad5s1d    989M    4.0K    910M     0%    /tmp2
/dev/ad4s1d    4.8G    755M    3.7G    17%    /usr
/dev/ad4s1g     20G    232K     18G     0%    /var

# mount
/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad4s1e on /home (ufs, local, soft-updates)
/dev/ad4s1h on /nsm1 (ufs, local, soft-updates)
/dev/ad5s1e on /nsm2 (ufs, local, soft-updates)
/dev/ad4s1f on /tmp (ufs, local, soft-updates)
/dev/ad5s1d on /tmp2 (ufs, local, soft-updates)
/dev/ad4s1d on /usr (ufs, local, soft-updates)
/dev/ad4s1g on /var (ufs, local, soft-updates)

# umount /nsm1
# umount /nsm2
# gconcat label -v nsm ad4s1h ad5s1e
Metadata value stored on ad4s1h.
Metadata value stored on ad5s1e.
Done.

# dmesg | tail
acd0: DVDR <Optiarc DVD+/-RW ND-3570A/104B> at ata0-master UDMA33
ad4: 38146MB <Seagate ST340014AS 8.05> at ata2-master SATA150
ad5: 76293MB <SAMSUNG HD080HJ/P ZH100-34> at ata2-slave SATA150
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad4s1a
GEOM_CONCAT: Device nsm created (id=1016589974).
GEOM_CONCAT: Disk ad4s1h attached to nsm.
GEOM_CONCAT: Disk ad5s1e attached to nsm.
GEOM_CONCAT: Device nsm activated.

>From here I can finish the process without a problem.

# newfs /dev/concat/nsm

# mkdir /nsm
# mount /dev/concat/nsm /nsm

# df -h
Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/ad4s1a        989M    129M    781M    14%    /
devfs              1.0K    1.0K      0B   100%    /dev
/dev/ad4s1e        989M     22K    910M     0%    /home
/dev/ad4s1f        989M     12K    910M     0%    /tmp
/dev/ad5s1d        989M    4.0K    910M     0%    /tmp2
/dev/ad4s1d        4.8G    755M    3.7G    17%    /usr
/dev/ad4s1g         20G    232K     18G     0%    /var
/dev/concat/nsm     77G    4.0K     71G     0%    /nsm

# mount
/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad4s1e on /home (ufs, local, soft-updates)
/dev/ad4s1f on /tmp (ufs, local, soft-updates)
/dev/ad5s1d on /tmp2 (ufs, local, soft-updates)
/dev/ad4s1d on /usr (ufs, local, soft-updates)
/dev/ad4s1g on /var (ufs, local, soft-updates)
/dev/concat/nsm on /nsm (ufs, local)

Thank you.

>Release-Note:
>Audit-Trail:
>Unformatted: