Create a gpart in a multipath container?

[John]

Hi Folks,

   I'm trying to create a set of partitions inside of a multipath
container. For this discussion, a drive in a shelf hanging off a
pair of controllers:

# camcontrol inquiry da0
pass2: <HP EH0146FAWJB HPDD> Fixed Direct Access SCSI-5 device 
pass2: Serial Number 3TB1BKGX00009036W9EN
pass2: 600.000MB/s transfers, Command Queueing Enabled
# camcontrol inquiry da25
pass27: <HP EH0146FAWJB HPDD> Fixed Direct Access SCSI-5 device 
pass27: Serial Number 3TB1BKGX00009036W9EN
pass27: 600.000MB/s transfers, Command Queueing Enabled

   The multipath container:

# gmultipath label  Z0 da0 da25
# gmultipath list
Geom name: Z0
Providers:
1. Name: multipath/Z0
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r0w0e0
Consumers:
1. Name: da0
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r0w0e0
2. Name: da25
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r0w0e0

Note, at this point I can create multipath containers from the rest of
the drives and create zfs volume with no problems.

However, I'd now like to create a set of partitions inside
of the multipath container.

# gpart create -s gpt multipath/Z0
multipath/Z0 created
# gpart add -s 1m -t freebsd-ufs -l Z0test multipath/Z0
multipath/Z0p1 added
# gpart add -t freebsd-zfs -l Z0 multipath/Z0

The created partition looks correct, but two additional
geoms have been created that are corrupt:

Geom name: multipath/Z0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 286749453
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: multipath/Z0p1
   Mediasize: 1048576 (1.0M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: f6a44058-eb72-11e0-8eb1-001e4f258317
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: Z0test
   length: 1048576
   offset: 17408
   type: freebsd-ufs
   index: 1
   end: 2081
   start: 34
2. Name: multipath/Z0p2
   Mediasize: 146814654464 (136G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1065984
   Mode: r0w0e0
   rawuuid: 0a064a37-eb73-11e0-8eb1-001e4f258317
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: Z0
   length: 146814654464
   offset: 1065984
   type: freebsd-zfs
   index: 2
   end: 286749453
   start: 2082
Consumers:
1. Name: multipath/Z0
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r0w0e0


These show up and are corrupt:

Geom name: da0
modified: false
state: CORRUPT
fwheads: 255
fwsectors: 63
last: 286749453
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 1048576 (1.0M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: f6a44058-eb72-11e0-8eb1-001e4f258317
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: Z0test
   length: 1048576
   offset: 17408
   type: freebsd-ufs
   index: 1
   end: 2081
   start: 34
2. Name: da0p2
   Mediasize: 146814654464 (136G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1065984
   Mode: r0w0e0
   rawuuid: 0a064a37-eb73-11e0-8eb1-001e4f258317
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: Z0
   length: 146814654464
   offset: 1065984
   type: freebsd-zfs
   index: 2
   end: 286749453
   start: 2082
Consumers:
1. Name: da0
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r0w0e0

And the same for da25.

 
Finally, these messages show up:

GEOM_MULTIPATH: adding da0 to Z0/ac33be7a-eb68-11e0-97dd-001e4f258317
GEOM_MULTIPATH: da0 now active path in Z0
GEOM: da0: the secondary GPT header is not in the last LBA.
GEOM_MULTIPATH: adding da25 to Z0/ac33be7a-eb68-11e0-97dd-001e4f258317
GEOM: da25: the secondary GPT header is not in the last LBA.
GEOM: da0: the secondary GPT header is not in the last LBA.
GEOM: da25: the secondary GPT header is not in the last LBA.
GEOM: da0: the secondary GPT header is not in the last LBA.

And after a reboot, the multipath container is gone, and this is
found for da0/da25. It can be recovered, but it seems to have
taken over the multipath container.

Geom name: da0
modified: false
state: CORRUPT
fwheads: 255
fwsectors: 63
last: 286749453
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 1048576 (1.0M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: 227634d0-eb6a-11e0-97dd-001e4f258317
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: Z0test
   length: 1048576
   offset: 17408
   type: freebsd-ufs
   index: 1
   end: 2081
   start: 34
2. Name: da0p2
   Mediasize: 146814654464 (136G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1065984
   Mode: r0w0e0
   rawuuid: 6b285348-eb6a-11e0-97dd-001e4f258317
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: Z0
   length: 146814654464
   offset: 1065984
   type: freebsd-zfs
   index: 2
   end: 286749453
   start: 2082
Consumers:
1. Name: da0
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r0w0e0

Apologies for the long-winded explanation. I hope it made sense.

Is there a way to make this work? Is there a better way to configure
this? I'd like the partitions to be protected by multipathing which
lead me to try this.  Am I missing something totally obvious?

I've been looking at the code and I'm thinking there is an issue between
a real physical disk container vs a partition and sizing.

Any comments are appreciated.

Thanks,
John