RE: DRAID - Expansion and other issues

From: Michael Jung (USER) <mikej_at_paymentallianceintl.com>
Date: Wed, 06 Apr 2022 14:05:22 UTC
Thanks Rich.

I hear what you are saying, but I added them like you would add a second vdev to a raidz1/z2 pool. None of the online talks nor documentation mention draid expansion, and I was under the impression that you could not expand a draid2:2d:10c:0s to draid2:2d:20c:0s for example.

I absolutely could be wrong in my assumption but do not see how to do the expansion other than how I did it.

--mikej

--------------------------------

I'm not an expert on draid, but you appear to have defined two draid vdevs with a ratio of 2 data disks to 2 parity disks for the internal "redundancy groups", so 50% overhead seems par for the course, much like if you had defined 4-disk raidz2 vdevs.

- Rich

On Wed, Apr 6, 2022 at 9:22 AM Michael Jung (USER) <mikej@paymentallianceintl.com<mailto:mikej@paymentallianceintl.com>> wrote:
Hi!

I started playing with draid some months ago and I have a problem I cannot figure out.

I started out with a single draid2:2d:10c:0s and life was good.  Ran some tests, added
a special device and life was better. Then I expanded the pool by adding another
draid2:2d:10c:0s.  Now the problem: the pool says it has ~18TB worth of space, the
file system only shows ~9TB. It did not auto-expand.

I’m currently on main-n253875-8e72f458c6d: but I initially spun this up just after
DRAID was brought Into the tree, at least when I became aware of it. I have tried
individually on lining devices in the pool with “-e”, exporting/importing the pool etc.
and basically every suggestion my “google foo” would lead me to.

I have no useful data on this pool and I can destroy and re-create it but I would
like to save some time and get opinions or fact on:

Can a draid pool be expanded after a special device has been added?  It would seem so
but the filesystem does not reflect that.?

What at is the correct way to wire down the following – I had problems after not doing
this since the 2.x days and would really want to wire down my draid devices now
that I have other pools.

I have pool construction all scripted but of course I want physical disk da<x> to always be
consistent. Then I easily test any construct someone comes up with.

root@draid:/home/mikej # camcontrol devlist -b
scbus0 on ata0 bus 0
scbus1 on ata1 bus 0
scbus2 on mpt0 bus 0
scbus3 on mps0 bus 0   <--
scbus4 on camsim0 bus 0
scbus-1 on xpt0 bus 0
root@draid:/home/mikej #


<dmesg>
mps0: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfd4fc000-0xfd4fffff,0xfd480000-0xfd4bffff irq 19 at device 0.0 on pci4
mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

<camcontrol>
<SEAGATE ST91000640SS AS09>        at scbus3 target 63 lun 0 (da42,pass35)
<SEAGATE ST91000640SS AS09>        at scbus3 target 67 lun 0 (da46,pass39)
<SEAGATE ST91000640SS AS08>        at scbus3 target 68 lun 0 (da47,pass40)

EXAMPLE: Is this correct?

hint.scbus.3.at<http://hint.scbus.3.at>=”mps0”

hint.da.42.at<http://hint.da.42.at>=”scbus3”
hint.da.42.target=”63”
hint.da.42.unit=”0”

hint.da.46.at<http://hint.da.46.at>=”scbus3”
hint.da.46.target=”67”
hint.da.46.unit=”0”

hint.da.47.at<http://hint.da.47.at>=”scbus3”
hint.da.47.target=”68”
hint.da.47.unit=”0"

I will try again, but this did not seem to work for me.  This is my home lab but it’s
now painful to spin down this host as it’s a iSCSI target for several machines for a
ESXi project I’m working on so I would like to put wired down SCSI devices to rest.

And yes, I have scheduled VM backups for my project to another data store that is not
this box ;-)

It’s also odd, that I did not get all GPT labels in the zpool. All disks have GPT labels.

I have added for the next reboot to loader.conf.

kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="1

This is an old Dell MD-1000 shelf I had laying around with a bunch of 1TB drives
which I thought would be perfect for playing around with draid.

mikej@draid:~ $ zpool get all tank | grep expand
tank  autoexpand                     on                             local
tank  expandsize                     -                              -
mikej@draid:~ $ zpool list -v tank
NAME                 SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank                18.5T  8.02T  10.4T        -         -     0%    43%  1.00x    ONLINE  -
  draid2:2d:10c:0s-0  9.03T  4.06T  4.97T        -         -     0%  44.9%      -    ONLINE
    gpt/da0p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d10p1           -      -      -        -         -      -      -      -    ONLINE
    da39p1              -      -      -        -         -      -      -      -    ONLINE
    da40p1              -      -      -        -         -      -      -      -    ONLINE
    da41p1              -      -      -        -         -      -      -      -    ONLINE
    da43p1              -      -      -        -         -      -      -      -    ONLINE
    da44p1              -      -      -        -         -      -      -      -    ONLINE
    da42p1              -      -      -        -         -      -      -      -    ONLINE
    da45p1              -      -      -        -         -      -      -      -    ONLINE
    gpt/d19p1           -      -      -        -         -      -      -      -    ONLINE
  draid2:2d:10c:0s-1  9.03T  3.96T  5.07T        -         -     0%  43.8%      -    ONLINE
    da46p1              -      -      -        -         -      -      -      -    ONLINE
    gpt/d11p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d12p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d13p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d14p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d15p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d16p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d17p1           -      -      -        -         -      -      -      -    ONLINE
    gpt/d18p1           -      -      -        -         -      -      -      -    ONLINE
    da18p1              -      -      -        -         -      -      -      -    ONLINE
special                 -      -      -        -         -      -      -      -         -
  mirror-3           398G  2.86G   395G        -         -     0%  0.71%      -    ONLINE
    gpt/special0        -      -      -        -         -      -      -      -    ONLINE
    gpt/special1        -      -      -        -         -      -      -      -    ONLINE
logs                    -      -      -        -         -      -      -      -         -
  mirror-2          15.5G   256K  15.5G        -         -     0%  0.00%      -    ONLINE
    gpt/slog0           -      -      -        -         -      -      -      -    ONLINE
    gpt/slog1           -      -      -        -         -      -      -      -    ONLINE

mikej@draid:~ $ zfs get mountpoint tank
NAME  PROPERTY    VALUE       SOURCE
tank  mountpoint  /tank       default


mikej@draid:~ $ df /tank
Filesystem  1K-blocks       Used      Avail Capacity  Mounted on
tank       9558892004 4301310224 5257581780    45%    /tank   <-- ~9TB not 18TB
mikej@draid:~ $

mikej@draid:~ $ zpool list
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ccache       9.50G  9.10G   406M        -         -    89%    95%  1.00x    ONLINE  -
raid-5400-1  6.28T  1.19T  5.09T        -         -     6%    19%  1.00x    ONLINE  -
tank         18.5T  8.02T  10.4T        -         -     0%    43%  1.00x    ONLINE  -           <--
zfsroot       103G  32.6G  70.4G        -         -    25%    31%  1.00x    ONLINE  -
mikej@draid:~ $

Thanks in advance for any comments or sugesstions.

--mikej




CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245




Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.