RE: DRAID - Expansion and other issues
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Wed, 06 Apr 2022 21:51:46 UTC
A big thank you too Rich for finally letting me have my Jimmy Neutron moment off list.
zpoolconcepts(7) (from current-14)
A dRAID with N disks of size X, D data disks per redundancy group, P parity level, and S
distributed hot spares can hold approximately (N-S)*(D/(D+P))*X bytes and can withstand P
devices failing without losing data.
So, if N is 10, D is 2, P is 2, S is 0, and X is 1T, we get:
(10-0) * ((2/(2+2)) * 1T)
= 10 * (2/4) * 1T
= 5T effective storage per vdev.
## RAIDZ2 for comparison
root@draid:/ # zpool destroy tank
root@draid:/ # zpool create tank raidz2 /dev/da11p1 /dev/da12p1 /dev/da13p1 /dev/da14p1 /dev/da15p1 /dev/da16p1 /dev/da17p1 /dev/da18p1 /dev/da19p1 /dev/da20p1 /dev/da36p1 /dev/da37p1 /dev/da38p1 /dev/da39p1 /dev/da40p1 /dev/da41p1 /dev/da42p1 /dev/da43p1 /dev/da44p1 /dev/da45p1
root@draid:/ # zpool add tank spare /dev/da46p1 /dev/da47p1
ot@draid:/ # df -h /tank
Filesystem Size Used Avail Capacity Mounted on
tank 16T 256K 16T 0% /tank <---- 16TB
root@draid:/ #
root@draid:/ # zpool list -v tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 18.1T 1.65M 18.1T - - 0% 0% 1.00x ONLINE -
raidz2-0 18.1T 1.65M 18.1T - - 0% 0.00% - ONLINE
da11p1 - - - - - - - - ONLINE
da12p1 - - - - - - - - ONLINE
da13p1 - - - - - - - - ONLINE
da14p1 - - - - - - - - ONLINE
da15p1 - - - - - - - - ONLINE
da16p1 - - - - - - - - ONLINE
da17p1 - - - - - - - - ONLINE
da18p1 - - - - - - - - ONLINE
da19p1 - - - - - - - - ONLINE
da20p1 - - - - - - - - ONLINE
da36p1 - - - - - - - - ONLINE
da37p1 - - - - - - - - ONLINE
da38p1 - - - - - - - - ONLINE
da39p1 - - - - - - - - ONLINE
da40p1 - - - - - - - - ONLINE
da41p1 - - - - - - - - ONLINE
da42p1 - - - - - - - - ONLINE
da43p1 - - - - - - - - ONLINE
da44p1 - - - - - - - - ONLINE
da45p1 - - - - - - - - ONLINE
spare - - - - - - - - -
da46p1 - - - - - - - - AVAIL
da47p1 - - - - - - - - AVAIL
root@draid:/ #
## Draid expansion with Distributed spares
[root@draid /]# zpool create tank draid2:8d:10c:0s /dev/da11p1 /dev/da12p1 /dev/da13p1 /dev/da14p1 /dev/da15p1 /dev/da16p1 /dev/da17p1 /dev/da18p1 /dev/da19p1 /dev/da20p1
[root@draid /]# zpool add tank draid2:8d:10c:0s /dev/da36p1 /dev/da37p1 /dev/da38p1 /dev/da39p1 /dev/da40p1 /dev/da41p1 /dev/da42p1 /dev/da43p1 /dev/da44p1 /dev/da45p1
[root@draid /]# zpool add tank spare /dev/da46p1 /dev/da47p1
[root@draid /]# df -h /tank
Filesystem Size Used Avail Capacity Mounted on
tank 14T 767K 14T 0% /tank <---- 14T
[root@draid /]# zpool list -v tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 18.1T 7.62M 18.1T - - 0% 0% 1.00x ONLINE -
draid2:8d:10c:0s-0 9.03T 6.09M 9.03T - - 0% 0.00% - ONLINE
da11p1 - - - - - - - - ONLINE
da12p1 - - - - - - - - ONLINE
da13p1 - - - - - - - - ONLINE
da14p1 - - - - - - - - ONLINE
da15p1 - - - - - - - - ONLINE
da16p1 - - - - - - - - ONLINE
da17p1 - - - - - - - - ONLINE
da18p1 - - - - - - - - ONLINE
da19p1 - - - - - - - - ONLINE
da20p1 - - - - - - - - ONLINE
draid2:8d:10c:0s-1 9.03T 1.52M 9.03T - - 0% 0.00% - ONLINE
da36p1 - - - - - - - - ONLINE
da37p1 - - - - - - - - ONLINE
da38p1 - - - - - - - - ONLINE
da39p1 - - - - - - - - ONLINE
da40p1 - - - - - - - - ONLINE
da41p1 - - - - - - - - ONLINE
da42p1 - - - - - - - - ONLINE
da43p1 - - - - - - - - ONLINE
da44p1 - - - - - - - - ONLINE
da45p1 - - - - - - - - ONLINE
spare - - - - - - - - -
da46p1 - - - - - - - - AVAIL
da47p1 - - - - - - - - AVAIL
[root@draid /]#
### One Pool with Distributed spares
[root@draid /]# zpool create tank draid2:18d:20c:0s /dev/da11p1 /dev/da12p1 /dev/da13p1 /dev/da14p1 /dev/da15p1 /dev/da16p1 /dev/da17p1 /dev/da18p1 /dev/da19p1 /dev/da20p1 /dev/da36p1 /dev/da37p1 /dev/da38p1 /dev/da39p1 /dev/da40p1 /dev/da41p1 /dev/da42p1 /dev/da43p1 /dev/da44p1 /dev/da45p1
[root@draid /]# zpool add tank spare /dev/da46p1 /dev/da47p1
[root@draid /]# df -h /tank
Filesystem Size Used Avail Capacity Mounted on
tank 14T 1.5M 14T 0% /tank <---- 14T
[root@draid /]# zpool list -v tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 18.1T 9.61M 18.1T - - 0% 0% 1.00x ONLINE -
draid2:18d:20c:0s-0 18.1T 9.61M 18.1T - - 0% 0.00% - ONLINE
da11p1 - - - - - - - - ONLINE
da12p1 - - - - - - - - ONLINE
da13p1 - - - - - - - - ONLINE
da14p1 - - - - - - - - ONLINE
da15p1 - - - - - - - - ONLINE
da16p1 - - - - - - - - ONLINE
da17p1 - - - - - - - - ONLINE
da18p1 - - - - - - - - ONLINE
da19p1 - - - - - - - - ONLINE
da20p1 - - - - - - - - ONLINE
da36p1 - - - - - - - - ONLINE
da37p1 - - - - - - - - ONLINE
da38p1 - - - - - - - - ONLINE
da39p1 - - - - - - - - ONLINE
da40p1 - - - - - - - - ONLINE
da41p1 - - - - - - - - ONLINE
da42p1 - - - - - - - - ONLINE
da43p1 - - - - - - - - ONLINE
da44p1 - - - - - - - - ONLINE
da45p1 - - - - - - - - ONLINE
spare - - - - - - - - -
da46p1 - - - - - - - - AVAIL
da47p1 - - - - - - - - AVAIL
[root@draid /]#
From: Rich [mailto:rincebrain@gmail.com]
Sent: Wednesday, April 6, 2022 9:43 AM
To: Michael Jung (USER) <mikej@paymentallianceintl.com>
Subject: Re: DRAID - Expansion and other issues
I'm not an expert on draid, but you appear to have defined two draid vdevs with a ratio of 2 data disks to 2 parity disks for the internal "redundancy groups", so 50% overhead seems par for the course, much like if you had defined 4-disk raidz2 vdevs.
- Rich
On Wed, Apr 6, 2022 at 9:22 AM Michael Jung (USER) <mikej@paymentallianceintl.com<mailto:mikej@paymentallianceintl.com>> wrote:
Hi!
I started playing with draid some months ago and I have a problem I cannot figure out.
I started out with a single draid2:2d:10c:0s and life was good. Ran some tests, added
a special device and life was better. Then I expanded the pool by adding another
draid2:2d:10c:0s. Now the problem: the pool says it has ~18TB worth of space, the
file system only shows ~9TB. It did not auto-expand.
I’m currently on main-n253875-8e72f458c6d: but I initially spun this up just after
DRAID was brought Into the tree, at least when I became aware of it. I have tried
individually on lining devices in the pool with “-e”, exporting/importing the pool etc.
and basically every suggestion my “google foo” would lead me to.
I have no useful data on this pool and I can destroy and re-create it but I would
like to save some time and get opinions or fact on:
Can a draid pool be expanded after a special device has been added? It would seem so
but the filesystem does not reflect that.?
What at is the correct way to wire down the following – I had problems after not doing
this since the 2.x days and would really want to wire down my draid devices now
that I have other pools.
I have pool construction all scripted but of course I want physical disk da<x> to always be
consistent. Then I easily test any construct someone comes up with.
root@draid:/home/mikej # camcontrol devlist -b
scbus0 on ata0 bus 0
scbus1 on ata1 bus 0
scbus2 on mpt0 bus 0
scbus3 on mps0 bus 0 <--
scbus4 on camsim0 bus 0
scbus-1 on xpt0 bus 0
root@draid:/home/mikej #
<dmesg>
mps0: <Avago Technologies (LSI) SAS2008> port 0x5000-0x50ff mem 0xfd4fc000-0xfd4fffff,0xfd480000-0xfd4bffff irq 19 at device 0.0 on pci4
mps0: Firmware: 20.00.04.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
<camcontrol>
<SEAGATE ST91000640SS AS09> at scbus3 target 63 lun 0 (da42,pass35)
<SEAGATE ST91000640SS AS09> at scbus3 target 67 lun 0 (da46,pass39)
<SEAGATE ST91000640SS AS08> at scbus3 target 68 lun 0 (da47,pass40)
EXAMPLE: Is this correct?
hint.scbus.3.at<http://hint.scbus.3.at>=”mps0”
hint.da.42.at<http://hint.da.42.at>=”scbus3”
hint.da.42.target=”63”
hint.da.42.unit=”0”
hint.da.46.at<http://hint.da.46.at>=”scbus3”
hint.da.46.target=”67”
hint.da.46.unit=”0”
hint.da.47.at<http://hint.da.47.at>=”scbus3”
hint.da.47.target=”68”
hint.da.47.unit=”0"
I will try again, but this did not seem to work for me. This is my home lab but it’s
now painful to spin down this host as it’s a iSCSI target for several machines for a
ESXi project I’m working on so I would like to put wired down SCSI devices to rest.
And yes, I have scheduled VM backups for my project to another data store that is not
this box ;-)
It’s also odd, that I did not get all GPT labels in the zpool. All disks have GPT labels.
I have added for the next reboot to loader.conf.
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="1
This is an old Dell MD-1000 shelf I had laying around with a bunch of 1TB drives
which I thought would be perfect for playing around with draid.
mikej@draid:~ $ zpool get all tank | grep expand
tank autoexpand on local
tank expandsize - -
mikej@draid:~ $ zpool list -v tank
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x ONLINE -
draid2:2d:10c:0s-0 9.03T 4.06T 4.97T - - 0% 44.9% - ONLINE
gpt/da0p1 - - - - - - - - ONLINE
gpt/d10p1 - - - - - - - - ONLINE
da39p1 - - - - - - - - ONLINE
da40p1 - - - - - - - - ONLINE
da41p1 - - - - - - - - ONLINE
da43p1 - - - - - - - - ONLINE
da44p1 - - - - - - - - ONLINE
da42p1 - - - - - - - - ONLINE
da45p1 - - - - - - - - ONLINE
gpt/d19p1 - - - - - - - - ONLINE
draid2:2d:10c:0s-1 9.03T 3.96T 5.07T - - 0% 43.8% - ONLINE
da46p1 - - - - - - - - ONLINE
gpt/d11p1 - - - - - - - - ONLINE
gpt/d12p1 - - - - - - - - ONLINE
gpt/d13p1 - - - - - - - - ONLINE
gpt/d14p1 - - - - - - - - ONLINE
gpt/d15p1 - - - - - - - - ONLINE
gpt/d16p1 - - - - - - - - ONLINE
gpt/d17p1 - - - - - - - - ONLINE
gpt/d18p1 - - - - - - - - ONLINE
da18p1 - - - - - - - - ONLINE
special - - - - - - - - -
mirror-3 398G 2.86G 395G - - 0% 0.71% - ONLINE
gpt/special0 - - - - - - - - ONLINE
gpt/special1 - - - - - - - - ONLINE
logs - - - - - - - - -
mirror-2 15.5G 256K 15.5G - - 0% 0.00% - ONLINE
gpt/slog0 - - - - - - - - ONLINE
gpt/slog1 - - - - - - - - ONLINE
mikej@draid:~ $ zfs get mountpoint tank
NAME PROPERTY VALUE SOURCE
tank mountpoint /tank default
mikej@draid:~ $ df /tank
Filesystem 1K-blocks Used Avail Capacity Mounted on
tank 9558892004 4301310224 5257581780 45% /tank <-- ~9TB not 18TB
mikej@draid:~ $
mikej@draid:~ $ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ccache 9.50G 9.10G 406M - - 89% 95% 1.00x ONLINE -
raid-5400-1 6.28T 1.19T 5.09T - - 6% 19% 1.00x ONLINE -
tank 18.5T 8.02T 10.4T - - 0% 43% 1.00x ONLINE - <--
zfsroot 103G 32.6G 70.4G - - 25% 31% 1.00x ONLINE -
mikej@draid:~ $
Thanks in advance for any comments or sugesstions.
--mikej
CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may
contain information that is privileged, confidential, and
exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying
of this communication is strictly prohibited. If you have
received this transmission in error, please notify us by
telephone at (502) 212-4000 or notify us at: PAI, Dept. 99,
2101 High Wickham Place, Suite 101, Louisville, KY 40245
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.