Bootenv containing several filesystems
Mike Clarke
jmc-freebsd2 at milibyte.co.uk
Wed Feb 10 16:37:38 UTC 2021
On Tuesday, 9 February 2021 19:10:34 GMT Michael Schuster wrote:
> On Tue, Feb 9, 2021 at 5:30 PM Mike Clarke <jmc-freebsd2 at milibyte.co.uk>
> wrote:
> > On Tuesday, 9 February 2021 09:53:27 GMT Matthew Seaman wrote:
> > > There's an important difference between beadm and bectl which seems
> > > relevant here. beadm defaults to accepting a tree of ZFSes as a boot
> > > environment, whereas bectl only applies to the ZFS at the top level of
> > > the boot environment unless you use the -r flag.
> >
> > That probably accounts for a discrepancy that I always see between beadm
> > list
> > and bectl list for my BE which has child datasets:
> >
> > curlew:/tmp% beadm list
> > BE Active Mountpoint Space Created
> > fbsd12.1y - - 1.9G 2020-12-20 20:52
> > fbsd12.2a - - 133.0M 2020-12-24 11:20
> > fbsd12.2b - - 18.5M 2021-01-02 09:50
> > fbsd12.2c - - 11.7M 2021-01-12 09:55
> > fbsd12.2d NR / 39.4G 2021-02-05 10:46
> > curlew:/tmp% bectl list
> > BE Active Mountpoint Space Created
> > fbsd12.1y - - 61.3M 2020-12-20 20:52
> > fbsd12.2a - - 6.97M 2020-12-24 11:20
> > fbsd12.2b - - 2.80M 2021-01-02 09:50
> > fbsd12.2c - - 5.91M 2021-01-12 09:55
> > fbsd12.2d NR / 39.5G 2021-02-05 10:46
>
> strangely, I don't see such a difference:
>
> bectl:
> BE_20210205_121021_CURRENT14 - - 81.6M 2021-02-05 12:10
> BE_20210205_181224_CURRENT14 - - 49.9M 2021-02-05 18:12
> BE_20210206_102540_CURRENT14 - - 153M 2021-02-06 10:25
> BE_20210206_175312_CURRENT14 NR / 30.9G 2021-02-06 17:53
> BE_20210208_204901_CURRENT_14 - - 31.9M 2021-02-08 20:49
>
> beadm:
> BE_20210205_121021_CURRENT14 - - 81.6M 2021-02-05 12:10
> BE_20210205_181224_CURRENT14 - - 49.9M 2021-02-05 18:12
> BE_20210206_102540_CURRENT14 - - 152.3M 2021-02-06 10:25
> BE_20210206_175312_CURRENT14 NR / 30.9G 2021-02-06 17:53
> BE_20210208_204901_CURRENT_14 - - 31.9M 2021-02-08 20:49
>
> as you can see, the difference is negligable ...
>
> is there some zpool or zfs property I need to set so that be(ctl|adm) (with
> appropriate options if need be) will create a recursive boot environment?
A possible explanation for the lack of any discrepancy with your system may
be that there might have been relatively small changes between your BEs which
were all created within a few days of each other. In my case there was a
significant change between fbsd12.1y and fbsd12.2a when I upgraded from 12.1
RELEASE to 12,2 RELEASE and there would have been a big change in the child
dataset for /usr which I suspect bectl is not including in the total.
All my BEs were created with beadm which is always recursive.
Most of the options for the datasets in my BEs are default ones apart from
noatime and compression on all datasets and noexec and nosuid for a few
datasets.
Here's the filesystems for a typical BE:
NAME USED AVAIL REFER MOUNTPOINT
ssd/ROOT/fbsd12.2d 39.5G 75.6G 1.54G /
ssd/ROOT/fbsd12.2d/usr 31.2G 75.6G 10.1G /usr
ssd/ROOT/fbsd12.2d/usr/ports 8.45G 75.6G 6.84G /usr/ports
ssd/ROOT/fbsd12.2d/usr/src 3.52G 75.6G 1.46G /usr/src
ssd/ROOT/fbsd12.2d/var 6.50G 75.6G 1.63G /var
ssd/ROOT/fbsd12.2d/var/db 3.13G 75.6G 1.93G /var/db
ssd/ROOT/fbsd12.2d/var/db/pkg 776M 75.6G 83.6M /var/db/pkg
ssd/ROOT/fbsd12.2d/var/empty 104K 75.6G 96K /var/empty
ssd/ROOT/fbsd12.2d/var/tmp 1.03G 75.6G 128K /var/tmp
Some datasets are created outside the BE to preserve data when switching
between BEs.
NAME USED AVAIL REFER MOUNTPOINT
home/DATA/var 2.87G 65.5G 31K none
home/DATA/var/cache 867M 65.5G 31K none
home/DATA/var/cache/pkg 867M 65.5G 866M /var/cache/pkg
home/DATA/var/db 1.76G 65.5G 31K none
home/DATA/var/db/mysql 1.76G 65.5G 1.01G /var/db/mysql
home/DATA/var/log 275M 749M 201M /var/log
home/DATA/var/mail 52K 65.5G 32K /var/mail
I've been experimenting by switching between BEs created with beadm and bectl
in an attempt to identify the reason for the discrepancies but I think that's
just muddied the water instead of throwing light on things.
After creating two BEs with 'bectl -r' and booting into the second one after
activating it with bectl I would have expected the space for the active BE,
bectl-test2, to be about 37.7G but this space is still attributed to the
previously active BE, fbsd12.2d.
The BEs are no longer displayed in date order by bectl, probably due to the
difference in date format for the snapshots used by beadm and bectl.
curlew:/home/mike% bectl list
BE Active Mountpoint Space Created
bectl-test - - 1.93M 2021-02-10 09:54
bectl-test2 NR / 1.82G 2021-02-10 10:10
fbsd12.1y - - 61.3M 2020-12-20 20:52
fbsd12.2a - - 6.97M 2020-12-24 11:20
fbsd12.2b - - 2.85M 2021-01-02 09:50
fbsd12.2c - - 5.91M 2021-01-12 09:55
fbsd12.2d - - 37.7G 2021-02-05 10:46
curlew:/home/mike% beadm list
BE Active Mountpoint Space Created
fbsd12.1y - - 1.9G 2020-12-20 20:52
fbsd12.2a - - 189.7M 2020-12-24 11:20
fbsd12.2b - - 84.5M 2021-01-02 09:50
fbsd12.2c - - 11.7M 2021-01-12 09:55
fbsd12.2d - - 37.7G 2021-02-05 10:46
bectl-test - - 2.2M 2021-02-10 09:54
bectl-test2 NR / 1.8G 2021-02-10 10:10
I then rebooted after using beadm to create and activate beadm-test. Both
bectl and beadm then showed 39.5G for the space attributed to the active BE
beadm-test.
curlew:/home/mike% bectl list
BE Active Mountpoint Space Created
beadm-test NR / 39.5G 2021-02-10 10:20
bectl-test - - 924K 2021-02-10 09:54
bectl-test2 - - 1.07M 2021-02-10 10:10
fbsd12.1y - - 61.3M 2020-12-20 20:52
fbsd12.2a - - 6.97M 2020-12-24 11:20
fbsd12.2b - - 2.85M 2021-01-02 09:50
fbsd12.2c - - 5.91M 2021-01-12 09:55
fbsd12.2d - - 1.18M 2021-02-05 10:46
curlew:/home/mike% beadm list
BE Active Mountpoint Space Created
fbsd12.1y - - 1.9G 2020-12-20 20:52
fbsd12.2a - - 189.7M 2020-12-24 11:20
fbsd12.2b - - 84.5M 2021-01-02 09:50
fbsd12.2c - - 11.7M 2021-01-12 09:55
fbsd12.2d - - 1.2M 2021-02-05 10:46
bectl-test - - 1.8M 2021-02-10 09:54
bectl-test2 - - 2.1M 2021-02-10 10:10
beadm-test NR / 39.5G 2021-02-10 10:20
The next step was to use bectl to reactivate bectl-test2 and reboot which
then left 37.7G attributed to the inactive beadm-test.
curlew:/home/mike% bectl list
BE Active Mountpoint Space Created
beadm-test - - 37.7G 2021-02-10 10:20
bectl-test - - 924K 2021-02-10 09:54
bectl-test2 NR / 1.82G 2021-02-10 10:10
fbsd12.1y - - 61.3M 2020-12-20 20:52
fbsd12.2a - - 6.97M 2020-12-24 11:20
fbsd12.2b - - 2.85M 2021-01-02 09:50
fbsd12.2c - - 5.91M 2021-01-12 09:55
fbsd12.2d - - 1.18M 2021-02-05 10:46
curlew:/home/mike% beadm list
BE Active Mountpoint Space Created
fbsd12.1y - - 1.9G 2020-12-20 20:52
fbsd12.2a - - 189.7M 2020-12-24 11:20
fbsd12.2b - - 84.5M 2021-01-02 09:50
fbsd12.2c - - 11.7M 2021-01-12 09:55
fbsd12.2d - - 1.2M 2021-02-05 10:46
bectl-test - - 1.8M 2021-02-10 09:54
bectl-test2 NR / 1.8G 2021-02-10 10:10
beadm-test - - 37.7G 2021-02-10 10:20
I then used beadm to reactivate bectl-test2 by using 'beadm activate beadm-
test' followed by 'beadm activate bectl-test2' and rebooted with the result
that both bectl and beadm then attributed 39.5G to the active BE, bectl-
test2.
curlew:/home/mike% bectl list
BE Active Mountpoint Space Created
beadm-test - - 1.45M 2021-02-10 10:20
bectl-test - - 924K 2021-02-10 09:54
bectl-test2 NR / 39.5G 2021-02-10 10:10
fbsd12.1y - - 61.3M 2020-12-20 20:52
fbsd12.2a - - 6.97M 2020-12-24 11:20
fbsd12.2b - - 2.85M 2021-01-02 09:50
fbsd12.2c - - 5.91M 2021-01-12 09:55
fbsd12.2d - - 1.18M 2021-02-05 10:46
curlew:/home/mike% beadm list
BE Active Mountpoint Space Created
fbsd12.1y - - 1.9G 2020-12-20 20:52
fbsd12.2a - - 189.7M 2020-12-24 11:20
fbsd12.2b - - 84.5M 2021-01-02 09:50
fbsd12.2c - - 11.7M 2021-01-12 09:55
fbsd12.2d - - 1.2M 2021-02-05 10:46
bectl-test - - 1.8M 2021-02-10 09:54
bectl-test2 NR / 39.5G 2021-02-10 10:10
beadm-test - - 2.5M 2021-02-10 10:20
Surprisingly using bectl to repeat the process with 'bectl activate beadm-
test' followed by 'bectl activate bectl-test2 resulted in 39.5G still being
attributed to bectl-test2.
Throughout all these tests bectl consistently showed the size of fbsd12.1y to
be 61.3M and beadm showed 1.9G.
So appears to be a significant difference in the way beadm and bectl account
for the space used by BEs with child datasets. Perhaps this only shows when
the BEs are not all created with the same program. I'll probably start using
bectl to create the next few BEs and see what happens to the space values
after the older beadm BEs get destroyed.
Before long I'll be deleting fbsd12.1y and it will be interesting to see
whether 1.9G or 61.3M get freed up if I use bectl to destroy it.
--
Mike Clarke
More information about the freebsd-questions
mailing list