FreeBSD 12.2-RELEASE-p9 trim: open failed: /dev/ada0: Operation not permitted

Fri Sep 3 20:30:51 UTC 2021

On 9/3/21 10:09 AM, Derek (freebsd lists) wrote:
> On 2021-08-29 10:26 p.m., David Christensen wrote:
>> I would like to trim the SSD, but trim(8) is not happy:
>>
>> 2021-08-29 19:22:36 toor at f1 ~
>> # trim -f /dev/ada0
>> trim /dev/ada0 offset 0 length 60022480896
>> trim: open failed: /dev/ada0: Operation not permitted
>>
> 
> I don't know if it's still relevant to the thread, but I feel like what 
> you are encountering here relates to what you are trying to trim.
> 
> It feels like if you trim the "raw device", the system wouldn't know 
> what is in use, and what isn't.
> 
> I feel like you should be trying to trim a filesystem on the device, 
> rather than the device itself, as the utility would be able to determine 
> what is currently allocated with the filesystem data.
> 
> It might be that a raw device could be trimmed, but perhaps there's some 
> safety there to stop it from wiping a device that is backing a mounted 
> filesystem.
> 
> Something to look at?
> 
> Derek

That is a good point -- the filesystem knows what blocks are in use, 
what blocks are not in use, and when blocks are removed from use. 
Somehow, trim(8) gets this information (from the kernel filesystem stack?).

RTFM trim(8):

SYNOPSIS
      trim [-Nfqv] [-[lo] offset[K|k|M|m|G|g|T|t]] [-r rfile] device ...

<snip>

      -l offset[K|k|M|m|G|g|T|t]

      -o offset[K|k|M|m|G|g|T|t]
              Specify the length -l of the region to trim or its offset 
-o from
              the beginning of the device.  The whole device is erased by
              default unless one or both of these options are presented.

<snip>

      Note that actual success of the operation depends of underlying device
      driver such as ada(4), da(4) and others.  Refer to corresponding 
manual
      pages for detail on possible caveats in low level support for ATA 
TRIM or
      SCSI UNMAP commands.

Note:

1.  The command line argument is *device*.

2.  The descriptions for the -l and -o options state trim(8) erases the 
whole *device*.

Here is another console session:

2021-09-03 12:07:27 toor at f2 ~
# freebsd-version ; uname -a
12.2-RELEASE-p10
FreeBSD f2.tracy.holgerdanske.com 12.2-RELEASE-p7 FreeBSD 
12.2-RELEASE-p7 GENERIC  amd64

2021-09-03 12:10:06 toor at f2 ~
# kldstat | egrep 'Id|zfs'
Id Refs Address                Size Name
  2    1 0xffffffff8247c000   3bad38 zfs.ko

2021-09-03 12:11:54 toor at f2 ~
# find /boot -iname '*zfs*'

2021-09-03 12:12:18 toor at f2 ~
# geom disk list ada0
Geom name: ada0
Providers:
1. Name: ada0
    Mediasize: 60022480896 (56G)
    Sectorsize: 512
    Mode: r3w3e9
    descr: INTEL SSDSC2CW060A3
    lunid: ****************
    ident: ******************
    rotationrate: 0
    fwsectors: 63
    fwheads: 16

2021-09-03 12:13:14 toor at f2 ~
# geom part show -p
=>       63  117231345    ada0  MBR  (56G)
          63       1985          - free -  (993K)
        2048   29360128  ada0s1  freebsd  [active]  (14G)
    29362176   87867392  ada0s2  freebsd  (42G)
   117229568       1840          - free -  (920K)

=>       0  29360128   ada0s1  BSD  (14G)
          0   4194304  ada0s1a  freebsd-zfs  (2.0G)
    4194304   4194304  ada0s1b  freebsd-swap  (2.0G)
    8388608  20971520  ada0s1d  freebsd-zfs  (10G)

2021-09-03 12:15:28 toor at f2 ~
# geli status
            Name  Status  Components
     ada0s1d.eli  ACTIVE  ada0s1d
mirror/swap.eli  ACTIVE  mirror/swap

2021-09-03 12:16:41 toor at f2 ~
# zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP 
HEALTH  ALTROOT
bootpool  1.88G   475M  1.41G        -         -     3%    24%  1.08x 
ONLINE  -
f2_zroot  9.50G  5.88G  3.62G        -         -    23%    61%  1.21x 
ONLINE  -

2021-09-03 12:17:34 toor at f2 ~
# zpool status
   pool: bootpool
  state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
   scan: scrub repaired 0 in 0 days 00:00:02 with 0 errors on Sat May 29 
16:18:50 2021
config:

	NAME        STATE     READ WRITE CKSUM
	bootpool    ONLINE       0     0     0
	  ada0s1a   ONLINE       0     0     0

errors: No known data errors

   pool: f2_zroot
  state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
   scan: scrub repaired 0 in 0 days 00:00:23 with 0 errors on Sat May 29 
16:19:12 2021
config:

	NAME           STATE     READ WRITE CKSUM
	f2_zroot       ONLINE       0     0     0
	  ada0s1d.eli  ONLINE       0     0     0

errors: No known data errors

2021-09-03 12:18:46 toor at f2 ~
# zfs get mountpoint bootpool f2_zroot
NAME      PROPERTY    VALUE       SOURCE
bootpool  mountpoint  /bootpool   local
f2_zroot  mountpoint  /f2_zroot   local

2021-09-03 12:19:06 toor at f2 ~
# trim -f -v /bootpool
trim /bootpool offset 0 length 3
trim: open failed: /bootpool: Is a directory

2021-09-03 12:20:00 toor at f2 ~
# trim -f -v /f2_zroot
trim /f2_zroot offset 0 length 2
trim: open failed: /f2_zroot: Is a directory

2021-09-03 12:21:49 toor at f2 ~
# trim -f -v ada0
trim ada0 offset 0 length 60022480896
trim: open failed: ada0: Operation not permitted

2021-09-03 12:22:52 toor at f2 ~
# trim -f -v ada0s1
trim ada0s1 offset 0 length 15032385536
trim: open failed: ada0s1: Operation not permitted

2021-09-03 12:23:55 toor at f2 ~
# trim -f -v ada0s1a
trim ada0s1a offset 0 length 2147483648
trim: open failed: ada0s1a: Operation not permitted

2021-09-03 12:24:57 toor at f2 ~
# trim -f -v ada0s1b
trim ada0s1b offset 0 length 2147483648
trim: open failed: ada0s1b: Operation not permitted

2021-09-03 12:25:58 toor at f2 ~
# trim -f -v ada0s1d
trim ada0s1d offset 0 length 10737418240
trim: open failed: ada0s1d: Operation not permitted

2021-09-03 12:26:57 toor at f2 ~
# man 4 ada | grep -i trim

2021-09-03 12:27:35 toor at f2 ~
# man 4 ada | grep -i unmap

2021-09-03 13:24:51 toor at f2 ~
# man 8 zpool | grep -i trim

2021-09-03 13:24:54 toor at f2 ~
# man 8 zpool | grep -i unmap

2021-09-03 13:24:55 toor at f2 ~
# man 8 zfs | grep -i trim

2021-09-03 13:25:07 toor at f2 ~
# man 8 zfs | grep -i unmap

My WAG:

1.  trim(8) is not integrated with ZFS.

2.  ZFS does not provide a TRIM configuration settings or userland tools.

3.  Given the disk was zeroed before installation and given the 
monotonically increasing sizes of the 'gzip -1' compressed raw images I 
take monthly, TRIM is not implemented by ZFS.

David