Re: newfs TRIM flag device support

On Friday, 16 February 2024 at 18:54, Ordinary Bit <ordinarybit@proton.me> wrote:

> On Friday, 16 February 2024 at 18:31, Ronald Klop <ronald-lists@klop.ws> wrote:
>
>> Van: Ordinary Bit <ordinarybit@proton.me>
>> Datum: vrijdag, 16 februari 2024 10:18
>> Aan: Mark Millard <marklmi@yahoo.com>
>> CC: "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org>
>> Onderwerp: Re: newfs TRIM flag device support
>>
>>> Sent with Proton Mail secure email.
>>>
>>> On Friday, 16 February 2024 at 14:02, Mark Millard <marklmi@yahoo.com> wrote:
>>>
>>>> On Feb 15, 2024, at 20:08, Ordinary Bit ordinarybit@proton.me wrote:
>>>>
>>>> > Hi!
>>>>
>>>>
>>>> Hello.
>>>>
>>>> > On Friday, 16 February 2024 at 11:41, Mark Millard marklmi@yahoo.com wrote:
>>>> >
>>>> > > [Only replying to lists I subscribe to.]
>>>> > >
>>>> > > On Feb 15, 2024, at 19:19, Ordinary Bit ordinarybit@proton.me wrote:
>>>> > >
>>>> > > > I'm reading the newfs manual https://man.freebsd.org/cgi/man.cgi?newfs(8) to be able to know about the TRIM flag. In the manual under -t parameter, it mentioned about "underlying device support", what exactly is this device?
>>>> > >
>>>> > > 2 contrasting examples:
>>>> > >
>>>> > > Example 0: Optane NVMe media (PCIe card or U.2, for example)
>>>> > >
>>>> > > Optane has no need of TRIM and, so, never supports TRIM.
>>>> > >
>>>> > > Example 1: microsd card media usage
>>>> > >
>>>> > > A microsd card in the normal type of microsd card slot on
>>>> > > Small Board Computers (normally) supports TRIM. Take the
>>>> > > same card and put it in a USB reader/writer and use it
>>>> > > via USB on the same system: no TRIM is supported by
>>>> > > FreeBSD over USB.
>>>> >
>>>> > So you mean to say that if I have a Rasperry Pi 3 or 4 now and then have my FreeBSD installed in a microSD card (for example, SanDisk Extreme card) with UFS/FFS filesytem in it with TRIM enabled parameter then is it going to recognize it? How to verify?
>>>> >
>>>> > > FYI:
>>>> > >
>>>> > > When the file system has TRIM enabled, FreeBSD put out a
>>>> > > notice if TRIM will not actually be used in the actual
>>>> > > context in use.
>>>> >
>>>> > Ok, got it. How to check this as well?
>>>>
>>>>
>>>> The console gets a message like:
>>>>
>>>> WARNING: /mnt: TRIM flag on fs but disk does not support TRIM
>>>>
>>>> when a mount is attempted (automatic or manually) for a
>>>> file system with the trim flag enabled but trim does not
>>>> end up active.
>>>>
>>>> So, for example:
>>>>
>>>> # dmesg -a | grep TRIM
>>>> WARNING: /mnt: TRIM flag on fs but disk does not support TRIM
>>>>
>>>> (This was a microsd card in a USB reader/writer that was not
>>>> used as the boot media: a separate manual mount was used.)
>>>>
>>>> The file system's TRIM flag status can be checked via use of
>>>> "tunefs -p . . .":
>>>>
>>>> # tunefs -p /mnt
>>>> tunefs: POSIX.1e ACLs: (-a) disabled
>>>> tunefs: NFSv4 ACLs: (-N) disabled
>>>> tunefs: MAC multilabel: (-l) disabled
>>>> tunefs: soft updates: (-n) enabled
>>>> tunefs: soft update journaling: (-j) disabled
>>>> tunefs: gjournal: (-J) disabled
>>>> tunefs: trim: (-t) enabled
>>>> tunefs: maximum blocks per file in a cylinder group: (-e) 4096
>>>> tunefs: average file size: (-f) 16384
>>>> tunefs: average number of files in a directory: (-s) 64
>>>> tunefs: minimum percentage of free space: (-m) 8%
>>>> tunefs: space to hold for metadata blocks: (-k) 6400
>>>> tunefs: optimization preference: (-o) time
>>>> tunefs: volume label: (-L)
>>>>
>>>> If the trim flag is enabled but the mount does not produce the
>>>> console message, then TRIM is active.
>>>>
>>>
>>> Oh, that's amazing! I try it with my SanDisk Ultra microSDXC 64GB and mount it in a USB reader and it shows enabled in the tunefs and is detected in the dmesg, the same as yours. Therefore, SanDisk Ultra microSD supports it!
>>>
>>> root@sd-extreme:~ # /sbin/newfs -U -t /dev/da0s2a
>>> /dev/da0s2a: 59000.9MB (120833920 sectors) block size 32768, fragment size 4096
>>> using 95 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
>>> with soft updates
>>> super-block backups (for fsck_ffs -b #) at:
>>> 192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880, 8963328, 10243776, 11524224, 12804672, 14085120,
>>> 15365568, 16646016, 17926464, 19206912, 20487360, 21767808, 23048256, 24328704, 25609152, 26889600, 28170048,
>>> 29450496, 30730944, 32011392, 33291840, 34572288, 35852736, 37133184, 38413632, 39694080, 40974528, 42254976,
>>> 43535424, 44815872, 46096320, 47376768, 48657216, 49937664, 51218112, 52498560, 53779008, 55059456, 56339904,
>>> 57620352, 58900800, 60181248, 61461696, 62742144, 64022592, 65303040, 66583488, 67863936, 69144384, 70424832,
>>> 71705280, 72985728, 74266176, 75546624, 76827072, 78107520, 79387968, 80668416, 81948864, 83229312, 84509760,
>>> 85790208, 87070656, 88351104, 89631552, 90912000, 92192448, 93472896, 94753344, 96033792, 97314240, 98594688,
>>> 99875136, 101155584, 102436032, 103716480, 104996928, 106277376, 107557824, 108838272, 110118720, 111399168,
>>> 112679616, 113960064, 115240512, 116520960, 117801408, 119081856, 120362304
>>>
>>> root@sd-extreme:~ # dmesg | grep TRIM
>>> WARNING: /mnt: TRIM flag on fs but disk does not support TRIM
>>>
>>> root@sd-extreme:~ # tunefs -p /mnt
>>> tunefs: POSIX.1e ACLs: (-a) disabled
>>> tunefs: NFSv4 ACLs: (-N) disabled
>>> tunefs: MAC multilabel: (-l) disabled
>>> tunefs: soft updates: (-n) enabled
>>> tunefs: soft update journaling: (-j) disabled
>>> tunefs: gjournal: (-J) disabled
>>> tunefs: trim: (-t) enabled
>>> tunefs: maximum blocks per file in a cylinder group: (-e) 4096
>>> tunefs: average file size: (-f) 16384
>>> tunefs: average number of files in a directory: (-s) 64
>>> tunefs: minimum percentage of free space: (-m) 8%
>>> tunefs: space to hold for metadata blocks: (-k) 6400
>>> tunefs: optimization preference: (-o) time
>>> tunefs: volume label: (-L)
>>>
>>> I plan to have TRIM flag enabled in the rootfs partition (/dev/ufs/rootfs) of my Raspberry Pi. Do you think of any implications when enabled? As shown below, it is disabled.
>>>
>>> root@sd-extreme:~ # tunefs -p /dev/ufs/rootfs
>>> tunefs: POSIX.1e ACLs: (-a) disabled
>>> tunefs: NFSv4 ACLs: (-N) disabled
>>> tunefs: MAC multilabel: (-l) disabled
>>> tunefs: soft updates: (-n) enabled
>>> tunefs: soft update journaling: (-j) disabled
>>> tunefs: gjournal: (-J) disabled
>>> tunefs: trim: (-t) disabled
>>> tunefs: maximum blocks per file in a cylinder group: (-e) 4096
>>> tunefs: average file size: (-f) 16384
>>> tunefs: average number of files in a directory: (-s) 64
>>> tunefs: minimum percentage of free space: (-m) 8%
>>> tunefs: space to hold for metadata blocks: (-k) 6400
>>> tunefs: optimization preference: (-o) time
>>> tunefs: volume label: (-L) rootfs
>>>
>>> BR,
>>> orbit
>>>
>>> ---------------------------------------------------------------
>>
>> Hi,
>>
>> I'm sorry to spoil the fun but this message says the device does *not* support trim.
>>
>> "root@sd-extreme:~ # dmesg | grep TRIM
>> WARNING: /mnt: TRIM flag on fs but disk does not support TRIM"
>>
>> But to give some hope. As Mark pointed out it is not only about the disk. The controller also needs to support it.
>>
>> As a real world example:
>>
>> I have an RPI3 with an SSD (supporting trim) connected via USB-to-SATA (not supporting trim). So the total system does not support trim.
>> $ cat /var/run/dmesg.boot | grep -E "umass0|da0|TRIM"
>> umass0 on uhub1
>> umass0: <GODO USB3.0 to SATA adapter, class 0/0, rev 2.10/1.00, addr 5> on usbus1
>> umass0: SCSI over Bulk-Only; quirks = 0x0100
>> umass0:0:0: Attached to scbus0
>> da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
>> da0: <LITEON C V3-DE256 0> Fixed Direct Access SPC-4 SCSI device
>> da0: Serial Number 2015100000D6
>> da0: 40.000MB/s transfers
>> da0: 244198MB (500118192 512 byte sectors)
>> da0: quirks=0x2<NO_6_BYTE>
>> WARNING: /: TRIM flag on fs but disk does not support TRIM
>>
>> $ sysctl kern.cam.da.0 | grep -E "trim|delete"
>> kern.cam.da.0.trim_ticks: 0
>> kern.cam.da.0.trim_goal: 0
>> kern.cam.da.0.trim_lbas: 0
>> kern.cam.da.0.trim_ranges: 0
>> kern.cam.da.0.trim_count: 0
>> kern.cam.da.0.delete_max: 65536
>> kern.cam.da.0.delete_method: NONE
>>
>> On an RPI4B I have an SSD (supporting trim) connected via USB-to-SATA (which does support trim also). Here trim works fine. (NB: this uses ZFS, but that does not matter)
>> $ dmesg | grep -E "umass0|da0"
>> umass0 on uhub0
>> umass0: <USB 3.0 Device USB 3.0 Device, class 0/0, rev 3.00/3.01, addr 3> on usbus0
>> da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
>> da0: <Micron 1 100 SATA 256GB 0301> Fixed Direct Access SPC-4 SCSI device
>> da0: Serial Number 000000004BA8
>> da0: 400.000MB/s transfers
>> da0: 244198MB (500118192 512 byte sectors)
>> da0: quirks=0x2<NO_6_BYTE>
>>
>> $ sysctl kern.cam.da.0 | grep -E "trim|delete"
>> kern.cam.da.0.trim_ticks: 0
>> kern.cam.da.0.trim_goal: 0
>> kern.cam.da.0.trim_lbas: 546841704
>> kern.cam.da.0.trim_ranges: 317280
>> kern.cam.da.0.trim_count: 301424
>> kern.cam.da.0.delete_max: 4294901760
>> kern.cam.da.0.delete_method: UNMAP
>>
>> BTW: enabling trim on the fs while the device does not support it does not matter and does no harm. Using the sysctl above you can see if the device actually uses trim.
>>
>> Running "gstat -d" while deleting a lot of files will also show if trim/delete actions are sent to the disk.
>>
>> So the combination of your USB reader + microSD do not support TRIM, but you can just try to put the microSD directly in the RPI4 and see what that does.
>>
>> Regards,
>> Ronald.
>
> Thanks for the callout! Got warmed-up when I saw the TRIM is enabled :-) but my ultimate goal is really the microSD card slot of my Raspberry Pi 3 or 4 and not with the USB reader. I'll create an image then see how it does.
>
> BR,
> orbit

Confirmed, the SanDisk Ultra microSD does not support TRIM using the microSD slot in both Raspberry Pi 3 and 4. Instead of building an image from scratch, I downloaded a FreeBSD 14.0 image and created a new partition (8GB in size) with UFS/FFS with TRIM enabled and mount it. The partition is /dev/mmcsd0s3. There's no delete activity observed with "gstat -d" when I deleted some files in it.

root@sd-extreme:~ # /sbin/gpart add -s 8G -t freebsd da0
da0s3 added
root@sd-extreme:~ # gpart show
=> 63 124735425 mmcsd0 MBR (59G)
63 1985 - free - (993K)
2048 102400 1 fat32lba [active] (50M)
104448 124626944 2 freebsd (59G)
124731392 4096 - free - (2.0M)

=> 0 124626944 mmcsd0s2 BSD (59G)
0 128 - free - (64K)
128 120833920 1 freebsd-ufs (58G)
120834048 3792896 2 freebsd-swap (1.8G)

=> 63 124735425 da0 MBR (59G)
63 1985 - free - (993K)
2048 102400 1 fat32lba [active] (50M)
104448 10381312 2 freebsd (5.0G)
10485760 16777216 3 freebsd (8.0G)
27262976 97472512 - free - (46G)

=> 0 10381312 da0s2 BSD (5.0G)
0 128 - free - (64K)
128 10381184 1 freebsd-ufs (4.9G)

=> 63 124735425 diskid/DISK-121220160204 MBR (59G)
63 1985 - free - (993K)
2048 102400 1 fat32lba [active] (50M)
104448 10381312 2 freebsd (5.0G)
10485760 16777216 3 freebsd (8.0G)
27262976 97472512 - free - (46G)

=> 0 10381312 diskid/DISK-121220160204s2 BSD (5.0G)
0 128 - free - (64K)
128 10381184 1 freebsd-ufs (4.9G)

root@sd-extreme:~ # ls -lah /dev/da0
/dev/da0 /dev/da0s1 /dev/da0s2 /dev/da0s2a /dev/da0s3

root@sd-extreme:~ # newfs -U -t /dev/da0s3
/dev/da0s3: 8192.0MB (16777216 sectors) block size 32768, fragment size 4096
using 14 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
with soft updates
super-block backups (for fsck_ffs -b #) at:
192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880, 8963328, 10243776, 11524224, 12804672, 14085120,
15365568, 16646016

root@sd-ultra:~ # gpart show -l
=> 63 124735425 mmcsd0 MBR (59G)
63 1985 - free - (993K)
2048 102400 1 (null) [active] (50M)
104448 10381312 2 (null) (5.0G)
10485760 16777216 3 (null) (8.0G)
27262976 97472512 - free - (46G)

=> 0 10381312 mmcsd0s2 BSD (5.0G)
0 128 - free - (64K)
128 10381184 1 (null) (4.9G)

root@sd-ultra:~ # mount -l
/dev/ufs/rootfs on / (ufs, local, soft-updates)
devfs on /dev (devfs)
/dev/msdosfs/EFI on /boot/efi (msdosfs, local, noatime)
tmpfs on /tmp (tmpfs, local)
/dev/mmcsd0s3 on /mnt (ufs, local, soft-updates)

root@sd-ultra:~ # gstat -d
dT: 1.065s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| mmcsd0
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| mmcsd0s1
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| mmcsd0s2
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| mmcsd0s3
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| msdosfs/EFI
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| mmcsd0s2a 0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0| ufs/rootfs
root@sd-ultra:~ # mount -l -v
/dev/ufs/rootfs on / (ufs, local, soft-updates, writes: sync 838 async 4316, reads: sync 2241 async 2697, fsid 89fc4d652aba17cd, vnodes: count 1264 )
devfs on /dev (devfs, fsid 00ff007171000000, vnodes: count 34 )
/dev/msdosfs/EFI on /boot/efi (msdosfs, local, noatime, writes: sync 1 async 0, reads: sync 8 async 0, fsid 5b00000032000000, vnodes: count 1 )
tmpfs on /tmp (tmpfs, local, fsid 01ff008787000000, vnodes: count 6 )
/dev/mmcsd0s3 on /mnt (ufs, local, soft-updates, writes: sync 2 async 7866, reads: sync 121491 async 0, fsid 4ee0c86506330890, vnodes: count 3 )

root@sd-ultra:~ # sysctl -a | grep trim
<118>Creating and/or trimming log files.
kern.cam.nda.max_trim: 256vfs.ffs.dotrimcons: 1

I'll proceed with plan B, to buy industrial grade microSD card having a garbage collection feature in the controller instead of this consumer grade card. Either TRIM will work or not there's still garbage collection I could benefit.

BR,
orbit

>