Re: {* 05.00 *}Re: Desperate with 870 QVO and ZFS

Am 06.04.22 um 18:34 schrieb egoitz@ramattack.net:

> Hi Stefan!
>
> Thank you so much for your answer!!. I do answer below in green bold for
> instance... for a better distinction....
>
> Very thankful for all your comments Stefan!!! :) :) :)
>
> Cheers!!
>
Hi,

glad to hear that it is useful information - I'll add comments below ...

> El 2022-04-06 17:43, Stefan Esser escribió:
>
>> ATENCION
>> ATENCION
>> ATENCION!!! Este correo se ha enviado desde fuera de la organizacion. No
>> pinche en los enlaces ni abra los adjuntos a no ser que reconozca el
>> remitente y sepa que el contenido es seguro.
>>
>> Am 06.04.22 um 16:36 schrieb egoitz@ramattack.net:
>>> Hi Rainer!
>>>
>>> Thank you so much for your help :) :)
>>>
>>> Well I assume they are in a datacenter and should not be a power outage....
>>>
>>> About dataset size... yes... our ones are big... they can be 3-4 TB easily each
>>> dataset.....
>>>
>>> We bought them, because as they are for mailboxes and mailboxes grow and
>>> grow.... for having space for hosting them...
>>
>> Which mailbox format (e.g. mbox, maildir, ...) do you use?
>>  
>> *I'm running Cyrus imap so sort of Maildir... too many little files
>> normally..... Sometimes directories with tons of little files....*

Assuming that many mails are much smaller than the erase block size of the SSD,
this may cause issues. (You may know the following ...)

For example, if you have message sizes of 8 KB and an erase block size of 64 KB
(just guessing), then 8 mails will be in an erase block. If half the mails are
deleted, then the erase block will still occupy 64 KB, but only hold 32 KB of
useful data (and the SSD will only be aware of this fact if TRIM has signaled
which data is no longer relevant). The SSD will copy several partially filled
erase blocks together in a smaller number of free blocks, which then are fully
utilized. Later deletions will repeat this game, and your data will be copied
multiple times until it has aged (and the user is less likely to delete further
messages). This leads to "write amplification" - data is internally moved
around and thus written multiple times.

Larger mails are less of an issue since they span multiple erase blocks, which
will be completely freed when such a message is deleted.

Samsung has a lot of experience and generally good strategies to deal with such
a situation, but SSDs specified for use in storage systems might be much better
suited for that kind of usage profile.

>>> We knew they had some speed issues, but those speed issues, we thought (as
>>> Samsung explains in the QVO site) they started after exceeding the speeding
>>> buffer this disks have. We though that meanwhile you didn't exceed it's
>>> capacity (the capacity of the speeding buffer) no speed problem arises. Perhaps
>>> we were wrong?.
>>
>> These drives are meant for small loads in a typical PC use case,
>> i.e. some installations of software in the few GB range, else only
>> files of a few MB being written, perhaps an import of media files
>> that range from tens to a few hundred MB at a time, but less often
>> than once a day.
>>  
>> *We move, you know... lots of little files... and lot's of different
>> concurrent modifications by 1500-2000 concurrent imap connections we have...*

I do not expect the read load to be a problem (except possibly when the SSD is
moving data from SLC to QLC blocks, but even then reads will get priority). But
writes and trims might very well overwhelm the SSD, especially when its getting
full. Keeping a part of the SSD unused (excluded from the partitions created)
will lead to a large pool of unused blocks. This will reduce the write
amplification - there are many free blocks in the "unpartitioned part" of the
SSD, and thus there is less urgency to compact partially filled blocks. (E.g.
if you include only 3/4 of the SSD capacity in a partition used for the ZPOOL,
then 1/4 of each erase block could be free due to deletions/TRIM without any
compactions required to hold all this data.)

Keeping a significant percentage of the SSD unallocated is a good strategy to
improve its performance and resilience.

>> As the SSD fills, the space available for the single level write
>> cache gets smaller
>>  
>> *The single level write cache is the cache these ssd drivers have, for
>> compensating the speed issues they have due to using qlc memory?. Do you
>> refer to that?. Sorry I don't understand well this paragraph.*

Yes, the SSD is specified to hold e.g. 1 TB at 4 bits per cell. The SLC cache
has only 1 bit per cell, thus a 6 GB SLC cache needs as many cells as 24 GB of
data in QLC mode.

A 100 GB SLC cache would reduce the capacity of a 1 TB SSD to 700 GB (600 GB in
150 tn QLC cells plus 100 GB in 100 tn SLC cells).

Therefore, the fraction of the cells used as an SLC cache is reduced when it
gets full (e.g. ~1 TB in ~250 tn QLC cells, plus 6 GB in 6 tn SLC cells).

And with less SLC cells available for short term storage of data the
probability of data being copied to QLC cells before the irrelevant messages
have been deleted is significantly increased. And that will again lead to many
more blocks with "holes" (deleted messages) in them, which then need to be
copied possibly multiple times to compact them.

>> (on many SSDs, I have no numbers for this
>> particular device), and thus the amount of data that can be
>> written at single cell speed shrinks as the SSD gets full.
>>  
>>
>>
>> I have just looked up the size of the SLC cache, it is specified
>> to be 78 GB for the empty SSD, 6 GB when it is full (for the 2 TB
>> version, smaller models will have a smaller SLC cache).
>>  
>> *Assuming you were talking about the cache for compensating speed we
>> previously commented, I should say these are the 870 QVO but the 8TB
>> version. So they should have the biggest cache for compensating the speed
>> issues...*

I have looked up the data: the larger versions of the 870 QVO have the same SLC
cache configuration as the 2 TB model, 6 GB minimum and up to 72 GB more if
there are enough free blocks.

>> But after writing those few GB at a speed of some 500 MB/s (i.e.
>> after 12 to 150 seconds), the drive will need several minutes to
>> transfer those writes to the quad-level cells, and will operate
>> at a fraction of the nominal performance during that time.
>> (QLC writes max out at 80 MB/s for the 1 TB model, 160 MB/s for the
>> 2 TB model.)
>>  
>> *Well we are in the 8TB model. I think I have understood what you wrote in
>> previous paragraph. You said they can be fast but not constantly, because
>> later they have to write all that to their perpetual storage from the cache.
>> And that's slow. Am I wrong?. Even in the 8TB model you think Stefan?.*

The controller in the SSD supports a given number of channels (e.g 4), each of
which can access a Flash chip independently of the others. Small SSDs often
have less Flash chips than there are channels (and thus a lower throughput,
especially for writes), but the larger models often have more chips than
channels and thus the performance is capped.

In the case of the 870 QVO, the controller supports 8 channels, which allows it
to write 160 MB/s into the QLC cells. The 1 TB model apparently has only 4
Flash chips and is thus limited to 80 MB/s in that situation, while the larger
versions have 8, 16, or 32 chips. But due to the limited number of channels,
the write rate is limited to 160 MB/s even for the 8 TB model.

If you had 4 * 2 TB instead, the throughput would be 4 * 160 MB/s in this limit.

>> *The main problem we are facing is that in some peak moments, when the
>> machine serves connections for all the instances it has, and only as said in
>> some peak moments... like the 09am or the 11am.... it seems the machine
>> becomes slower... and like if the disks weren't able to serve all they have
>> to serve.... In these moments, no big files are moved... but as we have
>> 1800-2000 concurrent imap connections... normally they are doing each one...
>> little changes in their mailbox. Do you think perhaps this disks then are
>> not appropriate for this kind of usage?-*

I'd guess that the drives get into a state in which they have to recycle lots
of partially free blocks (i.e. perform kind of a garbage collection) and then
three kinds of operations are competing with each other:

 1. reads (generally prioritized)
 2. writes (filling the SLC cache up to its maximum size)
 3. compactions of partially filled blocks (required to make free blocks
    available for re-use)

Writes can only proceed if there are sufficient free blocks, which on a filled
SSD with partially filled erase blocks means that operations of type 3. need to
be performed with priority to not stall all writes.

My assumption is that this is what you are observing under peak load.

>> And cheap SSDs often have no RAM cache (not checked, but I'd be
>> surprised if the QVO had one) and thus cannot keep bookkeeping date
>> in such a cache, further limiting the performance under load.
>>  
>> *This brochure
>> (https://semiconductor.samsung.com/resources/brochure/870_Series_Brochure.pdf
>> and the datasheet
>> https://semiconductor.samsung.com/resources/data-sheet/Samsung_SSD_870_QVO_Data_Sheet_Rev1.1.pdf)
>> sais if I have read properly, the 8TB drive has 8GB of ram?. I assume that
>> is what they call the turbo write cache?.*

No, the turbo write cache consists of the cells used in SLC mode (which can be
any cells, not only cells in a specific area of the flash chip).

The RAM is needed for fast lookup of the position of data for reads and of free
blocks for writes.

There is no simple relation between SSD "block number" (in the sense of a disk
block on some track of a magnetic disk) and its storage location on the Flash
chip. If an existing "data block" (what would be a sector on a hard disk drive)
is overwritten, it is instead written at the end of an "open" erase block, and
a pointer from that "block number" to the location on the chip is stored in an
index. This index is written to Flash storage and could be read from it, but it
is much faster to have a RAM with these pointers that can be accessed
independently of the Flash chips. This RAM is required for high transaction
rates (especially random reads), but it does not really help speed up writes.

>> And the resilience (max. amount of data written over its lifetime)
>> is also quite low - I hope those drives are used in some kind of
>> RAID configuration.
>>  
>> *Yep we use raidz-2*

Makes sense ... But you know that you multiply the amount of data written due
to the redundancy.

If a single 8 KB block is written, for example, 3 * 8 KB will written if you
take the 2 redundant copies into account.

>> The 870 QVO is specified for 370 full capacity
>> writes, i.e. 370 TB for the 1 TB model. That's still a few hundred
>> GB a day - but only if the write amplification stays in a reasonable
>> range ...
>>  
>> *Well yes... 2880TB in our case....not bad.. isn't it?*

I assume that 2880 TB is your total storage capacity? That's not too bad, in
fact. ;-)

This would be 360 * 8 TB ...

Even at 160 MB/s per 8 TB SSD this would allow for more than 50 GB/s of write
throughput (if all writes were evenly distributed).

Taking all odds into account, I'd guess that at least 10 GB/s can be
continuously written (if supported by the CPUs and controllers).

But this may not be true if the drive is simultaneously reading, trimming, and
writing ...


I have seen advice to not use compression in a high load scenario in some other
reply.

I tend to disagree: Since you seem to be limited when the SLC cache is
exhausted, you should get better performance if you compress your data. I have
found that zstd-2 works well for me (giving a significant overall reduction of
size at reasonable additional CPU load). Since ZFS allows to switch
compressions algorithms at any time, you can experiment with different
algorithms and levels.

One advantage of ZFS compression is that it applies to the ARC, too. And a
compression factor of 2 should easily be achieved when storing mail (not for
.docx, .pdf, .jpg files though). Having more data in the ARC will reduce the
read pressure on the SSDs and will give them more cycles for garbage
collections (which are performed in the background and required to always have
a sufficient reserve of free flash blocks for writes).

I'd give it a try - and if it reduces your storage requirements by 10% only,
then keep 10% of each SSD unused (not assigned to any partition). That will
greatly improve the resilience of your SSDs, reduce the write-amplification,
will allow the SLC cache to stay at its large value, and may make a large
difference to the effective performance under high load.

Regards, STefan

**