Re: Re: Re: Desperate with 870 QVO and ZFS

Hi Stefan, 

An extremely interesting answer and email. Extremely thankful for all
your deep explatanations...... They are like gold for us really.... 

I answer below and in blue bold for better distinction between your
lines and mine ones...

El 2022-04-06 23:49, Stefan Esser escribió:

> ATENCION: Este correo se ha enviado desde fuera de la organización. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro.
> 
> Am 06.04.22 um 18:34 schrieb egoitz@ramattack.net:
> 
>> Hi Stefan! 
>> 
>> Thank you so much for your answer!!. I do answer below in green bold for instance... for a better distinction.... 
>> 
>> Very thankful for all your comments Stefan!!! :) :) :) 
>> 
>> Cheers!!
> 
> Hi, 
> 
> glad to hear that it is useful information - I'll add comments below ... 
> 
> EXTREMELY HELPFUL INFORMATION REALLY! THANK YOU SO MUCH STEFFAN REALLY. VERY VERY THANKFUL FOR YOUR NICE HELP!. 
> 
> El 2022-04-06 17:43, Stefan Esser escribió: 
> 
> Am 06.04.22 um 16:36 schrieb egoitz@ramattack.net: Hi Rainer!
> 
> Thank you so much for your help :) :)
> 
> Well I assume they are in a datacenter and should not be a power outage....
> 
> About dataset size... yes... our ones are big... they can be 3-4 TB easily each
> dataset.....
> 
> We bought them, because as they are for mailboxes and mailboxes grow and
> grow.... for having space for hosting them... 
> Which mailbox format (e.g. mbox, maildir, ...) do you use? 
> 
> I'M RUNNING CYRUS IMAP SO SORT OF MAILDIR... TOO MANY LITTLE FILES NORMALLY..... SOMETIMES DIRECTORIES WITH TONS OF LITTLE FILES....

Assuming that many mails are much smaller than the erase block size of
the SSD, this may cause issues. (You may know the following ...) 

For example, if you have message sizes of 8 KB and an erase block size
of 64 KB (just guessing), then 8 mails will be in an erase block. If
half the mails are deleted, then the erase block will still occupy 64
KB, but only hold 32 KB of useful data (and the SSD will only be aware
of this fact if TRIM has signaled which data is no longer relevant). The
SSD will copy several partially filled erase blocks together in a
smaller number of free blocks, which then are fully utilized. Later
deletions will repeat this game, and your data will be copied multiple
times until it has aged (and the user is less likely to delete further
messages). This leads to "write amplification" - data is internally
moved around and thus written multiple times. 

STEFAN!! YOU ARE NICE!! I THINK THIS COULD EXPLAIN ALL OUR PROBLEM. SO,
WHY WE ARE HAVING THE MOST RANDOMNESS IN OUR PERFORMANCE DEGRADATION AND
THAT DOES NOT NECESSARILY HAS TO MATCH WITH THE MOST IO PEAK HOURS...
THAT I COULD CAUSE THAT PERFORMANCE DEGRADATION JUST BY DELETING A
COUPLE OF HUGE (PERHAPS 200.000 MAILS) MAIL FOLDERS IN A MIDDLE TRAFFIC
HOUR TIME!! 

THE PROBLEM IS THAT BY WHAT I KNOW, ERASE BLOCK SIZE OF AN SSD DISK IS
SOMETHING FIXED IN THE DISK FIRMWARE. I DON'T REALLY KNOW IF PERHAPS IT
COULD BE MODIFIED WITH SAMSUNG MAGICIAN OR THOSE KIND OF TOOL OF
SAMSUNG.... ELSE I DON'T REALLY SEE THE MANNER OF IMPROVING IT...
BECAUSE APART FROM THAT, YOU ARE DELETING A FILE IN RAIDZ-2 ARRAY... NO
JUST IN A DISK... I ASSUME ALIGNING CHUNK SIZE, WITH RECORD SIZE AND
WITH THE "SECRET" ERASE SIZE OF THE SSD, PERHAPS COULD BE SLIGHTLY
COMPENSATED?. 

Larger mails are less of an issue since they span multiple erase blocks,
which will be completely freed when such a message is deleted. 

I SEE I SEE STEFAN... 

Samsung has a lot of experience and generally good strategies to deal
with such a situation, but SSDs specified for use in storage systems
might be much better suited for that kind of usage profile. 

YES... AND THE DISKS FOR OUR PURPOSE... PERHAPS WEREN'T QVOS.... 

> We knew they had some speed issues, but those speed issues, we thought (as
> Samsung explains in the QVO site) they started after exceeding the speeding
> buffer this disks have. We though that meanwhile you didn't exceed it's
> capacity (the capacity of the speeding buffer) no speed problem arises. Perhaps
> we were wrong?. 
> These drives are meant for small loads in a typical PC use case,
> i.e. some installations of software in the few GB range, else only
> files of a few MB being written, perhaps an import of media files
> that range from tens to a few hundred MB at a time, but less often
> than once a day. 
> 
> WE MOVE, YOU KNOW... LOTS OF LITTLE FILES... AND LOT'S OF DIFFERENT CONCURRENT MODIFICATIONS BY 1500-2000 CONCURRENT IMAP CONNECTIONS WE HAVE...

I do not expect the read load to be a problem (except possibly when the
SSD is moving data from SLC to QLC blocks, but even then reads will get
priority). But writes and trims might very well overwhelm the SSD,
especially when its getting full. Keeping a part of the SSD unused
(excluded from the partitions created) will lead to a large pool of
unused blocks. This will reduce the write amplification - there are many
free blocks in the "unpartitioned part" of the SSD, and thus there is
less urgency to compact partially filled blocks. (E.g. if you include
only 3/4 of the SSD capacity in a partition used for the ZPOOL, then 1/4
of each erase block could be free due to deletions/TRIM without any
compactions required to hold all this data.) 

Keeping a significant percentage of the SSD unallocated is a good
strategy to improve its performance and resilience. 

WELL, WE HAVE ALLOCATED ALL THE DISK SPACE... BUT NOT USED... JUST
ALLOCATED.... YOU KNOW... WE DO A ZPOOL CREATE WITH THE WHOLE DISKS.....

>> As the SSD fills, the space available for the single level write
>> cache gets smaller 
>> 
>> THE SINGLE LEVEL WRITE CACHE IS THE CACHE THESE SSD DRIVERS HAVE, FOR COMPENSATING THE SPEED ISSUES THEY HAVE DUE TO USING QLC MEMORY?. DO YOU REFER TO THAT?. SORRY I DON'T UNDERSTAND WELL THIS PARAGRAPH.

Yes, the SSD is specified to hold e.g. 1 TB at 4 bits per cell. The SLC
cache has only 1 bit per cell, thus a 6 GB SLC cache needs as many cells
as 24 GB of data in QLC mode. 

OK, TRUE.... YES.... 

A 100 GB SLC cache would reduce the capacity of a 1 TB SSD to 700 GB
(600 GB in 150 tn QLC cells plus 100 GB in 100 tn SLC cells). 

AHH! YOU MEAN THAT SLC CAPACITY FOR SPEEDING UP THE QLC DISKS, IS
OBTAINED FROM EACH SINGLE LAYER OF THE QLC?. 

Therefore, the fraction of the cells used as an SLC cache is reduced
when it gets full (e.g. ~1 TB in ~250 tn QLC cells, plus 6 GB in 6 tn
SLC cells). 

SORRY I DON'T GET THIS LAST SENTENCE... DON'T UNDERSTAND IT BECAUSE I
DON'T REALLY KNOW THE MEANING OF TN... 

BUT I THINK I'M GETTING THE IDEA IF YOU SAY THAT EACH QLC LAYER, HAS
IT'S OWN SLC CACHE OBTAINED FROM THE DISK SPACE AVAIABLE FOR EACH QLC
LAYER.... 

And with less SLC cells available for short term storage of data the
probability of data being copied to QLC cells before the irrelevant
messages have been deleted is significantly increased. And that will
again lead to many more blocks with "holes" (deleted messages) in them,
which then need to be copied possibly multiple times to compact them. 

IF I CORRECT ABOVE, I THINK I GOT THE IDEA YES....

>> (on many SSDs, I have no numbers for this
>> particular device), and thus the amount of data that can be
>> written at single cell speed shrinks as the SSD gets full. 
>> 
>> I have just looked up the size of the SLC cache, it is specified
>> to be 78 GB for the empty SSD, 6 GB when it is full (for the 2 TB
>> version, smaller models will have a smaller SLC cache). 
>> 
>> ASSUMING YOU WERE TALKING ABOUT THE CACHE FOR COMPENSATING SPEED WE PREVIOUSLY COMMENTED, I SHOULD SAY THESE ARE THE 870 QVO BUT THE 8TB VERSION. SO THEY SHOULD HAVE THE BIGGEST CACHE FOR COMPENSATING THE SPEED ISSUES...

I have looked up the data: the larger versions of the 870 QVO have the
same SLC cache configuration as the 2 TB model, 6 GB minimum and up to
72 GB more if there are enough free blocks. 

OURS ONE IS THE 8TB MODEL SO I ASSUME IT COULD HAVE BIGGER LIMITS. THE
DISKS ARE MOSTLY EMPTY, REALLY.... SO... FOR INSTANCE.... 

ZPOOL LIST
NAME             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP 
DEDUP  HEALTH  ALTROOT
ROOT_DATASET             448G  2.29G   446G        -         -     1%   
 0%  1.00X  ONLINE  -
MAIL_DATASET  58.2T  11.8T  46.4T        -         -    26%    20% 
1.00X  ONLINE  - 

I SUPPOSE FRAGMENTATION AFFECTS TOO....

>> But after writing those few GB at a speed of some 500 MB/s (i.e.
>> after 12 to 150 seconds), the drive will need several minutes to
>> transfer those writes to the quad-level cells, and will operate
>> at a fraction of the nominal performance during that time.
>> (QLC writes max out at 80 MB/s for the 1 TB model, 160 MB/s for the
>> 2 TB model.) 
>> 
>> WELL WE ARE IN THE 8TB MODEL. I THINK I HAVE UNDERSTOOD WHAT YOU WROTE IN PREVIOUS PARAGRAPH. YOU SAID THEY CAN BE FAST BUT NOT CONSTANTLY, BECAUSE LATER THEY HAVE TO WRITE ALL THAT TO THEIR PERPETUAL STORAGE FROM THE CACHE. AND THAT'S SLOW. AM I WRONG?. EVEN IN THE 8TB MODEL YOU THINK STEFAN?.

The controller in the SSD supports a given number of channels (e.g 4),
each of which can access a Flash chip independently of the others. Small
SSDs often have less Flash chips than there are channels (and thus a
lower throughput, especially for writes), but the larger models often
have more chips than channels and thus the performance is capped. 

THIS IS TOTALLY LOGICAL. IF A QVO DISK WOULD OUTPERFORM BEST OR SIMILAR
THAN AN INTEL WITHOUT CONSEQUENCES.... WHO WAS GOING TO BUY A EXPENSIVE
INTEL ENTERPRISE?. 

In the case of the 870 QVO, the controller supports 8 channels, which
allows it to write 160 MB/s into the QLC cells. The 1 TB model
apparently has only 4 Flash chips and is thus limited to 80 MB/s in that
situation, while the larger versions have 8, 16, or 32 chips. But due to
the limited number of channels, the write rate is limited to 160 MB/s
even for the 8 TB model. 

TOTALLY LOGICAL STEFAN... 

If you had 4 * 2 TB instead, the throughput would be 4 * 160 MB/s in
this limit.

>> THE MAIN PROBLEM WE ARE FACING IS THAT IN SOME PEAK MOMENTS, WHEN THE MACHINE SERVES CONNECTIONS FOR ALL THE INSTANCES IT HAS, AND ONLY AS SAID IN SOME PEAK MOMENTS... LIKE THE 09AM OR THE 11AM.... IT SEEMS THE MACHINE BECOMES SLOWER... AND LIKE IF THE DISKS WEREN'T ABLE TO SERVE ALL THEY HAVE TO SERVE.... IN THESE MOMENTS, NO BIG FILES ARE MOVED... BUT AS WE HAVE 1800-2000 CONCURRENT IMAP CONNECTIONS... NORMALLY THEY ARE DOING EACH ONE... LITTLE CHANGES IN THEIR MAILBOX. DO YOU THINK PERHAPS THIS DISKS THEN ARE NOT APPROPRIATE FOR THIS KIND OF USAGE?-

I'd guess that the drives get into a state in which they have to recycle
lots of partially free blocks (i.e. perform kind of a garbage
collection) and then three kinds of operations are competing with each
other: 

 	* reads (generally prioritized)
 	* writes (filling the SLC cache up to its maximum size)
 	* compactions of partially filled blocks (required to make free blocks
available for re-use)

Writes can only proceed if there are sufficient free blocks, which on a
filled SSD with partially filled erase blocks means that operations of
type 3. need to be performed with priority to not stall all writes. 

My assumption is that this is what you are observing under peak load. 

IT COULD BE ALTHOUGH THE DISKS ARE NOT FILLED.... THE POOL ARE AT 20 OR
30% OF CAPACITY AND FRAGMENTATION FROM 20%-30% (AS ZPOOL LIST STATES).

>> And cheap SSDs often have no RAM cache (not checked, but I'd be
>> surprised if the QVO had one) and thus cannot keep bookkeeping date
>> in such a cache, further limiting the performance under load. 
>> 
>> THIS BROCHURE (HTTPS://SEMICONDUCTOR.SAMSUNG.COM/RESOURCES/BROCHURE/870_SERIES_BROCHURE.PDF AND THE DATASHEET HTTPS://SEMICONDUCTOR.SAMSUNG.COM/RESOURCES/DATA-SHEET/SAMSUNG_SSD_870_QVO_DATA_SHEET_REV1.1.PDF) SAIS IF I HAVE READ PROPERLY, THE 8TB DRIVE HAS 8GB OF RAM?. I ASSUME THAT IS WHAT THEY CALL THE TURBO WRITE CACHE?.

No, the turbo write cache consists of the cells used in SLC mode (which
can be any cells, not only cells in a specific area of the flash chip). 

I SEE I SEE.... 

The RAM is needed for fast lookup of the position of data for reads and
of free blocks for writes. 

OUR ONES... SEEM TO HAVE 8GB LPDDR4 OF RAM.... AS DATASHEET STATES.... 

There is no simple relation between SSD "block number" (in the sense of
a disk block on some track of a magnetic disk) and its storage location
on the Flash chip. If an existing "data block" (what would be a sector
on a hard disk drive) is overwritten, it is instead written at the end
of an "open" erase block, and a pointer from that "block number" to the
location on the chip is stored in an index. This index is written to
Flash storage and could be read from it, but it is much faster to have a
RAM with these pointers that can be accessed independently of the Flash
chips. This RAM is required for high transaction rates (especially
random reads), but it does not really help speed up writes. 

I SEE... I SEE.... I GOT IT... 

>> And the resilience (max. amount of data written over its lifetime)
>> is also quite low - I hope those drives are used in some kind of
>> RAID configuration. 
>> 
>> YEP WE USE RAIDZ-2

Makes sense ... But you know that you multiply the amount of data
written due to the redundancy. 

If a single 8 KB block is written, for example, 3 * 8 KB will written if
you take the 2 redundant copies into account. 

I SEE I SEE.... 

>> The 870 QVO is specified for 370 full capacity
>> writes, i.e. 370 TB for the 1 TB model. That's still a few hundred
>> GB a day - but only if the write amplification stays in a reasonable
>> range ... 
>> 
>> WELL YES... 2880TB IN OUR CASE....NOT BAD.. ISN'T IT?

I assume that 2880 TB is your total storage capacity? That's not too
bad, in fact. ;-) 

NO... THE TOTAL NUMBER OF WRITES YOU CAN DO....BEFORE THE DISK
"BREAKS".... 

LOL :) :)  ... WE ARE HAVING STORAGES OF 50TB DUE TO 8 DISKS OF 8TB IN
RAIDZ-2.... 

This would be 360 * 8 TB ... 

Even at 160 MB/s per 8 TB SSD this would allow for more than 50 GB/s of
write throughput (if all writes were evenly distributed). 

Taking all odds into account, I'd guess that at least 10 GB/s can be
continuously written (if supported by the CPUs and controllers). 

But this may not be true if the drive is simultaneously reading,
trimming, and writing ... 

I SEE.... IT'S EXTREMELY MISLEADING YOU KNOW... BECAUSE... YOU CAN COPY
FIVE MAILBOXES OF 50GB CONCURRENTLY FOR INSTANCE.... AND YOU FLOOD A
GIGABIT INTERFACE COPYING (OBVIOUSLY BECAUSE DISKS CAN KEEP THAT
THROUGHPUT)... BUT LATER.... YOU SEE... YOU ARE IN AN HOUR THAT
YESTERDAY, AND EVEN 4 DAYS BEFORE YOU HAVE NOT HAD ANY ISSUES... AND
THAT DAY... YOU SEE THE COMMENTED ISSUE... EVEN NOT BEING EXACTLY AT A
PEAK HOUR (PERHAPS IS TWO HOURS LATER THE PEAK HOUR EVEN)... OR... BUT I
WASN'T NOTICING ABOUT ALL THINGS YOU SAY IN THIS EMAIL.... 

I have seen advice to not use compression in a high load scenario in
some other reply. 

I tend to disagree: Since you seem to be limited when the SLC cache is
exhausted, you should get better performance if you compress your data.
I have found that zstd-2 works well for me (giving a significant overall
reduction of size at reasonable additional CPU load). Since ZFS allows
to switch compressions algorithms at any time, you can experiment with
different algorithms and levels. 

I SEE... YOU SAY COMPRESSION SHOULD BE ENABLED.... THE MAIN REASON
BECAUSE WE HAVE NOT ENABLED IT YET, IS FOR KEEPING THE SYSTEM THE MOST
NEAR POSSIBLE TO CONFIG DEFAULTS... YOU KNOW... FOR LATER BEING ABLE TO
ASK IN THIS MAILING LISTS IF WE HAVE AN ISSUE... BECAUSE YOU KNOW... IT
WOULD BE FAR MORE EASIER TO ASK ABOUT SOMETHING STRANGE YOU ARE SEEING
WHEN THAT STRANGE THING IS NEAR TO A WELL TESTED CONFIG, LIKE THE CONFIG
BY DEFAULT.... 

BUT NOW YOU SAY STEFAN... IF YOU SWITCH BETWEEN COMPRESSION ALGORITHMS
YOU WILL END UP WITH A MIX OF DIFFERENT FILES COMPRESSED IN A DIFFERENT
MANNER... THAT IS NOT A BIT DISASTER LATER?. DOESN'T AFFECT PERFORMANCE
IN SOME MANNER?. 

One advantage of ZFS compression is that it applies to the ARC, too. And
a compression factor of 2 should easily be achieved when storing mail
(not for .docx, .pdf, .jpg files though). Having more data in the ARC
will reduce the read pressure on the SSDs and will give them more cycles
for garbage collections (which are performed in the background and
required to always have a sufficient reserve of free flash blocks for
writes). 

WE WOULD USE I ASSUME THE LZ4... WHICH IS THE LESS "EXPENSIVE"
COMPRESSION ALGORITHM FOR THE CPU... AND I ASSUME TOO FOR AVOIDING DELAY
ACCESSING DATA... DO YOU RECOMMEND ANOTHER ONE?. DO YOU ALWAYS RECOMMEND
COMPRESSION THEN?. 

I'd give it a try - and if it reduces your storage requirements by 10%
only, then keep 10% of each SSD unused (not assigned to any partition).
That will greatly improve the resilience of your SSDs, reduce the
write-amplification, will allow the SLC cache to stay at its large
value, and may make a large difference to the effective performance
under high load. 

BUT WHEN YOU ENABLE COMPRESSION... ONLY GETS COMPRESSED THE NEW DATA
MODIFIED OR ENTERED. AM I WRONG?. 

BY THE WAY, WE HAVE MORE OR LESS 1/4 OF EACH DISK USED (12 TB ALLOCATED
IN A POLL STATED BY ZPOOL LIST, DIVIDED BETWEEN 8 DISKS OF 8TB...)... DO
YOU THINK WE COULD BE SUFFERING ON WRITE AMPLIFICATION AND SO... HAVING
A SO LITTLE DISK SPACE USED IN EACH DISK?. 

Regards, STefan 

HEY MATE, YOUR MAIL IS INCREDIBLE. IT HAS HELPED AS A LOT. CAN WE INVITE
YOU A CUP OF COFFEE OR A BEER THROUGH PAYPAL OR SIMILAR?. CAN I HELP YOU
IN SOME MANNER?. 

CHEERS!