From nobody Wed Apr 06 16:51:49 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 437DE1A983A4; Wed, 6 Apr 2022 16:51:55 +0000 (UTC) (envelope-from egoitz@ramattack.net) Received: from cu01208b.smtpx.saremail.com (cu01208b.smtpx.saremail.com [195.16.151.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KYVsk0NY1z4lVF; Wed, 6 Apr 2022 16:51:52 +0000 (UTC) (envelope-from egoitz@ramattack.net) Received: from www.saremail.com (unknown [194.30.0.183]) by sieve-smtp-backend01.sarenet.es (Postfix) with ESMTPA id 027BD60C6BF; Wed, 6 Apr 2022 18:51:49 +0200 (CEST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_be6199b7d5e868a9b11faca98e7631c5" Date: Wed, 06 Apr 2022 18:51:49 +0200 From: egoitz@ramattack.net To: Eugene Grosbein Cc: freebsd-fs@freebsd.org, freebsd-hackers@freebsd.org, Freebsd performance Subject: Re: {* 05.00 *}Re: Desperate with 870 QVO and ZFS In-Reply-To: References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> <665236B1-8F61-4B0E-BD9B-7B501B8BD617@ultra-secure.de> <0ef282aee34b441f1991334e2edbcaec@ramattack.net> <28e11d7ec0ac5dbea45f9f271fc28f06@ramattack.net> Message-ID: <7aa95cb4bf1fd38b3fce93bc26826042@ramattack.net> X-Sender: egoitz@ramattack.net User-Agent: Saremail webmail X-Rspamd-Queue-Id: 4KYVsk0NY1z4lVF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=reject) header.from=ramattack.net; spf=pass (mx1.freebsd.org: domain of egoitz@ramattack.net designates 195.16.151.183 as permitted sender) smtp.mailfrom=egoitz@ramattack.net X-Spamd-Result: default: False [-3.79 / 15.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:195.16.151.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[ramattack.net,reject]; FROM_NO_DN(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-fs,freebsd-hackers,freebsd-performance]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:3262, ipnet:195.16.128.0/19, country:ES]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --=_be6199b7d5e868a9b11faca98e7631c5 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Hi Eugene!!! Thank you so much really again mate :) :) :) About your recommendations... Eugene, if some of them wouldn't be working as expected, could we revert some or all of them or perhaps some of your recommendations below need to be definitive?. I do answer below in green bold for better distinction :) :) El 2022-04-06 18:14, Eugene Grosbein escribió: > ATENCION > ATENCION > ATENCION!!! Este correo se ha enviado desde fuera de la organizacion. No pinche en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y sepa que el contenido es seguro. > > 06.04.2022 22:30, egoitz@ramattack.net пишет: > >> One perhaps important note!! >> >> When this happens... almost all processes appear in top in the following state: >> >> txg state or >> >> txg-> >> >> bio.... >> >> perhaps should the the vfs.zfs.dirty_data_max, vfs.zfs.txg.timeout, vfs.zfs.vdev.async_write_active_max_dirty_percent be increased, decreased.... I'm afraid of doing some chage ana finally ending up with an inestable server.... I'm not an expert in handling these values.... >> >> Any recommendation?. > > 1) Make sure the pool has enough free space because ZFS can became crawling slow otherwise. > > THIS IS JUST AN EXAMPLE... BUT YOU CAN SEE ALL SIMILARLY.... > > ZPOOL LIST > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > ZROOT 448G 2.27G 446G - - 1% 0% 1.00X ONLINE - > MAIL_DATASET 58.2T 19.4T 38.8T - - 32% 33% 1.00X ONLINE - > > 2) Increase recordsize upto 1MB for file systems located in the pool > so ZFS is allowed to use bigger request sizes for read/write operations > > WE HAVE THE DEFAULT... SO 128K... > > 3) If you use compression, look if achieved compressratio worth it and > if not (<1.4 f.e.) then better disable compression to avoid its overhead; > > WE DON'T USE COMPRESSION AS IT'S NOT SET BY DEFAULT. SOME PEOPLE SAY YOU SHOULD HAVE IT ENABLED.... BUT.... JUST FOR AVOID HAVING SOME DATA COMPRESSED SOME OTHER NOT (IN CASE YOU ENABLE AND LATER DISABLE) AND FINALLY FOR AVOID ACCESSING TO INFORMATION WITH DIFFERENT CPU COSTS OF HANDLING... WE HAVE NOT TOUCHED COMPRESSION.... > > WE SHOULD SAY WE HAVE LOTS OF CPU... > > 4) try "zfs set redundant_metadata=most" to decrease amount of small writes to the file systems; > > OK.... > > 5) If you have good power supply and stable (non-crashing) OS, try increasing > sysctl vfs.zfs.txg.timeout from defaule 5sec, but do not be extreme (f.e. upto 10sec). > Maybe it will increase amount of long writes and decrease amount of short writes, that is good. > > WELL I HAVE SYNC IN DISABLED IN THE DATASETS... DO YOU STILL THINK IT'S GOOD TO CHANGE IT?. JUST A QUESTION OF PERSON WANTING TO LEARN :) . > > WHAT ABOUT THE VFS.ZFS.DIRTY_DATA_MAX AND THE VFS.ZFS.DIRTY_DATA_MAX_MAX, WOULD YOU INCREASE THEM FROM 4GB IT'S SET NOW?. > > THANKS A LOT EUGENE!!!! > CHEERS!! --=_be6199b7d5e868a9b11faca98e7631c5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

Hi Eugene!!!


Thank you so much really again mate  :) :) :)


About your recommendations... Eugene, if some of them wouldn't be workin= g as expected, could we revert some or all of them or perhaps some of your = recommendations below need to be definitive?.


I do answer below in green bold for better distinction :) :)



 


El 2022-04-06 18:14, Eugene Grosbein escribió:

= ATENCION
ATENCION
ATENCION!!! Este correo se ha enviado desde f= uera de la organizacion. No pinche en los enlaces ni abra los adjuntos a no= ser que reconozca el remitente y sepa que el contenido es seguro.
06.04.2022 22:30, egoitz@ramat= tack.net =D0=BF=D0=B8=D1=88=D0=B5=D1=82:
One perhaps important note!!


When = this happens... almost all processes appear in top in the following state:<= br />

txg state or

txg->

bio..= =2E.


perhaps should the the vfs.zfs.dirty_data_max, vfs= =2Ezfs.txg.timeout, vfs.zfs.vdev.async_write_active_max_dirty_percent be in= creased, decreased.... I'm afraid of doing some chage ana finally ending up= with an inestable server.... I'm not an expert in handling these values.= =2E..


Any recommendation?.

1) Make sure the pool has enough free space because ZFS can became c= rawling slow otherwise.
=  
= This is just an example... but you = can see all similarly....
=  
= zpool list
NAME     &nbs= p;       SIZE  ALLOC   FREE&nb= sp; CKPOINT  EXPANDSZ   FRAG    CAP  DED= UP  HEALTH  ALTROOT
zroot          = ;   448G  2.27G   446G    &nbs= p;   -         - &nb= sp;   1%     0%  1.00x  ONLINE = ; -
mail_datas= et  58.2T  19.4T  38.8T      &= nbsp; -         -   = 32%    33%  1.00x  ONLINE  -=


2) Increase recordsize upto 1MB for file systems locate= d in the pool
so ZFS is allowed to use bigger request sizes for read/= write operations
=  
= We have the default... so 128K...

3) If you use compression, look if achieved com= pressratio worth it and
if not (<1.4 f.e.) then better disable com= pression to avoid its overhead;
=  
= We don't use compression as it's no= t set by default. Some people say you should have it enabled.... but.... ju= st for avoid having some data compressed some other not (in case you enable= and later disable) and finally for avoid accessing to information with dif= ferent cpu costs of handling... we have not touched compression....<= /strong>
=  
= We should say we have lots of CPU= =2E..
=

4) try "zfs set redundant_metadata=3Dmost" to decrease amount= of small writes to the file systems;
=  
= Ok....
=

5) If you have good power supply and stable (non-crashing) OS= , try increasing
sysctl vfs.zfs.txg.timeout from defaule 5sec, but do= not be extreme (f.e. upto 10sec).
Maybe it will increase amount of l= ong writes and decrease amount of short writes, that is good.
=  
= Well I have sync in disabled in the= datasets... do you still think it's good to change it?. Just a question of= person wanting to learn :) .
=  
= What about the vfs.zfs.dirty_data_m= ax and the vfs.zfs.dirty_data_max_max, would you increase them from 4GB it'= s set now?.
=  
=  
=  
=  
=  
=  
=  
=  
=  
=  
=  
=  
= Thanks a lot Eugene!!!!
= Cheers!!
--=_be6199b7d5e868a9b11faca98e7631c5--