From nobody Sat Dec 23 12:00:44 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sy2nD52hZz54QHv for ; Sat, 23 Dec 2023 12:01:04 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "plan-b.pwste.edu.pl", Issuer "GEANT OV RSA CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sy2n95Zk9z4WVX for ; Sat, 23 Dec 2023 12:01:01 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b=VThSh9eT; spf=none (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl has no SPF policy when checking 2001:678:618::40) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl; dmarc=pass (policy=none) header.from=plan-b.pwste.edu.pl Received: from [IPV6:2a02:22e0:cf00:1ff:46f7:5375:dd57:c63f] (mzar@[IPv6:2a02:22e0:cf00:1ff:46f7:5375:dd57:c63f]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.17.2/8.17.2) with ESMTPSA id 3BNC0iTu080402 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Sat, 23 Dec 2023 13:00:48 +0100 (CET) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1703332848; bh=3UEV4eALbJigKtIs5AkDuhwhURennbDz6TX/QRwb4ck=; h=Date:Subject:To:References:From:In-Reply-To; b=VThSh9eT66HGEYsrKjDDIbKYgeJ5OWVelORBi/gys9FptWrTUVK+e0+8N5t7Q9WeJ hPJ02d1Bkgk0gWdwjZ0e/eTXX/MPgCsNcVjX5NLQcNlB2zXBEIsNhLY3XsTkb3K41A y9gyFSvQpsmDQeBMOlDwkdm6gukqCuZx3DvJEGyca15+HeB6KNFYS2vu/IMt85FFPj W6cViSAcxYBZ7NvRWH4Sjmm2vvseociVgPB04ycS0DjjfEw5ckhAHIvPfgcY1GV0kt 54sSn6LUYdCTF/vNt8FXVW1qKGh6Oj6xH+Eqpm6unEkuGjcnV0e9rMpgSeyPhK6Du8 Ro2LphppGWBYA== Message-ID: <3c13fd10-997e-40a7-823e-e37a4d481739@plan-b.pwste.edu.pl> Date: Sat, 23 Dec 2023 13:00:44 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Proposal: Disable compression of newsyslog by default To: freebsd-current@freebsd.org References: <8a585a40-0e78-4dbd-9701-aa0926ccde19@quip.cz> Content-Language: en-US From: Marek Zarychta Autocrypt: addr=zarychtam@plan-b.pwste.edu.pl; keydata= xsBNBFfi3cMBCADLecMTFXad4uDXqv3eRuB4qJJ8G9tzzFezeRnnwxOsPdytW5ES2z1ibSrR IsiImx6+PTqrAmXpTInxAi7yiZGdSiONRI4CCxKY9d1YFiNYT/2WyNXCekm9x29YeIU7x0JB Llbz0f/9HC+styBIu2H+PY/X98Clzm110CS+n/b9l1AtiGxTiVFj7/uavYAKxH6LNWnbkuc5 v8EVNc7NkEcl5h7Z9X5NEtzDxTOiBIFQ/kOT7LAtkYUPo1lqLeOM2DtWSXTXQgXl0zJI4iP1 OAu4qQYm2nXwq4b2AH9peknelvnt1mpfgDCGSKnhc26q6ibTfMwydp+tvUtQIQYpA6b9ABEB AAHNN01hcmVrIFphcnljaHRhIChQbGFuLWIpIDx6YXJ5Y2h0YW1AcGxhbi1iLnB3c3RlLmVk dS5wbD7CwHcEEwEIACEFAlfi4LkCGwMFCwkIBwIGFQgJCgsCBBYCAwECHgECF4AACgkQHZW8 vIFppoJXdgf8D9X3VRFSNaR9lthSx/+uqas17J3FJKBo1xMQsC2a+44vzNvYJSuPGLLJ+LW2 HPVazjP/BWZJbxOYpliY4zxNRU0YCp0BLIVLibc//yax+mE42FND/+NiIZhqJscl6MLPrSwo sIwXec4XYkldkyqW/xBbBYXoIkBqdKB9j5j42Npy1IV/RizOSdmvTWY27ir8e/yGMR1RLr4F 8P5K3OWTdlGy2H2F/3J8bIPBLG6FpaIyLQw4dHSx8V02PYqDxK1cNo2kAOnU8PnZL/AGuMOH iv3MN1VYL8ehcmpBBsrZGebQJxrjY2/5IaTSgp9xHYT70kshuU6Qb97vk1mOjNZxgc7ATQRX 4t3DAQgA10h6RCXuBLMHxq5B8X/ZIlj9sgLoeyfRdDZEc9rT2KUeUJVHDsbvOFf4/7F1ovWY hJbA6GK/LUZeHHTjnbZcH1uDYQeHly4UOLxeEvhGoz4JhS2C7JzN/uRnwbdOAUbJr8rUj/IY a7gk906rktsc/Ldrxrxh7O6WO0JCh2XO/p4pDfEwwB37g4xHprSab28ECYJ9JMbtA8Sy4M55 g3+GQ28FvSlGnx48OoGXU2BZdc1vZKSQmNOlikB+9/hDX8zdYWVfDaX1TLQ8Ib4+xTUmapza mV/bxIsaZRBw+jFjLQHhTbIMfPEU+4mxFDvTdbKPruKPqVf1ydgMnPZWngowdwARAQABwsBf BBgBCAAJBQJX4t3DAhsMAAoJEB2VvLyBaaaC6qkIAJs9sDPqrqW0bYoRfzY6XjDWQ59p9tJi v8aogxacQNCfAu+WkJ8PNVUtC1dlVcG5NnZ80gXzd1rc8ueIvXlvdanUt/jZd8jbb3gaDbK3 wh1yMCGBl/1fOJTyEGYv1CRojv97KK89KP5+r8x1P1iHcSrunlDNqGxTMydNCwBH23QcOM+m u4spKnJ/s0VRBkw3xoKBZfZza6fTQ4gTpAipjyk7ldOGBV+PvkKATdhK2yLwuWXhKbg/GRlD 1r5P0gxzSqfV4My+KJuc2EDcrqp1y0wOpE1m9iZqCcd0fup5f7HDsYlLWshr7NQl28f6+fQb sylq/j672BHXsdeqf/Ip9V4= In-Reply-To: <8a585a40-0e78-4dbd-9701-aa0926ccde19@quip.cz> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.69 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,none]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; XM_UA_NO_VERSION(0.01)[]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+]; R_SPF_NA(0.00)[no SPF record]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Sy2n95Zk9z4WVX X-Spamd-Bar: --- W dniu 23.12.2023 o 09:51, Miroslav Lachman pisze: > On 23/12/2023 08:18, Xin Li wrote: >> Hi, >> >> Inspired by D42961, I propose that we move forward with disabling the >> compression by default in newsyslog, as implemented in >> https://reviews.freebsd.org/D43169 >> >> Historically, newsyslog has compressed rotated log files to save disk >> space. This approach was valuable in the early days where storage >> space was limited.  However, the landscape has changed >> significantly.  Modern file systems, such as ZFS, now offer native >> compression capabilities. Additionally, the widespread availability >> of larger hard drives has diminished the necessity for additional >> compression.  Notably, the need to decompress log files for pattern >> searches poses a significant inconvenience, further questioning the >> utility of this legacy feature. >> >> In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log >> file is eligible for compression rather than directly enforcing it. >> It allows for a more flexible approach, wherein the actual >> compression method can be set to "none" or specified as one among >> bzip2, gzip, xz, or zstd. >> >> Therefore I would propose that we change the default compression >> setting to "none" in FreeBSD 15.0.  This change reflects our >> adaptation to the evolving technological environment and user needs.  >> It also aligns with the broader initiative to modernize our systems >> while maintaining flexibility and efficiency. >> >> I look forward to your thoughts and feedback on this proposal. Thank you for this change and future MFCs. Log files and an urge to process them differ a lot between setups. I guess some people would back completely disabling compression, and some would be against this proposal. There are also small VMs with limited storage formatted with UFS where current defaults: "-c legacy" and using bzip2 (J from newsyslog.conf) to compress logs seems to be perfectly suited. > > I don't think anything needs to be changed on newsyslog. Those who > want to disable compression can do so in the "default" newsyslog.conf > file. Why force this change in the newsyslog code? > I also don't think that the log file should not be compressed even on > a compressed filesystem. Compressed log files can be handled by tools > like zcat, zless, zgrep, etc. without first decompressing the log file. When newsyslog has to deal only with small/medium files and logs are just stored, if processed then very rarely, the compression method is not very relevant, but the situation changes if you have to deal with larger log files, which have to be processed after rotation. While testing this change, I have done a very simple benchmark with the largest log file which I have to rotate daily. It looks to me that having a compressed log file sometimes makes sense even if you have to process these logs in place. Please take a look at the result of running zcat over the compressed file and the timings for the original, uncompressed one. All the files are stored on ZFS lz4 compressed dataset lying on mirrored, not so fast, SSD drives. [r-b] /tmp# /usr/bin/time zcat flowd.0.zst > /dev/null         3.22 real         3.03 user         0.18 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.zst.orig  > /dev/null         3.40 real         3.24 user         0.15 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.gz > /dev/null         6.58 real         6.46 user         0.12 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.xz > /dev/null        18.15 real        16.97 user         1.18 sys [r-b] /tmp# /usr/bin/time zcat flowd.0.bz2 > /dev/null        45.47 real        45.33 user         0.12 sys [r-b] /tmp# /usr/bin/time cat flowd.0 > /dev/null         3.35 real         0.25 user         3.10 sys [r-b] /tmp# ls -l flowd.0* -rw-------  1 root wheel 3823756280 Dec 23 09:56 flowd.0 -rw-------  1 root wheel  195036235 Dec 23 09:56 flowd.0.bz2 -rw-------  1 root wheel  305250854 Dec 23 09:56 flowd.0.gz -rw-------  1 root wheel  180642412 Dec 23 09:56 flowd.0.xz -rw-------  1 root wheel  252470186 Dec 23 09:56 flowd.0.zst (zstd with --long --adapt -T0) -rw-------  1 root wheel  297184160 Dec 23 09:56 flowd.0.zst.orig (pre 906748d208d3 zstd options) > Text log files can still grow to large sizes, and if you have a daily > backup job over a long distance network, if you use a protocol without > own compression, it is still better to have compressed log files than > uncompressed on a compressed filesystem. To me, it's the same as > compressing large database dumps and not relying on filesystem > compression. > YMMV, but I really don't see any benefit of changing the newsyslog > code, just change defaults in newsyslog.conf. > There is a clear benefit for me, users who don't want their logs to be compressed, will have just to alter one line in /etc/crontab: # Rotate log files every hour, if necessary. 0    *    *    *    *    root    newsyslog -c none instead of changing /etc/newsyslog.conf line by line. Thank you for the new newsyslog(8) -c option providing this feature ! With kind regards -- Marek Zarychta