From nobody Sat Sep 02 20:11:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdQzB66dBz4s0kN for ; Sat, 2 Sep 2023 20:11:50 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdQz64Mrcz3fTv for ; Sat, 2 Sep 2023 20:11:46 +0000 (UTC) (envelope-from wojtek@puchar.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=puchar.net header.s=default header.b=eeDnbsGW; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net; dmarc=none Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.17.1) with ESMTPS id 382KBhqv005001 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 22:11:44 +0200 (CEST) (envelope-from wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1693685504; bh=qKtSOpQ2JqUrUTAeRrOnFHtrbLLxksUsamYd0pvxYiE=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=eeDnbsGW/BgpmBZ3KWivcRLDUcI+wxLH1fxXrEbgp5QHKQnyU7ipMzz4zLhZAXfc7 rk1qwxLsaqVowAe2Z+KbuAp/6doRlJzn5Y6Ckm8UoWGmwJM2hxcwRV2hwdiMtzmsDJ kdhpkk3vKPBLdUCdU3Uc+VNLw7ZilfQ8ngFkVp/Y= Received: from wojtek.intra (localhost [127.0.0.1]) by wojtek.intra (8.16.1/8.16.1) with ESMTP id 382KBhZU050851; Sat, 2 Sep 2023 22:11:43 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.intra (8.16.1/8.16.1/Submit) with ESMTP id 382KBgTj050848; Sat, 2 Sep 2023 22:11:42 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.intra: wojtek owned process doing -bs Date: Sat, 2 Sep 2023 22:11:42 +0200 (CEST) From: Wojciech Puchar To: Wei Hu cc: Guido Falsi , Mikhail Zakharov , "freebsd-hackers@freebsd.org" Subject: RE: Very slow scp performance comparing to Linux In-Reply-To: Message-ID: References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> <455dc69a-dd15-18d9-8e93-91bbff3997e4@madpilot.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_ALLOW(-0.20)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[madpilot.net,yahoo.com,freebsd.org]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; DKIM_TRACE(0.00)[puchar.net:+]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.1.144.90:from]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RdQz64Mrcz3fTv > The throughput on FreeBSD went up significantly from 50 MB/s to 630 MB/s with > NIC interface. Linux went up modestly form 550 MB/s to 660 MB/s. > > So, looks in the non-tmpfs case, the Linux ext4 filesystem does cache large amounts > of data in memory, much larger than FreeBSD ufs. > linux filesystem caching is just stupid, optimized to show up good in benchmarks. It just accepts writes as long as there is available memory, without writing anything to disk, then if it cannot do it anymore or some time passed, schedules huge amount of writes, often stalling whole system. This is how it was almost 20 years ago when i was still using linux, seems like didn't change much. Just think how much data loss and inconsistency would occur if there will be crash or power loss after "writing" 100000 files of 10GB which can easily be done in short time as linux just doesn't block writing process at all and fill memory. Since i started using FreeBSD UFS, many machines, lighter of heavier workload, always mixed workload, i never ever have more that a few files loss on power outage.