From nobody Fri Dec 22 09:37:42 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SxMfR10Zgz551hD for ; Fri, 22 Dec 2023 09:37:51 +0000 (UTC) (envelope-from SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int-backup.realworks.nl (smtp-relay-int-backup.realworks.nl [87.255.56.188]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SxMfQ5ttMz4Mky for ; Fri, 22 Dec 2023 09:37:50 +0000 (UTC) (envelope-from SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; none Received: from rwvirtual98.colo.realworks.nl (rwvirtual98.colo.realworks.nl [10.0.10.102]) by mailrelayint1.colo2.realworks.nl (Postfix) with ESMTP id 4SxMfH11GBz1Vq; Fri, 22 Dec 2023 10:37:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1703237863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=6a8XbPENsCiYh5VJZR4BGLkVktiNKR1gZHyNMOzETUI=; b=i9pH9N9N7MEVa7o41qfM7aBB8gkBYZk1cE4Q0NLxInibcCAoELcq+u9E7fpplhv0lZBWhg nUeIxrg6xGRa26a69VYwEIHGalw5FnOf9j8YkjUsPGBFNlo87vAcnhiqaJTKdSs46fWTEg +4gAEAhTAvwb0u+kreGBhnFbYND+zsnH7JcNXl7DeM9uiFdzeCMvMmoOeqVzlREBOQs9WE neZnIC5KhJTkU1rsMSuOqfEaoSra6SYA+jAsGT2bt0XbEZ/KH3faVxDuNkEDVppJIvHPQr vLsU6UrUI2Apb40v7sHLaSO0Jna+sneUFfkRyWsnyIbbGbQ605PdfTGpS0f2yA== Received: from rwvirtual98.colo.realworks.nl (localhost [127.0.0.1]) by rwvirtual98.colo.realworks.nl (Postfix) with ESMTP id DF7EAA02BB; Fri, 22 Dec 2023 10:37:42 +0100 (CET) Date: Fri, 22 Dec 2023 10:37:42 +0100 (CET) From: Ronald Klop To: void Cc: freebsd-fs Message-ID: <1137397689.3080.1703237862787@localhost> In-Reply-To: <1dbf8804-f1ac-4578-a538-889744d7de9a@app.fastmail.com> References: <1439231787.9333.1703184981914@localhost> <1dbf8804-f1ac-4578-a538-889744d7de9a@app.fastmail.com> Subject: Re: measuring swap partition speed List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3079_229342031.1703237861046" X-Mailer: Realworks (683.54) X-Originating-Host: from (84-105-120-103.cable.dynamic.v4.ziggo.nl [84.105.120.103]) by rwvirtual98 [10.0.10.102] with HTTP; Fri, 22 Dec 2023 10:37:42 +0100 Importance: Normal X-Priority: 3 (Normal) X-Originating-User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:38930, ipnet:87.255.32.0/19, country:NL] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SxMfQ5ttMz4Mky ------=_Part_3079_229342031.1703237861046 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: void Datum: vrijdag, 22 december 2023 01:05 Aan: freebsd-fs Onderwerp: Re: measuring swap partition speed > > On Thu, 21 Dec 2023, at 18:56, Ronald Klop wrote: > > A bit weird that a simple dd is so slow. > > > > Just a quick thought. > > Are your partitions aligned properly? > > no idea. How would I check? The system when installed, > the auto-zfs option was selected, block size set to 4k. The swap > was changed from the default 2gb to 12gb. geli encryption is > active for the zfs filesystem but not for swap. It's only the swap > that shows performance issues. > > > Is other IO going on on the same time? > > not really, the tests were done with cron jobs deactivated, almost completely idle, > swap was not being used at all. > > > What does gstat say about %util, queue length and all the other stats > > while running the dd? Or "iostat -x -d 1". > > I'll try that next, thanks > > > Could you try if another disk has the same issues? > > no spare or equivalent disk for this arch. > > > How is your disk connected? USB-to-SATA-adapter? Any output of dmesg? > > Regular usb3 connection. > > da0: 400.000MB/s transfers > da0: 953869MB (1953525168 512 byte sectors) > da0: quirks=0x2 > da0: Delete methods: > -- > > > > Can you provide more concrete information? Like: # usbconfig list ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA) ugen0.2: at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA) ugen0.3: at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA) ugen0.4: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA) ugen0.5: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA) And the output of "devinfo -v". That will give people much more information to work with instead of "anecdotal evidence" about your hardware being usb3. And the complete output of dmesg instead of only the "da0:" lines. There is more information in there which can be valuable for people to help you. About the alignment. You wrote in your first mail: "# gpart show => 40 1953525088 da0 GPT (932G) 40 532480 1 efi (260M) 532520 2008 - free - (1.0M) 534528 4194304 2 freebsd-swap (2.0G) 4728832 4194304 4 freebsd-swap (2.0G) 8923136 4194304 5 freebsd-swap (2.0G) 13117440 4194304 6 freebsd-swap (2.0G) 17311744 4194304 7 freebsd-swap (2.0G) 21506048 4194304 8 freebsd-swap (2.0G) 25700352 1927823360 3 freebsd-zfs (920G) 1953523712 1416 - free - (708K) " Your dd tests were on da0p4 which starts on block 4728832. Assuming "gpart list" shows "Sectorsize: 512". (4728832 * 512) mod 4096 = 0, so the partitiion is aligned on a 4KB page size. And the calculation is also 0 for 8KB (is this the page size of arm64?) which is good. AFAIK: this is properly aligned such that the hardware does not need to double write or read-before-write when paging out. Up until now I haven't seen anything which would explain the slow dd speed on of=/dev/da0p4 vs the quick dd on of=/mnt/test8k.bin. Regards, Ronald. ------=_Part_3079_229342031.1703237861046 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit

Van: void <void@f-m.fm>
Datum: vrijdag, 22 december 2023 01:05
Aan: freebsd-fs <freebsd-fs@freebsd.org>
Onderwerp: Re: measuring swap partition speed

On Thu, 21 Dec 2023, at 18:56, Ronald Klop wrote:
> A bit weird that a simple dd is so slow.
>
> Just a quick thought.
> Are your partitions aligned properly?

no idea. How would I check? The system when installed,
the auto-zfs option was selected, block size set to 4k. The swap
was changed from the default 2gb to 12gb. geli encryption is
active for the zfs filesystem but not for swap. It's only the swap
that shows performance issues.

> Is other IO going on on the same time?

not really, the tests were done with cron jobs deactivated, almost completely idle,
swap was not being used at all.

> What does gstat say about %util, queue length and all the other stats
> while running the dd? Or "iostat -x -d 1".

I'll try that next, thanks

> Could you try if another disk has the same issues?

no spare or equivalent disk for this arch.

> How is your disk connected? USB-to-SATA-adapter? Any output of dmesg?

Regular usb3 connection.

da0: 400.000MB/s transfers
da0: 953869MB (1953525168 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
da0: Delete methods: <NONE(*),ZERO>
-- 
 


Can you provide more concrete information? Like:

# usbconfig list
ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)

And the output of "devinfo -v". That will give people much more information to work with instead of "anecdotal evidence" about your hardware being usb3.
And the complete output of dmesg instead of only the "da0:" lines. There is more information in there which can be valuable for people to help you.

About the alignment. You wrote in your first mail:
"# gpart show

=>        40  1953525088  da0  GPT  (932G)
           40      532480    1  efi  (260M)
       532520        2008       - free -  (1.0M)
       534528     4194304    2  freebsd-swap  (2.0G)
      4728832     4194304    4  freebsd-swap  (2.0G)
      8923136     4194304    5  freebsd-swap  (2.0G)
     13117440     4194304    6  freebsd-swap  (2.0G)
     17311744     4194304    7  freebsd-swap  (2.0G)
     21506048     4194304    8  freebsd-swap  (2.0G)
     25700352  1927823360    3  freebsd-zfs  (920G)
   1953523712        1416       - free -  (708K)
"

Your dd tests were on da0p4 which starts on block 4728832.
Assuming "gpart list" shows "Sectorsize: 512".
(4728832 * 512) mod 4096 = 0, so the partitiion is aligned on a 4KB page size. And the calculation is also 0 for 8KB (is this the page size of arm64?) which is good.
AFAIK: this is properly aligned such that the hardware does not need to double write or read-before-write when paging out.

Up until now I haven't seen anything which would explain the slow dd speed on of=/dev/da0p4 vs the quick dd on of=/mnt/test8k.bin.

Regards,
Ronald.
  ------=_Part_3079_229342031.1703237861046--