gmirror: writes are faster than reads

[Schaich Alonso]

On Thu, 28 Nov 2013 21:47:17 +0100
Szalai András <szalai.bandi at gmail.com> wrote:

> Hi Guys,
> 
> Has somebody encountered (significantly) different read/write speeds
> when using gmirror?
> 
> I have 2xWD WD30EFRX RED drives which are configured as follows:
> 
> $ gmirror status
>         Name    Status  Components
> mirror/root  COMPLETE  ada0p2 (ACTIVE)
>                         ada1p2 (ACTIVE)
> mirror/data  COMPLETE  ada0p4 (ACTIVE)
>                         ada1p4 (ACTIVE)
> 
> mirror/root is mounted as the root fs (UFS2).
> 
> Doing write:
> 
> $ time dd if=/dev/zero of=/IMAGE bs=1024k count=`expr 4 \* 1024`
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes transferred in 29.326044 secs (146455733 bytes/sec)
> 
> Doing read:
> 
> $ time dd if=/IMAGE of=/dev/null bs=1024k count=`expr 4 \* 1024`
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes transferred in 48.821649 secs (87972598 bytes/sec)
> 
> As you can see, read is much slower than write (87 vs 146 MB/s). Why?
> 
> Any help would be appreciated.
> 
> Best regards,
> Andrew
> 
> PS: Partition layout (partitions are 4k aligned):
> 
> $ gpart show ada0 ada1
> =>        34  5860533101  ada0  GPT  (2.7T)
>            34           6        - free -  (3.0k)
>            40        1024     1  freebsd-boot  (512k)
>          1064    16777216     2  freebsd-ufs  (8.0G)
>      16778280    16777216     3  freebsd-swap  (8.0G)
>      33555496  5826977632     4  freebsd-ufs  (2.7T)
>    5860533128           7        - free -  (3.5k)
> 
> =>        34  5860533101  ada1  GPT  (2.7T)
>            34           6        - free -  (3.0k)
>            40        1024     1  freebsd-boot  (512k)
>          1064    16777216     2  freebsd-ufs  (8.0G)
>      16778280    16777216     3  freebsd-swap  (8.0G)
>      33555496  5826977632     4  freebsd-ufs  (2.7T)
>    5860533128           7        - free -  (3.5k)

Modern HDDs have both Command Queuing and Excessive Cache Memory. Using them
a spindle disk can cache multiple write requests and do them all in one
revolution. While multiple read requests can also be done at once chances
are the on-disk cache is not usefull (because the requested data is only
resident if it was accessed short before, and then it's also availible in
the kernel's larger Filesystem/GEOM caches which reside in main memory and
were consulted prior to disk accesses) and the GEOM layer might not have
issued them yet. IIRC the UFS subsystem will perform no read requests smaller
than 512kB by default, which means it does some readahead just in case the
issuing application wants to read more data soon - however you have used read
blocks which are exact multiples of 512kB, so there is no gain in this.

readahead is the buzzword for tuning large sequencial reads, and I had thought
there was a sysctl for it, though I can't find it now.

Alonso