Slow resilvering with mirrored ZIL

[Lawrence K. Chen, P.Eng.]

well, it would kind of make sense that the second drive resilvering slower than the first if its a 4K alignment issue...because now you have both misaligned reads and writes.  Plus it seems misaligned reads are worse.

When, I was first doing zpools on AF drives (I didn't know about how bad this problem was), they seemed fine at first....where things got really bad when I did a scrub...which should be all reads.  Redoing the zpool for aligned, made a scrub go more than 10 times faster.

Meanwhile, I've done zpools on non AF drives as 4k aligned and not 4k aligned.  I hope the latter doesn't come around and bite me.

Unfortunately, the only way to fix is create a new 4k aligned pool and copy over the old one....can't add a 4k aligned disk into a non-4k aligned pool.   Though its nicer to fix with ZFS than say with Linux.

Last system I redid, I split the mirror...created a new zpool on one disk, properly aligned.  zfs send/receive between the zpools.  blow away the old zpool.  export the new zpool, reimport it as the name of the old zpool.  Add other disk in as mirror.

Only snag was to update zpool.cache, which I think isn't necessary with 9.1?  I was running 9.0 when I was doing this.

FWIW, I ran into this problem with linux and mdadm as well....had an old raid1 lose a drive, knowing that new drive was AF...instead of copying partition table to new drive, I worked out a new partition table.  I had figured I would just replace resilver and carry on...and it was okay at first, but eventually it wasn't happy.  I first tried removing the old disk, repartitioning and readding...but, I guess there's other layers involved.  So eventually it got remade...though I opted to get another disk of the same size as the replacement I had gotten.  (the old array was 1.5TB, and the replacement disk I had gotten was 2TB.)

Next project....turn the system into FreeBSD & ZFS without losing any data, and without having any extra drives or systems to help....hmmm.  Suspect that it might not be possible without some intermediate steps/hardware....

----- Original Message -----
> 
> Not sure if new are 4k. Done nothing about that.
> But the SECOND drive, resilvering is SLOW. Not the first one.
> 
> As stated below. Those changes are introduced to the system.
> ALL new driver ARE identical, except S/N of cause :)
> 
> On 3 jul 2013, at 16:55, Steven Hartland <killing at multiplay.co.uk>
> wrote:
> 
> > 
> > ----- Original Message ----- From: "Daniel Kalchev"
> > <daniel at digsys.bg>
> > To: "mxb" <mxb at alumni.chalmers.se>
> > Cc: <freebsd-fs at freebsd.org>
> > Sent: Wednesday, July 03, 2013 3:40 PM
> > Subject: Re: Slow resilvering with mirrored ZIL
> > 
> > 
> >> On 03.07.13 16:36, mxb wrote:
> >>> Well, then my question persists - why I get so significant drop
> >>> of speed while resilvering second drive.
> >>> The only changes to the system are:
> >>> 
> >>> 1. Second partition for ZIL to create a mirror
> >>> 2. New disks are 7200rpm. old ones are 5400rpm.
> >>> 
> > 
> > Its not something like the old disks are 512byte sectors
> > where as the new ones are 4k?
> > 
> > It this is the case having already replaced one disk you've
> > killed performance as its having to do lots more work reading
> > none 4k aligned data?
> > 
> >   Regards
> >   steve
> >