Restructure a ZFS Pool

[Raimund Sacherer]

Hello, 

I have a ZFS pool consisting of 12 4TB drives, where 2 of those drives are dedicated as spares, as this was my first FreeBSD / ZFS server I did some things different than I would do it know and I want to rectify them:

# zpool status
  pool: tank
 state: ONLINE
  scan: scrub repaired 0 in 58h52m with 0 errors on Thu Sep 24 02:29:47 2015
config:

	NAME        STATE     READ WRITE CKSUM
	tank        ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	spares
	  da10      AVAIL   
	  da11      AVAIL   

errors: No known data errors


I had the pool fill up to over 80%, then I got it back to about 50-60%, but it feels more sluggish. I use a lot of NFS and we use it to backup some 5 million files in lots of sub-directorys (a/b/c/d/abcd...), besides other big files (SQL dump backups, bacula, etc.)

I said above sluggish because I do not have empirical data and I do not know exactly how to test the system correctly, but I read a lot and there seem to be suggestions that if you have NFS etc. that a independent ZIL helps with copy-on-write fragmentation. 

What I would like to know is if I can eliminate one Spare disk from the pool, and add it as a ZIL again, without having to shutdown/reboot the server? I am also thinking about swapping the spare 4TB disk for a small SSD, but that's immaterial to whether I can perform the change. 


Also I would appreciate it if someone has some pointers on how to test correctly so I see if there are real benefits before/after this operation.

Thank you,
Best
Ray