ZFS Write Lockup

[Dave Cundiff]

On Wed, Oct 5, 2011 at 5:52 PM, Daniel Staal <DStaal at usa.net> wrote:
> --As of October 4, 2011 2:43:45 AM -0400, Dave Cundiff is alleged to have
> said:
>
>> I don't know what triggers the problem but I know how to fix it. If I
>> perform a couple snapshot deletes the IO will come back in line every
>> single time. Fortunately I have LOTS of snapshots to delete.
>>
>> [root at san2 ~]# zfs list -r -t snapshot | wc -l
>>    5236
>> [root at san2 ~]# zfs list -r -t volume | wc -l
>>      17
>
> --As for the rest, it is mine.
>
> I have no good advice, but I have a thought.  ;)
>
> The thought is: Why so many snapshots?  And: How many other people have that
> many snapshots?  I know that ZFS is supposed to be able to handle huge
> numbers of snapshots (far more than a few thousand, from my understanding),
> but if it hasn't been used much in that config, there may be bugs lurking.
>
> You might try weeding through and figuring out if you can drop a good amount
> of those snapshots.  Also, try the filesystems list.  They may have better
> thoughts.
>
> Daniel T. Staal
>

Its for a backup service I've been working on. It takes a snapshot
hourly of all 17 zvols. I was planning on keeping them for a month.

I had the same thought about the snapshots and deleted them all
yesterday. It appears there is some issue with keeping that many. I
removed them all and the zvols are now functioning correctly. Its
strange that the large number didn't cause incremental slowdown. While
the snapshots were still there the IO was normal when it wasn't acting
up. Just it would have spurts of almost total lockup until I performed
a snapshot removal operation or 2.

Thanks,

-- 
Dave Cundiff
System Administrator
A2Hosting, Inc
http://www.a2hosting.com