Trim on gmirrored SSDs is slow and results in inresponsive system

[Steven Hartland]

On 05/03/2015 14:10, Robert Schulze wrote:
> Hi,
>
> Am 04.03.2015 um 17:09 schrieb Steven Hartland:
>> Transaction sizes look like they are the same but the tps from gmirror
>> is significantly lower.
>>
>> As iostat doesn't properly understand deletes it would still be nice to
>> see that gstat -d -p output as that will also give us queue depth.
> Since gstat doesnt just output row by row, I had to screenshot a running
> gstat and transcribe the values. Running gstat -b in a loop doesn't help
> here, since the system is inresponsive and won't start any program in
> time during the DELETE ops.
>
> So after deleting a 2GB file on a gmirror with two active SSDs I get:
>
> # gstat -dpfada -I3s
>
> dT: 3.189s  w: 3.000s  filter: ada
> L(q) ops/s r/s kBps ms/r w/s kBps ms/w  d/s   kBps ms/d %busy Name
> 0     3342   0    0  0.0   0    0  0.0 3342 106939 10.2  80.5| ada0
> 0     3342   0    0  0.0   0    0  0.0 3342 106939 10.0  82.0| ada1
>
> 9     2606   0    0  0.0   0    0  0.0 2606  83400 60.8  98.3| ada0
> 9     2606   0    0  0.0   0    0  0.0 2606  83400 56.9  98.7| ada1
>
> 46    1705   0    0  0.0   0    0  0.0 1705  54565 62.5  98.4| ada0
> 51    1704   0    0  0.0   0    0  0.0 1704  54512 62.9  97.6| ada1
>
> 22    1540   0    0  0.0   0    0  0.0 1540  49285 20.9  96.6| ada0
> 24    1541   0    0  0.0   0    0  0.0 1541  49317 21.8  96.0| ada1
>
> 2     1496   0    0  0.0   0    0  0.0 1496  47887 13.3  98.6| ada0
> 7     1495   0    0  0.0   0    0  0.0 1495  47855 14.1  98.9| ada1
>
> 11    1032   0    0  0.0   0    0  0.0 1032  33010 18.2  98.8| ada0
> 10    1033   0    0  0.0   0    0  0.0 1033  33070 15.4  98.1| ada1
>
> 12     775   0    0  0.0   0    0  0.0  775  24798 26.2  97.5| ada0
> 16     773   0    0  0.0   0    0  0.0  773  24746 29.6  98.5| ada1
>
> 65     743   0    0  0.0   0    0  0.0  743  23784 25.4  98.8| ada0
> 59     746   0    0  0.0   0    0  0.0  746  23887 24.4  99.0| ada1
These values look totally reasonable in terms a d/s.

How does this compare to the none gmirror case?
>> Try backing out r268816 and see if that has any impact.
> could you please explain what you mean by that?
>
Compile a version of the kernel with this change reverted.

Details of the change can be seen here.
https://svnweb.freebsd.org/base?view=revision&revision=268816

You can download the changes (only ata is relevant in your case) and 
apply the patch with reverse option.

Once you've done that recompile and re-install your kernel.

This feels like it might be an request ordering issue.

     Regards
     Steve