Re: git: 92e40a9b9241 - main - busdma_bounce: Batch bounce page free operations when possible.

On 5/6/22 4:59 AM, Bjoern A. Zeeb wrote:
> On Thu, 21 Apr 2022, John Baldwin wrote:
> 
>> The branch main has been updated by jhb:
>>
>> URL: https://cgit.FreeBSD.org/src/commit/?id=92e40a9b9241313b3d16543819ccd8d642bb11a5
>>
>> commit 92e40a9b9241313b3d16543819ccd8d642bb11a5
>> Author:     John Baldwin <jhb@FreeBSD.org>
>> AuthorDate: 2022-04-21 19:01:55 +0000
>> Commit:     John Baldwin <jhb@FreeBSD.org>
>> CommitDate: 2022-04-21 19:01:55 +0000
>>
>>     busdma_bounce: Batch bounce page free operations when possible.
>>
>>     Reviewed by:    imp
>>     Differential Revision:  https://reviews.freebsd.org/D34968
>> ---
>> sys/kern/subr_busdma_bounce.c | 69 +++++++++++++++++++++----------------------
>> 1 file changed, 34 insertions(+), 35 deletions(-)
>>
>> diff --git a/sys/kern/subr_busdma_bounce.c b/sys/kern/subr_busdma_bounce.c
>> index 243da8e9487f..f3699cf2ad27 100644
>> --- a/sys/kern/subr_busdma_bounce.c
>> +++ b/sys/kern/subr_busdma_bounce.c
>> @@ -382,55 +382,54 @@ add_bounce_page(bus_dma_tag_t dmat, bus_dmamap_t map, vm_offset_t vaddr,
>> }
>>
>> static void
>> -free_bounce_page(bus_dma_tag_t dmat, struct bounce_page *bpage)
>> +free_bounce_pages(bus_dma_tag_t dmat, bus_dmamap_t map)
>> {
> ...
>> 	mtx_lock(&bounce_lock);
>> -	STAILQ_INSERT_HEAD(&bz->bounce_page_list, bpage, links);
> ...
>> +	STAILQ_CONCAT(&bz->bounce_page_list, &map->bpages);
> 
> 
> This changes an aspect we had discussed in a different context of a
> change I had proposed to keep the pages ordered in order to fullfill
> multi-page nseg=1 requests better.

You could fix that by doing two CONCAT operations perhaps:

STAILQ_CONCAT(&map->bpages, &bz->bounce_page_list);
STAILQ_CONCAT(&bz->bounce_page_list, &map->bpages);

You'd definitely need a comment explaining why and ideally some use
case that shows a benefit.  That said, STAILQ_CONCAT are O(1) operations
that I think are 4 assignments each, so relatively cheap.

> In the new case your "possibly still cache hot" pages are no longer
> at the beginning of the bounce_page_list but at the end?
> 
> If this is no longer considered to be an issue, contigous page space
> still is, so I wondere if a change to keep the pages ordered would be
> accepted more likely now?

FWIW, this change actually keeps multi-segment page batches more
in order as they would retain their relative order in bz->bounce_page_list
on free rather than being reversed.

-- 
John Baldwin