The arm64 fork-then-swap-out-then-swap-in failures: a program source for exploring them

[Mark Millard]

On 2017-Apr-12, at 6:25 PM, Otacílio <otacilio.neto at bsd.com.br> wrote:

> Em 10/04/2017 17:15, Mark Millard escreveu:
>> On 2017-Apr-10, at 2:51 AM, Mark Millard <markmi at dsl-only.net> wrote:
>> 
>>> On 2017-Apr-9, at 5:10 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>> 
>>>> On 2017-Apr-9, at 10:24 AM, Mark Millard <markmi at dsl-only.net> wrote:
>>>> 
>>>>> On 2017-Apr-9, at 5:27 AM, Konstantin Belousov <kostikbel at gmail.com> wrote:
>>>>>> Hmm, could you try the following patch, I did not even compiled it.
>>>>> I'll try it later today.
>>>>> 
>>>>>> diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c
>>>>>> index 3d5756ba891..55aa402eb1c 100644
>>>>>> --- a/sys/arm64/arm64/pmap.c
>>>>>> +++ b/sys/arm64/arm64/pmap.c
>>>>>> @@ -2481,6 +2481,11 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot)
>>>>>> 		    sva += L3_SIZE) {
>>>>>> 			l3 = pmap_load(l3p);
>>>>>> 			if (pmap_l3_valid(l3)) {
>>>>>> +				if ((l3 & ATTR_SW_MANAGED) &&
>>>>>> +				    pmap_page_dirty(l3)) {
>>>>>> +					vm_page_dirty(PHYS_TO_VM_PAGE(l3 &
>>>>>> +					    ~ATTR_MASK));
>>>>>> +				}
>>>>>> 				pmap_set(l3p, ATTR_AP(ATTR_AP_RO));
>>>>>> 				PTE_SYNC(l3p);
>>>>>> 				/* XXX: Use pmap_invalidate_range */
>>>> 
>>>> Preliminary testing indicates that this fixes the
>>>> some-pages-become-zero problem for fork-then-swapout/in.
>>>> 
>>>> Thanks!
>>>> 
>>>> I'll see if a buildworld can go through without being stopped
>>>> by the type of issue. But that will take a while. (It is how
>>>> I originally ran into the problem(s) that others had been
>>>> reporting on the lists.)
>>> buildworld buildkernel completed non-stop for the first time
>>> on a BPI-M3 board.
>> I had been thinking of the BPI-M3 for other reasons
>> and typed that instead of the correct: Pine64+ 2GB.
>> (True elsewhere as well.) I do really mean arm64
>> here, not armv7.
>> 
>>> Looks good for a check-in to svn to me (head and stable/11).
>>> 
>>> This combined with 2017-Feb-15's -r313772's fix to the fork
>>> trampline code's updating of sp_el0 makes arm64 far more stable
>>> for my purposes.
>>> 
>>> -r313772 was never MFC'd to stable/11. In my view it should be.
>> ===
>> Mark Millard
>> markmi at dsl-only.net
>> 
> Dears
> 
> Will this patch be committed to HEAD?

It was:

Author: kib
Date: Mon Apr 10 15:32:26 2017
New Revision: 316679
URL: 
https://svnweb.freebsd.org/changeset/base/316679


Log:
  Do not lose dirty bits for removing PROT_WRITE on arm64.
  
  Arm64 pmap interprets accessed writable ptes as modified, since
  ARMv8.0 does not track Dirty Bit Modifier in hardware. If writable bit
  is removed, page must be marked as dirty for MI VM.
  
  This change is most important for COW, where fork caused losing
  content of the dirty pages which were not yet scanned by pagedaemon.
  
  Reviewed by:	alc, andrew
  Reported and tested by:	Mark Millard <markmi at dsl-only.net>
  PR:	217138, 217239
  Sponsored by:	The FreeBSD Foundation
  MFC after:	2 weeks

Modified:
  head/sys/arm64/arm64/pmap.c

Modified: head/sys/arm64/arm64/pmap.c
==============================================================================
--- head/sys/arm64/arm64/pmap.c	Mon Apr 10 12:35:58 2017	(r316678)
+++ head/sys/arm64/arm64/pmap.c	Mon Apr 10 15:32:26 2017	(r316679)
@@ -2481,6 +2481,11 @@ pmap_protect(pmap_t pmap, vm_offset_t sv
 		    sva += L3_SIZE) {
 			l3 = pmap_load(l3p);
 			if (pmap_l3_valid(l3)) {
+				if ((l3 & ATTR_SW_MANAGED) &&
+				    pmap_page_dirty(l3)) {
+					vm_page_dirty(PHYS_TO_VM_PAGE(l3 &
+					    ~ATTR_MASK));
+				}
 				pmap_set(l3p, ATTR_AP(ATTR_AP_RO));
 				PTE_SYNC(l3p);
 				/* XXX: Use pmap_invalidate_range */


There was a patch ( -r313772 ) committed to head back in Feb. for
interrupts sometimes trashing a special register during fork. It
takes both of these patches to get fork working reliably.

[stable/11 should eventually get both of these patches
so that fork becomes reliable there for aarch64 (armv8.0).]

===
Mark Millard
markmi at dsl-only.net