svn commit: r278472 - in head/sys: netinet netinet6

[Randall Stewart]

Gleb:

Ok here is the diff of the arp timer function that this changed made (238990):
****************************
 arptimer(void *arg)
 {
+       struct llentry *lle = (struct llentry *)arg;
        struct ifnet *ifp;
-       struct llentry   *lle;
-       int pkts_dropped;
+       size_t pkts_dropped;
 
-       KASSERT(arg != NULL, ("%s: arg NULL", __func__));
-       lle = (struct llentry *)arg;
+       if (lle->la_flags & LLE_STATIC) {
+               LLE_WUNLOCK(lle);
+               return;
+       }
+
        ifp = lle->lle_tbl->llt_ifp;
        CURVNET_SET(ifp->if_vnet);
+
+       if (lle->la_flags != LLE_DELETED) {
+               int evt;
+
+               if (lle->la_flags & LLE_VALID)
+                       evt = LLENTRY_EXPIRED;
+               else
+                       evt = LLENTRY_TIMEDOUT;
+               EVENTHANDLER_INVOKE(lle_event, lle, evt);
+       }
+
+       callout_stop(&lle->la_timer);
+
+       /* XXX: LOR avoidance. We still have ref on lle. */
+       LLE_WUNLOCK(lle);
        IF_AFDATA_LOCK(ifp);
        LLE_WLOCK(lle);
-       if (lle->la_flags & LLE_STATIC)
-               LLE_WUNLOCK(lle);
-       else {
-               if (!callout_pending(&lle->la_timer) &&
-                   callout_active(&lle->la_timer)) {
-                       callout_stop(&lle->la_timer);
-                       LLE_REMREF(lle);
 
-                       if (lle->la_flags != LLE_DELETED) {
-                               int evt;
-
-                               if (lle->la_flags & LLE_VALID)
-                                       evt = LLENTRY_EXPIRED;
-                               else
-                                       evt = LLENTRY_TIMEDOUT;
-                               EVENTHANDLER_INVOKE(lle_event, lle, evt);
-                       }
-
-                       pkts_dropped = llentry_free(lle);
-                       ARPSTAT_ADD(dropped, pkts_dropped);
-                       ARPSTAT_INC(timeouts);
-               } else {
-#ifdef DIAGNOSTIC
-                       struct sockaddr *l3addr = L3_ADDR(lle);
-                       log(LOG_INFO,
-                           "arptimer issue: %p, IPv4 address: \"%s\"\n", lle,
-                           inet_ntoa(
-                               ((const struct sockaddr_in *)l3addr)->sin_addr));
-#endif
-                       LLE_WUNLOCK(lle);
-               }
-       }
+       LLE_REMREF(lle);
+       pkts_dropped = llentry_free(lle);
        IF_AFDATA_UNLOCK(ifp);
+       ARPSTAT_ADD(dropped, pkts_dropped);
+       ARPSTAT_INC(timeouts);
        CURVNET_RESTORE();
 }
******************************

And I can see *what* the problem was.. If you look at the lines:
-               if (!callout_pending(&lle->la_timer) &&
-                   callout_active(&lle->la_timer)) {

This is the *incorrect* way to write this code it should have been:
              if (callout_pending(&lle->la_timer) &&
                  !callout_active(&lle->la_timer)) {

To properly do the MPSAFE thing.. take a look at the callout manual..
So the original author just mixed up the test.. 

The idea is after you get the lock you check if its pending again, if
so someone has restarted it.. and if its not active, then someone has
stopped it.

They check if it was *not* pending.. which is almost always the case
since the callout code removes the pending flag and thats what you want
to be there. So not pending would always be true.. 

I don’t see that this old code did the callout_deactive().. but I think
the callout_stop() does that I guess..

I think the problem was that the original code was wrong and that
the fix yes, avoided one race but put in another.

I do think a callout_async_drain is the right thing, but that will take a while
to get right. Peter Holm (thank you so much) has been pounding on this, if you
could find another test to add.. that would be great. I think this will work
the way it is..

R






On Feb 13, 2015, at 4:21 PM, Gleb Smirnoff <glebius at freebsd.org> wrote:

> 165863

--------
Randall Stewart
rrs at netflix.com
803-317-4952