ipv6 lock contention with parallel socket io

[Adrian Chadd]

On 30 December 2013 23:35, Adrian Chadd <adrian at freebsd.org> wrote:
> Hi,
>
> I've noticed a bunch of lock contention occurs when doing highly
> parallel (> 4096 sockets) TCP IPv6 traffic.
>
> The contention is here:
>
[snip]

>
> .. it's the IF_ADATA lock surrounding the lla_lookup() calls.
>
> Now:
>
> * is there any reason this isn't an rmlock?
> * the instance early on in nd6_output_lle() isn't taking the read
> lock, it's taking the full lock. Is there any reason for this?
>
> I don't have much experience or time to spend on optimising the ipv6
> code to scale better but this seems like one of those things that'll
> bite us in the ass as the amount of ipv6 deployed increases.
>
> Does anyone have any ideas/suggestions on how we could improve things?

This improves things quite a bit - from 1.9gbyte/sec @ 100% cpu usage
to 2.7gbyte/sec @ 85% CPU usage. It's not perfect - the lock
contention is still there - but it's much less of an issue now.

Are there any issues with it?

Index: sys/netinet6/nd6.c
===================================================================
--- sys/netinet6/nd6.c  (revision 259475)
+++ sys/netinet6/nd6.c  (working copy)
@@ -1891,9 +1891,9 @@
        flags = ((m != NULL) || (lle != NULL)) ? LLE_EXCLUSIVE : 0;
        if (ln == NULL) {
        retry:
-               IF_AFDATA_LOCK(ifp);
+               IF_AFDATA_RLOCK(ifp);
                ln = lla_lookup(LLTABLE6(ifp), flags, (struct sockaddr *)dst);
-               IF_AFDATA_UNLOCK(ifp);
+               IF_AFDATA_RUNLOCK(ifp);
                if ((ln == NULL) && nd6_is_addr_neighbor(dst, ifp))  {
                        /*
                         * Since nd6_is_addr_neighbor() internally
calls nd6_lookup(),


Thanks,



-a