pending changes for TOE support

[Kip Macy]

>
> Undoubtably this is true for high-end systems sporting PCIe interfaces with
> 10gbps cards, but we also run in other configurations.  A complaint we've had
> a fair amount in the last few years is that our work has increasingly targeted
> high-end systems where overhead is in cache misses, and decreasingly targeted
> low-end systems where overhead is in instruction count.  While you offer the
> opportunity to compile out some of this, I think we should try to make these
> things capable of being fast in both circumstances without multiple compile
> paths where it's easy.


Ok. That was the intent of the initial design. The design intent,
however, was that it would be compiled out on slower platforms -
sparc64, arm, mips etc.


> Ideally, we'd make the ifdef very, very small, as it's well-known that when we
> have two mutually exclusive code paths and one isn't compiled or used
> frequently, it rots.

Good point.


>
> >> I find myself wondering if the offload option should be a TCP socket option
> >> rather than a socket-layer option, but don't have strong feelings about
> >> this.
> >
> > The assumption is that, if you a) pay the extra for a version of the card
> > that supports TOE and b) you load the toe module (it isn't part of the NIC
> > driver) that you want your existing software to have its connections
> > offloaded. I don't know of any customers that want to modify their user
> > applications to selectively enable TOE.
>
> I think you mis-understood my question -- I was wondering whether it should be
> selectively disabled by an IP-layer socket option rather than a socket-layer
> socket option.


I did misunderstand, and yes a TCP_ socket option would probably be
more appropriate..


> > That is the current usage model. At some point we may tie it into
> > pf/ipf/ipfw to provide for offload policy. However, currently we offload all
> > connections on an interface until we run out of TCAM entries.
> >
> >> Are you currently anticipating enabling the capability by default?
> >
> > Yes, *if the TOE driver is loaded*.
>
> I have somewhat mixed feelings about this, and feel like there should be
> something eloquent to say about having functionality administratively enabled
> rather than a default when compiled in, but can't quite figure out a nice
> expression of that.  I think it might have something to do with expecting
> vendors of 10gbps cards to like shipping two modules for every device rather
> than one, and having the right behavior out of the box if they do ship just
> one.

This is easy enough to arrange, just as we have TSO and jumbo frames
that some drivers default to on and others default to off, we can have
the TOE capability enabled or disabled by default by the driver. I
think that if vendors left TOE disabled by default in the single
module case that this would address your concerns, would it not?


 -Kip