svn commit: r341682 - head/sys/sys

[Warner Losh]

On Mon, Dec 10, 2018 at 9:57 PM Scott Long <scottl at samsco.org> wrote:

>
>
> > On Dec 10, 2018, at 4:47 PM, Konstantin Belousov <kostikbel at gmail.com>
> wrote:
> >
> > On Mon, Dec 10, 2018 at 02:15:20PM -0800, John Baldwin wrote:
> >> On 12/8/18 7:43 PM, Warner Losh wrote:
> >>>
> >>>
> >>> On Sat, Dec 8, 2018, 8:36 PM Kevin Bowling <kevin.bowling at kev009.com
> <mailto:kevin.bowling at kev009.com> wrote:
> >>>
> >>>    On Sat, Dec 8, 2018 at 12:09 AM Mateusz Guzik <mjguzik at gmail.com
> <mailto:mjguzik at gmail.com>> wrote:
> >>>
> >>>>
> >>>> Fully satisfying solution would be that all architectures get 64-bit
> >>>> ops, even if in the worst case they end up taking a lock. Then
> >>>> subsystems would not have to ifdef on anything. However, there
> >>>> was some opposition to this proposal and I don't think this is
> >>>> important enough to push.
> >>>
> >>>    Mateusz,
> >>>
> >>>    Who is opposing this particular polyfill solution?  Scott Long
> brought
> >>>    up a situation in driver development where this would be useful as
> >>>    well.  The polyfills lower the cognitive load and #ifdef soup which
> >>>    are the right call here regardless of performance on toy ports.
> >>>
> >>>
> >>> I don't recall seeing the opposition either. It would have to be a
> global lock for all 64bit atomics.... but I think it would only be 2
> atomics on those architectures.
> >>
> >> It would have to be a spin lock, so in the case of unrl you would be
> trading
> >> an operation on one of N regular mutexes for a single spin lock that was
> >> also contested by other things.  This would be pretty crappy.  For
> drivers
> >> that aren't actually used on platforms without 32-bit atomics we can
> simply
> >> not build them in sys/modules/Makefile or not put them in GENERIC.  For
> >> something in the core kernel like unrl I think we will have to do what
> >> Mateusz has done here.
> >
> > It is worse. All atomics that acess the same location must use the same
> > lock. Otherwise, you could observe torn writes and out of thin air
> > values. Since you cannot know in advance which locations are acceses
> > by the locked variant, all freebsd atomics ops have to be switched to
> > locked variant on the architecture.
>
> 64bit atomics on I486 already suffer the risk of torn reads; the
> implementation
> merely does a CLI to protect against local preemption (though you could
> still
> get unlucky with an NMI).  I suppose you could argue that SMP isn’t really
> viable on I486 and therefore this fact is irrelevant, but it does
> illustrate
> precedence for having API completeness in a platform.
>

We haven't ever supported SMP on i486, to my knowledge. Certainly by the
5.x time frame with SMPng it wasn't there. The 64-bit ops that are there
are mostly to smoothly support some of the (now older) embedded boards. I
haven't looked at the SMP work smp did for 4.x, but IIRC, it wasn't even
supported there.


> Really, this isn’t that hard.  Part of the existing contract of using
> atomics is
> that you carefully evaluate all uses of the variable and decide when to use
> an atomic instruction.  Arguing that we can’t make this process automatic
> and foolproof for 64bit quantities, especially for a subset of subset of
> platforms/architectures, and therefore we should be even more of a
> difficult
> landmine, is not…. I don’t know what to say… sensical?
>

I think it's fine to say that 64-bit atomics need to be efficient on the
supported platforms (more on that below).


> 64bit operations are a reality for MI code in a modern OS, and I’m tired of
> having to tip-toe around them due to incomplete MD implementations.  The
> instructions have been available on Intel CPUs for 25 years!  My
> very strong preference is to have a complete and functional implementation
> of atomic.h for any architecture that is hooked up to the build.  We can
> then
> tackle the details of optimization and edge case refinement, just like we
> do
> with every other API and service that we work on.  It doesn’t have to be
> perfect to be useful, and at this point we’re providing neither perfection
> nor
> utility, just “buts” and “what ifs”.
>

I think you miss the point of discussion, at least on my part. I'm looking
at the MIPS side and asking the question whatever 32-bit SMP support we may
have had in the past should die. We only ever supported it on one
not-so-common board that's aged out of relevance (JZ4780 is the only one I
found that needs 32-bit MIPS SMP, and it's a 4 year old embedded board
that's not that relevant today and there's no successor products in the
market or as far as I can tell planned). We also have kernels that run in
32-bit mode on 64-bit hardware, but those provide little value and w
already transitioned our largest 64-bit mips platform away from that
support, so we can deorbit as well, I think. They give little value to the
project. some house keeping here is likely in order.

If that just leaves an odd PPC thing, then I think it's perfectly fine to
start conversations there as well about trimming that support with the
powerpc guys.

Going forward, I’m going to start using 64bit atomics where they’re prudent,
> instead of avoiding them due to this niche 32bit argument.  If that means
> more and more of what I do no longer compiles on a mips or a ppc32, then
> that’s a sacrifice that is fine with me.  It still creates extra
> development work,
> and having a uniformly available implementation would be much nicer.
>

To be clear, I'm not making the argument since I could dig up one place in
mips land that uses it, we should hold things back. I think the cost /
benefit ratio for mips at least is the same as we have for some of the
older arm stuff: we can implement it by disabling interrupts and declaring
by fiat we won't support what few SMP chips that might be out there on that
platform. FreeBSD/arm decided years ago we would never support the armv4
and armv5 SMP designs that were in the market place, for example. I think
it's fine that we do that today with at least the mips32 stuff. And if
people want to keep the mips32 UP stuff going, it's with the same CLI ; do
it; POPF trick we do on intel to keep i486 going. My argument is that we
need to find all the places that this would hurt, and we should make a
calm, dispassionate decision about that pain. So far, the pain brought up
on the 'hard to support end' suggests a minor trimming of what we support
is in order rather than a holding back of other gains that can be made. I
do plan on making this argument in the thread that's going on in mips@
right now, and hope to have it all squared away by the end of the week what
the plans will be.

We need to take a look at the trailing edge more aggressively. I'm looking
at super-old SCSI cards, PC Card, armv5 and a few others that we should
trim in 13. We've already circulated these ideas, and apart from very
specific objections met with general support. Maybe we should seriously
look at adding i486 to the list, as well as taking a hard look at 32-bit
platforms post 13 and planning for that transition starting today. Brooks
also has in motion this on the older nic card side of things too, so we'll
see a big trimming there in the coming months as well.  One thing the
project has been bad about has been collecting data on what's actually in
use and using that to more aggressively drive deprecation and removal of
old drivers and features from the system that are no longer relevant. So we
get stuck in arguments of someone has it, might be using it and since
there's not organized pushback that articulates the real cost this has to
the project, we accumulate a lot of drivers whose only changes have been
API changes (and we're not even sure those were done right). The 64-bit
atomics friction and grumpiness is but one symptom of this larger, deeper
problem. And unless we address that, these narrow issues that are easy to
talk to death will continue to plague our forward progress.

Warner