misc/93887: cpu_spinwait calls missing
Arthur Hartwig
arthur.hartwig at nokia.com
Mon Feb 27 18:40:17 PST 2006
The following reply was made to PR kern/93887; it has been noted by GNATS.
From: Arthur Hartwig <arthur.hartwig at nokia.com>
To: ext Kris Kennaway <kris at obsecurity.org>
Cc: freebsd-gnats-submit at FreeBSD.org
Subject: Re: misc/93887: cpu_spinwait calls missing
Date: Tue, 28 Feb 2006 12:32:08 +1000
This is a multi-part message in MIME format.
--------------090009090003010006010508
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
ext Kris Kennaway wrote:
>On Mon, Feb 27, 2006 at 06:42:08AM +0000, Arthur Hartwig wrote:
>
>
>>>Number: 93887
>>>Category: misc
>>>Synopsis: cpu_spinwait calls missing
>>>Confidential: no
>>>Severity: non-critical
>>>Priority: low
>>>Responsible: freebsd-bugs
>>>State: open
>>>Quarter:
>>>Keywords:
>>>Date-Required:
>>>Class: sw-bug
>>>Submitter-Id: current-users
>>>Arrival-Date: Mon Feb 27 06:50:04 GMT 2006
>>>Closed-Date:
>>>Last-Modified:
>>>Originator: Arthur Hartwig
>>>Release: 6.1-BETA1
>>>Organization:
>>>
>>>
>>Nokia
>>
>>
>>>Environment:
>>>
>>>
>>oz-net-11# uname -a
>>FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006 root at oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11 amd64
>>
>>
>>
>>
>>>Description:
>>>
>>>
>>There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops."
>>
>>Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action()
>>
>>
>
>Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
>restart_cpus()? They're used e.g. as preparation to break into/out of
>the debugger, and AFAICT they're not in the critical path for
>anything.
>
>Not sure about smp_rendezvous_action(), maybe that one is OK.
>
>Kris
>
Good question which didn't occur to me because I was coming from another
context. I was looking at using these functions to support a particular
piece of "hotswap" hardware. The hardware generates an interrupt to say
a PCI device has been removed (or plugged in). In the removal case some
work needs to be done to remap to main memory the memory address space
assigned to the device being removed so that if a device driver is
accessing the device's memory space it won't cause the system to die. On
a UP system this is "easy" in that because the hotswap interrupt is
running we know the driver for the removed device is not currently
running and even if it was running before the interrupt by the time it
runs again the memory remapping will have occurred and it will be safe
for the driver to access the memory space previously assigned to the
removed device.However on a MP system the driver could be executing
concurrently on another CPU and cause the system to die by accessing
memory that is no longer present.
I thought of using these functions (stop_cpus() etc) to momentarily
"pause" all other CPUs while adjusting the memory mapping. It would be
of advantage to cause other CPUs to pause as quickly as possible to
minimise the likelihood that the driver for the removed device will
attempt to access memory on the removed device. I have not been able to
find any data on how long I have between the interrupt to say a device
is being removed and the actual removal of the device. Nor have I any
data on how long a busy CPU of a hyperthreaded pair (for example) is
able to "lock out" its hyperthreaded mate.
The referenced functions are clearly adequate for their existing use by
the debugger so I accept that I might be better off making a customised
copy.
I still have doubts about the effectiveness of the approach using
stop_cpus() and co and I'm inclined to see if I can get away with saying
"you can't pull out a board untill you have disabled all the devices on
that board" but this is somewhat restricted compared with what users
have become accustomed to.
Arthur
--------------090009090003010006010508
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
ext Kris Kennaway wrote:
<blockquote cite="mid20060227232944.GA88341 at xor.obsecurity.org"
type="cite">
<pre wrap="">On Mon, Feb 27, 2006 at 06:42:08AM +0000, Arthur Hartwig wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">Number: 93887
Category: misc
Synopsis: cpu_spinwait calls missing
Confidential: no
Severity: non-critical
Priority: low
Responsible: freebsd-bugs
State: open
Quarter:
Keywords:
Date-Required:
Class: sw-bug
Submitter-Id: current-users
Arrival-Date: Mon Feb 27 06:50:04 GMT 2006
Closed-Date:
Last-Modified:
Originator: Arthur Hartwig
Release: 6.1-BETA1
Organization:
</pre>
</blockquote>
<pre wrap="">Nokia
</pre>
<blockquote type="cite">
<pre wrap="">Environment:
</pre>
</blockquote>
<pre wrap="">oz-net-11# uname -a
FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006 <a class="moz-txt-link-abbreviated" href="mailto:root at oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11">root at oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11</a> amd64
</pre>
<blockquote type="cite">
<pre wrap="">Description:
</pre>
</blockquote>
<pre wrap="">There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops."
Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action()
</pre>
</blockquote>
<pre wrap=""><!---->
Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
restart_cpus()? They're used e.g. as preparation to break into/out of
the debugger, and AFAICT they're not in the critical path for
anything.
Not sure about smp_rendezvous_action(), maybe that one is OK.
Kris</pre>
</blockquote>
Good question which didn't occur to me because I was coming from
another context. I was looking at using these functions to support a
particular piece of "hotswap" hardware. The hardware generates an
interrupt to say a PCI device has been removed (or plugged in). In the
removal case some work needs to be done to remap to main memory the
memory address space assigned to the device being removed so that if a
device driver is accessing the device's memory space it won't cause the
system to die. On a UP system this is "easy" in that because the
hotswap interrupt is running we know the driver for the removed device
is not currently running and even if it was running before the
interrupt by the time it runs again the memory remapping will have
occurred and it will be safe for the driver to access the memory space
previously assigned to the removed device.However on a MP system the
driver could be executing concurrently on another CPU and cause the
system to die by accessing memory that is no longer present.<br>
<br>
I thought of using these functions (stop_cpus() etc) to momentarily
"pause" all other CPUs while adjusting the memory mapping. It would be
of advantage to cause other CPUs to pause as quickly as possible to
minimise the likelihood that the driver for the removed device will
attempt to access memory on the removed device. I have not been able to
find any data on how long I have between the interrupt to say a device
is being removed and the actual removal of the device. Nor have I any
data on how long a busy CPU of a hyperthreaded pair (for example) is
able to "lock out" its hyperthreaded mate.<br>
<br>
The referenced functions are clearly adequate for their existing use by
the debugger so I accept that I might be better off making a customised
copy. <br>
<br>
I still have doubts about the effectiveness of the approach using
stop_cpus() and co and I'm inclined to see if I can get away with
saying "you can't pull out a board untill you have disabled all the
devices on that board" but this is somewhat restricted compared with
what users have become accustomed to.<br>
<br>
Arthur<br>
<br>
</body>
</html>
--------------090009090003010006010508--
More information about the freebsd-bugs
mailing list