misc/93887: cpu_spinwait calls missing

Mon Feb 27 18:40:17 PST 2006

The following reply was made to PR kern/93887; it has been noted by GNATS.

From: Arthur Hartwig <arthur.hartwig at nokia.com>
To: ext Kris Kennaway <kris at obsecurity.org>
Cc: freebsd-gnats-submit at FreeBSD.org
Subject: Re: misc/93887: cpu_spinwait calls missing
Date: Tue, 28 Feb 2006 12:32:08 +1000

 This is a multi-part message in MIME format.
 --------------090009090003010006010508
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit

 ext Kris Kennaway wrote:

 >On Mon, Feb 27, 2006 at 06:42:08AM +0000, Arthur Hartwig wrote:
 >  
 >
 >>>Number:         93887
 >>>Category:       misc
 >>>Synopsis:       cpu_spinwait calls missing
 >>>Confidential:   no
 >>>Severity:       non-critical
 >>>Priority:       low
 >>>Responsible:    freebsd-bugs
 >>>State:          open
 >>>Quarter:        
 >>>Keywords:       
 >>>Date-Required:
 >>>Class:          sw-bug
 >>>Submitter-Id:   current-users
 >>>Arrival-Date:   Mon Feb 27 06:50:04 GMT 2006
 >>>Closed-Date:
 >>>Last-Modified:
 >>>Originator:     Arthur Hartwig
 >>>Release:        6.1-BETA1
 >>>Organization:
 >>>      
 >>>
 >>Nokia
 >>    
 >>
 >>>Environment:
 >>>      
 >>>
 >>oz-net-11# uname -a
 >>FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006     root at oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11  amd64
 >>
 >>
 >>    
 >>
 >>>Description:
 >>>      
 >>>
 >>There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops."
 >>
 >>Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action()
 >>    
 >>
 >
 >Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
 >restart_cpus()?  They're used e.g. as preparation to break into/out of
 >the debugger, and AFAICT they're not in the critical path for
 >anything.
 >
 >Not sure about smp_rendezvous_action(), maybe that one is OK.
 >
 >Kris
 >
 Good question which didn't occur to me because I was coming from another 
 context. I was looking at using these functions to support a particular 
 piece of "hotswap" hardware. The hardware generates an interrupt to say 
 a PCI device has been removed (or plugged in). In the removal case some 
 work needs to be done to remap to main memory the memory address space 
 assigned to the device being removed so that if a device driver is 
 accessing the device's memory space it won't cause the system to die. On 
 a UP system this is "easy" in that because the hotswap interrupt is 
 running we know the driver for the removed device is not currently 
 running and even if it was running before the interrupt by the time it 
 runs again the memory remapping will have occurred and it will be safe 
 for the driver to access the memory space previously assigned to the 
 removed device.However on a MP system the driver could be executing 
 concurrently on another CPU and cause the system to die by accessing 
 memory that is no longer present.

 I thought of using these functions (stop_cpus() etc) to momentarily 
 "pause" all other CPUs while adjusting the memory mapping. It would be 
 of advantage to cause other CPUs to pause as quickly as possible to 
 minimise the likelihood that the driver for the removed device will 
 attempt to access memory on the removed device. I have not been able to 
 find any data on how long I have between the interrupt to say a device 
 is being removed and the actual removal of the device. Nor have I any 
 data on how long a busy CPU of a hyperthreaded pair (for example) is 
 able to "lock out" its hyperthreaded mate.

 The referenced functions are clearly adequate for their existing use by 
 the debugger so I accept that I might be better off making a customised 
 copy.

 I still have doubts about the effectiveness of the approach using 
 stop_cpus() and co and I'm inclined to see if I can get away with saying 
 "you can't pull out a board untill you have disabled all the devices on 
 that board" but this is somewhat restricted compared with what users 
 have become accustomed to.

 Arthur

 --------------090009090003010006010508
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: 7bit

 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
 <html>
 <head>
   <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
   <title></title>
 </head>
 <body bgcolor="#ffffff" text="#000000">
 ext Kris Kennaway wrote:
 <blockquote cite="mid20060227232944.GA88341 at xor.obsecurity.org"
  type="cite">
   <pre wrap="">On Mon, Feb 27, 2006 at 06:42:08AM +0000, Arthur Hartwig wrote:
   </pre>
   <blockquote type="cite">
     <blockquote type="cite">
       <pre wrap="">Number:         93887
 Category:       misc
 Synopsis:       cpu_spinwait calls missing
 Confidential:   no
 Severity:       non-critical
 Priority:       low
 Responsible:    freebsd-bugs
 State:          open
 Quarter:        
 Keywords:       
 Date-Required:
 Class:          sw-bug
 Submitter-Id:   current-users
 Arrival-Date:   Mon Feb 27 06:50:04 GMT 2006
 Closed-Date:
 Last-Modified:
 Originator:     Arthur Hartwig
 Release:        6.1-BETA1
 Organization:
       </pre>
     </blockquote>
     <pre wrap="">Nokia
     </pre>
     <blockquote type="cite">
       <pre wrap="">Environment:
       </pre>
     </blockquote>
     <pre wrap="">oz-net-11# uname -a
 FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006     <a class="moz-txt-link-abbreviated" href="mailto:root at oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11">root at oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11</a>  amd64

     </pre>
     <blockquote type="cite">
       <pre wrap="">Description:
       </pre>
     </blockquote>
     <pre wrap="">There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops."

 Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action()
     </pre>
   </blockquote>
   <pre wrap=""><!---->
 Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
 restart_cpus()?  They're used e.g. as preparation to break into/out of
 the debugger, and AFAICT they're not in the critical path for
 anything.

 Not sure about smp_rendezvous_action(), maybe that one is OK.

Kris</pre>
 </blockquote>
 Good question which didn't occur to me because I was coming from
 another context. I was looking at using these functions to support a
 particular piece of "hotswap" hardware. The hardware generates an
 interrupt to say a PCI device has been removed (or plugged in). In the
 removal case some work needs to be done to remap to main memory the
 memory address space assigned to the device being removed so that if a
 device driver is accessing the device's memory space it won't cause the
 system to die. On a UP system this is "easy" in that because the
 hotswap interrupt is running we know the driver for the removed device
 is not currently running and even if it was running before the
 interrupt by the time it runs again the memory remapping will have
 occurred and it will be safe for the driver to access the memory space
 previously assigned to the removed device.However on a MP system the
 driver could be executing concurrently on another CPU and cause the
 system to die by accessing memory that is no longer present. 
 
 I thought of using these functions (stop_cpus() etc) to momentarily
 "pause" all other CPUs while adjusting the memory mapping. It would be
 of advantage to cause other CPUs to pause as quickly as possible to
 minimise the likelihood that the driver for the removed device will
 attempt to access memory on the removed device. I have not been able to
 find any data on how long I have between the interrupt to say a device
 is being removed and the actual removal of the device. Nor have I any
 data on how long a busy CPU of a hyperthreaded pair (for example) is
 able to "lock out" its hyperthreaded mate. 
 
 The referenced functions are clearly adequate for their existing use by
 the debugger so I accept that I might be better off making a customised
 copy. 
 
 I still have doubts about the effectiveness of the approach using
 stop_cpus() and co and I'm inclined to see if I can get away with
 saying "you can't pull out a board untill you have disabled all the
 devices on that board" but this is somewhat restricted compared with
 what users have become accustomed to. 
 
 Arthur 
 
 </body>
 </html>

 --------------090009090003010006010508--