FreeBSD nfe driver and IPMI cards

[Pyun YongHyeon]

On Wed, Sep 12, 2007 at 05:21:13PM -0400, Robert Wojciechowski wrote:
 > >  > I'm the FreeBSD nfe driver from
 > >  >
 > http://www.f.csce.kyushu-u.ac.jp/~shigeaki/software/freebsd-nfe.html
 > >  > with FreeBSD 6-stable with good results for the most part. The only
 > >  > issue I've experienced is that during a detach/shutdown of if_nfe,
 > > the
 > >  > IPMI IP address I have set on my servers ceases to respond as well
 > > as
 > >  > the ability to manage the servers.
 > >  >
 > >  >
 > >  >
 > >  > I traced the problem down to nfe_stop() and the fact that it
 > > completely
 > >  > disables the Rx and Tx on the NIC. I have patched the driver to not
 > >  > disable the Rx/Tx and IPMI continues to work after a 'ifconfig nfe0
 > >  > down', 'shutdown -p now', etc.
 > >  >
 > >  >
 > >  >
 > >  > Does anyone have any comments on this change I've made and any
 > > possible
 > >  > side effects? Can this be included in the mainstream distribution
 > of
 > > the
 > > 
 > > Because MAC is still alive if's possible to recieve a packet. All DMA
 > > maps are unloaded and buffers are already freed in nfe_stop so it
 > > would cause panic I guess. But I'm not familiar with IPMI so I'm not
 > > sure.
 > 
 > Interesting, that is an issue that was also resolved by the forcedeth
 > driver in Linux by resetting in nv_close to prevent DMA into freed
 > memory.
 > 
 > > 
 > >  > nfe drivers (and updated in 7-CURRENT) without causing any adverse
 > >  > problems?
 > >  >
 > > 
 > > I have no experience on IPMI but the change you've made would not
 > > completely solve the issue. I guess supporting IPMI needs lots of
 > > more work including:
 > >  o Autodetect IPMI capability.
 > >  o Autodetect active IPMI session in device attach and don't blindly
 > >    reset MAC/PHY.
 > >  o Don't blindly stop Tx/Rx on device detach.
 > > Given that lack of publicly available datasheet for the hardware
 > > supporing IPMI would be severly limited. Fortunately Linux seems to
 > > have basic IPMI support in their forcedeth driver. Their code doesn't
 > > easy to read but you may see what should be done in driver. However
 > > I have no idea what we can do when active IPMI session is present in
 > > driver attach phase. Normally PHY driver would reset PHY hardware
 > > itself in driver attach which in turn would result in losing the IPMI
 > > connection.
 > > 
 > 
 > Since we have no idea how to auto-detect IPMI, I added a device sysctl
 > (enable_ipmi) to control the behavior I'm testing in-house right now.
 > 
 > If the enable_ipmi sysctl is enabled, it'll:
 > 
 >   * resets after disabling the Rx/Tx and before disabling interrupts

It also need to enable Tx/Rx bits related with IPMI.

 >   * re-enables the Rx/Tx after the ring buffers are freed, just like
 > forcedeth but also enabling the Tx for IPMI (Linux just enabled the Rx
 > for WOL)

You may only want to enable IPMI Tx/Rx. Blindly enabling entire MAC
generates more problems. forcedeth uses different bits for IPMI Rx/Tx.

 > 
 > I have attached my new patch to this email that does the above, guarded
 > by the sysctl.
 > 

Btw, it seems there is typo in the patch. You used sc->nfe_process_limit
instead of sc->nfe_enable_ipmi in enable_ipmi value range check. In
order to check IPMI capability you don't need '&' operator.

 > I have no idea how to handle the second case you mentioned, during
 > attach. It does indeed cause a disruption in IPMI, but only for a few
 > seconds. Where is the MAC and PHY reset and if it wasn't reset when
 > using IPMI, what problems could it cause?
 > 

It's somewhat complex. MAC reset is done by nfe(4) whereas all PHY
related handling is done by its PHY driver(ciphy(4), e1000phy(4),
rgephy(4), rlphy(4) etc) and link state change event can also
enable/disable MAC. See nfe_link_task().
Witout proper reset operation of PHY, some functions like automatic
MDI/MDI-X detection, wakeup from powerdown mode, special page register
settings wouldn't be activated which in turn result in operating at
legacy mode. In worst case the PHY wouldn't work at all and you may
see no link, i.e. "no career" message.

-- 
Regards,
Pyun YongHyeon