svn commit: r205614 - stable/7/sys/dev/msk

Tue Apr 6 13:46:29 UTC 2010

On Mon, 05-Apr-2010 at 20:06:42 +0200, Pyun YongHyeon wrote:
> On Mon, Apr 05, 2010 at 04:59:37PM +0200, Andre Albsmeier wrote:
> > On Wed, 24-Mar-2010 at 18:21:05 +0100, Pyun YongHyeon wrote:
> > > Author: yongari
> > > Date: Wed Mar 24 17:21:05 2010
> > > New Revision: 205614
> > > URL: http://svn.freebsd.org/changeset/base/205614
> > > 
> > > Log:
> > >   MFC r204545:
> > >     Remove taskqueue based interrupt handling. After r204541 msk(4)
> > >     does not generate excessive interrupts any more so we don't need
> > >     to have two copies of interrupt handler.
> > >     While I'm here remove two STAT_PUT_IDX register accesses in LE
> > >     status event handler. After r204539 msk(4) always sync status LEs
> > >     so there is no need to resort to reading STAT_PUT_IDX register to
> > >     know the end of status LE processing. Just trust status LE's
> > >     ownership bit.
> > 
> > This ruined the performance on my system heavily. I noticed it
> > when unpacking a local tar archive onto an NFS-mounted location
> > on an em(4)-based box. This archive is about 50MB of size with
> > a bit over 5600 files so files have an average size of 9 kB.
> > 
> > I also noticed the slowdown when doing rdist-based updates (again
> > lots of small files) onto the other box.
> > 
> > Just pumping bytes over the network shows no problems -- I can
> > transmit 100-105 MB/s and receive 95-100 MB/s when talking
> > to this em(4)-based box without problem (and as it was before).
> > 
> > When copying a few big files (several GBs of size) over NFS
> > I get something between 70 and 90 MB/s which is the same as
> > what I had got before.
> > 
> > If have made some tests to track down when the issues began.
> > Problems started with rev. 1.18.2.37 of if_msk.c but could
> > be alleviated by setting dev.mskc.0.int_holdoff to 1 or 0.
> > Things really got problematic with rev. 1.18.2.38 -- adjusting
> > dev.mskc.0.int_holdoff helped a lot but we are far from what
> > we had with 1.18.2.36 or earlier. I did 5 rounds of testing,
> > each with the same set of if_msk.c revisions and values for
> > int_holdoff (where appropriate) just to check reproducibility:
> > 
> > if_msk.c rev.   round1  round2  round3  round4  round5
> > --------------------------------------------------------
> > 1.18.2.34       17,115  18,408  17,977  16,412  19,170
> > 1.18.2.35       18,414  17,863  17,000  18,428  18,093
> > 1.18.2.36       19,631  18,167  18,105  18,401  17,995
> > 1.18.2.37       22,707  24,830  24,322  23,613  22,498
> >  int_holdoff=10 19,259  19,870  19,355  18,725  19,273
> >  int_holdoff=1  18,464  18,218  17,862  16,701  17,798
> >  int_holdoff=0  19,423  18,507  19,505  20,714  20,460
> > 1.18.2.38       57,169  53,394  58,721     not done
> >  int_holdoff=10 30,266  33,493  33,240  33,247  30,470
> >  int_holdoff=1  27,013  28,777  28,047  25,858  27,615
> >  int_holdoff=0  40,284  33,040  33,726  36,834  35,235
> > 
> > All this is on 
> > 
> > FreeBSD-7.3-STABLE
> > 
> > CPU: Intel(R) Core(TM)2 Quad CPU    Q9650  @ 3.00GHz (3001.18-MHz 686-class CPU)
> >   Origin = "GenuineIntel"  Id = 0x1067a  Family = 6  Model = 17  Stepping = 10
> > 
> > dev.mskc.0.%desc: Marvell Yukon 88E8053 Gigabit Ethernet
> > dev.msk.0.%desc: Marvell Technology Group Ltd. Yukon EC Id 0xb6 Rev 0x02
> > 
> > hw.msk.msi_disable was set to 1 but didn't change results
> > when commenting it out.
> > 
> > Any ideas or things I can try?
> > 
> 
> Could you narrow down which side(RX or TX) cause the issue you're
> seeing? From your description it's not clear whether msk(4) is used
> as sender or receiver.

Well, both. I will try to describe the setup more exactly:

On the msk(4)-box a locally residing tar file (48 MB size
containing 5600 files) is unpacked onto an NFS volume.
This NFS volume is mounted from another box which got
an em(4)-based NIC. I have now measured the amounts of
data being sent end received simply by using netstat:

About 62 MB are being sent out of the msk(4)-box to the
em(4)-based NFS box and about 22 MB are received on the
msk(4)-box from the em(4)-based NFS box.

I have now tried the reverse direction as well: The em(4)-
based box mounts an NFS volume from the msk(4)-box and
unpacks the same tar file (now the 62 MB are received on
the msk(4)-box and 22 MB are transmitted from the msk(4)-
box). The results are similar:

rev. 1.18.2.38:	48,243 seconds
rev. 1.18.2.36: 17,536 seconds

But I noticed another thing here at work: If I choose a
remote machine which uses myk(4) (not msk(4)) instead of
em(4) there are no performance issues noticable. Unfortu-
natley I can't test msk(4) on the remote side at the
moment... So the performance issues exist only when the
new msk driver is talking to an em-based NIC...

> As you know 1.18.2.38 removed taskqueue based interrupt handling so
> it could be culprit of the issue. But that revision also removed
> two register accesses in TX path so I'd like to know which one
> caused the issue. 

I have now tried rev. 1.18.2.38 with this patch (no idea if
this is right ;-)):

--- if_msk.c.1.18.2.38	2010-04-06 15:09:19.000000000 +0200
+++ if_msk.c.1.18.2.38.TRY	2010-04-06 15:38:13.000000000 +0200
@@ -3327,6 +3327,11 @@
 	uint32_t control, status;
 	int cons, len, port, rxprog;
 
+	int idx;
+	idx = CSR_READ_2(sc, STAT_PUT_IDX);
+	if (idx == sc->msk_stat_cons)
+		return (0);
+
 	/* Sync status LEs. */
 	bus_dmamap_sync(sc->msk_stat_tag, sc->msk_stat_map,
 	    BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE);
@@ -3407,7 +3412,7 @@
 	if (rxput[MSK_PORT_B] > 0)
 		msk_rxput(sc->msk_if[MSK_PORT_B]);
 
-	return (rxprog > sc->msk_process_limit ? EAGAIN : 0);
+	return (sc->msk_stat_cons != CSR_READ_2(sc, STAT_PUT_IDX));
 }
 
 static void

Now performance seems to be the same as with the older
driver (at least here at work) and in both directions!
Some numbers:

em0 writes to rev. 1.18.2.36:				20 seconds
em0 writes to rev. 1.18.2.38:				50 seconds
em0 writes to rev. 1.18.2.38 with patch from above:	23 seconds
same as before but with int_holdoff: 100 -> 1:		20 seconds

rev. 1.18.2.36 writes to em0:				22 seconds
rev. 1.18.2.38 writes to em0:				40 seconds
rev. 1.18.2.38 with patch from above writes to em0:	21 seconds
same as before but with int_holdoff: 100 -> 1:		20 seconds

It seems that these two CSR_READ_2s really help ;-).

As I said, this is at work and with slightly different machines.
I will try things at home later but I am rather confident of
receiving good results there as well...

Thanks,

	-Andre
