svn commit: r263682 - in projects/rrs_mqueue: . sys/conf sys/dev/bxe sys/dev/cxgb sys/dev/cxgbe sys/dev/e1000 sys/dev/fdt sys/dev/ixgbe sys/dev/mxge sys/dev/oce sys/dev/virtio/network sys/dev/vxge ...
Randall Stewart
rrs at FreeBSD.org
Mon Mar 24 11:43:39 UTC 2014
Author: rrs
Date: Mon Mar 24 11:43:35 2014
New Revision: 263682
URL: http://svnweb.freebsd.org/changeset/base/263682
Log:
Ok first downward drop of my old mqueue patch that
I had laying on an old version of head (plus a little
read me to describe what I am doing in this project branch).
I am not sure, so far, if this even compiles yet. It used
to but that was a large leap ago.. I will do a large make
to bring this machine up to current and make sure everything
compiles (which will probably be my next commit here :-)
Sponsored by: Adara Networks
Added:
projects/rrs_mqueue/README_RRS
projects/rrs_mqueue/sys/net/drbr.c (contents, props changed)
projects/rrs_mqueue/sys/net/drbr.h (contents, props changed)
Modified:
projects/rrs_mqueue/sys/conf/files
projects/rrs_mqueue/sys/dev/bxe/bxe.c
projects/rrs_mqueue/sys/dev/bxe/bxe.h
projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h
projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c
projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c
projects/rrs_mqueue/sys/dev/cxgbe/adapter.h
projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c
projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c
projects/rrs_mqueue/sys/dev/e1000/if_em.c
projects/rrs_mqueue/sys/dev/e1000/if_em.h
projects/rrs_mqueue/sys/dev/e1000/if_igb.c
projects/rrs_mqueue/sys/dev/e1000/if_igb.h
projects/rrs_mqueue/sys/dev/fdt/fdt_common.c
projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c
projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h
projects/rrs_mqueue/sys/dev/ixgbe/ixv.c
projects/rrs_mqueue/sys/dev/ixgbe/ixv.h
projects/rrs_mqueue/sys/dev/mxge/if_mxge.c
projects/rrs_mqueue/sys/dev/mxge/if_mxge_var.h
projects/rrs_mqueue/sys/dev/oce/oce_hw.c
projects/rrs_mqueue/sys/dev/oce/oce_if.c
projects/rrs_mqueue/sys/dev/oce/oce_if.h
projects/rrs_mqueue/sys/dev/oce/oce_mbox.c
projects/rrs_mqueue/sys/dev/oce/oce_queue.c
projects/rrs_mqueue/sys/dev/virtio/network/if_vtnet.c
projects/rrs_mqueue/sys/dev/virtio/network/if_vtnetvar.h
projects/rrs_mqueue/sys/dev/vxge/vxge.c
projects/rrs_mqueue/sys/dev/vxge/vxge.h
projects/rrs_mqueue/sys/kern/kern_mbuf.c
projects/rrs_mqueue/sys/kern/subr_bufring.c
projects/rrs_mqueue/sys/kern/subr_bus.c
projects/rrs_mqueue/sys/net/if_var.h
projects/rrs_mqueue/sys/netinet/if_ether.c
projects/rrs_mqueue/sys/ofed/drivers/net/mlx4/en_tx.c
projects/rrs_mqueue/sys/ofed/drivers/net/mlx4/mlx4_en.h
projects/rrs_mqueue/sys/sys/buf_ring.h
Added: projects/rrs_mqueue/README_RRS
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ projects/rrs_mqueue/README_RRS Mon Mar 24 11:43:35 2014 (r263682)
@@ -0,0 +1,57 @@
+Welcome to my little side project. It has taken
+me a while to get around to finding the time to
+put this back in.. whew.
+
+What we have here currently is some basic changes that were
+done at Adara Networks to the way the lower level stack works
+with device drivers. Basically all mbufs have a priority
+field (cosqos), this field gets set to the highest number
+by default (usually queue 7 by default if we have
+8 queues defined 0-7). Now whenever a driver goes
+to send out data, it starts at queue 0, and works
+its way out. This means that *yes* you can get
+packet starvation of things placed in queue 7 while
+other things are going to queue 0.
+
+There are also some new methods added to if_net that
+deal with managment of the queues/rings. This is
+the start of where I plan on putting an AQM module
+in place. Right now its just drop-tail if the queue
+gets full, at some point I will try to add in both
+codel and possibly Cisco's Pi. Pi will need to be a
+klm since C has patents on it (the typical C patent
+of you can have a free license unless you sue me).
+Codel is open created by Van Jacobson and Kathy Nichols
+only it was done in Linux first, so I will have to
+read the latest RFC's and try to do a clean implementation
+of that so we have no GNU contamination (this may take
+a while depending on how much time I can pull away
+from Adara, or if I can justify doing it as an
+Adara project).
+
+Now once I have all this settled in to head and know it
+all compiles (not sure since I was merging old code from
+quite a while ago that this branch yet compiles :-D), I
+want to move on and make the IP packet processing stack
+lock-less. What I mean by that is I would like to see
+*no* locks in the packet forwarding path. Only when you
+change things (radix trees etc) will locks be applied.
+How I will do this is by adding a garbage collector entity
+that "defers" removal of things long enough for packets
+being forwarded (during a change) to be out of the system
+*before* removal of the old entity. This sounds a bit
+strange but it works quite well as long as one can
+bound the time period needed before the garbage
+collector gets active.
+
+This will come later assuming I get time.. of course
+my life as been full of ENOTIME errors, but hopefully
+is slowing down here a bit :-)
+
+You are welcome to play in here and contribute. You can
+email me at rrs at freebsd.org (or rrs at lakerest.net.. thats
+where the mail goes anyway :D)
+
+Best wishes
+Randall Stewart
+
Modified: projects/rrs_mqueue/sys/conf/files
==============================================================================
--- projects/rrs_mqueue/sys/conf/files Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/conf/files Mon Mar 24 11:43:35 2014 (r263682)
@@ -3081,6 +3081,7 @@ net/bridgestp.c optional bridge | if_b
net/flowtable.c optional flowtable inet | flowtable inet6
net/ieee8023ad_lacp.c optional lagg
net/if.c standard
+net/drbr.c standard
net/if_arcsubr.c optional arcnet
net/if_atmsubr.c optional atm
net/if_bridge.c optional bridge inet | if_bridge inet
Modified: projects/rrs_mqueue/sys/dev/bxe/bxe.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/bxe/bxe.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/bxe/bxe.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -5946,10 +5946,11 @@ bxe_tx_mq_start_locked(struct bxe_softc
struct bxe_fastpath *fp,
struct mbuf *m)
{
- struct buf_ring *tx_br = fp->tx_br;
+ struct drbr_ring *tx_br = fp->tx_br;
struct mbuf *next;
int depth, rc, tx_count;
uint16_t tx_bd_avail;
+ uint8_t qused;
rc = tx_count = 0;
@@ -5966,25 +5967,16 @@ bxe_tx_mq_start_locked(struct bxe_softc
BXE_FP_TX_LOCK_ASSERT(fp);
- if (m == NULL) {
- /* no new work, check for pending frames */
- next = drbr_dequeue(ifp, tx_br);
- } else if (drbr_needs_enqueue(ifp, tx_br)) {
- /* have both new and pending work, maintain packet order */
- rc = drbr_enqueue(ifp, tx_br, m);
- if (rc != 0) {
- fp->eth_q_stats.tx_soft_errors++;
- goto bxe_tx_mq_start_locked_exit;
- }
- next = drbr_dequeue(ifp, tx_br);
- } else {
- /* new work only and nothing pending */
- next = m;
+ if (m != NULL) {
+ rc = drbr_enqueue(ifp, tx_br, m);
+ if (rc != 0) {
+ fp->eth_q_stats.tx_soft_errors++;
+ goto bxe_tx_mq_start_locked_exit;
+ }
}
/* keep adding entries while there are frames to send */
- while (next != NULL) {
-
+ while ((next = drbr_peek(ifp, fp->tx_br, &qused)) != NULL) {
/* the mbuf now belongs to us */
fp->eth_q_stats.mbuf_alloc_tx++;
@@ -5996,19 +5988,22 @@ bxe_tx_mq_start_locked(struct bxe_softc
rc = bxe_tx_encap(fp, &next);
if (__predict_false(rc != 0)) {
fp->eth_q_stats.tx_encap_failures++;
- if (next != NULL) {
- /* mark the TX queue as full and save the frame */
- ifp->if_drv_flags |= IFF_DRV_OACTIVE;
- /* XXX this may reorder the frame */
- rc = drbr_enqueue(ifp, tx_br, next);
- fp->eth_q_stats.mbuf_alloc_tx--;
- fp->eth_q_stats.tx_frames_deferred++;
- }
-
+ if (next == NULL) {
+ drbr_advance(ifp, fp->tx_br, qused);
+ } else {
+ drbr_putback(ifp, fp->tx_br, next, qused);
+ /*
+ * Mark the TX queue as full and save
+ * the frame.
+ */
+ ifp->if_drv_flags |= IFF_DRV_OACTIVE;
+ fp->eth_q_stats.mbuf_alloc_tx--;
+ fp->eth_q_stats.tx_frames_deferred++;
+ }
/* stop looking for more work */
break;
}
-
+ drbr_advance(ifp, fp->tx_br, qused);
/* the transmit frame was enqueued successfully */
tx_count++;
@@ -6087,7 +6082,6 @@ bxe_mq_flush(struct ifnet *ifp)
{
struct bxe_softc *sc = ifp->if_softc;
struct bxe_fastpath *fp;
- struct mbuf *m;
int i;
for (i = 0; i < sc->num_queues; i++) {
@@ -6102,9 +6096,7 @@ bxe_mq_flush(struct ifnet *ifp)
if (fp->tx_br != NULL) {
BLOGD(sc, DBG_LOAD, "Clearing fp[%02d] buf_ring\n", fp->index);
BXE_FP_TX_LOCK(fp);
- while ((m = buf_ring_dequeue_sc(fp->tx_br)) != NULL) {
- m_freem(m);
- }
+ drbr_flush(ifp, fp->tx_br);
BXE_FP_TX_UNLOCK(fp);
}
}
@@ -6505,12 +6497,9 @@ bxe_free_fp_buffers(struct bxe_softc *sc
#if __FreeBSD_version >= 800000
if (fp->tx_br != NULL) {
- struct mbuf *m;
/* just in case bxe_mq_flush() wasn't called */
- while ((m = buf_ring_dequeue_sc(fp->tx_br)) != NULL) {
- m_freem(m);
- }
- buf_ring_free(fp->tx_br, M_DEVBUF);
+ drbr_flush(sc->ifnet, fp->tx_br);
+ drbr_free(fp->tx_br, M_DEVBUF);
fp->tx_br = NULL;
}
#endif
@@ -6771,8 +6760,7 @@ bxe_alloc_fp_buffers(struct bxe_softc *s
fp = &sc->fp[i];
#if __FreeBSD_version >= 800000
- fp->tx_br = buf_ring_alloc(BXE_BR_SIZE, M_DEVBUF,
- M_NOWAIT, &fp->tx_mtx);
+ fp->tx_br = drbr_alloc(M_DEVBUF, M_NOWAIT, &fp->tx_mtx);
if (fp->tx_br == NULL) {
BLOGE(sc, "buf_ring alloc fail for fp[%02d]\n", i);
goto bxe_alloc_fp_buffers_error;
Modified: projects/rrs_mqueue/sys/dev/bxe/bxe.h
==============================================================================
--- projects/rrs_mqueue/sys/dev/bxe/bxe.h Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/bxe/bxe.h Mon Mar 24 11:43:35 2014 (r263682)
@@ -69,6 +69,7 @@ __FBSDID("$FreeBSD$");
#include <net/if_vlan_var.h>
#include <net/zlib.h>
#include <net/bpf.h>
+#include <net/drbr.h>
#include <netinet/in.h>
#include <netinet/ip.h>
@@ -734,7 +735,7 @@ struct bxe_fastpath {
#if __FreeBSD_version >= 800000
#define BXE_BR_SIZE 4096
- struct buf_ring *tx_br;
+ struct drbr_ring *tx_br;
#endif
}; /* struct bxe_fastpath */
Modified: projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h
==============================================================================
--- projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/cxgb/cxgb_adapter.h Mon Mar 24 11:43:35 2014 (r263682)
@@ -252,7 +252,7 @@ struct sge_txq {
bus_dma_tag_t entry_tag;
struct mbuf_head sendq;
- struct buf_ring *txq_mr;
+ struct drbr_ring *txq_mr;
struct ifaltq *txq_ifq;
struct callout txq_timer;
struct callout txq_watchdog;
Modified: projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/cxgb/cxgb_main.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -66,6 +66,7 @@ __FBSDID("$FreeBSD$");
#include <net/if_media.h>
#include <net/if_types.h>
#include <net/if_vlan_var.h>
+#include <net/drbr.h>
#include <netinet/in_systm.h>
#include <netinet/in.h>
@@ -2361,7 +2362,7 @@ cxgb_tick_handler(void *arg, int count)
drops = 0;
for (j = pi->first_qset; j < pi->first_qset + pi->nqsets; j++)
- drops += sc->sge.qs[j].txq[TXQ_ETH].txq_mr->br_drops;
+ drops += drbr_get_dropcnt(sc->sge.qs[j].txq[TXQ_ETH].txq_mr);
ifp->if_snd.ifq_drops = drops;
ifp->if_oerrors =
Modified: projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/cxgb/cxgb_sge.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -61,6 +61,7 @@ __FBSDID("$FreeBSD$");
#include <net/bpf.h>
#include <net/ethernet.h>
#include <net/if_vlan_var.h>
+#include <net/drbr.h>
#include <netinet/in_systm.h>
#include <netinet/in.h>
@@ -1684,7 +1685,7 @@ cxgb_transmit_locked(struct ifnet *ifp,
{
struct port_info *pi = qs->port;
struct sge_txq *txq = &qs->txq[TXQ_ETH];
- struct buf_ring *br = txq->txq_mr;
+ struct drbr_ring *br = txq->txq_mr;
int error, avail;
avail = txq->size - txq->in_use;
@@ -1980,7 +1981,7 @@ t3_free_qset(adapter_t *sc, struct sge_q
reclaim_completed_tx(q, 0, TXQ_ETH);
if (q->txq[TXQ_ETH].txq_mr != NULL)
- buf_ring_free(q->txq[TXQ_ETH].txq_mr, M_DEVBUF);
+ drbr_free(q->txq[TXQ_ETH].txq_mr, M_DEVBUF);
if (q->txq[TXQ_ETH].txq_ifq != NULL) {
ifq_delete(q->txq[TXQ_ETH].txq_ifq);
free(q->txq[TXQ_ETH].txq_ifq, M_DEVBUF);
@@ -2430,8 +2431,8 @@ t3_sge_alloc_qset(adapter_t *sc, u_int i
q->port = pi;
q->adap = sc;
- if ((q->txq[TXQ_ETH].txq_mr = buf_ring_alloc(cxgb_txq_buf_ring_size,
- M_DEVBUF, M_WAITOK, &q->lock)) == NULL) {
+ if ((q->txq[TXQ_ETH].txq_mr = drbr_alloc(M_DEVBUF, M_WAITOK,
+ &q->lock)) == NULL) {
device_printf(sc->dev, "failed to allocate mbuf ring\n");
goto err;
}
@@ -3523,9 +3524,9 @@ t3_add_configured_sysctls(adapter_t *sc)
CTLTYPE_STRING | CTLFLAG_RD, &qs->rspq,
0, t3_dump_rspq, "A", "dump of the response queue");
- SYSCTL_ADD_UQUAD(ctx, txqpoidlist, OID_AUTO, "dropped",
+/* RRS FIXME SYSCTL_ADD_UQUAD(ctx, txqpoidlist, OID_AUTO, "dropped",
CTLFLAG_RD, &qs->txq[TXQ_ETH].txq_mr->br_drops,
- "#tunneled packets dropped");
+ "#tunneled packets dropped");*/
SYSCTL_ADD_UINT(ctx, txqpoidlist, OID_AUTO, "sendqlen",
CTLFLAG_RD, &qs->txq[TXQ_ETH].sendq.qlen,
0, "#tunneled packets waiting to be sent");
Modified: projects/rrs_mqueue/sys/dev/cxgbe/adapter.h
==============================================================================
--- projects/rrs_mqueue/sys/dev/cxgbe/adapter.h Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/cxgbe/adapter.h Mon Mar 24 11:43:35 2014 (r263682)
@@ -445,7 +445,7 @@ struct sge_txq {
struct ifnet *ifp; /* the interface this txq belongs to */
bus_dma_tag_t tx_tag; /* tag for transmit buffers */
- struct buf_ring *br; /* tx buffer ring */
+ struct drbr_ring *br; /* tx buffer ring */
struct tx_sdesc *sdesc; /* KVA of software descriptor ring */
struct mbuf *m; /* held up due to temporary resource shortage */
Modified: projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/cxgbe/t4_main.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -54,6 +54,7 @@ __FBSDID("$FreeBSD$");
#include <net/if.h>
#include <net/if_types.h>
#include <net/if_dl.h>
+#include <net/drbr.h>
#include <net/if_vlan_var.h>
#if defined(__i386__) || defined(__amd64__)
#include <vm/vm.h>
@@ -1284,7 +1285,7 @@ cxgbe_transmit(struct ifnet *ifp, struct
struct port_info *pi = ifp->if_softc;
struct adapter *sc = pi->adapter;
struct sge_txq *txq = &sc->sge.txq[pi->first_txq];
- struct buf_ring *br;
+ struct drbr_ring *br;
int rc;
M_ASSERTPKTHDR(m);
@@ -1326,7 +1327,7 @@ cxgbe_transmit(struct ifnet *ifp, struct
*/
TXQ_LOCK_ASSERT_OWNED(txq);
- if (drbr_needs_enqueue(ifp, br) || txq->m) {
+ if (txq->m) {
/* Queued for transmission. */
@@ -1352,7 +1353,6 @@ cxgbe_qflush(struct ifnet *ifp)
struct port_info *pi = ifp->if_softc;
struct sge_txq *txq;
int i;
- struct mbuf *m;
/* queues do not exist if !PORT_INIT_DONE. */
if (pi->flags & PORT_INIT_DONE) {
@@ -1360,8 +1360,7 @@ cxgbe_qflush(struct ifnet *ifp)
TXQ_LOCK(txq);
m_freem(txq->m);
txq->m = NULL;
- while ((m = buf_ring_dequeue_sc(txq->br)) != NULL)
- m_freem(m);
+ drbr_flush(ifp, txq->br);
TXQ_UNLOCK(txq);
}
}
@@ -4125,7 +4124,7 @@ cxgbe_tick(void *arg)
drops = s->tx_drop;
for_each_txq(pi, i, txq)
- drops += txq->br->br_drops;
+ drops += drbr_get_dropcnt(txq->br);
ifp->if_oqdrops = drops;
ifp->if_oerrors = s->tx_error_frames;
@@ -6616,7 +6615,7 @@ sysctl_wcwr_stats(SYSCTL_HANDLER_ARGS)
static inline void
txq_start(struct ifnet *ifp, struct sge_txq *txq)
{
- struct buf_ring *br;
+ struct drbr_ring *br;
struct mbuf *m;
TXQ_LOCK_ASSERT_OWNED(txq);
@@ -7857,7 +7856,6 @@ t4_ioctl(struct cdev *dev, unsigned long
txq->txpkt_wrs = 0;
txq->txpkts_wrs = 0;
txq->txpkts_pkts = 0;
- txq->br->br_drops = 0;
txq->no_dmamap = 0;
txq->no_desc = 0;
}
Modified: projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/cxgbe/t4_sge.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$");
#include <net/ethernet.h>
#include <net/if.h>
#include <net/if_vlan_var.h>
+#include <net/drbr.h>
#include <netinet/in.h>
#include <netinet/ip.h>
#include <netinet/ip6.h>
@@ -1799,9 +1800,10 @@ t4_eth_tx(struct ifnet *ifp, struct sge_
struct port_info *pi = (void *)ifp->if_softc;
struct adapter *sc = pi->adapter;
struct sge_eq *eq = &txq->eq;
- struct buf_ring *br = txq->br;
+ struct drbr_ring *br = txq->br;
struct mbuf *next;
int rc, coalescing, can_reclaim;
+ uint8_t qused;
struct txpkts txpkts;
struct sgl sgl;
@@ -1828,8 +1830,7 @@ t4_eth_tx(struct ifnet *ifp, struct sge_
if (__predict_false(eq->flags & EQ_DOOMED)) {
m_freem(m);
- while ((m = buf_ring_dequeue_sc(txq->br)) != NULL)
- m_freem(m);
+ drbr_flush(ifp, br);
return (ENETDOWN);
}
@@ -1844,7 +1845,7 @@ t4_eth_tx(struct ifnet *ifp, struct sge_
next = m->m_nextpkt;
m->m_nextpkt = NULL;
- if (next || buf_ring_peek(br))
+ if (next || drbr_peek(ifp, br, &qused))
coalescing = 1;
rc = get_pkt_sgl(txq, &m, &sgl, coalescing);
@@ -2868,7 +2869,7 @@ alloc_txq(struct port_info *pi, struct s
txq->sdesc = malloc(eq->cap * sizeof(struct tx_sdesc), M_CXGBE,
M_ZERO | M_WAITOK);
- txq->br = buf_ring_alloc(eq->qsize, M_CXGBE, M_WAITOK, &eq->eq_lock);
+ txq->br = drbr_alloc(M_CXGBE, M_WAITOK, &eq->eq_lock);
rc = bus_dma_tag_create(sc->dmat, 1, 0, BUS_SPACE_MAXADDR,
BUS_SPACE_MAXADDR, NULL, NULL, 64 * 1024, TX_SGL_SEGS,
@@ -2923,8 +2924,8 @@ alloc_txq(struct port_info *pi, struct s
SYSCTL_ADD_UQUAD(&pi->ctx, children, OID_AUTO, "txpkts_pkts", CTLFLAG_RD,
&txq->txpkts_pkts, "# of frames tx'd using txpkts work requests");
- SYSCTL_ADD_UQUAD(&pi->ctx, children, OID_AUTO, "br_drops", CTLFLAG_RD,
- &txq->br->br_drops, "# of drops in the buf_ring for this queue");
+/* SYSCTL_ADD_UQUAD(&pi->ctx, children, OID_AUTO, "br_drops", CTLFLAG_RD,
+ &txq->br->br_drops, "# of drops in the buf_ring for this queue");*/
SYSCTL_ADD_UINT(&pi->ctx, children, OID_AUTO, "no_dmamap", CTLFLAG_RD,
&txq->no_dmamap, 0, "# of times txq ran out of DMA maps");
SYSCTL_ADD_UINT(&pi->ctx, children, OID_AUTO, "no_desc", CTLFLAG_RD,
@@ -2953,7 +2954,7 @@ free_txq(struct port_info *pi, struct sg
if (txq->txmaps.maps)
t4_free_tx_maps(&txq->txmaps, txq->tx_tag);
- buf_ring_free(txq->br, M_CXGBE);
+ drbr_free(txq->br, M_CXGBE);
if (txq->tx_tag)
bus_dma_tag_destroy(txq->tx_tag);
Modified: projects/rrs_mqueue/sys/dev/e1000/if_em.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/e1000/if_em.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/e1000/if_em.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -67,6 +67,7 @@
#include <net/if_arp.h>
#include <net/if_dl.h>
#include <net/if_media.h>
+#include <net/drbr.h>
#include <net/if_types.h>
#include <net/if_vlan_var.h>
@@ -273,6 +274,9 @@ static int em_is_valid_ether_addr(u8 *)
static int em_sysctl_int_delay(SYSCTL_HANDLER_ARGS);
static void em_add_int_delay_sysctl(struct adapter *, const char *,
const char *, struct em_int_delay_info *, int, int);
+static void em_max_bytes(struct ifnet *, uint64_t max);
+static struct drbr_ring *em_get_ring(struct ifnet *ifp, int num);
+static int em_ring_query(struct ifnet *ifp, struct mbuf *);
/* Management and WOL Support */
static void em_init_manageability(struct adapter *);
static void em_release_manageability(struct adapter *);
@@ -897,6 +901,37 @@ em_resume(device_t dev)
return bus_generic_resume(dev);
}
+void
+em_max_bytes(struct ifnet *ifp, uint64_t max)
+{
+ struct adapter *adapter = ifp->if_softc;
+ adapter->ring_bytes_max = max;
+}
+
+struct drbr_ring *
+em_get_ring(struct ifnet *ifp, int num)
+{
+ struct adapter *adapter = ifp->if_softc;
+ struct tx_ring *txr;
+ if (num >= adapter->num_queues) {
+ return (NULL);
+ }
+ if (adapter->tx_rings) {
+ txr = &adapter->tx_rings[num];
+ return (txr->br);
+ } else {
+ return (NULL);
+ }
+}
+
+int
+em_ring_query(struct ifnet *ifp, struct mbuf *m)
+{
+ struct adapter *adapter = ifp->if_softc;
+ struct tx_ring *txr;
+ txr = &adapter->tx_rings[0];
+ return(drbr_is_on_ring(txr->br, m));
+}
#ifdef EM_MULTIQUEUE
/*********************************************************************
@@ -913,6 +948,7 @@ em_mq_start_locked(struct ifnet *ifp, st
struct adapter *adapter = txr->adapter;
struct mbuf *next;
int err = 0, enq = 0;
+ uint8_t qused;
if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) !=
IFF_DRV_RUNNING || adapter->link_active == 0) {
@@ -929,20 +965,26 @@ em_mq_start_locked(struct ifnet *ifp, st
}
/* Process the queue */
- while ((next = drbr_peek(ifp, txr->br)) != NULL) {
+ while ((next = drbr_peek(ifp, txr->br, &qused)) != NULL) {
if ((err = em_xmit(txr, &next)) != 0) {
if (next == NULL)
- drbr_advance(ifp, txr->br);
+ drbr_advance(ifp, txr->br, qused);
else
- drbr_putback(ifp, txr->br, next);
+ drbr_putback(ifp, txr->br, next, qused);
break;
}
- drbr_advance(ifp, txr->br);
+ drbr_advance(ifp, txr->br, qused);
+ atomic_add_long(&txr->bytes_on_ring,
+ (uint64_t)next->m_pkthdr.len);
enq++;
ifp->if_obytes += next->m_pkthdr.len;
if (next->m_flags & M_MCAST)
ifp->if_omcasts++;
ETHER_BPF_MTAP(ifp, next);
+ if (adapter->ring_bytes_max &&
+ (txr->bytes_on_ring >= adapter->ring_bytes_max)) {
+ break;
+ }
if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0)
break;
}
@@ -991,8 +1033,7 @@ em_qflush(struct ifnet *ifp)
for (int i = 0; i < adapter->num_queues; i++, txr++) {
EM_TX_LOCK(txr);
- while ((m = buf_ring_dequeue_sc(txr->br)) != NULL)
- m_freem(m);
+ drbr_flush(ifp, txr->br);
EM_TX_UNLOCK(txr);
}
if_qflush(ifp);
@@ -2984,6 +3025,9 @@ em_setup_interface(device_t dev, struct
ifp->if_softc = adapter;
ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;
ifp->if_ioctl = em_ioctl;
+ ifp->if_maxbytes = em_max_bytes;
+ ifp->if_getdrbr_ring = em_get_ring;
+ ifp->if_mbuf_on_ring = em_ring_query;
#ifdef EM_MULTIQUEUE
/* Multiqueue stack interface */
ifp->if_transmit = em_mq_start;
@@ -3222,7 +3266,7 @@ em_allocate_queues(struct adapter *adapt
}
#if __FreeBSD_version >= 800000
/* Allocate a buf ring */
- txr->br = buf_ring_alloc(4096, M_DEVBUF,
+ txr->br = drbr_alloc(M_DEVBUF,
M_WAITOK, &txr->tx_mtx);
#endif
}
@@ -3272,7 +3316,7 @@ err_tx_desc:
free(adapter->rx_rings, M_DEVBUF);
rx_fail:
#if __FreeBSD_version >= 800000
- buf_ring_free(txr->br, M_DEVBUF);
+ drbr_free(txr->br, M_DEVBUF);
#endif
free(adapter->tx_rings, M_DEVBUF);
fail:
@@ -3396,6 +3440,7 @@ em_setup_transmit_ring(struct tx_ring *t
/* Set number of descriptors available */
txr->tx_avail = adapter->num_tx_desc;
+ txr->bytes_on_ring = 0;
txr->queue_status = EM_QUEUE_IDLE;
/* Clear checksum offload context. */
@@ -3579,7 +3624,7 @@ em_free_transmit_buffers(struct tx_ring
}
#if __FreeBSD_version >= 800000
if (txr->br != NULL)
- buf_ring_free(txr->br, M_DEVBUF);
+ drbr_free(txr->br, M_DEVBUF);
#endif
if (txr->tx_buffers != NULL) {
free(txr->tx_buffers, M_DEVBUF);
@@ -3876,6 +3921,8 @@ em_txeof(struct tx_ring *txr)
++processed;
if (tx_buffer->m_head) {
+ atomic_subtract_long(&txr->bytes_on_ring,
+ (u_long)tx_buffer->m_head->m_pkthdr.len);
bus_dmamap_sync(txr->txtag,
tx_buffer->map,
BUS_DMASYNC_POSTWRITE);
@@ -5332,7 +5379,7 @@ em_add_hw_stats(struct adapter *adapter)
queue_node = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, namebuf,
CTLFLAG_RD, NULL, "Queue Name");
queue_list = SYSCTL_CHILDREN(queue_node);
-
+ drbr_add_sysctl_stats(dev, queue_list, txr->br);
SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "txd_head",
CTLTYPE_UINT | CTLFLAG_RD, adapter,
E1000_TDH(txr->me),
Modified: projects/rrs_mqueue/sys/dev/e1000/if_em.h
==============================================================================
--- projects/rrs_mqueue/sys/dev/e1000/if_em.h Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/e1000/if_em.h Mon Mar 24 11:43:35 2014 (r263682)
@@ -298,8 +298,9 @@ struct tx_ring {
u8 last_hw_tucso;
u8 last_hw_tucss;
#if __FreeBSD_version >= 800000
- struct buf_ring *br;
+ struct drbr_ring *br;
#endif
+ volatile u_long bytes_on_ring;
/* Interrupt resources */
bus_dma_tag_t txtag;
void *tag;
@@ -346,6 +347,7 @@ struct rx_ring {
/* Our adapter structure */
struct adapter {
struct ifnet *ifp;
+ uint64_t ring_bytes_max;
struct e1000_hw hw;
/* FreeBSD operating-system-specific structures. */
Modified: projects/rrs_mqueue/sys/dev/e1000/if_igb.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/e1000/if_igb.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/e1000/if_igb.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -72,6 +72,7 @@
#include <net/if_arp.h>
#include <net/if_dl.h>
#include <net/if_media.h>
+#include <net/drbr.h>
#include <net/if_types.h>
#include <net/if_vlan_var.h>
@@ -216,6 +217,9 @@ static void igb_reset(struct adapter *);
static int igb_setup_interface(device_t, struct adapter *);
static int igb_allocate_queues(struct adapter *);
static void igb_configure_queues(struct adapter *);
+static void igb_max_bytes(struct ifnet *, uint64_t max);
+static struct drbr_ring *igb_get_ring(struct ifnet *ifp, int num);
+static int igb_ring_query(struct ifnet *ifp, struct mbuf *m);
static int igb_allocate_transmit_buffers(struct tx_ring *);
static void igb_setup_transmit_structures(struct adapter *);
@@ -883,6 +887,42 @@ igb_resume(device_t dev)
return bus_generic_resume(dev);
}
+void
+igb_max_bytes(struct ifnet *ifp, uint64_t max)
+{
+ struct adapter *adapter = ifp->if_softc;
+ adapter->ring_bytes_max = max;
+
+}
+
+struct drbr_ring *
+igb_get_ring(struct ifnet *ifp, int num)
+{
+ struct adapter *adapter = ifp->if_softc;
+ struct tx_ring *txr;
+
+ if (num >= adapter->num_queues) {
+ return (NULL);
+ }
+ if (adapter->tx_rings) {
+ txr = &adapter->tx_rings[num];
+ return (txr->br);
+ } else {
+ return (NULL);
+ }
+}
+
+int
+igb_ring_query(struct ifnet *ifp, struct mbuf *m)
+{
+ struct adapter *adapter = ifp->if_softc;
+ struct tx_ring *txr;
+ /* For this hack, we only use 0, since adara stuff
+ * sends out on queue 0 always.
+ */
+ txr = &adapter->tx_rings[0];
+ return(drbr_is_on_ring(txr->br, m));
+}
#ifdef IGB_LEGACY_TX
@@ -1003,6 +1043,7 @@ igb_mq_start_locked(struct ifnet *ifp, s
struct adapter *adapter = txr->adapter;
struct mbuf *next;
int err = 0, enq = 0;
+ uint8_t qused;
IGB_TX_LOCK_ASSERT(txr);
@@ -1012,22 +1053,24 @@ igb_mq_start_locked(struct ifnet *ifp, s
/* Process the queue */
- while ((next = drbr_peek(ifp, txr->br)) != NULL) {
+ while ((next = drbr_peek(ifp, txr->br, &qused)) != NULL) {
if ((err = igb_xmit(txr, &next)) != 0) {
if (next == NULL) {
/* It was freed, move forward */
- drbr_advance(ifp, txr->br);
+ drbr_advance(ifp, txr->br, qused);
} else {
/*
* Still have one left, it may not be
* the same since the transmit function
* may have changed it.
*/
- drbr_putback(ifp, txr->br, next);
+ drbr_putback(ifp, txr->br, next, qused);
}
break;
}
- drbr_advance(ifp, txr->br);
+ drbr_advance(ifp, txr->br, qused);
+ atomic_add_long(&txr->bytes_on_ring,
+ (u_long)next->m_pkthdr.len);
enq++;
ifp->if_obytes += next->m_pkthdr.len;
if (next->m_flags & M_MCAST)
@@ -1035,6 +1078,11 @@ igb_mq_start_locked(struct ifnet *ifp, s
ETHER_BPF_MTAP(ifp, next);
if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0)
break;
+ if (adapter->ring_bytes_max &&
+ (txr->bytes_on_ring >= adapter->ring_bytes_max)) {
+ break;
+ }
+
}
if (enq > 0) {
/* Set the watchdog */
@@ -1072,12 +1120,10 @@ igb_qflush(struct ifnet *ifp)
{
struct adapter *adapter = ifp->if_softc;
struct tx_ring *txr = adapter->tx_rings;
- struct mbuf *m;
for (int i = 0; i < adapter->num_queues; i++, txr++) {
IGB_TX_LOCK(txr);
- while ((m = buf_ring_dequeue_sc(txr->br)) != NULL)
- m_freem(m);
+ drbr_flush(ifp, txr->br);
IGB_TX_UNLOCK(txr);
}
if_qflush(ifp);
@@ -3119,6 +3165,9 @@ igb_setup_interface(device_t dev, struct
#ifndef IGB_LEGACY_TX
ifp->if_transmit = igb_mq_start;
ifp->if_qflush = igb_qflush;
+ ifp->if_maxbytes = igb_max_bytes;
+ ifp->if_getdrbr_ring = igb_get_ring;
+ ifp->if_mbuf_on_ring = igb_ring_query;
#else
ifp->if_start = igb_start;
IFQ_SET_MAXLEN(&ifp->if_snd, adapter->num_tx_desc - 1);
@@ -3363,7 +3412,7 @@ igb_allocate_queues(struct adapter *adap
}
#ifndef IGB_LEGACY_TX
/* Allocate a buf ring */
- txr->br = buf_ring_alloc(igb_buf_ring_size, M_DEVBUF,
+ txr->br = drbr_alloc(M_DEVBUF,
M_WAITOK, &txr->tx_mtx);
#endif
}
@@ -3423,7 +3472,7 @@ err_tx_desc:
free(adapter->rx_rings, M_DEVBUF);
rx_fail:
#ifndef IGB_LEGACY_TX
- buf_ring_free(txr->br, M_DEVBUF);
+ drbr_free(txr->br, M_DEVBUF);
#endif
free(adapter->tx_rings, M_DEVBUF);
tx_fail:
@@ -3541,6 +3590,7 @@ igb_setup_transmit_ring(struct tx_ring *
/* Set number of descriptors available */
txr->tx_avail = adapter->num_tx_desc;
+ txr->bytes_on_ring = 0;
bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map,
BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE);
@@ -3682,7 +3732,7 @@ igb_free_transmit_buffers(struct tx_ring
}
#ifndef IGB_LEGACY_TX
if (txr->br != NULL)
- buf_ring_free(txr->br, M_DEVBUF);
+ drbr_free(txr->br, M_DEVBUF);
#endif
if (txr->tx_buffers != NULL) {
free(txr->tx_buffers, M_DEVBUF);
@@ -4016,6 +4066,8 @@ igb_txeof(struct tx_ring *txr)
if (buf->m_head) {
txr->bytes +=
buf->m_head->m_pkthdr.len;
+ atomic_subtract_long(&txr->bytes_on_ring,
+ (uint64_t)buf->m_head->m_pkthdr.len);
bus_dmamap_sync(txr->txtag,
buf->map,
BUS_DMASYNC_POSTWRITE);
@@ -5639,7 +5691,7 @@ igb_add_hw_stats(struct adapter *adapter
queue_node = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, namebuf,
CTLFLAG_RD, NULL, "Queue Name");
queue_list = SYSCTL_CHILDREN(queue_node);
-
+ drbr_add_sysctl_stats(dev, queue_list, txr->br);
SYSCTL_ADD_PROC(ctx, queue_list, OID_AUTO, "interrupt_rate",
CTLFLAG_RD, &adapter->queues[i],
sizeof(&adapter->queues[i]),
Modified: projects/rrs_mqueue/sys/dev/e1000/if_igb.h
==============================================================================
--- projects/rrs_mqueue/sys/dev/e1000/if_igb.h Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/e1000/if_igb.h Mon Mar 24 11:43:35 2014 (r263682)
@@ -309,12 +309,14 @@ struct tx_ring {
IGB_QUEUE_DEPLETED = 8,
} queue_status;
u32 txd_cmd;
- bus_dma_tag_t txtag;
char mtx_name[16];
#ifndef IGB_LEGACY_TX
- struct buf_ring *br;
+ struct drbr_ring *br;
struct task txq_task;
#endif
+ bus_dma_tag_t txtag;
+ volatile u_long bytes_on_ring;
+
u32 bytes; /* used for AIM */
u32 packets;
/* Soft Stats */
@@ -371,17 +373,17 @@ struct adapter {
struct device *dev;
struct cdev *led_dev;
- struct resource *pci_mem;
- struct resource *msix_mem;
- int memrid;
-
+ struct resource *pci_mem;
+ struct resource *msix_mem;
+ uint64_t ring_bytes_max;
+ int memrid;
/*
* Interrupt resources: this set is
* either used for legacy, or for Link
* when doing MSIX
*/
- void *tag;
- struct resource *res;
+ void *tag;
+ struct resource *res;
struct ifmedia media;
struct callout timer;
Modified: projects/rrs_mqueue/sys/dev/fdt/fdt_common.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/fdt/fdt_common.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/fdt/fdt_common.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -182,7 +182,6 @@ fdt_is_compatible(phandle_t node, const
compat += l;
len -= l;
}
-
return (rv);
}
@@ -544,15 +543,18 @@ fdt_get_phyaddr(phandle_t node, device_t
if (OF_getencprop(node, "phy-handle", (void *)&phy_handle,
sizeof(phy_handle)) <= 0)
return (ENXIO);
-
phy_node = OF_xref_phandle(phy_handle);
+ device_printf(dev, "phy-handle:0x%x phy_ihandle:0x%x phy_node:0x%x\n",
+ (uint32_t)phy_handle, (uint32_t)phy_ihandle,
+ (uint32_t)phy_node);
if (OF_getprop(phy_node, "reg", (void *)&phy_reg,
sizeof(phy_reg)) <= 0)
return (ENXIO);
+ device_printf(dev, "reg:0x%x\n", (uint32_t)phy_reg);
*phy_addr = fdt32_to_cpu(phy_reg);
-
+ device_printf(dev, "tran to reg:0x%x\n", (uint32_t)*phy_addr);
/*
* Search for softc used to communicate with phy.
*/
Modified: projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -845,7 +845,8 @@ ixgbe_mq_start_locked(struct ifnet *ifp,
struct adapter *adapter = txr->adapter;
struct mbuf *next;
int enqueued = 0, err = 0;
-
+ uint8_t qused;
+
if (((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) ||
adapter->link_active == 0)
return (ENETDOWN);
@@ -858,18 +859,18 @@ ixgbe_mq_start_locked(struct ifnet *ifp,
if (next != NULL)
err = drbr_enqueue(ifp, txr->br, next);
#else
- while ((next = drbr_peek(ifp, txr->br)) != NULL) {
+ while ((next = drbr_peek(ifp, txr->br, &qused)) != NULL) {
if ((err = ixgbe_xmit(txr, &next)) != 0) {
if (next == NULL) {
- drbr_advance(ifp, txr->br);
+ drbr_advance(ifp, txr->br, qused);
} else {
- drbr_putback(ifp, txr->br, next);
+ drbr_putback(ifp, txr->br, next, qused);
}
#endif
break;
}
#if __FreeBSD_version >= 901504
- drbr_advance(ifp, txr->br);
+ drbr_advance(ifp, txr->br, qused);
#endif
enqueued++;
/* Send a copy of the frame to the BPF listener */
@@ -917,12 +918,10 @@ ixgbe_qflush(struct ifnet *ifp)
{
struct adapter *adapter = ifp->if_softc;
struct tx_ring *txr = adapter->tx_rings;
- struct mbuf *m;
for (int i = 0; i < adapter->num_queues; i++, txr++) {
IXGBE_TX_LOCK(txr);
- while ((m = buf_ring_dequeue_sc(txr->br)) != NULL)
- m_freem(m);
+ drbr_flush(ifp, txr->br);
IXGBE_TX_UNLOCK(txr);
}
if_qflush(ifp);
@@ -2887,7 +2886,7 @@ ixgbe_allocate_queues(struct adapter *ad
}
#ifndef IXGBE_LEGACY_TX
/* Allocate a buf ring */
- txr->br = buf_ring_alloc(IXGBE_BR_SIZE, M_DEVBUF,
+ txr->br = drbr_alloc(M_DEVBUF,
M_WAITOK, &txr->tx_mtx);
if (txr->br == NULL) {
device_printf(dev,
@@ -3249,7 +3248,7 @@ ixgbe_free_transmit_buffers(struct tx_ri
}
#ifdef IXGBE_LEGACY_TX
if (txr->br != NULL)
- buf_ring_free(txr->br, M_DEVBUF);
+ drbr_free(txr->br, M_DEVBUF);
#endif
if (txr->tx_buffers != NULL) {
free(txr->tx_buffers, M_DEVBUF);
Modified: projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h
==============================================================================
--- projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/ixgbe/ixgbe.h Mon Mar 24 11:43:35 2014 (r263682)
@@ -58,6 +58,7 @@
#include <net/ethernet.h>
#include <net/if_dl.h>
#include <net/if_media.h>
+#include <net/drbr.h>
#include <net/bpf.h>
#include <net/if_types.h>
@@ -313,7 +314,7 @@ struct tx_ring {
bus_dma_tag_t txtag;
char mtx_name[16];
#ifndef IXGBE_LEGACY_TX
- struct buf_ring *br;
+ struct drbr_ring *br;
struct task txq_task;
#endif
#ifdef IXGBE_FDIR
Modified: projects/rrs_mqueue/sys/dev/ixgbe/ixv.c
==============================================================================
--- projects/rrs_mqueue/sys/dev/ixgbe/ixv.c Mon Mar 24 11:19:25 2014 (r263681)
+++ projects/rrs_mqueue/sys/dev/ixgbe/ixv.c Mon Mar 24 11:43:35 2014 (r263682)
@@ -603,6 +603,7 @@ ixv_mq_start_locked(struct ifnet *ifp, s
struct adapter *adapter = txr->adapter;
struct mbuf *next;
int enqueued, err = 0;
+ uint8_t qused;
if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) !=
IFF_DRV_RUNNING || adapter->link_active == 0) {
@@ -623,16 +624,16 @@ ixv_mq_start_locked(struct ifnet *ifp, s
}
*** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
More information about the svn-src-projects
mailing list