socsvn commit: r289212 - soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve
stefano at FreeBSD.org
stefano at FreeBSD.org
Tue Aug 4 15:22:55 UTC 2015
Author: stefano
Date: Tue Aug 4 15:22:52 2015
New Revision: 289212
URL: http://svnweb.FreeBSD.org/socsvn/?view=rev&rev=289212
Log:
[bhyve/ptnetmap] add guest/host(kthread) notification and cleanup
1) notification from ptnetmap kthread to guest VM (interrupt/irq)
The bhyve user-space, during netmap configuration, passes
the ioctl parameters (fd, com, data) to ptnetmap kthreads.
The ptnetmap kthread (attached to bhyve process) uses
kern_ioctl() with these parameters to notify the guest VM.
2) notification from guest VM to ptnetmap kthread
(write into the specific register)
Use new IOCTL on vmm.ko (VM_IO_REG_HANDLER) to catch write/read
on specific I/O address and send notification.
With this new IOCTL, when the guest writes on specific register,
the vmm.ko invokes wakeup() on the parameter (chan) specified
from bhyve user-space application.
The same parameter is passed to the ptnetmap kthreads that
call tsleep(chan, ...) to wait the event.
Added:
soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/README.network
Modified:
soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c
soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c
soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h
soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h
Added: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/README.network
==============================================================================
--- /dev/null 00:00:00 1970 (empty, because file is newly added)
+++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/README.network Tue Aug 4 15:22:52 2015 (r289212)
@@ -0,0 +1,122 @@
+Network frontends and backends in bhyve
+
+20140804
+
+The following is a proposal on how to implement network frontends (FE)
+and backends (BE) in bhyve, and how to interface them.
+
+We need to address the following requirements:
+1. Depending on the type of FE and BE, one may need zero or more
+ additional threads (iothreads) in charge of sending or receiving
+ packets, implementing one side of paravirtualized devices, etc.
+
+ The requirements for an intermediate thread are the following:
+
+ --- transmit ---
+ Frontend Backend Operations
+ non-PV non block* (1) all in VCPU
+ other cases (2) notify_tx() in VCPU, I/O in thread
+
+ case 1: VCPU executes (autonomously or on kick_tx)
+ fe->fe2be(); // fill the ring
+ be->transmit();
+ fe->fe2be(); // reclaim buffers
+
+ case 2: VCPU executes (autonomously)
+ be->notify_tx();
+
+ iothread executes
+ for (;;) {
+ <loop or wait notify_tx>
+ fe->fe2be(); // fill the ring
+ be->transmit(); // e.g. txsync, write
+ fe->fe2be(); // reclaim buffers
+ }
+
+ non blocking means that either the BE tx function is non
+ blocking, or that we have an independent notification mechanism
+ (e.g. an event loop, kevent, epoll...) to notify the FE when
+ transmit will be possible and issue a kick_tx() to the VCPU
+
+ --- receive ---
+ Backend Operations
+ non block* (1) all in VCPU
+ other cases (2) all in thread
+
+ case 1: VCPU executes (autonomously or on kick_rx)
+ if (needed) fe->be2fe(); // release buffers from FE
+ be->receive(); // read or rxsync
+ if (needed) fe->be2fe(); // pass packets to FE
+ main thread does fe->kick_rx() when data available
+
+ case 2: i/o thread executes (autonomously)
+ for (;;) {
+ <loop or wait notify_rx>
+ fe->be2fe(); // reclaim buffers from FE
+ be->receive(); // read or rxsync
+ fe->be2fe(); // pass packets to FE and kick_rx
+ }
+ /* note, in netmap be->receive() may be empty because poll()
+ * will already fill the ring with packets.
+ */
+
+ same as above, non blocking means that the BE will use an
+ existing mechanism to notify the FE when receive will be possible
+ and issue a kick_rx() to the VCPU
+
+2. for performance reasons, it is important that the interface between
+ FE and BE support batched transfers and asynchronous completion
+ notifications.
+
+ Given that the netmap API has the features required by #2, and that
+ netmap backends (and possibly even frontends) are going to be used
+ with bhyve, we suggest to standardize netmap as the FE-BE API
+
+In practice, the FE will need to call some BE-supplied function
+during its operation, and the same goes for the BE.
+To implement this, both entities export some information
+through descriptors visible to the other entity.
+
+Frontend descriptor
+
+ struct fe_desc {
+ ...
+ int (*be2fe)(struct fe_desc *, struct netmap_ring *);
+ int (*fe2be)(struct fe_desc *, struct netmap_ring *);
+ ...
+ }
+
+ fe->be2fe()
+ is invoked by the BE (when receiving from the network)
+ to move packets from the netmap ring into the FE and to release
+ completed buffers from the FE back into the netmap ring.
+ The amount of data moved can be determined by the BE by comparing
+ the ring pointers before and after the call.
+
+ Before returning, be2fe() will likely issue a kick to the VCPU
+ (e.g. an interrupt) to notify the availability of new data.
+
+ The return value may be used to indicate events that cannot
+ be represented by the ring pointers.
+
+ The function runs in the context of a thread "owned" by the BE,
+ and should implement protection against concurrent
+ activities on the FE's data structures, whereas the netmap_ring
+ is only used by this function during its execution.
+
+ fe->fe2be()
+ is it needed ?
+
+Backend descriptors
+
+ struct be_desc {
+ ...
+ int (*notify_tx)(struct re_desc *, struct netmap_ring *);
+ int (*notify_rx)(struct re_desc *, struct netmap_ring *);
+ ...
+ }
+
+ be->notify_tx()
+ is invoked by the ...
+
+
Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c
==============================================================================
--- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c Tue Aug 4 15:11:36 2015 (r289211)
+++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/net_backends.c Tue Aug 4 15:22:52 2015 (r289212)
@@ -46,7 +46,7 @@
#include <assert.h>
#include "mevent.h"
-#include "dev/virtio/network/virtio_net.h"
+#include <dev/virtio/network/virtio_net.h>
#include "net_backends.h"
#include <sys/linker_set.h>
@@ -443,30 +443,38 @@
return 0;
}
+/* used by netmap and ptnetmap */
static int
-netmap_init(struct net_backend *be, const char *devname,
- net_backend_cb_t cb, void *param)
+netmap_commom_init(struct net_backend *be, struct netmap_priv *priv, uint32_t nr_flags,
+ const char *devname, net_backend_cb_t cb, void *param)
{
const char *ndname = "/dev/netmap";
- struct netmap_priv *priv = NULL;
+ struct nmreq req;
char tname[40];
- priv = calloc(1, sizeof(struct netmap_priv));
- if (priv == NULL) {
- WPRINTF(("Unable alloc netmap private data\n"));
- return -1;
- }
-
strncpy(priv->ifname, devname, sizeof(priv->ifname));
priv->ifname[sizeof(priv->ifname) - 1] = '\0';
- priv->nmd = nm_open(priv->ifname, NULL, NETMAP_NO_TX_POLL, NULL);
+ memset(&req, 0, sizeof(req));
+ req.nr_flags |= nr_flags;
+
+ priv->nmd = nm_open(priv->ifname, &req, NETMAP_NO_TX_POLL, NULL);
if (priv->nmd == NULL) {
WPRINTF(("Unable to nm_open(): device '%s', "
"interface '%s', errno (%s)\n",
ndname, devname, strerror(errno)));
goto err_open;
}
+#if 0
+ /* check parent (nm_desc with the same allocator already mapped) */
+ parent_nmd = netmap_find_parent(nmd);
+ /* mmap or inherit from parent */
+ if (nm_mmap(nmd, parent_nmd)) {
+ error_report("failed to mmap %s: %s", netmap_opts->ifname, strerror(errno));
+ nm_close(nmd);
+ return -1;
+ }
+#endif
priv->tx = NETMAP_TXRING(priv->nmd->nifp, 0);
priv->rx = NETMAP_RXRING(priv->nmd->nifp, 0);
@@ -476,7 +484,6 @@
priv->rx_continue = 0;
be->fd = priv->nmd->fd;
- be->priv = priv;
/* Create a thread for netmap poll. */
pthread_create(&priv->evloop_tid, NULL, netmap_evloop_thread, (void *)be);
@@ -486,6 +493,32 @@
return 0;
err_open:
+ return -1;
+}
+
+static int
+netmap_init(struct net_backend *be, const char *devname,
+ net_backend_cb_t cb, void *param)
+{
+ const char *ndname = "/dev/netmap";
+ struct netmap_priv *priv = NULL;
+ char tname[40];
+
+ priv = calloc(1, sizeof(struct netmap_priv));
+ if (priv == NULL) {
+ WPRINTF(("Unable alloc netmap private data\n"));
+ return -1;
+ }
+
+ if (netmap_commom_init(be, priv, 0, devname, cb, param)) {
+ goto err;
+ }
+
+ be->priv = priv;
+
+ return 0;
+
+err:
free(priv);
return -1;
@@ -708,9 +741,9 @@
/*
* The ptnetmap backend
*/
-#include <stddef.h>
-#include "net/netmap.h"
-#include "dev/netmap/netmap_virt.h"
+#include <stddef.h> /* IFNAMSIZ */
+#include <net/netmap.h>
+#include <dev/netmap/netmap_virt.h>
#include "ptnetmap.h"
struct ptnbe_priv {
@@ -721,10 +754,8 @@
unsigned long acked_features; /* ptnetmap acked features */
};
-
-
/* The virtio-net features supported by ptnetmap. */
-#define VIRTIO_NET_F_PTNETMAP 0x2000000 /* ptnetmap available */
+#define VIRTIO_NET_F_PTNETMAP 0x2000000 /* (25) ptnetmap available */
#define PTNETMAP_FEATURES VIRTIO_NET_F_PTNETMAP
@@ -737,10 +768,6 @@
static uint64_t
ptnbe_set_features(struct net_backend *be, uint64_t features)
{
- if (features & PTNETMAP_FEATURES) {
- WPRINTF(("ptnbe_set_features\n"));
- }
-
return 0;
}
@@ -763,48 +790,20 @@
WPRINTF(("Unable alloc netmap private data\n"));
return -1;
}
- WPRINTF(("ptnbe_init(): devname '%s'\n", devname));
npriv = &priv->up;
- strncpy(npriv->ifname, devname + PTNETMAP_NAME_HDR, sizeof(npriv->ifname));
- npriv->ifname[sizeof(npriv->ifname) - 1] = '\0';
- memset(&req, 0, sizeof(req));
- req.nr_flags |= NR_PASSTHROUGH_HOST;
-
- npriv->nmd = nm_open(npriv->ifname, &req, NETMAP_NO_TX_POLL, NULL);
- if (npriv->nmd == NULL) {
- WPRINTF(("Unable to nm_open(): device '%s', "
- "interface '%s', errno (%s)\n",
- ndname, devname, strerror(errno)));
- goto err_open;
- }
-#if 0
- /* check parent (nm_desc with the same allocator already mapped) */
- parent_nmd = netmap_find_parent(nmd);
- /* mmap or inherit from parent */
- if (nm_mmap(nmd, parent_nmd)) {
- error_report("failed to mmap %s: %s", netmap_opts->ifname, strerror(errno));
- nm_close(nmd);
- return -1;
+ if (netmap_commom_init(be, npriv, NR_PTNETMAP_HOST, devname + PTNETMAP_NAME_HDR, cb, param)) {
+ goto err;
}
-#endif
-
-
- npriv->tx = NETMAP_TXRING(npriv->nmd->nifp, 0);
- npriv->rx = NETMAP_RXRING(npriv->nmd->nifp, 0);
- npriv->cb = cb;
- npriv->cb_param = param;
-
- be->fd = npriv->nmd->fd;
be->priv = priv;
ptnbe_init_ptnetmap(be);
return 0;
-err_open:
+err:
free(priv);
return -1;
@@ -816,6 +815,7 @@
struct ptnbe_priv *priv = be->priv;
if (priv) {
+ ptnetmap_delete(&priv->ptns);
nm_close(priv->up.nmd);
}
be->fd = -1;
@@ -903,8 +903,7 @@
if (priv->created)
return 0;
- /* TODO: ioctl to start kthreads */
-#if 1
+ /* ioctl to start ptnetmap kthreads */
memset(&req, 0, sizeof(req));
strncpy(req.nr_name, priv->up.ifname, sizeof(req.nr_name));
req.nr_version = NETMAP_API;
@@ -916,8 +915,6 @@
priv->up.ifname, strerror(errno));
} else
priv->created = 1;
-#endif
-
return err;
}
@@ -929,15 +926,15 @@
struct nmreq req;
int err;
+ if (!priv->created)
+ return 0;
+
if (!(priv->acked_features & NET_PTN_FEATURES_BASE)) {
printf("ptnetmap features not acked\n");
return EFAULT;
}
- if (!priv->created)
- return 0;
-
- /* TODO: ioctl to stop kthreads */
+ /* ioctl to stop ptnetmap kthreads */
memset(&req, 0, sizeof(req));
strncpy(req.nr_name, priv->up.ifname, sizeof(req.nr_name));
req.nr_version = NETMAP_API;
@@ -956,6 +953,8 @@
.name = "ptnetmap|ptvale",
.init = ptnbe_init,
.cleanup = ptnbe_cleanup,
+ .send = netmap_send,
+ .recv = netmap_receive,
.get_features = ptnbe_get_features,
.set_features = ptnbe_set_features,
.get_ptnetmap = ptnbe_get_ptnetmap,
Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c
==============================================================================
--- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c Tue Aug 4 15:11:36 2015 (r289211)
+++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_ptnetmap_memdev.c Tue Aug 4 15:22:52 2015 (r289212)
@@ -1,3 +1,29 @@
+/*
+ * Copyright (C) 2015 Stefano Garzarella (stefano.garzarella at gmail.com)
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
@@ -17,22 +43,9 @@
#include "pci_emul.h"
#include "ptnetmap.h"
-
-/* ptnetmap memdev PCI-ID and PCI-BARS XXX-ste: remove*/
-#define PTN_MEMDEV_NAME "ptnetmap-memdev"
-#define PTNETMAP_PCI_VENDOR_ID 0x3333 /* XXX-ste: set vendor_id */
-#define PTNETMAP_PCI_DEVICE_ID 0x0001
-#define PTNETMAP_IO_PCI_BAR 0
-#define PTNETMAP_MEM_PCI_BAR 1
-
-/* ptnetmap memdev register */
-/* 32 bit r/o */
-#define PTNETMAP_IO_PCI_FEATURES 0
-/* 32 bit r/o */
-#define PTNETMAP_IO_PCI_MEMSIZE 4
-/* 16 bit r/o */
-#define PTNETMAP_IO_PCI_HOSTID 8
-#define PTNEMTAP_IO_SIZE 10
+#include <net/if.h> /* IFNAMSIZ */
+#include <net/netmap.h>
+#include <dev/netmap/netmap_virt.h>
struct ptn_memdev_softc {
struct pci_devinst *pi; /* PCI device instance */
@@ -129,8 +142,8 @@
return 0;
if (baridx == PTNETMAP_MEM_PCI_BAR) {
- printf("ptnetmap_memdev: MEM read\n");
- printf("ptnentmap_memdev: mem_read - offset: %lx size: %d ret: %lx\n", offset, size, ret);
+ printf("ptnetmap_memdev: unexpected MEM read - offset: %lx size: %d ret: %lx\n",
+ offset, size, ret);
return 0; /* XXX */
}
@@ -149,7 +162,6 @@
break;
}
- printf("ptnentmap_memdev: io_read - offset: %lx size: %d ret: %llu\n", offset, size,(unsigned long long)ret);
return ret;
}
@@ -164,22 +176,16 @@
return;
if (baridx == PTNETMAP_MEM_PCI_BAR) {
- printf("ptnetmap_memdev: MEM write\n");
- return; /* XXX */
+ printf("ptnetmap_memdev: unexpected MEM write - offset: %lx size: %d value: %lx\n",
+ offset, size, value);
+ return;
}
- /* XXX probably should do something better than just assert() */
- assert(baridx == PTNETMAP_IO_PCI_BAR);
-
switch (offset) {
-
default:
printf("ptnentmap_memdev: write io reg unexpected\n");
break;
}
-
-
- printf("ptnentmap_memdev: io_write - offset: %lx size: %d val: %lx\n", offset, size, value);
}
static int
@@ -187,12 +193,9 @@
{
int ret;
- printf("ptnetmap_memdev: configuring\n");
-
if (sc->pi == NULL || sc->mem_ptr == NULL)
return 0;
-
/* init iobar */
ret = pci_emul_alloc_bar(sc->pi, PTNETMAP_IO_PCI_BAR, PCIBAR_IO, PTNEMTAP_IO_SIZE);
if (ret) {
@@ -209,10 +212,6 @@
return ret;
}
- printf("ptnetmap_memdev: pci_addr: %llx, mem_size: %llu, mem_ptr: %p\n",
- (unsigned long long) sc->pi->pi_bar[PTNETMAP_MEM_PCI_BAR].addr,
- (unsigned long long) sc->mem_size, sc->mem_ptr);
-
ret = vm_map_user_buf(sc->pi->pi_vmctx, sc->pi->pi_bar[PTNETMAP_MEM_PCI_BAR].addr,
sc->mem_size, sc->mem_ptr);
if (ret) {
@@ -220,8 +219,6 @@
return ret;
}
- printf("ptnetmap_memdev: configured\n");
-
return 0;
}
@@ -232,8 +229,6 @@
uint64_t size;
int ret;
- printf("ptnetmap_memdev: loading\n");
-
sc = ptn_memdev_find_empty_pi();
if (sc == NULL) {
sc = ptn_memdev_create();
@@ -260,8 +255,6 @@
goto err;
}
- printf("ptnetmap_memdev: loaded\n");
-
return (0);
err:
ptn_memdev_delete(sc);
@@ -274,7 +267,6 @@
{
struct ptn_memdev_softc *sc;
int ret;
- printf("ptnetmap_memdev: attaching\n");
/* if a device with the same mem_id is already attached, we are done */
if (ptn_memdev_find_memid(mem_id)) {
@@ -295,17 +287,13 @@
sc->mem_size = mem_size;
sc->mem_id = mem_id;
- printf("ptnetmap_memdev_attach: mem_id: %u, mem_size: %lu, mem_ptr: %p\n", mem_id,
- (unsigned long) mem_size, mem_ptr);
-
- /* TODO: configure device BARs */
+ /* configure device BARs */
ret = ptn_memdev_configure(sc);
if (ret) {
printf("ptnetmap_memdev: configure error\n");
goto err;
}
- printf("ptnetmap_memdev: attached\n");
return 0;
err:
Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h
==============================================================================
--- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h Tue Aug 4 15:11:36 2015 (r289211)
+++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/pci_virtio_ptnetmap.h Tue Aug 4 15:22:52 2015 (r289212)
@@ -1,12 +1,41 @@
+/*
+ * Copyright (C) 2015 Stefano Garzarella (stefano.garzarella at gmail.com)
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
#ifndef __PCI_VIRTIO_PTNETMAP_H__
#define __PCI_VIRTIO_PTNETMAP_H__
-/* XXX-ste: move in other file and split in .c .h? */
#ifdef BHYVE_VIRTIO_PTNETMAP
+#include <machine/vmm.h>
+#include <machine/vmm_dev.h> /* VM_LAPIC_MSI */
+#include <vmmapi.h>
+
+#include "ptnetmap.h"
/* ptnetmap virtio register BASE */
#define PTNETMAP_VIRTIO_IO_BASE sizeof(struct virtio_net_config)
-#include "ptnetmap.h"
static void
ptnetmap_configure_csb(struct vmctx *ctx, struct paravirt_csb** csb, uint32_t csbbal,
@@ -23,7 +52,6 @@
* The CSB is then remapped if the new pointer is != 0
*/
if (*csb) {
- /* TODO: unmap */
*csb = NULL;
}
if (base) {
@@ -56,8 +84,6 @@
/* extend cfgsize. virtio creates PCIBAR for us */
vc->vc_cfgsize += PTNEMTAP_VIRTIO_IO_SIZE;
-
- printf("ptnetmap-virtio init END\n");
}
static int
@@ -75,17 +101,12 @@
printf("ERROR ptnetmap: csb not initialized\n");
return ret;
}
+ /* share netmap_if info to the guest through CSB */
csb->nifp_offset = ptns->offset;
csb->num_tx_rings = ptns->num_tx_rings;
csb->num_rx_rings = ptns->num_rx_rings;
csb->num_tx_slots = ptns->num_tx_slots;
csb->num_rx_slots = ptns->num_rx_slots;
- printf("txr %u rxr %u txd %u rxd %u nifp_offset %u\n",
- csb->num_tx_rings,
- csb->num_rx_rings,
- csb->num_tx_slots,
- csb->num_rx_slots,
- csb->nifp_offset);
return ret;
}
@@ -95,6 +116,12 @@
{
struct ptnetmap_state *ptns = sc->ptn.state;
struct paravirt_csb *csb = sc->ptn.csb;
+ struct pci_devinst *pi;
+ struct vmctx *vmctx;
+ struct vqueue_info *vq;
+ struct msix_table_entry *mte;
+ struct iovec iov[1];
+ uint16_t idx;
int ret;
if (sc->ptn.up) {
@@ -107,22 +134,59 @@
return -1;
}
- /* TODO-ste: add support for multiqueue */
+ /* TODO: add support for multiqueue */
+ pi = sc->vsc_vs.vs_pi;
+ vmctx = pi->pi_vmctx;
- /* TODO: Stop processing guest/host IO notifications in qemu.
+ /* Configure the RX ring */
+ sc->ptn.cfg.rx_ring.irqfd = vm_get_fd(vmctx);
+ sc->ptn.cfg.rx_ioctl.com = VM_LAPIC_MSI;
+ vq = &sc->vsc_queues[VTNET_RXQ];
+ mte = &pi->pi_msix.table[vq->vq_msix_idx];
+ sc->ptn.cfg.rx_ioctl.data.msg = mte->msg_data;
+ sc->ptn.cfg.rx_ioctl.data.addr = mte->addr;
+ /* push fake-elem in the rx queue to enable interrupts */
+ if (vq_getchain(vq, &idx, iov, 1, NULL) > 0) {
+ vq_relchain(vq, idx, 0);
+ }
+ /* enable rx notification from guest */
+ vq->vq_used->vu_flags &= ~VRING_USED_F_NO_NOTIFY;
+ /*
+ * Stop processing guest/host IO notifications in bhyve.
* Start processing them in ptnetmap.
*/
-
-
- /* Configure the RX ring */
- sc->ptn.cfg.rx_ring.ioeventfd = -1;
- sc->ptn.cfg.rx_ring.irqfd = -1;
+ ret = vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0,
+ 0xFFFFFFFF, VTNET_RXQ, VM_IO_REGH_KWEVENTS, (void *) vq);
+ if (ret != 0) {
+ printf("ERROR ptnetmap: vm_io_reg_handler %d\n", ret);
+ goto err_reg_rx;
+ }
+ sc->ptn.cfg.rx_ring.ioeventfd = (uint64_t) vq;
/* Configure the TX ring */
- sc->ptn.cfg.tx_ring.ioeventfd = -1;
- sc->ptn.cfg.tx_ring.irqfd = -1;
-
- /* TODO: push fake-elem in the tx/rx queue to enable interrupts */
+ sc->ptn.cfg.tx_ring.irqfd = vm_get_fd(vmctx);
+ sc->ptn.cfg.tx_ioctl.com = VM_LAPIC_MSI;
+ vq = &sc->vsc_queues[VTNET_TXQ];
+ mte = &pi->pi_msix.table[vq->vq_msix_idx];
+ sc->ptn.cfg.tx_ioctl.data.msg = mte->msg_data;
+ sc->ptn.cfg.tx_ioctl.data.addr = mte->addr;
+ /* push fake-elem in the tx queue to enable interrupts */
+ if (vq_getchain(vq, &idx, iov, 1, NULL) > 0) {
+ vq_relchain(vq, idx, 0);
+ }
+ /* enable tx notification from guest */
+ vq->vq_used->vu_flags &= ~VRING_USED_F_NO_NOTIFY;
+ /*
+ * Stop processing guest/host IO notifications in bhyve.
+ * Start processing them in ptnetmap.
+ */
+ ret = vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0,
+ 0xFFFFFFFF, VTNET_TXQ, VM_IO_REGH_KWEVENTS, (void *) vq);
+ if (ret != 0) {
+ printf("ERROR ptnetmap: vm_io_reg_handler %d\n", ret);
+ goto err_reg_tx;
+ }
+ sc->ptn.cfg.tx_ring.ioeventfd = (uint64_t) vq;
/* Initialize CSB */
sc->ptn.cfg.csb = sc->ptn.csb;
@@ -131,7 +195,8 @@
sc->ptn.csb->guest_need_rxkick = 1;
sc->ptn.csb->host_need_rxkick = 1;
- sc->ptn.cfg.features = PTNETMAP_CFG_FEAT_CSB | PTNETMAP_CFG_FEAT_EVENTFD;
+ sc->ptn.cfg.features = PTNETMAP_CFG_FEAT_CSB | PTNETMAP_CFG_FEAT_EVENTFD |
+ PTNETMAP_CFG_FEAT_IOCTL;
/* Configure the net backend. */
ret = ptnetmap_create(sc->ptn.state, &sc->ptn.cfg);
@@ -143,22 +208,38 @@
return (0);
err_ptn_create:
+ vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0,
+ 0xFFFFFFFF, VTNET_TXQ, VM_IO_REGH_DELETE, 0);
+err_reg_tx:
+ vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0,
+ 0xFFFFFFFF, VTNET_RXQ, VM_IO_REGH_DELETE, 0);
+err_reg_rx:
return (ret);
}
static int
pci_vtnet_ptnetmap_down(struct pci_vtnet_softc *sc)
{
+ struct pci_devinst *pi;
+ struct vmctx *vmctx;
int ret;
if (!sc->ptn.state || !sc->ptn.up) {
return (0);
}
- sc->ptn.up = 0;
+ pi = sc->vsc_vs.vs_pi;
+ vmctx = pi->pi_vmctx;
+
/*
- * TODO: Start processing guest/host IO notifications in qemu.
+ * Start processing guest/host IO notifications in bhyve.
*/
+ vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0,
+ 0xFFFFFFFF, VTNET_RXQ, VM_IO_REGH_DELETE, 0);
+ vm_io_reg_handler(vmctx, pi->pi_bar[0].addr + VTCFG_R_QNOTIFY, 0,
+ 0xFFFFFFFF, VTNET_TXQ, VM_IO_REGH_DELETE, 0);
+
+ sc->ptn.up = 0;
return (ptnetmap_delete(sc->ptn.state));
}
@@ -177,53 +258,50 @@
memcpy(&sc->ptn.reg[offset], &value, size);
switch (offset) {
- case PTNETMAP_VIRTIO_IO_PTFEAT:
- val = (uint32_t *)(sc->ptn.reg + offset);
- ret = (sc->ptn.features &= *val);
- ptnetmap_ack_features(sc->ptn.state, sc->ptn.features);
- printf("ptnetmap acked features: %x\n", sc->ptn.features);
-
- sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTFEAT] = ret;
+ case PTNETMAP_VIRTIO_IO_PTFEAT:
+ val = (uint32_t *)(sc->ptn.reg + offset);
+ ret = (sc->ptn.features &= *val);
+ ptnetmap_ack_features(sc->ptn.state, sc->ptn.features);
+
+ sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTFEAT] = ret;
+ break;
+ case PTNETMAP_VIRTIO_IO_PTCTL:
+ val = (uint32_t *)(sc->ptn.reg + offset);
+
+ ret = EINVAL;
+ switch(*val) {
+ case NET_PARAVIRT_PTCTL_CONFIG:
+ ret = pci_vtnet_ptnetmap_get_mem(sc);
break;
- case PTNETMAP_VIRTIO_IO_PTCTL:
- val = (uint32_t *)(sc->ptn.reg + offset);
-
- ret = EINVAL;
- switch(*val) {
- case NET_PARAVIRT_PTCTL_CONFIG:
- ret = pci_vtnet_ptnetmap_get_mem(sc);
- break;
- case NET_PARAVIRT_PTCTL_REGIF:
- ret = pci_vtnet_ptnetmap_up(sc);
- break;
- case NET_PARAVIRT_PTCTL_UNREGIF:
- ret = pci_vtnet_ptnetmap_down(sc);
- break;
- case NET_PARAVIRT_PTCTL_HOSTMEMID:
- ret = ptnetmap_get_hostmemid(sc->ptn.state);
- break;
- case NET_PARAVIRT_PTCTL_IFNEW:
- case NET_PARAVIRT_PTCTL_IFDELETE:
- case NET_PARAVIRT_PTCTL_FINALIZE:
- case NET_PARAVIRT_PTCTL_DEREF:
- ret = 0;
- break;
- }
- printf("PTSTS - ret %d\n", ret);
- sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTSTS] = ret;
+ case NET_PARAVIRT_PTCTL_REGIF:
+ ret = pci_vtnet_ptnetmap_up(sc);
break;
- case PTNETMAP_VIRTIO_IO_CSBBAH:
+ case NET_PARAVIRT_PTCTL_UNREGIF:
+ ret = pci_vtnet_ptnetmap_down(sc);
break;
- case PTNETMAP_VIRTIO_IO_CSBBAL:
- ptnetmap_configure_csb(sc->vsc_vs.vs_pi->pi_vmctx, &sc->ptn.csb, *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAL)),
- *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAH)));
+ case NET_PARAVIRT_PTCTL_HOSTMEMID:
+ ret = ptnetmap_get_hostmemid(sc->ptn.state);
break;
- default:
+ case NET_PARAVIRT_PTCTL_IFNEW:
+ case NET_PARAVIRT_PTCTL_IFDELETE:
+ case NET_PARAVIRT_PTCTL_FINALIZE:
+ case NET_PARAVIRT_PTCTL_DEREF:
+ ret = 0;
break;
+ }
+ sc->ptn.reg[PTNETMAP_VIRTIO_IO_PTSTS] = ret;
+ break;
+ case PTNETMAP_VIRTIO_IO_CSBBAH:
+ break;
+ case PTNETMAP_VIRTIO_IO_CSBBAL:
+ ptnetmap_configure_csb(sc->vsc_vs.vs_pi->pi_vmctx, &sc->ptn.csb,
+ *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAL)),
+ *((uint32_t *)(sc->ptn.reg + PTNETMAP_VIRTIO_IO_CSBBAH)));
+ break;
+ default:
+ break;
}
- printf("ptnentmap_vtnet: io_write - offset: %d size: %d val: %u\n", offset, size, value);
-
return (0);
}
@@ -240,14 +318,14 @@
memcpy(value, &sc->ptn.reg[offset], size);
#if 0
switch (offset) {
- case PTNETMAP_VIRTIO_IO_PTFEAT:
- case PTNETMAP_VIRTIO_IO_PTSTS:
- break;
- default:
- break;
+ case PTNETMAP_VIRTIO_IO_PTFEAT:
+ case PTNETMAP_VIRTIO_IO_PTSTS:
+ break;
+ default:
+ printf("pci_vtnet_ptnentmap: write io reg unexpected\n");
+ break;
}
#endif
- printf("ptnentmap_vtnet: io_read - offset: %d size: %d ret: %u\n", offset, size, *value);
return (0);
}
Modified: soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h
==============================================================================
--- soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h Tue Aug 4 15:11:36 2015 (r289211)
+++ soc2015/stefano/ptnetmap/stable/10/usr.sbin/bhyve/ptnetmap.h Tue Aug 4 15:22:52 2015 (r289212)
@@ -1,3 +1,29 @@
+/*
+ * Copyright (C) 2015 Stefano Garzarella (stefano.garzarella at gmail.com)
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
#ifndef __PTNETMAP_H__
#define __PTNETMAP_H__
More information about the svn-soc-all
mailing list