From nobody Mon Jan 17 16:22:14 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A2092196D4DE; Mon, 17 Jan 2022 16:22:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jcxxy3MlBz3DSS; Mon, 17 Jan 2022 16:22:14 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642436534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DurRF6MmjdeX20wBbEI122c4fka0FMoGIyyCk5jUBkA=; b=jNpOsZNg1HGpLHiSXR6n9D8yaV628kZqK2AI3gY6Bwa/UCpAHbkMaieKZkBYjKgf3kCGJ3 bggc0nuW7zaCjgUH+OHxVBhCtF6PKtVebdzpOXUtri3EtaITIGvNHCgBP223C0mgEmq/ct f3dYQeq4GxD0giovlc/fUhpNznj/Jn495K64Llk+pR2ofxz0niwy6QGishsZPoxBVpECk1 CeI9gbJcp25swrNy6nC1/2Tfx+Vs3DWk7uiqmxoGsa8OJqSEc0+LlAQgDWS1zjPIK2fBiC LzFqNKtnuae48c19JE5tLD8XfVTjsNxDZjBZBSkmeHL9Rdos1z19bhTD743thQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 37FC51121A; Mon, 17 Jan 2022 16:22:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 20HGMEjd016063; Mon, 17 Jan 2022 16:22:14 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 20HGMEC2016062; Mon, 17 Jan 2022 16:22:14 GMT (envelope-from git) Date: Mon, 17 Jan 2022 16:22:14 GMT Message-Id: <202201171622.20HGMEC2016062@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Emmanuel Vadot Subject: git: 104c83a999fa - stable/13 - bhyve: enumerate BARs by size List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: manu X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 104c83a999fa459fd59f63374d7145b156ba76d1 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642436534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DurRF6MmjdeX20wBbEI122c4fka0FMoGIyyCk5jUBkA=; b=EtK/+F4DrwJcWzHGKdgnVN4Ui7Ljsx/VBIQHjCpN3z+r7L8cDD/CuO40ATAgPDGBdiO9Sq m3nXpgF4a/8pAncEciCeZMmX9vJx3SSpzS4tZ6m/3FpR/O3cbTIcTmcrJK7cgseTxsUwwG Bxg82TrL1Fxybq7HMh9U20gnvtBXSF2pjOzF59A7yaFfPH3SfZjUYeXdt58qYBpn7tkXTk dEnvCuTVXY1mtEp51YksqsrqAlrwKrcLm3VnveEYJ2Mp1gRyqD1b0PBUffg6zFRs7FHJTw kJhi/EuTO47FPKuTL+DHE/NnFT7UDZOgrd6idrdw3Hz5++DaNpQbDYRnTimh7w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642436534; a=rsa-sha256; cv=none; b=ZYZeh/TNRCwwpNunZvQAINAOHZfvSAT6h++c6mrMn1az05K20VbSyEg9NUaD6SWmWcDHGT hdjYP6xmlvdUnOdx9hqNDrqF5dQHRK6YlBpcfWwpbUe+nDuhTv4rNT60KmanPIDMKiYc71 lpaDVRrEYTCQDGlE6I1ZXVDCoVl54K1jlVLXB3rDemunmMExcjzZ7Lf1Vsio/DSCuNvH8s NT4AGmylBtfQaMttKLvmN9KR4Uco51VXFL/uCXU6kPnJpLJGRVx41LuW9u7SZOJCojADJK uuhR5BpakE7J3bT2sZJ9FPnjnYi9rz1VBv4VJXSNrea4XtfNwr3jTeyZf4OrlA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by manu: URL: https://cgit.FreeBSD.org/src/commit/?id=104c83a999fa459fd59f63374d7145b156ba76d1 commit 104c83a999fa459fd59f63374d7145b156ba76d1 Author: Corvin Köhne AuthorDate: 2022-01-03 13:16:59 +0000 Commit: Emmanuel Vadot CommitDate: 2022-01-17 12:53:10 +0000 bhyve: enumerate BARs by size E.g. Framebuffers can require large space and BARs need to be aligned by their size. If BARs aren't allocated by size, it'll cause much fragmentation of the MMIO space. Reduce fragmentation by ordering the BAR allocation on their size to reduce the risk of OUT_OF_MMIO_SPACE issues. Reviewed by: markj MFC after: 2 weeks Sponsored by: Beckhoff Automation GmbH & Co. KG Differential Revision: https://reviews.freebsd.org/D28278 (cherry picked from commit 01f9362ef4eb14b041ccdf935fccf0f794074258) --- usr.sbin/bhyve/pci_emul.c | 105 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 94 insertions(+), 11 deletions(-) diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_emul.c index 0b46bcc6200e..4365cda234b3 100644 --- a/usr.sbin/bhyve/pci_emul.c +++ b/usr.sbin/bhyve/pci_emul.c @@ -105,6 +105,16 @@ static uint64_t pci_emul_membase32; static uint64_t pci_emul_membase64; static uint64_t pci_emul_memlim64; +struct pci_bar_allocation { + TAILQ_ENTRY(pci_bar_allocation) chain; + struct pci_devinst *pdi; + int idx; + enum pcibar_type type; + uint64_t size; +}; +TAILQ_HEAD(pci_bar_list, pci_bar_allocation) pci_bars = TAILQ_HEAD_INITIALIZER( + pci_bars); + #define PCI_EMUL_IOBASE 0x2000 #define PCI_EMUL_IOLIMIT 0x10000 @@ -639,10 +649,6 @@ int pci_emul_alloc_bar(struct pci_devinst *pdi, int idx, enum pcibar_type type, uint64_t size) { - int error; - uint64_t *baseptr, limit, addr, mask, lobits, bar; - uint16_t cmd, enbit; - assert(idx >= 0 && idx <= PCI_BARMAX); if ((size & (size - 1)) != 0) @@ -657,17 +663,88 @@ pci_emul_alloc_bar(struct pci_devinst *pdi, int idx, enum pcibar_type type, size = 16; } + /* + * To reduce fragmentation of the MMIO space, we allocate the BARs by + * size. Therefore, don't allocate the BAR yet. We create a list of all + * BAR allocation which is sorted by BAR size. When all PCI devices are + * initialized, we will assign an address to the BARs. + */ + + /* create a new list entry */ + struct pci_bar_allocation *const new_bar = malloc(sizeof(*new_bar)); + memset(new_bar, 0, sizeof(*new_bar)); + new_bar->pdi = pdi; + new_bar->idx = idx; + new_bar->type = type; + new_bar->size = size; + + /* + * Search for a BAR which size is lower than the size of our newly + * allocated BAR. + */ + struct pci_bar_allocation *bar = NULL; + TAILQ_FOREACH(bar, &pci_bars, chain) { + if (bar->size < size) { + break; + } + } + + if (bar == NULL) { + /* + * Either the list is empty or new BAR is the smallest BAR of + * the list. Append it to the end of our list. + */ + TAILQ_INSERT_TAIL(&pci_bars, new_bar, chain); + } else { + /* + * The found BAR is smaller than our new BAR. For that reason, + * insert our new BAR before the found BAR. + */ + TAILQ_INSERT_BEFORE(bar, new_bar, chain); + } + + /* + * pci_passthru devices synchronize their physical and virtual command + * register on init. For that reason, the virtual cmd reg should be + * updated as early as possible. + */ + uint16_t enbit = 0; + switch (type) { + case PCIBAR_IO: + enbit = PCIM_CMD_PORTEN; + break; + case PCIBAR_MEM64: + case PCIBAR_MEM32: + enbit = PCIM_CMD_MEMEN; + break; + default: + enbit = 0; + break; + } + + const uint16_t cmd = pci_get_cfgdata16(pdi, PCIR_COMMAND); + pci_set_cfgdata16(pdi, PCIR_COMMAND, cmd | enbit); + + return (0); +} + +static int +pci_emul_assign_bar(struct pci_devinst *const pdi, const int idx, + const enum pcibar_type type, const uint64_t size) +{ + int error; + uint64_t *baseptr, limit, addr, mask, lobits, bar; + switch (type) { case PCIBAR_NONE: baseptr = NULL; - addr = mask = lobits = enbit = 0; + addr = mask = lobits = 0; break; case PCIBAR_IO: baseptr = &pci_emul_iobase; limit = PCI_EMUL_IOLIMIT; mask = PCIM_BAR_IO_BASE; lobits = PCIM_BAR_IO_SPACE; - enbit = PCIM_CMD_PORTEN; break; case PCIBAR_MEM64: /* @@ -689,14 +766,12 @@ pci_emul_alloc_bar(struct pci_devinst *pdi, int idx, enum pcibar_type type, mask = PCIM_BAR_MEM_BASE; lobits = PCIM_BAR_MEM_SPACE | PCIM_BAR_MEM_64; } - enbit = PCIM_CMD_MEMEN; break; case PCIBAR_MEM32: baseptr = &pci_emul_membase32; limit = PCI_EMUL_MEMLIMIT32; mask = PCIM_BAR_MEM_BASE; lobits = PCIM_BAR_MEM_SPACE | PCIM_BAR_MEM_32; - enbit = PCIM_CMD_MEMEN; break; default: printf("pci_emul_alloc_base: invalid bar type %d\n", type); @@ -732,9 +807,6 @@ pci_emul_alloc_bar(struct pci_devinst *pdi, int idx, enum pcibar_type type, pci_set_cfgdata32(pdi, PCIR_BAR(idx + 1), bar >> 32); } - cmd = pci_get_cfgdata16(pdi, PCIR_COMMAND); - if ((cmd & enbit) != enbit) - pci_set_cfgdata16(pdi, PCIR_COMMAND, cmd | enbit); register_bar(pdi, idx); return (0); @@ -1189,6 +1261,7 @@ init_pci(struct vmctx *ctx) bi->membase32 = pci_emul_membase32; bi->membase64 = pci_emul_membase64; + /* first run: init devices */ for (slot = 0; slot < MAXSLOTS; slot++) { si = &bi->slotinfo[slot]; for (func = 0; func < MAXFUNCS; func++) { @@ -1228,6 +1301,16 @@ init_pci(struct vmctx *ctx) } } + /* second run: assign BARs and free list */ + struct pci_bar_allocation *bar; + struct pci_bar_allocation *bar_tmp; + TAILQ_FOREACH_SAFE(bar, &pci_bars, chain, bar_tmp) { + pci_emul_assign_bar(bar->pdi, bar->idx, bar->type, + bar->size); + free(bar); + } + TAILQ_INIT(&pci_bars); + /* * Add some slop to the I/O and memory resources decoded by * this bus to give a guest some flexibility if it wants to