From nobody Sat Aug 06 18:06:34 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M0VlZ5fjrz4YQv8; Sat, 6 Aug 2022 18:06:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M0VlZ59LJz3cFw; Sat, 6 Aug 2022 18:06:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659809194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=95yCTr2e1bvK6x1wkNBbTd6PI3Wxma8mu8BEDhNvNVc=; b=NhrBX1iyPS7jPUMxwxkqeAdzajmPyPM3uHZq/aqOjEwyLtD1pOgy+NvuwcFsUBEd9GF//a HL+kTq5U556GR3n6E0AlEfRtMRG+GLh1cM7jYeerWzTvYW+BL747ksM7FihRfKwWmE7wnZ lJpL6xEu62J45/tUU8/NheP/WD8XUzbgFNLB48CnL4waSaW8DP2TfVcKJ64B2VVKTS8XLa v4BiCZauCKdsQi9pxjxH7WuaGHcZ6t73/pWd0JpVc+hEIBpvwkWKDWevV9tVmKHga1sCSm C3zm2WnFJrCUm/azLuUpbB8sXZDQ/zQHYZCH6ZoZLNSsG6YiQHF2UPvWS6qTCA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4M0VlZ44bnzhvf; Sat, 6 Aug 2022 18:06:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 276I6YSs005624; Sat, 6 Aug 2022 18:06:34 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 276I6YCh005623; Sat, 6 Aug 2022 18:06:34 GMT (envelope-from git) Date: Sat, 6 Aug 2022 18:06:34 GMT Message-Id: <202208061806.276I6YCh005623@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: 7f46deccbed7 - main - x86/iommu: Reduce the number of queued invalidation interrupts List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 7f46deccbed74436b62f8fd02655ff4ad89f1023 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1659809194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=95yCTr2e1bvK6x1wkNBbTd6PI3Wxma8mu8BEDhNvNVc=; b=IyuGhb4UGXcnuSSOVwHq97gdZ5JBRiyW7tTbo8nI8KkyGzgtYdvCmvkGHkQI2kp2GMkPdb l6vjo5n5b7hPcTN83PFdrKcIOGFH80Y4AWnoDY4ZQ4gA9AtrNcIB6SXW0iofHzX5gul1S0 7xvS+MB4un2z4Uos2tqZcXUTd5Es6Yd48Qmc0GFzH0pMHE8hpdiE0LbpFbpZIyfq4BBfGo /0+8X7ax+r2BMqMLuxy3RJync7Xpr9mAqiUp/KXxWCI/duQmbrejdS3HLNH13Q8EnM5vbF AneO5Z9xEifhQUnTRyD4azJQfZzs6WVGfSMNhYDsL/8cRj0le6CQCzw/Cjiwlg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1659809194; a=rsa-sha256; cv=none; b=uN0QRivi4RYOMx++XCtGJUIuztuSQ0GfmAHdubA43cvhVwzs9HA6wpG+WOVNUG2RMc/AyH 0Ct97IbsCRutUOvmIy2seeyAUtt3imwNfbPDhwtguUFzi3e40GsSunx1tz9QP9Gz8Rckka lnjzVS774T/dZmVRIdDp7DAwlmIr+Z6aE8y6QHHUbFd3+T/v0ayFLoP2/tWDA7r1x94UBH b/8Qp7tW8EIT6gSDIiir38lJvsBE/x3/PrAZMsGuYWvwd+J+DMiJ8LhzGLh9wOqQfYLGJy mBaLRvfOb110luN+WaxpUTK8VY1VAL0J8q41+79/u84AyqFAc15jrYfkQ1xfcg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=7f46deccbed74436b62f8fd02655ff4ad89f1023 commit 7f46deccbed74436b62f8fd02655ff4ad89f1023 Author: Alan Cox AuthorDate: 2022-07-31 19:28:30 +0000 Commit: Alan Cox CommitDate: 2022-08-06 18:05:58 +0000 x86/iommu: Reduce the number of queued invalidation interrupts Restructure dmar_qi_task() so as to reduce the number of invalidation completion interrupts. Specifically, because processing completed invalidations in dmar_qi_task() can take quite some time, don't reenable completion interrupts until processing has completed a first time. Then, check a second time after reenabling completion interrupts, so that any invalidations that complete just before interrupts are reenabled do not linger until a future invalidation might raise an interrupt. (Recent changes have made checking for completed invalidations cheap; no locking is required.) Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D36054 --- sys/x86/iommu/intel_qi.c | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/sys/x86/iommu/intel_qi.c b/sys/x86/iommu/intel_qi.c index baaf5b472a2c..8a8e656083e3 100644 --- a/sys/x86/iommu/intel_qi.c +++ b/sys/x86/iommu/intel_qi.c @@ -411,14 +411,34 @@ dmar_qi_intr(void *arg) return (FILTER_HANDLED); } +static void +dmar_qi_drain_tlb_flush(struct dmar_unit *unit) +{ + struct iommu_map_entry *entry, *head; + + for (head = unit->tlb_flush_head;; head = entry) { + entry = (struct iommu_map_entry *) + atomic_load_acq_ptr((uintptr_t *)&head->tlb_flush_next); + if (entry == NULL || + !dmar_qi_seq_processed(unit, &entry->gseq)) + break; + unit->tlb_flush_head = entry; + iommu_gas_free_entry(head); + if ((entry->flags & IOMMU_MAP_ENTRY_RMRR) != 0) + iommu_gas_free_region(entry); + else + iommu_gas_free_space(entry); + } +} + static void dmar_qi_task(void *arg, int pending __unused) { struct dmar_unit *unit; - struct iommu_map_entry *entry, *head; uint32_t ics; unit = arg; + dmar_qi_drain_tlb_flush(unit); /* * Request an interrupt on the completion of the next invalidation @@ -428,23 +448,16 @@ dmar_qi_task(void *arg, int pending __unused) if ((ics & DMAR_ICS_IWC) != 0) { ics = DMAR_ICS_IWC; dmar_write4(unit, DMAR_ICS_REG, ics); - } - for (;;) { - head = unit->tlb_flush_head; - entry = (struct iommu_map_entry *) - atomic_load_acq_ptr((uintptr_t *)&head->tlb_flush_next); - if (entry == NULL) - break; - if (!dmar_qi_seq_processed(unit, &entry->gseq)) - break; - unit->tlb_flush_head = entry; - iommu_gas_free_entry(head); - if ((entry->flags & IOMMU_MAP_ENTRY_RMRR) != 0) - iommu_gas_free_region(entry); - else - iommu_gas_free_space(entry); + /* + * Drain a second time in case the DMAR processes an entry + * after the first call and before clearing DMAR_ICS_IWC. + * Otherwise, such entries will linger until a later entry + * that requests an interrupt is processed. + */ + dmar_qi_drain_tlb_flush(unit); } + if (unit->inv_seq_waiters > 0) { /* * Acquire the DMAR lock so that wakeup() is called only after