From nobody Wed Dec 22 12:42:48 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 454DF1907701 for ; Wed, 22 Dec 2021 12:42:52 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JJtJq4rbHz3rHg for ; Wed, 22 Dec 2021 12:42:51 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4CC78903 for ; Wed, 22 Dec 2021 12:42:51 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: Date: Wed, 22 Dec 2021 14:42:48 +0200 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 Content-Language: en-US To: FreeBSD Current From: Andriy Gapon Subject: observations on Ryzen 5xxx (Zen 3) processors Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1640176971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mBwlxTZ498Q5wdiQ3+jw/TNwRr0Ewc+MckE9LoALdRc=; b=D34GaINae49uZMtorFWBQ75Q9jtmSDUKaTXC0/iiyyD8cMgq283qr7P8HUKM/PBv0C1sIA gwu6AEiZ1GLd03BBUyko1x//MwD+0o6Mv/wVi39W8yddweKHZZVrhZXVymnZTFUQVNDkjl tSRQ0g6buf0Lkcw1WYzQUNihAIsE4qlF3muqB5FYT3IAjjK4esa8D7GVl3AFIsZYM4qKjA s67s4/S1e4BAAAZAAc+1Pg/h0De+2vyW/nlp0wuDTejPILrAUWveclskXekfq+SEOhDPks 6brZI+hluUa+ceNG8vqWdybcw5d4xK6zDCNfDVO8EY4RuoyaiEZuAL1YFube8Q== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1640176971; a=rsa-sha256; cv=none; b=Rw2Toa9pinGzrCU/y1Fmy/FPdburpF6Fhb/AUxVypUg99IK53dhs1pF2WhhjhhYHBOXtNz OgpZH99c20HSTXPElDGkD29BNyTs1LXAVJ7PlDaU1+Nl1nXzhvBPrL8b8voNqw8mEumBRN 4zZ/hY9YR+I1lUj/JMfOhPGSQYVgaJPf8Yl1TGi9oHkp57HeLMSvj4E0SMlkoOsVnv8bjC aSlcLq4BhAZaIWsrMInY4F2jQerTcaQO0aVarQlCyHR64uvXowFiqpjV/Nh1LbUGOO52Fa LD+46kkHSb3e1KXWxV8SxDpWskSFvAf0c7y+PPA8Z+XCDVMey+nvXLBIoJWnTA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N There have been some reports on strange / unexpected things with Ryzen 5xxx processors. I think I have seen 5950X, 5900X and 5800X mentioned, not sure about others. Since I have 5800X myself I looked into a couple of issues that have straightforward demonstrators. I would like to share my findings and observations on those issues. Issue 1. High wake-up latency for CPU idle states. This seems to be related to the so called CC6 idle state. The official information on it is very sparse. The state is not explicitly exposed to the OS, at least, though ACPI interfaces that FreeBSD currently supports. In my tests I see that if all logical processors enter an idle state then an external interrupt can be delayed by 500+ us. Specifically, I observed this with an MSI-X interrupt from a discrete network chip. Interrupts from internal components seem to be affected as well, but to a lesser degree. The deep state in question can be entered regardless of whether C2 (via I/O) is enabled, C1 (via hlt) is sufficient. In fact, with machdep.idle=hlt it works the same. The state is not entered if at least one logical CPU is not idle. The state is not entered if machdep.idle=mwait is used. Apparently, the processors do not attempt to automatically enter as deep idle modes with mwait as they do with hlt. Finally, the state is not entered if zenstates.py utility is used to disable C6 / CC6 state via an undocumented (publicly) MSR. For me personally that state does not cause any annoyances but anyone who experiences problems related to "stuttering", "jitter", latency might want to look into this. Issue 2. Uneven performance of CPU intensive tasks, especially with SCHED_ULE, when SMT is enabled. I found out that at least on my hardware all even numbered logical CPUs can perform much better than odd numbered logical CPUs. It seems that hardware threads within a core are not equal. Maybe this is related to ability to use boosted frequencies, but maybe something else, I am not sure. From a brief look at the ULE code it looks that the selection of a hw thread within a core is intentionally random when all other things are equal. I suspect that the hardware + firmware may actually describe that performance disparity via ACPI CPPC (_CPC object, etc), but right now we do not support querying that or making use of it. It would interesting to see if other owners of similar processors can confirm or provide counter-examples to my observations. Simple tests for issue 1: - ping a host attached to the same switch (so, with very low expected latency) - ping 127.0.0.1 For issue 2: take some CPU intensive single-threaded task and bind it (with cpuset -l) to different logical CPUs. Multiple such tasks can be run concurrently on different logical CPUs. References: - https://forums.freebsd.org/threads/variable-ping-latency-on-ryzen-setup.82791/ - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256594 - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254040 - https://github.com/r4m0n/ZenStates-Linux - https://github.com/meowthink/ZenStates-FreeBSD -- has a bug - https://github.com/avg-I/ZenStates-FreeBSD -- has a fix - https://www.kernel.org/doc/html/latest/admin-guide/acpi/cppc_sysfs.html - https://static.linaro.org/connect/lvc21/presentations/lvc21-219.pdf - https://uefi.org/specs/ACPI/6.4/14_Platform_Communications_Channel/Platform_Comm_Channel.html -- Andriy Gapon