amdgpu panics

Sat Apr 4 09:54:53 UTC 2020

On 12/03/2020 16:34, John Kennedy wrote:
> On Thu, Mar 12, 2020 at 12:22:21AM +0000, Grzegorz Junka wrote:
>> On 10/03/2020 19:46, Hans Petter Selasky wrote:
>>> On 2020-03-10 20:29, Grzegorz Junka wrote:
>>>> On 09/03/2020 23:04, Hans Petter Selasky wrote:
>>>>> On 2020-03-10 00:03, Grzegorz Junka wrote:
>>>>>> I've upgraded my system to 12.1. I have recompiled all ports
>>>>>> with poudriere using jail 12.1. As soon as "amdgpu" kernel
>>>>>> module is loaded the system panics. Tried with both,
>>>>>> "drm-kmo" and "drm-fbsd12.0-kmod". Any ideas?
>>>>> Are the kernel sources in the jail matched with your system?
>>>> Sorry, I don't understand your question. Why does it matter? The
>>>> jail was created by poudriere. Does the installed package use
>>>> sources from my system?
>>> All kernel module packages use sources from the build system!
>> The base system has also been updated to 12.1 if that's what you are asking.
>> I am not sure if the sources are the same. Both the host and the poudriere
>> jail are 12.1. Do they need to be exact to the patch version?
>>
>> I understood that the drm-kmod or drm-fsbsd12.0-kmod are build as packages
>> by poudriere in the poudriere jail. Are you saying that they are build with
>> sources from the host? How is that possible?
>    This may have absolutely nothing to do with your setup.  I can only say that
> I've been where you are in the past, what I've done, and that it has made me
> happy and solved my problems.
>
>    First off, packages are great.  I can't quantify that, but I know that the
> small percentage of the time that they don't work for some reason is really
> annoying to me (and arguably a personal problem).  (:  But here is how I've
> channeled that into something productive, even if it isn't something that
> everybody can do.  If everybody could do it, we wouldn't have a strong demand
> for packages (even ignoring some economy of scale/ease of use arguments).
>
>    First off, I have a local poudriere setup, and compile all my packages from
> source.  I have some non-default options that I prefer, occasionally some odd
> platforms (arm64, etc) and run closer to the bleeding edge (I tend to ride
> the pre- into OS releases, use the master (vs quarterly) ports, etc.  So I'm
> pretty sure I run into all the problems that the ports/src (and other reasonable)
> people expect.  Compiling from source dodges a huge number of those.  I didn't
> have problems with pkg, X11 during -1.20 (except the FIXDRM/DRI issue for one
> graphics card, and only showed up with firefox with certain site graphics), the
> input problems some people have had, etc.  I've been very pleased to not have
> those problems, although a bunch of them had nothing to do with how something
> was compiled vs life moving on and things changing (udev/dev/event changes, etc).
> Lets face it, X11 is annoying.  I have a very stripped down X11 environment
> and I think it is at least 253 packages, even more if you have X11 for reasons
> (like firefox browser).  Kudos to a lot of nameless people keeping that whole
> mess working  as well as it does.
>
>    That "life" choice has been REALLY painful on RPI3/arm64, but not painful
> at all on amd64 type systems, your mileage may vary.  It has certainly given
> me an understanding on why some packages on FreeBSD's repo take a while to
> get out.  I'm only building ~500 packages, and they're building all of them.
> Watching a readline upgrade trigger hundreds of packages to rebuild hurts,
> but it helps solve the mixing-source-and-package-can-hurt problems.
>
>    In any case, on my poudriere system, I upgrade the kernel jail (poudriere
> jail -u ...) once per patch on release systems and maybe once a week (or CVE,
> or when they bump __FreeBSD_version) when I'm tracking stable.  Basically
> when I think that the kernel ABI might have changed or when some security
> related piece of code might need to be baked into the packages.  The ultimate
> thing might be once per kernel upgrade, but that is overkill so I try to
> compensate with the weekly sync, just in case.  AFAIK, the ports builders
> keep theirs at the -release level (?), which is pretty good for most things
> but fails in a few cases (and here is where some of the readers are).
>
>    In my case, I created the kernel source jail like this:
>
> 	poudriere jail -c -j 12-1 -v 12.1 -m src=/usr/src
>
>    It just has whatever source when you create it or when you update it.  I
> didn't try to create it with a null method, but maybe that would work.  If
> you upgrade your local kernel but not your jail, then yes, you can have some
> kernel ABI issues that might bite you like you've seen (kernel panics).
>
>    I haven't had a problem with video drivers myself, but what drove that
> behavior used to be virtual box.
>
>    Probably daily I pull down the source tree and let poudriere decide what
> ports need to be rebuilt.  They're always built with kernel OS that is really
> close to what is actually running.
>
>    On top of all of THAT, I have this in my /etc/make.conf:
>
> 	# be extra paranoid about drivers being compiled for running kernel
> 	PORTS_MODULES +=	graphics/drm-kmod
> 	PORTS_MODULES +=	graphics/drm-fbsd12.0-kmod
>
>    No problem with that, but doesn't work as cleanly with x11/nvidia-driver.
> AFAIK, this package is built from the local (vs poudriere jail) sources, so
> it should help with boot-panic (after kernel upgrade) type situations but
> might confuse you-the-human when figuring out where those binaries came from
> down the road (I upgrade the package, now it fails on some future reboot).
>
>    So, every time I build that kernel/world, I get the graphics modules compiled
> at that instant.  In my case, /usr/ports is a git repo, and I've set up
> poudriere for ports like this:
>
> 	poudriere ports -c -m null -M /usr/ports -p master
>
>    So I've tried to minimize any port source sync problems I might have.

Thank you John for the comprehensive explanation. It took me a while to 
go through all the details, then again to recompile the ports and try to 
reinstall all packages.

What i discovered in the meantime is that it's not an isolated problem:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241787

https://forums.freebsd.org/threads/upgrading-to-freebsd-12-1-release-resolving-an-issue-with-drm-fbsd12-0-kmod.72895/

On my system I indeed had the jail at a different patch level than the 
host system, although they were all running 12.1-RELEASE. I updated the 
host and the jail to 12.1-RELEASE-p3. Poudriere noticed the updated jail 
and deleted and recompiled all 2000+ packages. Then I upgraded the 
system on which I wanted to install the packages to 12.1-RELEASE-p3 too. 
Then I deleted drm-fbsd12.0-kmod and installed drm-kmod. It reinstalled 
drm-fbsd12.0-kmod.

The result? Blank screen!!!

I start as single or normal user then do:

kldload amdgpu

I see the driver is loading various graphics kernel modules then the 
screen goes blank and the whole system hangs. No panic is shown, no 
restart, just hungs. Any SSH sessions to the system become stale. Only 
hard reset is able to restart it.

This is really frustrating and a really bad user experience. I wouldn't 
be surprised if the remained desktop users moved to Linux or other 
FreeBSD forks if they haven't already.

The only option left I see is to also compile the kernel myself from 
sources.

I haven't yet tried to add /usr/src to the jail as you suggested nor use 
ports from Git. I guess the first shouldn't matter if I updated the host 
and the jail at the same time to the same patch level? Is it possible to 
add /usr/src to an existing jail with poudriere so that I don't have 
recompile all the packages again?

As for the second, this also shouldn't matter since the drm-mod sources 
aren't kept in ports, just the metadata to fetch them?

I am tempted to give up FreeBSD for now and come back in 6 months to 
retry on 12.2 when it's out.

GrzegorzJ