ARM Graviton AWS Processor (AMI Image)

[Michał Krawczyk]

Hi,

we have solution for two of the issues with ENA,

The double resource activation can be fixed by this patch:
https://reviews.freebsd.org/D19655

And the issue with reloading module/reconfiguring it by using ifconfig
can be fixed by using this patch:
https://reviews.freebsd.org/D19641

Please check if that fixes it for you.

Thanks,
Michał


śr., 20 mar 2019 o 04:35 Colin Percival <cperciva at tarsnap.com> napisał(a):
>
> Hi ENA people,
>
> Any luck figuring out what was going on with the ENA driver here
> with interrupt activation?
>
> Colin Percival
>
> On 3/11/19 1:35 AM, Marcin Wojtas wrote:
> > +FreeBSD ENA maintainers
> >
> > W dniu pon., 11.03.2019 o 00:40 Greg V <greg at unrelenting.technology>
> > napisał(a):
> >
> >>
> >> On Tue, Jan 22, 2019 at 9:39 PM, Greg V <greg at unrelenting.technology>
> >> wrote:
> >>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer <martin at bmalum.com>
> >>> wrote:
> >>>> My question is if there are any plans yet to support the Graviton
> >>>> ARM  instances of AWS?
> >>>>
> >>>> We have a heavy load on FreeBSD and would also use the ARM
> >>>> instances.  Are there any other interested parties?
> >>>
> >>> I have tried this. It should work very well in theory, e.g. the
> >>> network card driver (if_ena) compiles with no changes for aarch64,
> >>> and in fact NetBSD has ported this driver and is up and running on
> >>> these instances: https://dmesgd.nycbug.org/index.cgi?do=view&id=4623
> >>>
> >>> But my result with FreeBSD was: nothing on the console after
> >>> loader.efi hands control to the kernel.
> >>> […]
> >>
> >> Hello everyone, big update:
> >>
> >> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS!
> >>
> >> https://dmesgd.nycbug.org/index.cgi?do=view&id=4813
> >>
> >> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now:
> >>
> >> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland)
> >>
> >> -----
> >>
> >> So, what I had to do / what should be done / how others can help get
> >> this into a finished state:
> >>
> >> 1. Serial console:
> >>    - I fixed it: https://reviews.freebsd.org/D19507
> >>    - (I learned some things about UARTs and their support in FreeBSD,
> >> should write a blog post about that)
> >>
> >> 2. aarch64 build configuration:
> >>    - if_ena network driver module should be enabled:
> >> https://reviews.freebsd.org/D18372
> >>    - NVMe driver should be enabled in the GENERIC kernel config (device
> >> nvme, device nvd)
> >>    - BTW, why not also go with hw.nvme.use_nvd="0" by default on
> >> aarch64, IIRC that was done on powerpc64
> >>
> >> 3. VM image build system:
> >>    - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was
> >> MBR with EFI (???)): https://reviews.freebsd.org/D18371
> >>    - bsdec2-image-upload --arm64 flag should be supported: included
> >> above ^^
> >>    - ec2.conf: amazon-ssm-agent shouldn't be installed when building
> >> for aarch64 TARGET, since that's written in Go, and Go isn't ported to
> >> FreeBSD/aarch64 yet:
> >>
> >> https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aad01689bc
> >> (untested patch, I actually used an interactive shell in between the
> >> image build commands)
> >>    - qemu-aarch64-static should be used for preinstalling pkgs when
> >> chrooting into the image: rough version included above ^^
> >>
> >> 4. ENA (Elastic Network Adapter) driver:
> >>    - it works
> >>    - except there's something funky with interrupt activation, and it
> >> hits panic("Attempt to double activation of resource id: %u\n", res_id)
> >> (for the management IRQ) on boot, so I applied the obvious silly
> >> workaround of "don't panic":
> >>
> >> https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba63392262938e62c
> >>    - but still, it doesn't properly reactivate interrupts (and the box
> >> becomes unreachable over the net) after going down and up again —
> >> guess what does that on boot? dhclient applying the big jumbo MTU —
> >> so I set dhclient.conf to reject MTU changes:
> >>
> >> https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6d8c870f7f
> >>
> >>
> >> Would be great if someone familiar with interrupts and stuff could help
> >> debug the ena driver and make it work without these hacks :)
> --
> Colin Percival
> Security Officer Emeritus, FreeBSD | The power to serve
> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid