ARM Graviton AWS Processor (AMI Image)

[Greg V]

On Tue, Jan 22, 2019 at 9:39 PM, Greg V <greg at unrelenting.technology> 
wrote:
> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer <martin at bmalum.com> 
> wrote:
>> My question is if there are any plans yet to support the Graviton 
>> ARM instances of AWS?
>> 
>> We have a heavy load on FreeBSD and would also use the ARM 
>> instances. Are there any other interested parties?
> 
> I have tried this. It should work very well in theory, e.g. the 
> network card driver (if_ena) compiles with no changes for aarch64, 
> and in fact NetBSD has ported this driver and is up and running on 
> these instances: https://dmesgd.nycbug.org/index.cgi?do=view&id=4623
> 
> But my result with FreeBSD was: nothing on the console after 
> loader.efi hands control to the kernel.
> […]

Hello everyone, big update:

FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS!

https://dmesgd.nycbug.org/index.cgi?do=view&id=4813

And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now:

ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland)

-----

So, what I had to do / what should be done / how others can help get 
this into a finished state:

1. Serial console:
   - I fixed it: https://reviews.freebsd.org/D19507
   - (I learned some things about UARTs and their support in FreeBSD, 
should write a blog post about that)

2. aarch64 build configuration:
   - if_ena network driver module should be enabled: 
https://reviews.freebsd.org/D18372
   - NVMe driver should be enabled in the GENERIC kernel config (device 
nvme, device nvd)
   - BTW, why not also go with hw.nvme.use_nvd="0" by default on 
aarch64, IIRC that was done on powerpc64

3. VM image build system:
   - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was 
MBR with EFI (???)): https://reviews.freebsd.org/D18371
   - bsdec2-image-upload --arm64 flag should be supported: included 
above ^^
   - ec2.conf: amazon-ssm-agent shouldn't be installed when building 
for aarch64 TARGET, since that's written in Go, and Go isn't ported to 
FreeBSD/aarch64 yet: 
https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aad01689bc 
(untested patch, I actually used an interactive shell in between the 
image build commands)
   - qemu-aarch64-static should be used for preinstalling pkgs when 
chrooting into the image: rough version included above ^^

4. ENA (Elastic Network Adapter) driver:
   - it works
   - except there's something funky with interrupt activation, and it 
hits panic("Attempt to double activation of resource id: %u\n", res_id) 
(for the management IRQ) on boot, so I applied the obvious silly 
workaround of "don't panic": 
https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba63392262938e62c
   - but still, it doesn't properly reactivate interrupts (and the box 
becomes unreachable over the net) after going down and up again — 
guess what does that on boot? dhclient applying the big jumbo MTU — 
so I set dhclient.conf to reject MTU changes: 
https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6d8c870f7f


Would be great if someone familiar with interrupts and stuff could help 
debug the ena driver and make it work without these hacks :)