ARM Graviton AWS Processor (AMI Image)
Greg V
greg at unrelenting.technology
Sun Mar 10 22:07:40 UTC 2019
On Tue, Jan 22, 2019 at 9:39 PM, Greg V <greg at unrelenting.technology>
wrote:
> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer <martin at bmalum.com>
> wrote:
>> My question is if there are any plans yet to support the Graviton
>> ARM instances of AWS?
>>
>> We have a heavy load on FreeBSD and would also use the ARM
>> instances. Are there any other interested parties?
>
> I have tried this. It should work very well in theory, e.g. the
> network card driver (if_ena) compiles with no changes for aarch64,
> and in fact NetBSD has ported this driver and is up and running on
> these instances: https://dmesgd.nycbug.org/index.cgi?do=view&id=4623
>
> But my result with FreeBSD was: nothing on the console after
> loader.efi hands control to the kernel.
> […]
Hello everyone, big update:
FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS!
https://dmesgd.nycbug.org/index.cgi?do=view&id=4813
And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now:
ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland)
-----
So, what I had to do / what should be done / how others can help get
this into a finished state:
1. Serial console:
- I fixed it: https://reviews.freebsd.org/D19507
- (I learned some things about UARTs and their support in FreeBSD,
should write a blog post about that)
2. aarch64 build configuration:
- if_ena network driver module should be enabled:
https://reviews.freebsd.org/D18372
- NVMe driver should be enabled in the GENERIC kernel config (device
nvme, device nvd)
- BTW, why not also go with hw.nvme.use_nvd="0" by default on
aarch64, IIRC that was done on powerpc64
3. VM image build system:
- GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was
MBR with EFI (???)): https://reviews.freebsd.org/D18371
- bsdec2-image-upload --arm64 flag should be supported: included
above ^^
- ec2.conf: amazon-ssm-agent shouldn't be installed when building
for aarch64 TARGET, since that's written in Go, and Go isn't ported to
FreeBSD/aarch64 yet:
https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aad01689bc
(untested patch, I actually used an interactive shell in between the
image build commands)
- qemu-aarch64-static should be used for preinstalling pkgs when
chrooting into the image: rough version included above ^^
4. ENA (Elastic Network Adapter) driver:
- it works
- except there's something funky with interrupt activation, and it
hits panic("Attempt to double activation of resource id: %u\n", res_id)
(for the management IRQ) on boot, so I applied the obvious silly
workaround of "don't panic":
https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba63392262938e62c
- but still, it doesn't properly reactivate interrupts (and the box
becomes unreachable over the net) after going down and up again —
guess what does that on boot? dhclient applying the big jumbo MTU —
so I set dhclient.conf to reject MTU changes:
https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6d8c870f7f
Would be great if someone familiar with interrupts and stuff could help
debug the ena driver and make it work without these hacks :)
More information about the freebsd-arm
mailing list