[Bug 253261] AWS EC2 disk I/O performance is extremely bad

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253261

            Bug ID: 253261
           Summary: AWS EC2 disk I/O performance is extremely bad
           Product: Base System
           Version: 12.2-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Many People
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: raj at gusw.net

I am trying to move all my AWS systems back from Amazon Linux to FreeBSD. But I
encounter massive slowness.

People have reported about slow ENA network driver before. 
Even worse is the performance of the NVMe nvd disks. This is so bad that my
attempt to go with FreeBSD is now completely dead. And it makes me really sad.

FreeBSD was once "the power to serve" and now it feels like I'm running Ultrix
on a VAX for the fun of it but for any real work I need Linux. So sad!

The issue can be reproduced by anyone on AWS. Just get a FreeBSD 12 (latest)
instance and allocate a gp2 of 1 TB or a gp3 of smaller size for which you
specify 3000 IOPS and 250 MB/s IO capacity. Let's say that disk is /dev/nvd2

Now do this:

dd if=/dev/zero of=/dev/nvd2 bs=100M status=progress

and you see that it's writing with the whopping speed of 70 MB/s. 

That used to be good, but it is no longer good. Compare Amazon Linux doing the
same thing at 300 MB/s.

Now, when you put a file system over it, zfs or ufs, then instantly the
performance gets better:

newfs /dev/nvd2
mount /dev/nvd2 /mnt
dd if=/dev/zero of=/mnt/test bs=100M status=progress

now that works at about 250 MB/s. Decent. So, problem solved?

No! It turns out if I create a PostgreSQL database over this setup, then again
there is massive delay on the read and write and throughput will drop to even
worse than 70 MB/s. Creating one index takes 10 times as long as that same on
the Linux system.

The dd issue is probably the best manifestation for fixing at least part of the
problem. 

Is this about tuning? Even if it was, there is a severe problem with how the
system is set up by default. But I'm afraid it's not even about tuning so bad
is it.

-- 
You are receiving this mail because:
You are the assignee for the bug.