NVMe performance 4x slower than expected

[Tobias Oberstein]

>>>> Something is severely wrong. It seems, there might be multiple issues
>>>> (not only NVMe). And this is after already running with 3 patches to
>>>> make it even boot.
>>> The speed of dd already looks wrong.
>>
>> Yes.
> Try some simple checks for the performance anomalies.  E.g., verify the
> raw memory bandwidth, check that it is in the expected range.  Quick search
> popped up the following tool, e.g.:
> https://zsmith.co/bandwidth.html

Here are results for those 2 machines (single socket 4 core E3, quad
socket 48 core E7):

https://github.com/oberstet/scratchbox/blob/master/freebsd/cruncher/results/freebsd_memperf.md

https://raw.githubusercontent.com/oberstet/scratchbox/master/freebsd/cruncher/results/bw_4core.bmp

https://raw.githubusercontent.com/oberstet/scratchbox/master/freebsd/cruncher/results/bw_48core.bmp