[Bug 276463] Linux guest uses only 503G of memory when using UEFI loader

From: <bugzilla-noreply_at_freebsd.org>
Date: Fri, 19 Jan 2024 16:33:05 UTC

            Bug ID: 276463
           Summary: Linux guest uses only 503G of memory when using UEFI
           Product: Base System
           Version: 13.2-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Keywords: bhyve
          Severity: Affects Only Me
          Priority: ---
         Component: bhyve
          Assignee: virtualization@FreeBSD.org
          Reporter: antranigv@freebsd.am

Created attachment 247783
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=247783&action=edit
Information about the Linux Guest

We have a massive system (details below) which runs a massive VM. The host runs
on FreeBSD while the VM guest is Ubuntu Linux.

Host Information:
root@genomic:~ # sysctl hw.model
hw.model: AMD EPYC 7702 64-Core Processor                
root@genomic:~ # sysctl hw.ncpu
hw.ncpu: 256
root@genomic:~ # sysctl hw.physmem
hw.physmem: 2185595944960
root@genomic:~ # sysctl hw.realmem
hw.realmem: 2200361238528

The VM uses 1.5TB of RAM, here's the VM configuration:
# vm info comp0
Virtual Machine: comp0
  state: stopped
  datastore: default
  loader: uefi
  uuid: 2893e12c-637d-11ee-af94-7cc255269cb8
  cpu: 240
  cpu-topology: sockets=2, cores=60, threads=2
  memory: 1536G

    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:0e:4b:00
    fixed-device: -

    number: 0
    device-type: file
    emulation: virtio-blk
    options: -
    system-path: /usr/local/vm/comp0/disk0.img
    bytes-size: 216895848448 (202.000G)
    bytes-used: 190077076480 (177.023G)

and here's the bhyve arguments according to vm-bhyve:

Jan 19 15:40:32:  [bhyve options: -c 240,sockets=2,cores=60,threads=2 -m 1536G
-Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U
2893e12c-637d-11ee-af94-7cc255269cb8 -u -S]
Jan 19 15:40:32:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s
4:0,virtio-blk,/usr/local/vm/comp0/disk0.img -s
5:0,virtio-blk,/dev/zvol/zscratch/scratch -s
6:0,virtio-net,tap0,mac=58:9c:fc:0e:4b:00 -s 7:0,passthru,68/0/81 -s

We've been using the GRUB loader for a while and decided to move to the UEFI
loader. After the migration we noticed that Linux shows only 503GB of RAM from
the 1.5TB allocated. We didn't have such issue when using the grub loader.

I'm attaching a tarball which has information of the Linux system, including
the following files.

linux_uefi_iomem.txt — /proc/iomem
linux_uefi_meminfo.txt — /proc/meminfo
linux_uefi_vmstat.txt — /proc/vmstat
linux_uefi_dmidecode.txt — dmidecode
linux_uefi_free.txt — free
linux_uefi_dmesg.txt — dmesg
linux_uefi_cpuinfo.txt — /proc/cpuinfo

We've also noticed that inside of TianoCore's UEFI menu, it shows the 1.5TB of
RAM properly (check screenshot at: https://notes.bsd.am/bhyve_uefi_menu.png )

Not sure if this is an issue with Linux or bhyve's UEFI loader. any tips are

You are receiving this mail because:
You are the assignee for the bug.