Re: bhyve disk performance issue
- Reply: Matthew Grooms : "Re: bhyve disk performance issue"
- In reply to: Matthew Grooms : "Re: bhyve disk performance issue"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sun, 18 Feb 2024 15:21:10 UTC
On 2/17/24 15:53, Matthew Grooms wrote: > On 2/16/24 12:00, Matthew Grooms wrote: >> On 2/16/24 11:42, Chuck Tuffli wrote: >>> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>>> >>>> Hi All, >>>> >>>> >>>> I'm in the middle of a project that involves building out a handful >>>> of servers to host virtual Linux instances. Part of that includes >>>> testing bhyve to see how it performs. The intent is to compare host >>>> storage options such as raw vs zvol block devices and ufs vs zfs >>>> disk images using hardware raid vs zfs managed disks. It would also >>>> involve >>>> >>>> >>> … >>>> >>>> Here is a list of a few other things I'd like to try: >>>> >>>> >>>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>>> 2) Downgrading the host to 13.2-RELEASE >>> >>> FWIW we recently did a similar exercise and saw significant >>> performance differences on ZFS backed disk images when comparing >>> 14.0 and 13.2. We didn’t have time to root cause the difference, so >>> it could simply be some tuning difference needed for 14. >> >> Hi Chuck, >> >> That's very helpful feedback. I'll start by downgrading the host to >> 13.2 and report back here. >> > > Unfortunately same story with 13.2. I'm going to try and downgrade to > 12.4 and see if it gets any better. > > ================================================================================ > Begin @ Sat Feb 17 11:00:01 CST 2024 > > Version 2.00 ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > localhost.lo 63640M 690k 99 1.5g 97 727m 78 950k 99 1.3g 68 > +++++ +++ > Latency 11759us 29114us 8098us 8649us 23413us > 4540us > Version 2.00 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > Latency 7791us 131us 1671us 464us 15us 1811us > > End @ Sat Feb 17 11:03:13 CST 2024 > ================================================================================ > Begin @ Sat Feb 17 11:10:01 CST 2024 > > Version 2.00 ------Sequential Output------ --Sequential Input- > --Random- > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > localhost.lo 63640M 667k 99 449m 99 313m 94 940k 99 398m 99 > 16204 563 > Latency 12147us 1079us 24470us 8795us 17853us > 4384us > Version 2.00 ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- > files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > /sec %CP > 16 0 93 +++++ +++ +++++ +++ 0 96 +++++ +++ > +++++ +++ > Latency 118us 159us 269us 164us 54us 844us > > End @ Sat Feb 17 11:18:43 CST 2024 > I wasn't able to get a working 12.4 system in place due to lack of packages. However, I did fire up a FreeBSD 14 VM and let it run overnight on the same SSD array. It consistently ran at a much higher speed for 50+ runs @ 10m intervals ... ================================================================================ Begin @ Sun Feb 18 15:00:00 UTC 2024 Version 1.98 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP freebsd.shrew.l 64G 628k 99 1.6g 98 831m 60 1278k 99 1.1g 42 +++++ +++ Latency 13447us 68490us 207ms 7187us 195ms 17665us Version 1.98 ------Sequential Create------ --------Random Create-------- freebsd.shrew.lab -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency 18225us 18us 28us 18812us 18us 25us End @ Sun Feb 18 15:03:11 UTC 2024 I used identical options to run both that VM and the Linux VM I've been testing. The backing store for each VM has a 1TB partition and the guest interfaces are NVME. Now I'm really scratching my head. Chuck, were you testing disk performance in Linux VMs or only FreeBSD? Anyone have ideas on why Linux disk performance would drop off a cliff over time? Thanks, Thanks, -Matthew