Limited number of vale switch useable with bhyve ?
Olivier Cochard-Labbé
olivier at cochard.me
Mon Mar 14 12:52:47 UTC 2016
Hi,
Thanks to btw's patch I can now use bhyve and vale.
But I'm hitting a limitation:
I want to start 5 VM connected by a full-meshed network.
This mean:
- VM 1 has 4 interfaces toward each other 4 VMs
- and so one for all others 4 VMs
This give these 5 command lines:
bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \
-s 2:0,virtio-net,vale12:BSDRP_1,mac=58:9c:fc:01:02:01 \
-s 2:1,virtio-net,vale13:BSDRP_1,mac=58:9c:fc:01:03:01 \
-s 2:2,virtio-net,vale14:BSDRP_1,mac=58:9c:fc:01:04:01 \
-s 2:3,virtio-net,vale15:BSDRP_1,mac=58:9c:fc:01:05:01 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_1 \
-l com1,/dev/nmdm1A BSDRP_1
bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \
-s 2:0,virtio-net,vale15:BSDRP_5,mac=58:9c:fc:01:05:05 \
-s 2:1,virtio-net,vale25:BSDRP_5,mac=58:9c:fc:02:05:05 \
-s 2:2,virtio-net,vale35:BSDRP_5,mac=58:9c:fc:03:05:05 \
-s 2:3,virtio-net,vale45:BSDRP_5,mac=58:9c:fc:04:05:05 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_5 \
-l com1,/dev/nmdm5A BSDRP_5
bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc \
-s 2:0,virtio-net,vale12:BSDRP_2,mac=58:9c:fc:01:02:02 \
-s 2:1,virtio-net,vale23:BSDRP_2,mac=58:9c:fc:02:03:02 \
-s 2:2,virtio-net,vale24:BSDRP_2,mac=58:9c:fc:02:04:02 \
-s 2:3,virtio-net,vale25:BSDRP_2,mac=58:9c:fc:02:05:02 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_2 \
-l com1,/dev/nmdm2A BSDRP_2
bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc
-s 2:0,virtio-net,vale13:BSDRP_3,mac=58:9c:fc:01:03:03 \
-s 2:1,virtio-net,vale23:BSDRP_3,mac=58:9c:fc:02:03:03 \
-s 2:2,virtio-net,vale34:BSDRP_3,mac=58:9c:fc:03:04:03 \
-s 2:3,virtio-net,vale35:BSDRP_3,mac=58:9c:fc:03:05:03 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_3 \
-l com1,/dev/nmdm3A BSDRP_3
bhyve -c 1 -m 256M -A -H -P -s 0:0,hostbridge -s 0:1,lpc
-s 2:0,virtio-net,vale14:BSDRP_4,mac=58:9c:fc:01:04:04 \
-s 2:1,virtio-net,vale24:BSDRP_4,mac=58:9c:fc:02:04:04 \
-s 2:2,virtio-net,vale34:BSDRP_4,mac=58:9c:fc:03:04:04 \
-s 2:3,virtio-net,vale45:BSDRP_4,mac=58:9c:fc:04:05:04 \
-s 1:0,virtio-blk,/tmp/BSDRP/BSDRP_4
-l com1,/dev/nmdm4A BSDRP_4
But host system complains it didn't reach to allocate memory,:
743.527464 nm_open [608] NIOCREGIF failed: Cannot allocate memory
vale35:BSDRP_5
743.527470 nm_open [608] NIOCREGIF failed: Cannot allocate memory
vale45:BSDRP_4
open of netmap device vale35:BSDRP_5 failed
open of netmap device vale45:BSDRP_4 failed
743.561324 nm_open [608] NIOCREGIF failed: Cannot allocate memory
vale45:BSDRP_5
open of netmap device vale45:BSDRP_5 failed
743.561359 nm_open [608] NIOCREGIF failed: Cannot allocate memory
vale35:BSDRP_3
open of netmap device vale35:BSDRP_3 failed
And the vtnet interfaces that failed to allocate memory are correctly in
"status: no carrier mode" in the guests.
But the host system has plenty of free RAM:
root at lame5:~ # top -b
last pid: 2242; load averages: 0.09, 0.24, 0.22 up 0+04:24:44
13:23:20
64 processes: 1 running, 63 sleeping
Mem: 556M Active, 94M Inact, 5645M Wired, 243G Free
ARC: 1273M Total, 461M MFU, 805M MRU, 16K Anon, 3117K Header, 3332K Other
Swap: 4096M Total, 4096M Free
Can I increase some sysctl dev.netmap for allowing more memory to be used
by netmap/vale ?
I've tried to multiply per 10 dev.netmap.buf_num, but no changes.
Thanks
More information about the freebsd-virtualization
mailing list