Virtualization on FreeBSD Host and Oracle VirtualBox Guests with VDE Networking: Supratim Sanyal's Computing Blog

Supratim Sanyal's Blog: Oracle VirtualBox Guest Networking using VDE / VDE2 (Virtual Distributed Ethernet)

I described the basic setup for VirtualBox virtualization on FreeBSD 12-RELEASE as a host in an earlier post. Here are some notes on setting up VDE (Virtual Distributed Ethernet) guest networking with Oracle VirtualBox hypervisor version 5.2.32 running on FreeBSD 12.0-RELEASE-p10.

vde2 User-mode virtual ethernet infrastructure was installed from FreeBSD binary package repository using "sudo pkg install vde2". Keep in mind vde2 should be installed, not the "vde" package which is also available.

VirtualBox was built from the FreeBSD ports sources using "cd /usr/ports/emulators/virtualbox-ose/ && make install clean" with VDE networking option selected from the configuration screens produced by the build script.

The following setup script vde-switch.sh is invoked at boot time from /etc/rc.local to set the VDE switch and tap interfaces up:

	#!/bin/sh

	# ------------------------------
	# vde-switch.sh
	# Setup VDE switch and bridged tap interfaces
	# FreeBSD 12-RELEASE
	# Uses vde2 package
	#
	# *****
	# MUST RUN AS ROOT!
	# *****
	#
	# More information:
	# http://bit.ly/freebsd-virtualbox
	#
	# Basically does this:
	#
	# +---------+ +----+
	# \|VDE \|---\|tap0\| (VDE tap, has local IP)
	# \|SWITCH \| +----+
	# \|"vde-ip" \| +-----------------------------------+
	# \| \|===========\| +-----------------------------------+
	# +---------+ \| +-----------------------------------+
	# \| \| Oracle VirtualBox Virtual Machines\|
	# +----+ +-----------------------------------+
	# \|tap1\| (VDE Plug)
	# +----+
	# \|
	# \| +--------+
	# +----------\|bridge \|
	# \|bridge0 \|
	# +--------+
	# \| +--------+
	# +--------\| tap2 \| <--> QEMU SliTaz Linux
	# \| +--------+
	# \|
	# \| +--------+
	# +--------\| tap3 \| <--> QEMU Debian/HP parisc
	# \| +--------+
	# \|
	# \| +--------+
	# +--------\| tap4 \| <--> QEMU HPUX 11.11/HP parisc
	# \| +--------+
	# \|
	# \| +--------+
	# +--------\| tap5 \| <--> QEMU XENIX 386
	# \| +--------+
	# \|
	# \| +--------+
	# +--------\| tap6 \| <--> QEMU HPUX 10.20/HP parisc
	# +--------+
	# \|
	# \| +--------+
	# +--------\| tap7 \| <--> Future Emulator
	# +--------+
	# \|
	# \| +--------+
	# +--------\| tap8 \| <--> Future Emulator
	# +--------+
	# \|
	# \| +--------+
	# +--------\| tap9 \| <--> Future Emulator
	# +--------+
	#
	# ---
	# /boot/loader.conf needs:
	# if_bridge_load="YES"
	# if_tap_load="YES"
	# kqemu_load="YES"
	# vboxdrv_load="YES"
	#
	# --
	# /etc/rc.conf needs:
	# kqemu_enable="YES"
	# vboxnet_enable="YES"
	#
	# ---
	# Example usage of tap2 interface from QEMU (SliTaz Linux virtual machine):
	# qemu-system-x86_64 -m 1024 -hda slitaz.qcow2.4GB.disk -boot c -net nic -net tap,ifname=tap2,script=no -nographic -vnc :0
	#
	# ---
	# Example usage of VDE switch in Oracle VirtualBox compiled with VDE support:
	# vboxmanage modifyvm TrueOS --nic1 generic
	# vboxmanage modifyvm TrueOS --nicgenericdrv1 VDE
	# vboxmanage modifyvm TrueOS --nicproperty1 network=/tmp/vde-ip.ctl
	#
	# ---
	# Supratim Sanyal
	# supratim at riseup dot net
	# http://tuklusan.decsystem.org/
	# ------------------------------

	ifconfig tap0 create
	chmod 0666 /dev/tap0
	ifconfig tap0 10.42.2.252/24 promisc up

	ifconfig tap1 create
	chmod 0666 /dev/tap1
	ifconfig tap1 0.0.0.0 promisc up

	/usr/local/bin/vde_switch -t tap0 -s /tmp/vde-ip.ctl -m 666 --mgmt /tmp/vde-ip.mgmt --mgmtmode 666 --daemon --fstp
	/usr/local/bin/vde_plug2tap -s /tmp/vde-ip.ctl -m 666 -d tap1

	sysctl net.link.tap.user_open=1
	sysctl net.link.tap.up_on_open=1

	# ---
	# Now create a bridge, and bridge vdeplug "tap1" to additional "tap2" ... "tapx" interfaces for
	# qemu or other emulators that use tap
	# ---

	ifconfig tap2 create
	ifconfig tap3 create
	ifconfig tap4 create
	ifconfig tap5 create
	ifconfig tap6 create
	ifconfig tap7 create
	ifconfig tap8 create
	ifconfig tap9 create
	chmod 0666 /dev/tap2
	chmod 0666 /dev/tap3
	chmod 0666 /dev/tap4
	chmod 0666 /dev/tap5
	chmod 0666 /dev/tap6
	chmod 0666 /dev/tap7
	chmod 0666 /dev/tap8
	chmod 0666 /dev/tap9
	ifconfig tap2 0.0.0.0 promisc up
	ifconfig tap3 0.0.0.0 promisc up
	ifconfig tap4 0.0.0.0 promisc up
	ifconfig tap5 0.0.0.0 promisc up
	ifconfig tap6 0.0.0.0 promisc up
	ifconfig tap7 0.0.0.0 promisc up
	ifconfig tap8 0.0.0.0 promisc up
	ifconfig tap9 0.0.0.0 promisc up

	ifconfig bridge0 create
	ifconfig bridge0 addm tap1 up
	ifconfig bridge0 addm tap2
	ifconfig bridge0 addm tap3
	ifconfig bridge0 addm tap4
	ifconfig bridge0 addm tap5
	ifconfig bridge0 addm tap6
	ifconfig bridge0 addm tap7
	ifconfig bridge0 addm tap8
	ifconfig bridge0 addm tap9

	sync;sync;sync
	# EOF

view raw vde-switch FreeBSD.sh hosted with ❤ by GitHub

It is to be noted that on FreeBSD, rc.local should source /etc/rc.conf first, i.e. the following should appear first before invoking other startup scripts:
. /etc/rc.conf

Once the VDE switch is created, the Oracle VirtualBox virtual machines are configured to use the VDE switch. This is accomplished by a series of "vboxmanage modifyvm" commands, as in the example below which configured my IBM OS/2 Warp 4.5 for eBusiness virtual machine:

$ vboxmanage modifyvm OS2-Warp-4.5-Server --nic1 generic
$ vboxmanage modifyvm OS2-Warp-4.5-Server --nicgenericdrv1 VDE
$ vboxmanage modifyvm OS2-Warp-4.5-Server --nicproperty1 network=/tmp/vde-ip.ctl

To identify the VDE switch to be used for virtual machine networking, the "network=" specification in the last command above uses the same socket specified in the "-s" switch when setting up the VDE switch at boot time.

After configuring the VirtualBox virtual appliance for VDE using the commands, the VirtualBox GUI settings show the networking configuration as the screenshot at the top of this post.

It may be possible to use the GUI and choose the VDE switch-related parameters graphically, but I have not tried that personally.

VDE networking for VirtualBox appliances works flawlessly once configured. I have multiple Virtualization hosts and connect all the VirtualBox virtual machines across all of them to one logical Ethernet switch using VDE's Ethernet-over-SSL tunneling capability. A typical command line to set up a Ethernet-over-SSL tunnel to another VM host looks like:

/usr/local/bin/dpipe /usr/local/bin/vde_plug /tmp/vde-ip.ctl = ssh vde0@$REMOTE vde_plug /tmp/vde-ip.ctl

For the above to work, obviously password-less SSH login to the "vde0" account on the remote host specified by the REMOTE environment variable had to be set up.

I also usually enable VDE switch's fstp (Fast Spanning Tree Protocol) on all of my VDE switches in the hope of avoiding Ethernet packet flooding-related Network issues. It appears once in a while the VDE port that is the end-point of the SSL tunnel to the remote VDE switch is designated "root" sporadically, potentially impacting connectivity to other VDE switches. When this happens, I manually change the SSL tunnel source port to and edge port using the "fstp/setedge" command from unixterm (the control program for VDE switch). Here is a typical example of such a session, where port 3 had transitioned to a "root" port which I force back to edge port:

$ unixterm /tmp/vde-ip.mgmt

VDE switch V.2.3.2

vde$ port/allprint

0000 DATA END WITH '.'

Port 0001 untagged_vlan=0000 ACTIVE - Unnamed Allocatable

Current User: NONE Access Control: (User: NONE - Group: NONE)

-- endpoint ID 0007 module tuntap : tap0

Port 0002 untagged_vlan=0000 ACTIVE - Unnamed Allocatable

Current User: root Access Control: (User: NONE - Group: NONE)

-- endpoint ID 0003 module unix prog : vde_plug: user=root PID=1043

Port 0003 untagged_vlan=0000 ACTIVE - Unnamed Allocatable