Properly setting up FreeBSD bridge to communicate with bhyve tap device member
Finally getting around to setting IP address on host bridge on home server so routing works properly. While here also troubleshoot DHCP on Ubuntu LXD bridged devices not getting DHCP IPv4 addresses.
For a while now, after adding a bridge device for VMs, I've simply added the default host physical NIC to the vm-bridge device automatically set up by vm-byhve. This is wrong, it still works for IPv4 but routes aren't properly propagated for IPv6 for VM tap devices attached to the bridge. In FreeBSD 16.0 this won't be allowed by default any more.
You're supposed to configure host IP address on the bridge device itself as per bridge(4):
To allow the host to communicate with bridge members, IP addresses should
be assigned to the if_bridge interface itself, not to the bridge's member
interfaces. Attempting to assign an IP address to a bridge member
interface, or add a member interface with an assigned IP address to a
bridge, will return an EINVAL (“Invalid argument”) error. For
compatibility with older releases where this was permitted, setting the
sysctl(8) variable net.link.bridge.member_ifaddrs to 1 will permit this
configuration. This sysctl variable will be removed in FreeBSD 16.0.
Setting up manual bridge
The new bridge0 device will now be the default router for the home network.As per bridge(4) manual, you need to enable auto_linklocal for ipv6, it isn't set automatically for bridge device.
This is straightforward and well documented. Following lines added to /etc/rc.conf
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.0.1 netmask 255.255.255.0 addm igc0 up"
ifconfig_bridge0_ipv6="inet6 -ifdisabled auto_linklocal -accept_rtadv"
ifconfig_bridge0_alias0="inet6 fdd5:xxxx:xxxx::1 prefixlen 64"
vm-bhyve switch
For <path to vm dir>/.config/system.conf I needed to change the switch to manual and reference bridge0 device. vm-bhyve will automatically add tap devices for each VM to the bridge on startup.
switch_list="bridge0"
type_bridge0="manual"
bridge_bridge0="bridge0"
And then for VM's I want on this bridge, to also change it appropriately in each vm's conf file.
network0_switch="bridge0"
Now all my VMs on FreeBSD are able to discover IPv6 route to the host.
Ubuntu LXC/docker DHCP issues
Not sure why my system never had an issue before, but apparently docker or lxd loads the br_netfilter module that enabled ipfilter for bridge traffic. Like my home server, my workstation also has a bridge0 device, I connect a few of the LXC containers I want available on my home network. And this was causing LXC and VMs attached to the bridge on my Linux workstation to not get DHCP announcements. A quick fix was to disable, this but will revisit this later as there must be good reasons for a change to this default setting.
In /etc/sysctl.d/99-bridge-firewall.conf:
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-arptables = 0