RE: bhyve terribly slow when using tagged vlan

From: <jason_at_tubnor.net>
Date: Fri, 14 Jan 2022 23:34:23 UTC
Try:

ifconfig ixl1 -lro -tso

Observations that these options active on a stacked VLAN interface causes
issues. Typically when a VLAN interface is pushed into a bridge, these
should be stripped, but are not.

Cheers,

Jason.

> -----Original Message-----
> From: owner-freebsd-virtualization@freebsd.org <owner-freebsd-
> virtualization@freebsd.org> On Behalf Of Daniel Braniss
> Sent: Friday, 14 January 2022 9:50 PM
> To: freebsd-virtualization@freebsd.org
> Subject: bhyve terribly slow when using tagged vlan
> 
> hi,
> the host is a very new Dell PowerEdge R6525 running FreeBSD 13.0, I have
> connected ixl0 to an untagged link and ixl1 to a tagged link
> 
> I have created 2 bhyve clients, identical, except for the links, one is
using ixl0
> and the other ixl1.340
> 
> the untagged boots diskless in about 40 seconds, while the one using the
> tagged one takes 12 minutes!
> 
> I compared their dmesgs and they are identical (except for minor diffs
like
> mac address)
> 
> any ideas?
> 
> danny
> 
> some more data:
> 
> sa# uname -a
> FreeBSD sa 13.0-STABLE FreeBSD 13.0-STABLE #4 huji-13-736697896: Mon Dec
> 13 14:03:04 IST 2021     danny@hms-00:/home/obj/hms-
> 00/h/rnd/git/stable/13/amd64.amd64/sys/HUJI  amd64
> sa# ifconfig
> ixl0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> 
> options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,
> VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
>        ether b4:96:91:97:08:14
>        inet 132.65.116.32 netmask 0xfffff000 broadcast 132.65.127.255
>        media: Ethernet autoselect (10Gbase-T <full-duplex>)
>        status: active
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> ixl1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> 
> options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMB
> O_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,R
> XCSUM_IPV6,TXCSUM_IPV6,NOMAP>
>        ether b4:96:91:97:08:15
>        media: Ethernet autoselect (10Gbase-T <full-duplex>)
>        status: active
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 
> options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_H
> WCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>        ether b0:7b:25:dd:ff:9c
>        media: Ethernet autoselect
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
> 
> options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_H
> WCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
>        ether b0:7b:25:dd:ff:9d
>        media: Ethernet autoselect
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> 
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV
> 6>
>        inet6 ::1 prefixlen 128
>        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
>        inet 127.0.0.1 netmask 0xff000000
>        groups: lo
>        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ixl1.340:
> flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
>        options=4200401<RXCSUM,LRO,RXCSUM_IPV6,NOMAP>
>        ether b4:96:91:97:08:15
>        inet 132.65.80.251 netmask 0xfffff000 broadcast 132.65.95.255
>        groups: vlan
>        vlan: 340 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl1
>        media: Ethernet autoselect (10Gbase-T <full-duplex>)
>        status: active
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        ether b0:7b:25:e5:dc:57
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> vm-servers: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
>        ether 0a:91:de:0e:f6:26
>        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>                ifmaxaddr 0 port 10 priority 128 path cost 2000000
>        member: ixl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>                ifmaxaddr 0 port 1 priority 128 path cost 2000
>        groups: bridge vm-switch viid-d5539@
>        nd6 options=9<PERFORMNUD,IFDISABLED>
> vm-system: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
>        ether 42:5b:f8:8f:72:00
>        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> 		                ifmaxaddr 0 port 11 priority 128 path cost
2000000
>        member: ixl1.340 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>                ifmaxaddr 0 port 6 priority 128 path cost 2000
>        groups: bridge vm-switch viid-54b53@
>        nd6 options=9<PERFORMNUD,IFDISABLED>
> tap0:
> flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
>        description: vmnet-bhv-08-0-servers
>        options=80000<LINKSTATE>
>        ether 58:9c:fc:10:82:13
>        groups: tap vm-port
>        media: Ethernet autoselect
>        status: active
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>        Opened by PID 9357
> tap1:
> flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
>        description: vmnet-bhv-09-0-system
>        options=80000<LINKSTATE>
>        ether 58:9c:fc:10:ff:a2
>        groups: tap vm-port
>        media: Ethernet autoselect
>        status: active
>        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>