[Bug 278573] arm64: virtio-net-pci not work on qemu pcie.1 with gicv3 and its=on

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278573

            Bug ID: 278573
           Summary: arm64: virtio-net-pci not work on qemu pcie.1 with
                    gicv3 and its=on
           Product: Base System
           Version: 14.0-STABLE
          Hardware: arm64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: arm
          Assignee: freebsd-arm@FreeBSD.org
          Reporter: none@tmpfs.dev

Created attachment 250217
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=250217&action=edit
pciconf -lv outout

The virtio-net-pci network interface (eg. vtnet1) placed on not-root pcie bus
is unusable on arm64 FreeBSD as a qemu guest with virt platform, gic-version=3
and its=on.

There are 3 requirements to produce this bug.
1. The GIC version must be 3, "gic-version=3"; (vtnet works fine with
gic-version=2)
2. The ITS must be on, "its=on"; (vtnet works fine with its=off).
3. The virtio-net-pci must be placed on a QEMU PCI-PCI bridge instead of the
pcie.0 HOST-PCI root bus. virtio-net-pci works fine when it is placed at
virtio_pci1@pci0:0:4:0 but it becomes unusable on virtio_pci2@pci0:1:5:0.

This is an example command to produce this bug.

sudo qemu-system-aarch64 \
 -m 1024 \
 -nographic \
 -drive if=pflash,format=raw,file=efi.img,readonly=on \
 -drive if=pflash,format=raw,file=varstore.img \
 -device pci-bridge,id=pcie.1,chassis_nr=1,bus=pcie.0,addr=0xa \
 -device qemu-xhci,id=qemu-xhci,bus=pcie.0,addr=3 \
 -drive
if=none,format=qcow2,file=FreeBSD-14.0-RELEASE-arm64-aarch64.qcow2,id=hd0 \
 -device virtio-blk-pci,bus=pcie.0,addr=2,drive=hd0 \
 -netdev type=tap,id=net0,ifname=tap0,script=no,downscript=no \
 -device
virtio-net-pci,mac=02:01:07:00:00:10,netdev=net0,bus=pcie.0,addr=4,id=net0 \
 -netdev type=tap,id=net1,ifname=tap1,script=no,downscript=no \
 -device
virtio-net-pci,mac=02:01:07:00:00:11,netdev=net1,bus=pcie.1,addr=5,id=net1 \
 -cpu max -M virt,gic-version=3,its=on

To use KVM, replace `-cpu max -M virt,gic-version=3,its=on` with `-cpu host -M
virt,gic-version=3,accel=kvm,its=on`. KVM does not affect the result (make sure
your host is GICv3).

Manually set up tap0 and tap1 and add them to a host bridge br0.
(host) ifconfig br0 192.168.3.1/24
(vm) ifconfig vtnet0 up
(vm) ifconfig vtnet0 192.168.3.10/24
(vm) ifconfig vtnet1 up
(vm) ifconfig vtnet1 192.168.3.11/24

Actual result:
There will be two network interfaces vtnet0 and vtnet1. vtnet0 on
virtio_pci1@pci0:0:4:0 works fine, but vtnet1 on virtio_pci2@pci0:1:5:0 is
usable. You can ping 192.168.3.10 and get replies, but ping 192.168.3.11 gets
no reply.

This bug is reproducible on the following FreeBSD images:
- FreeBSD-13.2-RELEASE-arm64-aarch64.qcow2
- FreeBSD-14.0-RELEASE-arm64-aarch64.qcow2
- FreeBSD-15.0-CURRENT-arm64-aarch64-20240404-112783ebbc31-269103-disc1.iso

With the exactly same qemu configuration (except system image, of course), all
two network adapters works fine in Linux guest: Linux debian 6.1.0-20-arm64 #1
SMP Debian 6.1.85-1 (2024-04-11) aarch64 GNU/Linux.

I have tested on the two host platforms:
- qemu-system-aarch64 QEMU emulator version 8.2.2 on Linux ArchLinux
6.8.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 17 Apr 2024 15:20:28 +0000 x86_64
GNU/Linux
- qemu-system-aarch64 QEMU emulator version 8.1.5 (pve-qemu-kvm_8.1.5-4) on
Linux armbian 6.6.16-current-rockchip64 #1 SMP PREEMPT Fri Feb 23 08:25:28 UTC
2024 aarch64 GNU/Linux

About efi.img and varstore.img:
efi image is from qemu-efi-aarch64_2024.02-2_all.deb (you can get it from
https://packages.debian.org/sid/all/qemu-efi-aarch64/download), and expand it
to 64M, according to
https://ubuntu.com/server/docs/boot-arm64-virtual-machines-on-qemu.
varstore.img is 64M of zeros.

The descriptions of "gic-version=3" and "its=on" are available at
https://qemu-project.gitlab.io/qemu/system/arm/virt.html.

-- 
You are receiving this mail because:
You are the assignee for the bug.