svn commit: r361944 - in head/sys/dev/virtio: . network

[Jessica Clarke]

On 14 Jun 2020, at 22:22, Tom Jones <thj at freebsd.org> wrote:
> On Sun, Jun 14, 2020 at 09:56:03PM +0100, Jessica Clarke wrote:
>> On 14 Jun 2020, at 20:51, Tom Jones <thj at freebsd.org> wrote:
>>> On Mon, Jun 08, 2020 at 09:51:36PM +0000, Jessica Clarke wrote:
>>>> Author: jrtc27
>>>> Date: Mon Jun  8 21:51:36 2020
>>>> New Revision: 361944
>>>> URL: https://svnweb.freebsd.org/changeset/base/361944
>>>> 
>>>> Log:
>>>> virtio: Support non-legacy network device and queue
>>>> 
>>>> The non-legacy interface always defines num_buffers in the header,
>>>> regardless of whether VIRTIO_NET_F_MRG_RXBUF, just leaving it unused. We
>>>> also need to ensure our virtqueue doesn't filter out VIRTIO_F_VERSION_1
>>>> during negotiation, as it supports non-legacy transports just fine. This
>>>> fixes network packet transmission on TinyEMU.
>>>> 
>>>> Reviewed by:	br, brooks (mentor), jhb (mentor)
>>>> Approved by:	br, brooks (mentor), jhb (mentor)
>>>> Differential Revision:	https://reviews.freebsd.org/D25132
>>>> 
>>>> Modified:
>>>> head/sys/dev/virtio/network/if_vtnet.c
>>>> head/sys/dev/virtio/network/if_vtnetvar.h
>>>> head/sys/dev/virtio/virtio.c
>>>> head/sys/dev/virtio/virtqueue.c
>>>> 
>>> 
>>> Hi Jessica,
>>> 
>>> After updating my current bhyve vm today (on a 12.1 host), networking no longer
>>> works. Reverting this commit seems to resolve the issue. I think vtnet is not
>>> passing enough data up to the ip layer.
>>> 
>>> If I capture on the tap interface for the vm I see arp requests and arp
>>> replies, however kern.msgbuf is full of: 
>>> 
>>> <5>arp: short packet received on vtnet0
>>> 
>>> and netstat does not see any replies to arp requests:
>>> 
>>> root at freebsd-current:~ # netstat -s -p arp
>>> arp:
>>>       11 ARP requests sent
>>>       0 ARP requests failed to sent
>>>       0 ARP replies sent
>>>       0 ARP requests received
>>>       0 ARP replies received
>>>       0 ARP packets received
>>>       24 total packets dropped due to no ARP entry
>>>       2 ARP entrys timed out
>>>       0 Duplicate IPs seen
>>> 
>>> If I set up an arp entry manually I can see ICMP echo requests and responses on
>>> the tap interface, but the vm does not see the responses. 
>>> 
>>> root at freebsd-current:~ # netstat -s -p ip
>>> ip:
>>>       7 total packets received
>>>       0 bad header checksums
>>>       0 with size smaller than minimum
>>>       7 with data size < data length
>>>       0 with ip length > max ip packet size
>>>       0 with header length < data size
>>>       0 with data length < header length
>>> 
>>> The line
>>> 
>>>       7 with data size < data length
>>> 
>>> makes me think that vtnet is truncating packets. 
>>> 
>>> markj pointed me at this bug in irc which might also be related:
>>> 
>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247242
>> 
>> Hi Tom,
>> Sorry about that; it seems bhyve hits the "legacy and no MrgRxBuf"
>> case. Could you please try the patch below?
>> 
>> Jess
>> 
> 
> This changed fixed the issue for me. Please feel free to add 
> 
> Tested By: thj 
> 
> when you commit.

Great, thanks for the report.

> In testing I this lor went by, I wonder if this is something you care about:
> 
> acquiring duplicate lock of same type: "vtnet0-rx0"
> 1st vtnet0-rx0 @ /usr/home/tj/code/freebsd/projects/review-D25220/sys/dev/virtio/network/if_vtnet.c:1780
> 2nd vtnet0-rx0 @ /usr/home/tj/code/freebsd/projects/review-D25220/sys/kern/subr_taskqueue.c:281
> stack backtrace:
> #0 0xffffffff80c32881 at witness_debugger+0x71
> #1 0xffffffff80ba3e54 at __mtx_lock_flags+0x94
> #2 0xffffffff80c24bd2 at taskqueue_enqueue+0x42
> #3 0xffffffff80a1af99 at vtnet_rxq_tq_intr+0xb9
> #4 0xffffffff80c2520a at taskqueue_run_locked+0xaa
> #5 0xffffffff80c26284 at taskqueue_thread_loop+0x94
> #6 0xffffffff80b830e0 at fork_exit+0x80
> #7 0xffffffff81040eae at fork_trampoline+0xe

Hm, I think that's just a false-positive, because if_vtnet constructs
the taskqueue using the same name as its own internal mutexes. Though
the locking around vtnet_rx_vq_intr and vtnet_rxq_tq_intr is a bit
fishy given they're rather similar yet inconsistent. I would imagine
rxq->vtnrx_stats.vrxs_rescheduled is supposed to be protected by that
mutex, but wouldn't like to say whether taskqueue_enqueue needs to be.
Vincenzo, you recently touched code around there, perhaps you could be
persuaded to have a quick look?..

Jess