From nobody Tue Sep 17 21:21:46 2024 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X7ZV2357Xz5W0J8 for ; Tue, 17 Sep 2024 21:21:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X7ZV21fQQz4HCl for ; Tue, 17 Sep 2024 21:21:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1726608106; a=rsa-sha256; cv=none; b=Sbt5icVONXsA5gVT6zrcmOwVZiHr5LBLR/J0Vt9TajmhM5JkI93rHKaBbPcaig/s5FPr3I 2phjXVi2LS1xKgkf5ZCI+DpOuZzRtVmTIjyJdiMiAInRc8/s6hrV6Fn0TNcNGyvq4zC1qe biAtpz+CWZ9SU8SN5m+vnVthTRdwlxLpPb7/tv4qJlyJJ11HbNMa4T7NRy8F+8By7sd+ZN jJkWXvRAQJGk8ElGE/PNUqhq27mjozbsKQFuAS4YEmd9p1ybuG73vgT3rof1fcqHGBNrdv PS7m5rTBs022Er2UFxE9WAdKhNviS6yAsRBBSkECB9uDVvFoTMTjdxi6CJidTA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1726608106; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=D5jCXyt2MkOgpeaGVTlq3tMShqxVen9sSGckZNDBxEY=; b=OlNAtRgp+hTq+3nlCrXgQVQ0mFpKYikKdAFwPLLHOw4E8b/bcuOEtD4Z0Z0FkJ1DiLo0pd 68v3S5LJPC2Hp05/PboqGxkwfbQmYO6i0KenkxkQCXH2czx+Sj9hejTxy+CGMRBk9jvBlt 79ZI7VlpKs7nzP9nbMKY5wdUBFvLZ2dNcVQOIXMI2hLFROI2QgPCqyjhuo/gueulN8nIJJ brAqEmfH5kDkN7q0nX1+u4EubL22I9NcrKY2Afjd4q7KKYmKP8udPB8zCHWipnwU2Vh2N1 sa18f1BZv8AwPnWzIP6J2llyE/Y6sYOy5o12Uc9B4OmRiP8iADCfq2wykewi9A== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X7ZV21G91zv7j for ; Tue, 17 Sep 2024 21:21:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 48HLLkFV099640 for ; Tue, 17 Sep 2024 21:21:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 48HLLkOC099639 for bugs@FreeBSD.org; Tue, 17 Sep 2024 21:21:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 281560] gve (4) uma deadlock during high tcp throughput Date: Tue, 17 Sep 2024 21:21:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 14.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: shailend@google.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D281560 Bug ID: 281560 Summary: gve (4) uma deadlock during high tcp throughput Product: Base System Version: 14.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: shailend@google.com In https://reviews.freebsd.org/D46690 I am adding support for a newer queue format to gve that supports ~200Gbps of throughput.=20=20 While stress testing these changes I ran into a deadlock that I'm unable to find a root cause for. This is reproduced by having two VMs sending bidirectional tcp traffic to each other, and the deadlock sets in after a f= ew hours.=20=20 I am listing the stack traces of the driver taskqueues after the deadlock s= ets in for two different occurrences of the deadlock: ``` [root@FreeBSD14 ~]# for locktid in $(procstat -ta | grep gve | grep lock | = cut -d" " -f6); do procstat -k $locktid | grep $locktid; done 0 100416 kernel gve0 rxq 0 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100417 kernel gve0 rxq 1 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100418 kernel gve0 rxq 2 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100419 kernel gve0 rxq 3 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100420 kernel gve0 rxq 4 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100421 kernel gve0 rxq 5 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_active=20 0 100422 kernel gve0 rxq 6 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100423 kernel gve0 rxq 7 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100424 kernel gve0 rxq 8 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100425 kernel gve0 rxq 9 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_active=20 0 100426 kernel gve0 rxq 10 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100427 kernel gve0 rxq 11 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100428 kernel gve0 rxq 12 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done=20 0 100429 kernel gve0 rxq 13 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100430 kernel gve0 rxq 14 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done tcp_lro_flush_all=20 0 100431 kernel gve0 rxq 15 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_rx_done tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100434 kernel gve0 txq 1 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100448 kernel gve0 txq 8 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100456 kernel gve0 txq 12 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100458 kernel gve0 txq 13 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 ```=20=20 ``` [root@FreeBSD14 ~]# for locktid in $(procstat -ta | grep gve | grep lock | = cut -d" " -f6); do procstat -k $locktid | grep $locktid; done 0 100433 kernel gve0 rxq 0 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100434 kernel gve0 rxq 1 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100435 kernel gve0 rxq 2 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100436 kernel gve0 rxq 3 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100437 kernel gve0 rxq 4 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100438 kernel gve0 rxq 5 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_default_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo=20 0 100439 kernel gve0 rxq 6 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100440 kernel gve0 rxq 7 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100441 kernel gve0 rxq 8 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_= all gve_rx_cleanup_tq_dqo taskqueue_run_locked=20 0 100442 kernel gve0 rxq 9 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_= all gve_rx_cleanup_tq_dqo taskqueue_run_locked=20 0 100443 kernel gve0 rxq 10 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100444 kernel gve0 rxq 11 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100445 kernel gve0 rxq 12 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100446 kernel gve0 rxq 13 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100447 kernel gve0 rxq 14 mi_switch __rw_wlock_h= ard _rw_wlock_cookie _inp_smr_lock in_pcblookup_hash_smr in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_= all gve_rx_cleanup_tq_dqo taskqueue_run_locked=20 0 100448 kernel gve0 rxq 15 mi_switch __mtx_lock_s= leep __mtx_lock_flags cache_alloc cache_alloc_retry tcp_m_copym tcp_default_outp= ut tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_all=20 0 100457 kernel gve0 txq 4 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100463 kernel gve0 txq 7 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100465 kernel gve0 txq 8 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100467 kernel gve0 txq 9 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100469 kernel gve0 txq 10 mi_switch __mtx_lock_s= leep __mtx_lock_flags zone_put_bucket cache_free uma_zfree_arg mb_free_ext m_free m_freem gve_handle_packet_completion gve_tx_cleanup_tq_dqo taskqueue_run_lo= cked taskqueue_thread_loop fork_exit fork_trampoline=20 ```=20=20 I tried to look into any incorrect kpi use in the driver (https://reviews.freebsd.org/D46690) but couldn't find any offenses. It is rather straightforward: in the Rx paths it inputs cluster mbufs and m_freems transmitted mbufs in the Tx path. Thus I'm hesitantly suspecting some race condition in the uma stack given the traces.=20=20 I reproduced the issue with INVARIANTS and WITNESS but they didn't catch anything.=20 The issue also reproduces with a slightly different datapath mode in the dr= iver (https://reviews.freebsd.org/D46691). In this mode, it is not cluster mbufs that are input in the Rx path usually, but rather mbufs with external stora= ge attached. The stack traces of the driver taskqueues after the deadlock look like so: ``` [root@FreeBSD13 ~]# for locktid in $(procstat -ta | grep gve | grep lock | = cut -d" " -f6); do procstat -k $locktid | grep $locktid; done 0 100451 kernel gve0 rxq 1 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100452 kernel gve0 rxq 2 mi_switch __mtx_lock_s= leep cache_alloc cache_alloc_retry m_get2 gve_rx_cleanup_tq_dqo taskqueue_run_lo= cked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100453 kernel gve0 rxq 3 mi_switch __mtx_lock_s= leep cache_alloc cache_alloc_retry m_get2 gve_rx_cleanup_tq_dqo taskqueue_run_lo= cked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100454 kernel gve0 rxq 4 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100455 kernel gve0 rxq 5 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100457 kernel gve0 rxq 7 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100458 kernel gve0 rxq 8 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100459 kernel gve0 rxq 9 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg mb_free_ext m_free m_freem tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush=20 0 100460 kernel gve0 rxq 10 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100461 kernel gve0 rxq 11 mi_switch __mtx_lock_s= leep cache_alloc cache_alloc_retry m_get2 gve_rx_cleanup_tq_dqo taskqueue_run_lo= cked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100462 kernel gve0 rxq 12 mi_switch __mtx_lock_s= leep cache_alloc cache_alloc_retry tcp_m_copym tcp_output tcp_do_segment tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_input tcp_lro_flush tcp_lro_flush_= all gve_rx_cleanup_tq_dqo=20 0 100463 kernel gve0 rxq 13 mi_switch __mtx_lock_s= leep cache_alloc cache_alloc_retry m_get2 gve_rx_cleanup_tq_dqo taskqueue_run_lo= cked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100464 kernel gve0 rxq 14 mi_switch __rw_wlock_h= ard in_pcblookup_hash in_pcblookup_mbuf tcp_input_with_port tcp_input ip_input netisr_dispatch_src ether_demux ether_nh_input netisr_dispatch_src ether_in= put tcp_lro_flush tcp_lro_flush_all gve_rx_cleanup_tq_dqo taskqueue_run_locked taskqueue_thread_loop fork_exit=20 0 100465 kernel gve0 rxq 15 mi_switch __mtx_lock_s= leep cache_alloc cache_alloc_retry m_get2 gve_rx_cleanup_tq_dqo taskqueue_run_lo= cked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100466 kernel gve0 txq 0 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100468 kernel gve0 txq 1 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100470 kernel gve0 txq 2 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100472 kernel gve0 txq 3 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100474 kernel gve0 txq 4 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100480 kernel gve0 txq 7 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100482 kernel gve0 txq 8 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100484 kernel gve0 txq 9 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100486 kernel gve0 txq 10 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100488 kernel gve0 txq 11 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100490 kernel gve0 txq 12 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100492 kernel gve0 txq 13 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100494 kernel gve0 txq 14 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 0 100496 kernel gve0 txq 15 mi_switch __mtx_lock_s= leep zone_put_bucket cache_free uma_zfree_arg m_free m_freem gve_tx_cleanup_tq_d= qo taskqueue_run_locked taskqueue_thread_loop fork_exit fork_trampoline=20 ``` I've reproduced this in 14-1 and 13-1.=20 Any guidance or help in uncovering the source of this deadlock would be very helpful! --=20 You are receiving this mail because: You are the assignee for the bug.=