From nobody Thu Aug 31 18:48:57 2023 X-Original-To: freebsd-wireless@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rc9Dk6P61z4rTfw for ; Thu, 31 Aug 2023 18:49:10 +0000 (UTC) (envelope-from jeremy.m.cox@gmail.com) Received: from mail-lj1-x233.google.com (mail-lj1-x233.google.com [IPv6:2a00:1450:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rc9Dj2gTkz4Yq2 for ; Thu, 31 Aug 2023 18:49:09 +0000 (UTC) (envelope-from jeremy.m.cox@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=AYw2utd2; spf=pass (mx1.freebsd.org: domain of jeremy.m.cox@gmail.com designates 2a00:1450:4864:20::233 as permitted sender) smtp.mailfrom=jeremy.m.cox@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2bccda76fb1so21720821fa.2 for ; Thu, 31 Aug 2023 11:49:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693507747; x=1694112547; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=jgE9rR/l9Rj7jgU27Niuq6o47qycnU2s37ZZNZvoOig=; b=AYw2utd2FgAkxuGQJX0PS9huakIgtYDc8MEomKT9KoyOOLDY6v9CUje9FB2HvobXzb OtljpxXMXAp4mm9OxATNWCaBqJrd0w9n1RR4cvsLFXtAlns4wDGHlk62Oe0v5tA4cn/d S6XW/gAcPlZ+I83M0cihv7aDm3TZx2xOXY6+5MNLnJL8XGQIu75AHTrdzE3ZuAzfatrO ghvQtnpk8IMYeFo+10zKHg3W98DzxqqZptIRJu+PYmh2CRJANSvYglTH2qpYmOH0J2Eh c7Wid7CeDty0U6D+n8Ar+kjS6l0qGrNnbaxdBkJaisOf8TCUbkUBdU8MiCQtjRJrHixZ oOaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693507747; x=1694112547; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=jgE9rR/l9Rj7jgU27Niuq6o47qycnU2s37ZZNZvoOig=; b=fpPjPHCBj6CSCMiOYAxl57VUdjuJ90+3iGaonB4tWjtqHVBl5/hQue9LIEt/rjb645 EqDRHSnU7UwJBqXXHQ0jM2N97JVFnWWHM6eySwVzBzqaT3b676+9qaOy1XLuMQP1nk3i VkhNkTV95lNt0CHgPOhxcw3hBiFccOwcWKwhxAZxHNX9Quvev1gaCNRP8nTSdWkrtxzS mdj7Y1mEGNBAQeNlZzKAkmrTSXDZS7scWv9vmJHJFQdZbdYA1iZSgUjeVVWj5dUu32Wv Am9edJ80287nmof7wIEu5FVIfuQtFB73Nte32Lvsw1ZGZygVM8O19N4LS9cc2F6Va6cj 24ZQ== X-Gm-Message-State: AOJu0YypoO9d+o3qN9fL8syqQYrwOd/AfUN0a8ZxcbuiG0exZ91Q9NFG YJM7UAj/2p15JsjDPhot/uXh+5r1FNyk9kvOgty3bzPO9k3+Jw== X-Google-Smtp-Source: AGHT+IFzPaCWMsquo/VPp74KpZIbsKyS81mwD3fwzJDHDgeb6J58mDcaE/rx3PlH8I0Law52r5Xyp4u/qapLyAgHwG0= X-Received: by 2002:ac2:5f93:0:b0:4fe:3a57:7c98 with SMTP id r19-20020ac25f93000000b004fe3a577c98mr76402lfe.32.1693507746545; Thu, 31 Aug 2023 11:49:06 -0700 (PDT) List-Id: Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-wireless List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-wireless@freebsd.org X-BeenThere: freebsd-wireless@freebsd.org MIME-Version: 1.0 From: Jeremy Date: Thu, 31 Aug 2023 13:48:57 -0500 Message-ID: Subject: rtw88 kernel panic with 14 stable and recent 13.2 stable To: freebsd-wireless@freebsd.org Content-Type: multipart/alternative; boundary="000000000000a844cd06043c7fa7" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.99 / 15.00]; URI_COUNT_ODD(1.00)[13]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.99)[-0.992]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-wireless@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::233:from]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; TAGGED_FROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[freebsd-wireless@freebsd.org] X-Rspamd-Queue-Id: 4Rc9Dj2gTkz4Yq2 --000000000000a844cd06043c7fa7 Content-Type: text/plain; charset="UTF-8" Hi all, After attempting a clean install of the 14 stable snapshot my rtw88 wireless card stopped working and the kernel panicked. It did connect to the wifi router properly and got an IP address, but any attempt to fetch distfiles or do a pkg bootstrap resulted in a kernel panic. So I attempted a clean install of a recent 13.2 stable snapshot (from August 25 2023) and it did the same thing. When I did a fresh install of 13.2 release, it started working again. I'm using a desktop with a Ryzen 4600G CPU and an RT8822CE wireless pci card. The kernel panics with a custom built kernelor the GENERIC kernel. The kernel panic: FreeBSD riotskates 13.2-STABLE FreeBSD 13.2-STABLE #0: Wed Aug 30 18:06:54 CDT 2023 root@riotskates:/usr/obj/usr/src/amd64.amd64/sys/VENUS amd64 panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666b000 GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD] Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.2". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666b000 cpuid = 8 time = 1693437565 KDB: stack backtrace: #0 0xffffffff8078181b at kdb_backtrace+0x6b #1 0xffffffff80734c02 at vpanic+0x152 #2 0xffffffff80734aa3 at panic+0x43 #3 0xffffffff80a33b1a at vm_fault+0x12ea #4 0xffffffff80a32751 at vm_fault_trap+0xb1 #5 0xffffffff80adee11 at trap_pfault+0x1f1 #6 0xffffffff80ab8618 at calltrap+0x8 #7 0xffffffff809954ad at linux_work_fn+0xed #8 0xffffffff80796687 at taskqueue_run_locked+0x187 #9 0xffffffff80797983 at taskqueue_thread_loop+0xc3 #10 0xffffffff806f10e2 at fork_exit+0x82 #11 0xffffffff80ab968e at fork_trampoline+0xe Uptime: 1m36s Dumping 701 out of 15684 MB:..3%..12%..21%..32%..42%..51%..62%..71%..83%..92% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=) at /usr/src/sys/kern/kern_shutdown.c:396 #2 0xffffffff807347bf in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484 #3 0xffffffff80734c6f in vpanic (fmt=, ap=ap@entry=0xfffffe01025e6b10) at /usr/src/sys/kern/kern_shutdown.c:923 #4 0xffffffff80734aa3 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:847 #5 0xffffffff80a33b1a in vm_fault_lookup (fs=0xfffffe01025e6b80) at /usr/src/sys/vm/vm_fault.c:842 #6 vm_fault (map=, vaddr=vaddr@entry=18446741879088656384, fault_type=2 '\002', fault_flags=fault_flags@entry=0, m_hold=m_hold@entry=0x0) at /usr/src/sys/vm/vm_fault.c:1477 #7 0xffffffff80a32751 in vm_fault_trap (map=, vaddr=vaddr@entry=18446741879088656392, fault_type=, fault_flags=fault_flags@entry=0, signo=0x0, ucode=0x0) at /usr/src/sys/vm/vm_fault.c:662 #8 0xffffffff80adee11 in trap_pfault (frame=0xfffffe01025e6d00, usermode=false, signo=, ucode=) at /usr/src/sys/amd64/amd64/trap.c:846 #9 #10 __skb_unlink (skb=0xfffffe0106e9f000, head=0xfffffe0104bfd0c0) at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:616 #11 skb_unlink (skb=0xfffffe0106e9f000, head=0xfffffe0104bfd0c0) at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:624 #12 rtw_c2h_work (work=) at /usr/src/sys/contrib/dev/rtw88/main.c:276 #13 0xffffffff809954ad in linux_work_fn (context=0xfffffe0104bfd0f8, pending=) at /usr/src/sys/compat/linuxkpi/common/src/linux_work.c:299 #14 0xffffffff80796687 in taskqueue_run_locked ( queue=queue@entry=0xfffff80006678600) at /usr/src/sys/kern/subr_taskqueue.c:514 #15 0xffffffff80797983 in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:826 #16 0xffffffff806f10e2 in fork_exit ( callout=0xffffffff807978c0 , arg=0xfffff80001a78640, frame=0xfffffe01025e6f40) at /usr/src/sys/kern/kern_fork.c:1094 #17 #18 0x4b061c04e8080000 in ?? () Backtrace stopped: Cannot access memory at address 0x6213040a04e8 (kgdb) The only things that jumped out at me were the __skb_unlink and skb_unlink in frames 10 and 11. Frame 12 mentions something about rtw_c2h_work, but the file main.c was last modified before 13.2 release came out. The other culprit, /usr/src/sys/compat/linuxkpicommon/include/linux/skbuff.h has been modified more recently. So I reverted stable back to commit 1b33afb7e88b2e36db2083c488de41dbe097ef49 on stable 13 and rebuilt and reinstalled it, and the wireless card seems to work again. I am able to use pkg update and fetch distfiles without a kernel panic. Essentially on the stable 13 branch commits: de40bc6f3d6bf8ab1a4b546630cc847a3b8c5113 LinuxKPI: skbuff.h: fix -Warray-bounds warnings and b30d9f2e5204b475cf79150542b758c3dab14e70 LinuxKPI: skbuff.h: add more (skeleton) functions used by wireless drivers were reverted (along with everything else after June 25th 2023) and it seems to work. I'm no kernel expert and there were quite a few commits for linux emulation and linuxkpi between then and now. But at least the wireless card is working again. BTW, I have to manually download these sources from github and copy them to a thumb drive because "git clone"ing a branch doesn't seem to work at all, it just downloads for 30 seconds and just sits and does nothing after that. Thanks for your time, Jeremy Cox ReplyForward --000000000000a844cd06043c7fa7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi all,

After attempting a clean install of the 14 st= able snapshot my rtw88 wireless card stopped working and the kernel panicke= d. It did connect to the wifi router properly and got an IP address, but an= y attempt to fetch distfiles or do a pkg bootstrap resulted in a kernel pan= ic. So I attempted a clean install of a recent 13.2 stable snapshot (from A= ugust 25 2023) and it did the same thing.

When I d= id a fresh install of 13.2 release, it started working again. I'm using= a desktop with a Ryzen 4600G CPU and an RT8822CE wireless pci card. The ke= rnel panics with a custom built kernelor the GENERIC kernel.=C2=A0

The kernel panic:

FreeBSD riotska= tes 13.2-STABLE FreeBSD 13.2-STABLE #0: Wed Aug 30 18:06:54 CDT 2023 =C2=A0= =C2=A0 root@riotskates:/usr/obj/usr/src/amd64.amd64/sys/VENUS =C2=A0amd64<= br>
panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666= b000

GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]
Copyright (C) 202= 3 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or la= ter <http= ://gnu.org/licenses/gpl.html>
This is free software: you are free= to change and redistribute it.
There is NO WARRANTY, to the extent perm= itted by law.
Type "show copying" and "show warranty"= ; for details.
This GDB was configured as "x86_64-portbld-freebsd13= .2".
Type "show configuration" for configuration details.=
For bug reporting instructions, please see:
<https://www.gnu.org/softwa= re/gdb/bugs/>.
Find the GDB manual and other documentation resour= ces online at:
=C2=A0 =C2=A0 <http://www.gnu.org/software/gdb/docum= entation/>.

For help, type "help".
Type "ap= ropos word" to search for commands related to "word"...
R= eading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib= /debug//boot/kernel/kernel.debug...

Unread portion of the kernel mes= sage buffer:
panic: vm_fault_lookup: fault on nofault entry, addr: 0xfff= ffe010666b000
cpuid =3D 8
time =3D 1693437565
KDB: stack backtrace= :
#0 0xffffffff8078181b at kdb_backtrace+0x6b
#1 0xffffffff80734c02 a= t vpanic+0x152
#2 0xffffffff80734aa3 at panic+0x43
#3 0xffffffff80a33= b1a at vm_fault+0x12ea
#4 0xffffffff80a32751 at vm_fault_trap+0xb1
#5= 0xffffffff80adee11 at trap_pfault+0x1f1
#6 0xffffffff80ab8618 at calltr= ap+0x8
#7 0xffffffff809954ad at linux_work_fn+0xed
#8 0xffffffff80796= 687 at taskqueue_run_locked+0x187
#9 0xffffffff80797983 at taskqueue_thr= ead_loop+0xc3
#10 0xffffffff806f10e2 at fork_exit+0x82
#11 0xffffffff= 80ab968e at fork_trampoline+0xe
Uptime: 1m36s
Dumping 701 out of 1568= 4 MB:..3%..12%..21%..32%..42%..51%..62%..71%..83%..92%

__curthread (= ) at /usr/src/sys/amd64/include/pcpu_aux.h:55
55 __asm("movq %%gs:%= P1,%0" : "=3Dr" (td) : "n" (offsetof(struct pcpu,<= br>(kgdb) #0 =C2=A0__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:= 55
#1 =C2=A0doadump (textdump=3D<optimized out>)
=C2=A0 =C2=A0 = at /usr/src/sys/kern/kern_shutdown.c:396
#2 =C2=A00xffffffff807347bf in = kern_reboot (howto=3D260)
=C2=A0 =C2=A0 at /usr/src/sys/kern/kern_shutdo= wn.c:484
#3 =C2=A00xffffffff80734c6f in vpanic (fmt=3D<optimized out&= gt;,
=C2=A0 =C2=A0 ap=3Dap@entry=3D0xfffffe01025e6b10) at /usr/src/sys/k= ern/kern_shutdown.c:923
#4 =C2=A00xffffffff80734aa3 in panic (fmt=3D<= unavailable>)
=C2=A0 =C2=A0 at /usr/src/sys/kern/kern_shutdown.c:847<= br>#5 =C2=A00xffffffff80a33b1a in vm_fault_lookup (fs=3D0xfffffe01025e6b80)=
=C2=A0 =C2=A0 at /usr/src/sys/vm/vm_fault.c:842
#6 =C2=A0vm_fault (m= ap=3D<optimized out>, vaddr=3Dvaddr@entry=3D18446741879088656384,
= =C2=A0 =C2=A0 fault_type=3D2 '\002', fault_flags=3Dfault_flags@entr= y=3D0,
=C2=A0 =C2=A0 m_hold=3Dm_hold@entry=3D0x0) at /usr/src/sys/vm/vm_= fault.c:1477
#7 =C2=A00xffffffff80a32751 in vm_fault_trap (map=3D<opt= imized out>,
=C2=A0 =C2=A0 vaddr=3Dvaddr@entry=3D18446741879088656392= , fault_type=3D<optimized out>,
=C2=A0 =C2=A0 fault_flags=3Dfault_= flags@entry=3D0, signo=3D0x0, ucode=3D0x0)
=C2=A0 =C2=A0 at /usr/src/sys= /vm/vm_fault.c:662
#8 =C2=A00xffffffff80adee11 in trap_pfault (frame=3D0= xfffffe01025e6d00,
=C2=A0 =C2=A0 usermode=3Dfalse, signo=3D<unavailab= le>, ucode=3D<unavailable>)
=C2=A0 =C2=A0 at /usr/src/sys/amd64= /amd64/trap.c:846
#9 =C2=A0<signal handler called>
#10 __skb_un= link (skb=3D0xfffffe0106e9f000, head=3D0xfffffe0104bfd0c0)
=C2=A0 =C2=A0= at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:616
#11 s= kb_unlink (skb=3D0xfffffe0106e9f000, head=3D0xfffffe0104bfd0c0)
=C2=A0 = =C2=A0 at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:624#12 rtw_c2h_work (work=3D<optimized out>)
=C2=A0 =C2=A0 at /usr/s= rc/sys/contrib/dev/rtw88/main.c:276
#13 0xffffffff809954ad in linux_work= _fn (context=3D0xfffffe0104bfd0f8,
=C2=A0 =C2=A0 pending=3D<optimized= out>)
=C2=A0 =C2=A0 at /usr/src/sys/compat/linuxkpi/common/src/linux= _work.c:299
#14 0xffffffff80796687 in taskqueue_run_locked (
=C2=A0 = =C2=A0 queue=3Dqueue@entry=3D0xfffff80006678600)
=C2=A0 =C2=A0 at /usr/s= rc/sys/kern/subr_taskqueue.c:514
#15 0xffffffff80797983 in taskqueue_thr= ead_loop (arg=3D<optimized out>)
=C2=A0 =C2=A0 at /usr/src/sys/ker= n/subr_taskqueue.c:826
#16 0xffffffff806f10e2 in fork_exit (
=C2=A0 = =C2=A0 callout=3D0xffffffff807978c0 <taskqueue_thread_loop>,
=C2= =A0 =C2=A0 arg=3D0xfffff80001a78640, frame=3D0xfffffe01025e6f40)
=C2=A0 = =C2=A0 at /usr/src/sys/kern/kern_fork.c:1094
#17 <signal handler call= ed>
#18 0x4b061c04e8080000 in ?? ()
Backtrace stopped: Cannot acce= ss memory at address 0x6213040a04e8
(kgdb)

The = only things that jumped out at me were the __skb_unlink and skb_unlink in f= rames 10 and 11. Frame 12 mentions something about rtw_c2h_work, but the fi= le main.c was last modified before 13.2 release came out. The other culprit= ,=C2=A0 /usr/src/sys/compat/linuxkpicommon/include/linux/skbuff.h has been = modified more recently. So I reverted stable back to commit 1b33afb7e88b2e3= 6db2083c488de41dbe097ef49 on stable 13 and rebuilt and reinstalled it, and = the wireless card seems to work again. I am able to use pkg update and fetc= h distfiles without a kernel panic.=C2=A0

Esse= ntially on the stable 13 branch commits:

de40bc6f3= d6bf8ab1a4b546630cc847a3b8c5113=C2=A0=C2=A0LinuxKPI: skbuff.h: fix -War= ray-bounds warnings=C2=A0 =C2=A0=C2=A0= and

b30d9f2e5204b475cf79150542b758c3dab14e70=C2=A0=C2= =A0Lin= uxKPI: skbuff.h: add more (skeleton) functions used by wireless drivers

were reverted (along with everything else after June 25th 2023) and = it seems to work. I'm no kernel expert and there were quite a few commi= ts for linux emulation and linuxkpi between then and now. But at least the = wireless card is working again. BTW, I have to manually download these sour= ces from github and copy them to a thumb drive because "git clone"= ;ing a branch doesn't seem to work at all, it just downloads for 30 sec= onds and just sits and does nothing after that.=C2=A0
Jeremy Cox
Reply= Forward
--000000000000a844cd06043c7fa7--