[Bug 268934] [ena] Counters are alloced after they are available for reading which can cause a kernel crash

From: <bugzilla-noreply_at_freebsd.org>
Date: Fri, 13 Jan 2023 16:16:47 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268934

            Bug ID: 268934
           Summary: [ena] Counters are alloced after they are available
                    for reading which can cause a kernel crash
           Product: Base System
           Version: 13.1-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: ghuckriede@blackberry.com

Created attachment 239448
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=239448&action=edit
Potential Fix

The kernel panics because the stats are allocated in ena_attach()
[@ena.c:3686].
However ena_get_counters() is registered by ena_setup_ifnet()[@ena.c:2402] in
ena_attach() [@ena.c:3663].  Once ether_ifattach() [@ena.c:2434] is done, the
interface is available and can counters can be read before they are allocated.

N.B. Line numbers refer to the following version of the file.
https://cgit.freebsd.org/src/tree/sys/dev/ena/ena.c @ blob
c091091fed206a949b11eb751a4d990d66fa181f 

A potential fix that creates the counters before calling ether_attach() has
been attached.
N.B. The EC2 instance is not setup to build, so the provided patch is not
tested.


Steps to Reproduce: 
root@freebsd:~ # cat ./dump.sh
#!/bin/sh
while true
do
netstat -I ena0
done
root@freebsd:~ # cat ./reset.sh
#!/bin/sh
while true
do
devctl disable ena0
devctl enable ena0
done
root@freebsd:~ # ./dump.sh &
root@freebsd:~ # ./reset.sh &
<SNIP>
ena0: link is UP
ena0: Link is down
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0
    0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Jan 13 15:50:46 freebsd dhclient[1577]: ena0: not found
Jan 13 15:50:46 ena0: detached
freebsd dhclient[1577]: exiting.
ena0: <ENA adapter>Name    Mtu Netw mem 0x80008000ork       Addres-0x8000bfff
irq 37 at device 5.0 on pci0
s              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller
version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Netwena0: LLQ is not supported. Fallback to host mode policy.
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller
version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Iena0: detached
pkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter>Name    Mtu Netw mem 0x80008000-0x8000bfff irq 37 at device
5.0 on pci0
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller
version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0Name    Mtu
Netw on pci0
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller
version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
ena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1
implementation version 1
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0
    0
Name    Mtu Network       Address              Ipkts Ierrs Idropena0: Link is
down
    Opkts Oerrs  Coll
ena0: link is UP
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0
    0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0*  1500 <Link#1>      02:55:a9:7c:3f:bb        0     0     0        0     0
    0
Jan 13 15:50:48 freebsd dhclient[1678]: ena0: not found
Jan 13 15:50:48 freebsd dhclient[1678]: exiting.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: detached
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0 on pci0
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA device
version: 0.10
ena0: ena_com_validate_version() [TID:100093]: ENA controller version: 0.0.1
implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Netwena0: detached
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: <ENA adapter> mem 0x80008000-0x8000bfff irq 37 at device 5.0Name    Mtu
Netw on pci0
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: ena_com_validate_version() [TID:100093]: ENA device version: 0.10
Name    Mtu Netwena0: ena_com_validate_version() [TID:100093]: ENA controller
version: 0.0.1 implementation version 1
ork       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ena0: LLQ is not supported. Fallback to host mode policy.
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
ena0: Link is down
 Coll
ena0: Link is down
ena0*  1500 <Linena0: Link is down
k#1>      02:55:a9:7c:3f:bb        0     0     0        0     0     0
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs
 Coll
<SNIP>


Actual Results:
Fatal data abort:
  x0:                0
  x1:                0
  x2:               d8
  x3: ffff0000da508284
  x4: ffff0000da5081a0
  x5: ffff00009ad620d8
  x6:                0
  x7:                0
  x8:                0
  x9:                0
 x10:                0
 x11:                1
 x12: ffff000000e5a250
 x13:                3
 x14:                3
 x15:                0
 x16: ffff000001280d28
 x17: ffff00000050c088
 x18: ffff0000da508250
 x19: ffff0000da508308
 x20: ffffa0001404a000
 x21:                0
 x22:                0
 x23:               d8
 x24: ffffa00001518390
 x25:               18
 x26:               98
 x27: ffff000000e6c000
 x28: ffff00009ad62000
 x29: ffff0000da508250
  sp: ffff0000da508250
  lr: ffff0000005ed6d0
 elr: ffff00000050c0d8
spsr:         80400045
 far:                0
 esr:         96000007
panic: vm_fault failed: ffff00000050c0d8
cpuid = 0
time = 1673625050
KDB: stack backtrace:
#0 0xffff00000051646c at kdb_backtrace+0x60
#1 0xffff0000004c24c0 at vpanic+0x174
#2 0xffff0000004c2348 at panic+0x44
#3 0xffff0000007f48c0 at data_abort+0x204
#4 0xffff0000007d5010 at handle_el1h_sync+0x10
#5 0xffff0000005ed6cc at if_data_copy+0x7c
#6 0xffff0000005ed6cc at if_data_copy+0x7c
#7 0xffff000000625384 at sysctl_iflist+0xe8
#8 0xffff0000006251e0 at sysctl_rtsock+0x26c
#9 0xffff0000004d4634 at sysctl_root_handler_locked+0x118
#10 0xffff0000004d3aa4 at sysctl_root+0x218
#11 0xffff0000004d4094 at userland_sysctl+0x18c
#12 0xffff0000004d3ec8 at sys___sysctl+0x68
#13 0xffff0000007f3e90 at do_el0_sync+0x560
#14 0xffff0000007d50fc at handle_el0_sync+0x38
Uptime: 25m32s
N.B. The kenel dump was not created on the target after reboot, and therefore
not included.


Build Date & Hardware:
Target is an AWS EC2 instance with an EC2 serial console connection
root@freebsd:~ # uname -a
FreeBSD freebsd 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC arm64

-- 
You are receiving this mail because:
You are the assignee for the bug.