How approach debugging a kernel crash?

Mon Feb 27 05:35:14 UTC 2017

Hi,

I am trying to write a custom boot loader for ARM, to replace u-boot and
ubldr.

As I'm working through this, I keep getting kernel crashes.  I've got the
kernel debugger enabled, but doing a backtrace doesn't reveal any useful
information.

How does one go about figuring out exactly what caused an exception?  I
need to know where the kernel crashed so I can figure out what piece of
hardware I haven't set up correctly.

The back trace is just a bunch of abort stuff, and ends in the message
"Unable to unwind into user mode".

I've quoted the backtrace below, and also my kernel message.

Mostly I'm looking for suggestions on how to go about finding the location
of the crash, as I expect to be doing this a lot this week :-)

Thanks!

Lee

db> bt
Tracing pid 0 tid 100000 td 0xc08f8470
db_trace_self() at db_trace_self
         pc = 0xc0669b44  lr = 0xc014c288 (db_hex2dec+0x1f4)
         sp = 0xffff0cb0  fp = 0xffff0cc8
db_hex2dec() at db_hex2dec+0x1f4
         pc = 0xc014c288  lr = 0xc014becc (db_command_loop+0x2f4)
         sp = 0xffff0cd0  fp = 0xffff0d70
         r4 = 0x00000001  r5 = 0x00000000
         r6 = 0xc0704ae6 r10 = 0xc08f6f98
db_command_loop() at db_command_loop+0x2f4
         pc = 0xc014becc  lr = 0xc014bc4c (db_command_loop+0x74)
         sp = 0xffff0d78  fp = 0xffff0d88
         r4 = 0xc06cfe7d  r5 = 0xc06e1e0e
         r6 = 0xc08f6f84  r7 = 0xffff0fa0
         r8 = 0xc08ead98  r9 = 0xc0791060
        r10 = 0xc08ead9c
db_command_loop() at db_command_loop+0x74
         pc = 0xc014bc4c  lr = 0xc014f084 (db_fetch_ksymtab+0x2e8)
         sp = 0xffff0d90  fp = 0xffff0ea8
         r4 = 0x00000807  r5 = 0x00000000
         r6 = 0xc08f6f90 r10 = 0xc08ead9c
db_fetch_ksymtab() at db_fetch_ksymtab+0x2e8
         pc = 0xc014f084  lr = 0xc0341870 (kdb_trap+0x180)
         sp = 0xffff0eb0  fp = 0xffff0ed8
         r4 = 0x00000000  r5 = 0x00000807
         r6 = 0xc08eadb8 r10 = 0xc08ead9c
kdb_trap() at kdb_trap+0x180
         pc = 0xc0341870  lr = 0xc06908b4 (abort_handler+0x678)
         sp = 0xffff0ee0  fp = 0xffff0f00
         r4 = 0xffff0fa0  r5 = 0x00000013
         r6 = 0xffff1030  r7 = 0x00000007
         r8 = 0x00000807  r9 = 0xc08f8470
        r10 = 0xffff0fa0
abort_handler() at abort_handler+0x678
         pc = 0xc06908b4  lr = 0xc0690600 (abort_handler+0x3c4)
         sp = 0xffff0f08  fp = 0xffff0f98
         r4 = 0x00000001  r5 = 0x00000007
         r6 = 0x00000000  r7 = 0x00000807
         r8 = 0x00000013 r10 = 0xffff0fa0
abort_handler() at abort_handler+0x3c4
         pc = 0xc0690600  lr = 0xc066c42c (exception_exit)
         sp = 0xffff0fa0  fp = 0xc0a13e70
         r4 = 0x00000000  r5 = 0xc08f8808
         r6 = 0x00000001  r7 = 0x00000000
         r8 = 0xc08f890c  r9 = 0xc08f8908
        r10 = 0x00002802
exception_exit() at exception_exit
         pc = 0xc066c42c  lr = 0x1000019c (0x1000019c)
         sp = 0xffff1034  fp = 0xc0a13e70
         r0 = 0xc066c534  r1 = 0xc0a0b000
         r2 = 0xffff107c  r3 = 0x20010193
         r4 = 0x00000000  r5 = 0xc08f8808
         r6 = 0x00000001  r7 = 0x00000000
         r8 = 0xc08f890c  r9 = 0xc08f8908
        r10 = 0x00002802 r12 = 0xfefefeff
data_abort_entry() at data_abort_entry+0x30
         pc = 0xc066c534  lr = 0x1000019c (0x1000019c)
         sp = 0xffff1034  fp = 0xc0a13e70
Unable to unwind into user mode

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p1 #27 r309723M: Sat Feb 25 18:51:15 EST 2017
    builder at abe:/usr/home/builder/projects/fbsd_11.0.1/obj/arm.armv6/usr/home/builder/projects/fbsd_11.0.1/src/sys/AXSACM
arm
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM
3.8.0)
VT: init without driver.
CPU: Cortex A9-r3 rev 0 (Cortex-A core)
 Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext
 WB enabled LABT branch prediction disabled
LoUU:2 LoC:2 LoUIS:2
Cache level 1:
 32KB/32B 4-way data cache WB Read-Alloc Write-Alloc
 32KB/32B 4-way instruction cache Read-Alloc
real memory  = 535822336 (511 MB)
avail memory = 513486848 (489 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
random: entropy device external interface
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
simplebus1: <Flattened device tree simple bus> on ofwbus0
simplebus2: <Flattened device tree simple bus> on ofwbus0
l2cache0: <PL310 L2 cache controller> mem 0xf02000-0xf02fff on simplebus0
l2cache0: cannot allocate IRQ, not using interrupt
l2cache0: Part number: 0x3, release: 0x8
l2cache0: L2 Cache enabled: 512KB/32B 8 ways
gic0: <ARM Generic Interrupt Controller> mem
0xf01000-0xf01fff,0xf00100-0xf001ff on simplebus0
gic0: pn 0x390, arch 0x1, rev 0x2, implementer 0x43b irqs 96
mp_tmr0: <ARM MPCore Timers> mem 0xf00200-0xf002ff,0xf00600-0xf0061f on
simplebus0
Timecounter "MPCore" frequency 325000000 Hz quality 800
Event timer "MPCore" frequency 325000000 Hz quality 1000
zy7_slcr0: <Zynq-7000 slcr block> mem 0-0xfff on simplebus0
zy7_devcfg0: <Zynq devcfg block> mem 0x7000-0x7fff on simplebus0
uart0: <Cadence UART> mem 0x1000-0x1fff on simplebus1
uart0: console (-1,n,8,1)
ehci0: <Zynq-7000 EHCI USB 2.0 controller> mem 0x2000-0x2fff on simplebus1
usbus0: EHCI version 1.0
usbus0: stop timeout
usbus0 on ehci0
gpio0: <Zynq-7000 GPIO driver> mem 0xa000-0xafff on simplebus1
gpiobus0: <GPIO bus> on gpio0
gpioc0: <GPIO controller> on gpio0
cgem0: <Cadence CGEM Gigabit Ethernet Interface> mem 0xb000-0xbfff on
simplebus1
miibus0: <MII bus> on cgem0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 0 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT-FDX, 1000baseT-FDX-master, auto
rgephy1: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT-FDX, 1000baseT-FDX-master, auto
cgem0: no mac address found, assigning random: 62:73:64:b9:65:d2
cgem0: Ethernet address: 62:73:64:b9:65:d2
sdhci_fdt0: <Zynq-7000 generic fdt SDHCI controller> mem 0x100000-0x100fff
on simplebus1
sdhci_fdt0: 1 slot(s) allocated
mmc0: <MMC/SD bus> on sdhci_fdt0
sdhci_fdt1: <Zynq-7000 generic fdt SDHCI controller> mem 0x101000-0x101fff
on simplebus1
sdhci_fdt1: 1 slot(s) allocated
mmc1: <MMC/SD bus> on sdhci_fdt1
cryptosoft0: <software crypto>
Fatal kernel mode data abort: 'Translation Fault (L2)' on write
trapframe: 0xffff0fa0
FSR=00000807, FAR=ffff1030, spsr=20010193
r0 =c066c534, r1 =c0a0b000, r2 =ffff107c, r3 =20010193
r4 =00000000, r5 =c08f8808, r6 =00000001, r7 =00000000
r8 =c08f890c, r9 =c08f8908, r10=00002802, r11=c0a13e70
r12=fefefeff, ssp=ffff1034, slr=1000019c, pc =c066c534

[ thread pid 0 tid 100000 ]
Stopped at      data_abort_entry+0x30:  str     r0, [r13, -#0x004]!