Boot time panic on recent i386 build after SMP: AP CPU #1 Launched

[Jason Unovitch]

Evening,
I have a Soekris 6501 running i386 11-CURRENT that I had seen panic at 
boot when updating this weekend.  It'll panic right after the "SMP: AP 
CPU #1 Launched" and go into a long loop of __mtx_lock_spin_flags, 
i8254_get_timecount, binuptime, _mtx_lock_spin_cookie.  An abbreviated 
output is at the bottom of the email while the full output is linked below.

- My currently working build's uname -a:
FreeBSD xju-rtr 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r284194: Tue Jun  9 
23:38:40 UTC 2015 
root at xts-bsd:/usr/obj/nanobsd.soekris/i386.i386/usr/src/sys/IPSEC  i386

- The misbehaving builds:
FreeBSD 11.0-CURRENT #0 r285155: Sun Jul  5 07:35:36 UTC 2015
 
root at xts-bsd.:/usr/obj/nanobsd.soekris/i386.i386/usr/src/sys/GENERIC i386
- Full boot log: http://pastebin.com/QmzpnZdf

FreeBSD 11.0-CURRENT #0 r285179: Mon Jul  6 00:10:45 UTC 2015
 
root at xts-bsd.:/usr/obj/nanobsd.soekris/i386.i386/usr/src/head/sys/GENERIC-NODEBUG 
i386
- Full boot log: http://pastebin.com/j9ZRdK3V

I tested 10.2-PRERELEASE and didn't see any issues so this appears to be 
confined to CURRENT for me.  I was experimenting with trying out a 
GENERIC kernel since IPSEC is available there now but otherwise I was 
just doing a routine update.  For the time being I just reverted to the 
prior NanoBSD image slice.

I'm going to start to a binary search to nail down where the issue came 
about.  I'll test each evening after work with a new image but this is 
my home router/wifi ap so there a minor inconvenience while I do so and 
it will take a few days.  I'll report what I find.

Any ideas on where to start would be appreciated!  Thanks for all the 
good work you do!

Jason

-- Truncated panic output follows (full is at links above):

SMP: AP CPU #1 Launched!
Timecounter "TSC" frequency 1600030024 Hz quality 1000

Fatal double fault:
eip = 0xc0bfe063
esp = 0xe1fbc000
ebp = 0xe1fbc000
cpuid = 1; apic id = 01
panic: double fault
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper(c137c62d,c1229194,ff,0,c15c1f1c,...) at 
db_trace_self_wrapper+0x2a/frame 0xc171fc20
kdb_backtrace(c15381f0,1,c152f513,c171fcdc,1,...) at 
kdb_backtrace+0x2d/frame 0xc171fc88
vpanic(c152f513,c171fcdc,c171fcdc,c171fcf4,c1200186,...) at 
vpanic+0x117/frame 0xc171fcbc
panic(c152f513,1,1,1,e1fbc000,...) at panic+0x1b/frame 0xc171fcd0
dblfault_handler() at dblfault_handler+0xa6/frame 0xc171fcd0
--- trap 0x17, eip = 0xc0bfe063, esp = 0xe1fbc000, ebp = 0xe1fbc000 ---
lockstat_nsecs(aa55aa55,aa55aa55,aa55aa55,aa55aa55,aa55aa55,...) at 
lockstat_nsecs+0x3/frame 0xe1fbc000
_mtx_lock_spin_cookie(c1774d64,c6efc960,0,0,0,...) at 
_mtx_lock_spin_cookie+0x42/frame 0xe1fbc064
getit(c7163400,aa55aa55,aa55aa55,aa55aa55,aa55aa55,...) at 
getit+0x9d/frame 0xe1fbc08c
i8254_get_timecount(c7119118,9751ad54,44,c6efc960,c6efc960,...) at 
i8254_get_timecount+0x89/frame 0xe1fbc0b8
binuptime(e1fbc0e8,aa55aa55,b,34d2b2cb,3fd28fa4,...) at 
binuptime+0x46/frame 0xe1fbc0d8

... Repeat.... over and over and over with steady increments to memory 
addresses...

lockstat_nsecs(0,13669f,e1fbdb40,c0bcb188,c1757704,...) at 
lockstat_nsecs+0x16/frame 0xe1fbdb00
thread_lock_flags_(c6efc960,0,c137adf2,87a,e1fadcac,...) at 
thread_lock_flags_+0x24f/frame 0xe1fbdb68
sched_preempt(c6efc960,0,1,c176cf80,0,...) at sched_preempt+0x65/frame 
0xe1fbdb90
ipi_bitmap_handler(8,28,28,0,c176cf80,...) at 
ipi_bitmap_handler+0x4d/frame 0xe1fbdbb0
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2f/frame 0xe1fbdbb0
--- interrupt, eip = 0xc121a893, esp = 0xe1fbdbf8, ebp = 0xe1fbdc04 ---
cpu_idle_acpi(7e8f33c7,0,c16f4780,12,c16f4808,...) at 
cpu_idle_acpi+0x43/frame 0xe1fbdc04
cpu_idle(0,e1fbdc78,aa55aa55,aa55aa55,aa55aa55,...) at 
cpu_idle+0x9d/frame 0xe1fbdc20
sched_idletd(0,e1fbdce8,aa55aa55,aa55aa55,aa55aa55,...) at 
sched_idletd+0x341/frame 0xe1fbdca4
fork_exit(c0c5fa40,0,e1fbdce8) at fork_exit+0xa2/frame 0xe1fbdcd4
fork_trampoline() at fork_trampoline+0x8/frame 0xe1fbdcd4
--- trap 0, eip = 0, esp = 0xe1fbdd20, ebp = 0 ---
KDB: enter: panic
[ thread pid 11 tid 100005 ]
Stopped at      kdb_enter+0x3d: movl    $0,kdb_why