Help my SMP is broken???

[John Baldwin]

On Tuesday 13 July 2004 09:22 am, Willem Jan Withagen wrote:
> > I'm trying to boot my dual opteron box with a recent kernel.
> > Without ACPI is boots single processor, with ACPI I get the stuff
> > downbelow.
> >
> > SMAP type=01 base=0000000000000000 len=00000000000a0000
> > SMAP type=02 base=00000000000f0000 len=0000000000010000
> > SMAP type=02 base=00000000fec00000 len=0000000001400000
> > SMAP type=02 base=000000007fef0000 len=0000000000010000
> > SMAP type=01 base=0000000000100000 len=000000007fde0000
> > SMAP type=03 base=000000007fee3000 len=000000000000d000
> > SMAP type=04 base=000000007fee0000 len=0000000000003000
> > Copyright (c) 1992-2004 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> >         The Regents of the University of California. All rights reserved.
> > FreeBSD 5.2-CURRENT #57: Fri Jul  9 10:46:05 CEST 2004
> >     root at opteron.digiware.nl:/usr/obj/home2/src/sys/OPTERON.amd64
> > WARNING: WITNESS option enabled, expect reduced performance.
> > WARNING: DIAGNOSTIC option enabled, expect reduced performance.
> > Preloaded elf kernel "/boot/kernel/kernel" at 0xffffffff806f1000.
> > Calibrating clock(s) ... i8254 clock: 1193255 Hz
> > CLK_USE_I8254_CALIBRATION not specified - using default frequency
> > Timecounter "i8254" frequency 1193182 Hz quality 0
> > Calibrating TSC clock ... TSC clock: 1603648612 Hz
> > CPU: AMD Opteron(tm) Processor 242 (1603.65-MHz K8-class CPU)
> >   Origin = "AuthenticAMD"  Id = 0xf51  Stepping = 1
>
> Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA
>,CM
>
> > OV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
> >   AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow+,3DNow>
> > L1 2MB data TLB: 8 entries, fully associative
> > L1 2MB instruction TLB: 8 entries, fully associative
> > L1 4KB data TLB: 32 entries, fully associative
> > L1 4KB instruction TLB: 32 entries, fully associative
> > L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative
> > L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way
> > associative L2 2MB unified TLB: 0 entries, disabled/not present
> > L2 4KB data TLB: 512 entries, 4-way associative
> > L2 4KB instruction TLB: 512 entries, 4-way associative
> > L2 unified cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 16-way
> > associative real memory  = 2146304000 (2046 MB)
> > Physical memory chunk(s):
> > 0x0000000000001000 - 0x000000000009cfff, 638976 bytes (156 pages)
> > 0x0000000000777000 - 0x000000007c2b7fff, 2075398144 bytes (506689 pages)
> > avail memory = 2062643200 (1967 MB)
> > ACPI APIC Table: <VIAK8  AWRDACPI>
> > panic: AP #1 (PHY# 1) failed!
> > cpuid = 0;
> > Stack backtrace:
> > backtrace() at backtrace+0x17
> > panic() at panic+0x1d2
> > start_all_aps() at start_all_aps+0x187
> > cpu_mp_start() at cpu_mp_start+0x199
> > mp_start() at mp_start+0x54
> > mi_startup() at mi_startup+0xb8
> > btext() at btext+0x2c
> > Debugger("panic")
> > Stopped at      Debugger+0x4d:  xchgl   %ebx,0x27d461
>
> I'm trying to figure this one out.....
> So I added a printf into init_secondary. This is namely the place where a
> variable (mp_naps) is incremented to make it out of the waiting loop in
> start_all_aps.
>
> BUT we never get to init_secondary, which I expected to be called from an
> interrupt routine. ini_secondary is called from an asm-sequence in mpboot.S
> which suggest that it is the first thing to be called once we get into
> 64bit mode???
>
> Is there anywhere a description on how booting is done on UP or SMP (let
> alone SMP amd64??)

Basically, when a CPU is started up, it starts up in 8086-style real mode 
executing the code in mpboot.s.  That code brings the computer into protected 
mode and eventually into the same virtual addresses used by the kernel (the 
mpboot code is dual-mapped to allow for this transition).  At that point the 
AP can safely call into the kernel and it calls init_secondary().  If the AP 
faults during the early boot code it will basically die and never get into 
init_secondary().  Also, if the IPI is somehow lost then it will never get 
into init_secondary().

-- 
John Baldwin <jhb at FreeBSD.org>  <><  http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve"  =  http://www.FreeBSD.org