GEOM: ada0: the secondary GPT header is not in the last LBA or random: unblocking device. when using more than 2 cores

Sun Jul 16 10:16:17 UTC 2017

I change the zone on AWS (eu-central-1a) and got the same behaviour,
instantes with 1 core work but with 2 cores they panic, I was available to
get more details, in this time I was using UFS instead of ZFS on root:

Setting up harvesting:
[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
spin lock 0xffffffff80db45c0 (smp rendezvous) held by 0xfffff80004378560
(tid 100074) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff804f69a7 at kdb_backtrace+0x67
#1 0xffffffff804b9666 at vpanic+0x186
#2 0xffffffff804b94d3 at panic+0x43
#3 0xffffffff8049da60 at __mtx_trylock_spin_flags+0
#4 0xffffffff807bd2d6 at smp_targeted_tlb_shootdown+0xd6
#5 0xffffffff807bd62c at smp_masked_invlpg+0x4c
#6 0xffffffff807558b2 at pmap_invalidate_page+0x142
#7 0xffffffff8075f2a9 at pmap_ts_referenced+0x709
#8 0xffffffff8073ac9c at vm_pageout+0xcbc
#9 0xffffffff80482345 at fork_exit+0x75
#10 0xffffffff8074b4be at fork_trampoline+0xe
Uptime: 1m0s
Rebooting...
cpu_reset: Stopping other CPUs
timeout stopping cpus
cpu_reset: Restarting BSP
cpu_reset: Failed to restart BSP

Full output:

- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
/boot/kernel/kernel text=0x6876f0 - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
\ data=0x76348+0x37d168 | / - \ syms=[0x8+0xa1e38| / - \ |
+0x8+0x9d58c/ - \ | / ]
- \ | / - \ | / - \ |

Booting [/boot/kernel/kernel]...
/ - \ | / Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-PRERELEASE #0 r321036: Sat Jul 15 21:52:50 UTC 2017
    devops at fabrik-de1.127.network:/fabrik/aws-nozfs/host/obj/usr/src/sys/FABRIKAWS
amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on
LLVM 4.0.0)
VT: init without driver.
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (2394.50-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x728<BMI1,AVX2,BMI2,ERMS,INVPCID>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "XenVMMXenVMM"
real memory  = 4294967296 (4096 MB)
avail memory = 4131692544 (3940 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
random: entropy device external interface
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on
pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem
0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
vgapci0: Boot video device
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem
0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
Timecounters tick every 1.000 msec
nvme cam probe device init
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 02:0e:56:1c:1c:d3
xenbusb_back0: <Xen Backend Devices> on xenstore0
xn0: backend features: feature-sg feature-gso-tcp4
xbd0: 8192MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
GEOM: ada0: the secondary GPT header is not in the last LBA.
Growing root partition to fill device
ada0 recovered
ada0p3 resized
super-block backups (for fsck_ffs -b #) at:
 2097600, 2621952, 3146304, 3670656, 4195008, 4719360, 5243712, 5768064,
 6292416, 6816768, 7341120, 7865472, 8389824, 8914176, 9438528, 9962880,
 10487232, 11011584, 11535936, 12060288, 12584640, 13108992, 13633344, 14157696
Setting hostuuid: ec259844-260d-90a9-bfa5-5157bc239b6b.
Setting hostid: 0x52ee9fe5.
Starting file system checks:
/dev/gpt/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/gpt/rootfs: clean, 1648359 free (39 frags, 206040 blocks, 0.0%
fragmentation)
Mounting local filesystems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
random: unblocking device.
32-bit compatibility ldconfig path:
Setting hostname: fabrik.
Setting up harvesting:
[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
spin lock 0xffffffff80db45c0 (smp rendezvous) held by
0xfffff80004378560 (tid 100074) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 1
KDB: stack backtrace:
#0 0xffffffff804f69a7 at kdb_backtrace+0x67
#1 0xffffffff804b9666 at vpanic+0x186
#2 0xffffffff804b94d3 at panic+0x43
#3 0xffffffff8049da60 at __mtx_trylock_spin_flags+0
#4 0xffffffff807bd2d6 at smp_targeted_tlb_shootdown+0xd6
#5 0xffffffff807bd62c at smp_masked_invlpg+0x4c
#6 0xffffffff807558b2 at pmap_invalidate_page+0x142
#7 0xffffffff8075f2a9 at pmap_ts_referenced+0x709
#8 0xffffffff8073ac9c at vm_pageout+0xcbc
#9 0xffffffff80482345 at fork_exit+0x75
#10 0xffffffff8074b4be at fork_trampoline+0xe
Uptime: 1m0s
Rebooting...
cpu_reset: Stopping other CPUs
timeout stopping cpus
cpu_reset: Restarting BSP
cpu_reset: Failed to restart BSP

On Sun, Jul 16, 2017 at 10:59 AM, Nicolas Embriz <nbari at tequila.io> wrote:
>
> Hi, this is the only output I have from AWS (system log), important to
mention that this only happens when using more than 2 cores (t2.medium) for
example, when using 1 core it works.
>
> Output:
>
> / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \
| / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
- \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/kernel/kernel
text=0x6876f0 | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | /
data=0x76348+0x37d168 - \ | syms=[0x8+0xa1e38/ - +0x8+0x9d58c\ | / ]
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
\ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ |
/ - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ /boot/kernel/zfs.ko |
/ - \ | / - \ | / size 0x2e8938 at 0xfba000
> loading required module 'opensolaris'
> - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / -
\ | / /boot/kernel/opensolaris.ko size 0xaab0 at 0x12a3000
>
>
> Booting [/boot/kernel/kernel]...
> - \ | / - \ | / - \ | / - \ Copyright (c) 1992-2017 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.1-PRERELEASE #0 r321034: Sat Jul 15 20:44:15 UTC 2017
>     devops at fabrik-de1.127.network:/fabrik/aws/host/obj/usr/src/sys/FABRIKAWS
amd64
> FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on
LLVM 4.0.0)
> VT: init without driver.
> XEN: Hypervisor version 4.2 detected.
> CPU: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz (2400.05-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
>
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
>
Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
>   AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
>   AMD Features2=0x21<LAHF,ABM>
>   Structured Extended Features=0x728<BMI1,AVX2,BMI2,ERMS,INVPCID>
>   XSAVE Features=0x1<XSAVEOPT>
> Hypervisor: Origin = "XenVMMXenVMM"
> real memory  = 4294967296 (4096 MB)
> avail memory = 4128489472 (3937 MB)
> Event timer "LAPIC" quality 100
> ACPI APIC Table: <Xen HVM>
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s)
> ioapic0: Changing APIC ID to 1
> MADT: Forcing active-low polarity and level trigger for SCI
> ioapic0 <Version 1.1> irqs 0-47 on motherboard
> random: entropy device external interface
> random: registering fast source Intel Secure Key RNG
> random: fast provider: "Intel Secure Key RNG"
> nexus0
> aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
> cryptosoft0: <software crypto> on motherboard
> acpi0: <Xen> on motherboard
> acpi0: Power Button (fixed)
> acpi0: Sleep Button (fixed)
> cpu0: <ACPI CPU> on acpi0
> cpu1: <ACPI CPU> on acpi0
> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 62500000 Hz quality 950
> attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> Event timer "RTC" frequency 32768 Hz quality 0
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
> acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> isab0: <PCI-ISA bridge> at device 1.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel PIIX3 WDMA2 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on pci0
> ata0: <ATA channel> at channel 0 on atapci0
> ata1: <ATA channel> at channel 1 on atapci0
> pci0: <bridge> at device 1.3 (no driver attached)
> vgapci0: <VGA-compatible display> mem
0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
> vgapci0: Boot video device
> xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem
0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart0: console (9600,n,8,1)
> xenpv0: <Xen PV bus> on motherboard
> granttable0: <Xen Grant-table Device> on xenpv0
> xen_et0: <Xen PV Clock> on xenpv0
> Event timer "XENTIMER" frequency 1000000000 Hz quality 950
> Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
> xenstore0: <XenStore> on xenpv0
> evtchn0: <Xen event channel user-space device> on xenpv0
> privcmd0: <Xen privileged interface user-space device> on xenpv0
> debug0: <Xen debug handler> on xenpv0
> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is
present;
>             to enable, add "vfs.zfs.prefetch_disable=0" to
/boot/loader.conf.
> ZFS filesystem version: 5
> ZFS storage pool version: features support (5000)
> Timecounters tick every 1.000 msec
> nvme cam probe device init
> xenballoon0: <Xen Balloon Device> on xenstore0
> xctrl0: <Xen Control Device> on xenstore0
> xs_dev0: <Xenstore user-space device> on xenstore0
> xenbusb_front0: <Xen Frontend Devices> on xenstore0
> xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
> xn0: Ethernet address: 0a:9a:53:0d:ff:6b
> xenbusb_back0: <Xen Backend Devices> on xenstore0
> xn0: backend features: feature-sg feature-gso-tcp4
> xbd0: 18432MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
> xbd0: attaching as ada0
> SMP: AP CPU #1 Launched!
> Trying to mount root from zfs:zroot/ROOT/default []...
> GEOM: ada0: the secondary GPT header is not in the last LBA.
>
>
>
>
> On Sun, Jul 16, 2017 at 9:49 AM, Nicolas Embriz <nbari at tequila.io> wrote:
>>
>> Hi,
>>
>> I am trying this in Amazon AWS, the problem is that I can’t do a dmesg
>> or get more logs because when choosing a t2.medium or any instance
>> with more than 2 cores, the image gets stuck on the boot process and
>> therefore I can’t login.
>>
>> I have a working image with FreeBSD 11.0-stable:
>> https://github.com/fabrik-red/images/releases/download/11.0/disk.tar.gz
>>
>> Using this kernel:
>> https://github.com/fabrik-red/images/blob/11.0/fabrik.kernel and made
>> using this script:
>> https://github.com/fabrik-red/images/blob/11.0/fabrik.sh
>>
>> I know is probably not useful this info but so far the only difference
>> is that I updated the sources and now while doing the same thing with
>> FreeBSD 11.1-prelrelease I am getting this strange behaviour.
>>
>> When using only one single cpu core the image works but when using
>> more thant 2 cores the image just don’t boot.
>>
>> Any ideas of what else I may add so that I could get more info at
>> least on the boot screen that is the only thing I get from AWS while
>> booting?
>>
>> Thanks in advance.
>>
>> Regards.
>
>