[Bug 269581] provisioning arm64 Azure image in Azure getting stuck intermittently.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269581

            Bug ID: 269581
           Summary: provisioning arm64 Azure image in Azure getting stuck
                    intermittently.
           Product: Base System
           Version: CURRENT
          Hardware: arm64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: schakrabarti@microsoft.com

Sometime during provisioning arm64 Azure image in Azure, the boot is getting
stuck at Starting file system checks:
Invalid signature in boot block: 0000

Attached whole boot log for reference:
Loading /boot/loader.conf.local
|/-\|c/-Loading kernel...
\|/-\|/boot/kernel/kernel text=0x2a8 text=0x88c8d0 /-text=0x26c62c
\data=0x1cb138 data=0x0+0x354000 |0x8+0x146418+0x8+0x16e88c
Loading configured modules...
/-/boot/kernel/mlx5en.ko text=0x12eb1 text=0x17d8c data=0x3048+0x8
0x8+0x40b0+0x8+0x2c8a
loading required module 'mlx5'
\|/boot/kernel/mlx5.ko text=0x14274 text=0x24f8c data=0x2380+0x4a0
0x8+0x59e8+0x8+0x450f
loading required module 'mlxfw'
/-/boot/kernel/mlxfw.ko text=0x143a text=0x16b4 data=0x2e8
0x8+0x7f8+0x8+0x47f
loading required module 'xz'
\/boot/kernel/xz.ko text=0xc10 text=0x2660 data=0x308+0xc00
0x8+0x978+0x8+0x478
loading required module 'linuxkpi'
|//boot/kernel/linuxkpi.ko text=0x11cad text=0x1baa0 data=0x2f30+0x2920
0x8+0x8af0+0x8+0x6854
loading required module 'backlight'
-\/boot/kernel/backlight.ko text=0xc66 text=0x598 data=0x3b0+0x28
0x8+0x7c8+0x8+0x44c
loading required module 'iicbb'
|//boot/kernel/iicbb.ko text=0x18d8 text=0x1348 data=0x570+0x4
0x8+0xca8+0x8+0x845
-/boot/kernel/zfs.ko text=0xe5adc \text=0x21c814 data=0x2a8b0+0xab684
0x8+0x348a8+0x8+0x2dfec
|//boot/kernel/mlx4en.ko text=0x7d0c text=0xda74 data=0xab8+0xc
0x8+0x2460+0x8+0x19a7
loading required module 'mlx4'
-\/boot/kernel/mlx4.ko text=0x11004 text=0x36ccc data=0x1be0+0x79
0x8+0x50e8+0x8+0x3f3b
|/can't find '/etc/hostid'
-\can't find '/boot/entropy'

Booting [/boot/kernel/kernel]...
|No valid device tree blob found!
WARNING! Trying to fire up the kernel, but no device tree blob found!
EFI framebuffer information:
addr, size 0x40000000, 0x800000
dimensions 1024 x 768
stride 1024
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
---<<BOOT>>---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n260584-edfc7d6e5b5b: Sun Feb 5 16:23:49 UTC 2023
root@build-current:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git
llvmorg-14.0.5-0-gc12386ae247c)
WARNING: WITNESS option enabled, expect reduced performance.
SRAT: Ignoring memory at addr 0x1100000000
SRAT: Ignoring memory at addr 0x10000000000
SRAT: Ignoring memory at addr 0x20000000000
SRAT: Ignoring memory at addr 0x40000000000
SRAT: Ignoring memory at addr 0x80000000000
SRAT: Ignoring memory at addr 0x100000000000
SRAT: Ignoring memory at addr 0x200000000000
SRAT: Ignoring memory at addr 0x400000000000
SRAT: Ignoring memory at addr 0x800000000000
VT(efifb): resolution 1024x768
module scmi already present!
module firmware already present!
real memory = 68719308800 (65535 MB)
avail memory = 66970431488 (63867 MB)
Starting CPU 1 (1)
Starting CPU 2 (2)
Starting CPU 3 (3)
Starting CPU 4 (4)
Starting CPU 5 (5)
Starting CPU 6 (6)
Starting CPU 7 (7)
Starting CPU 8 (8)
Starting CPU 9 (9)
Starting CPU 10 (a)
Starting CPU 11 (b)
Starting CPU 12 (c)
Starting CPU 13 (d)
Starting CPU 14 (e)
Starting CPU 15 (f)
Starting CPU 16 (100)
Starting CPU 17 (101)
Starting CPU 18 (102)
Starting CPU 19 (103)
Starting CPU 20 (104)
Starting CPU 21 (105)
Starting CPU 22 (106)
Starting CPU 23 (107)
Starting CPU 24 (108)
Starting CPU 25 (109)
Starting CPU 26 (10a)
Starting CPU 27 (10b)
Starting CPU 28 (10c)
Starting CPU 29 (10d)
Starting CPU 30 (10e)
Starting CPU 31 (10f)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device
because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
MAP 3ec85000 mode 2 pages 41
MAP 3fd2e000 mode 2 pages 48
MAP 3fd5e000 mode 2 pages 36
MAP effed000 mode 0 pages 1
kbd0 at kbdmux0
acpi0: <VRTUAL MICROSFT>
acpi0: Could not update all GPEs: AE_NOT_CONFIGURED
psci0: <ARM Power State Co-ordination Interface Driver> on acpi0
gic0: <ARM Generic Interrupt Controller v3.0> iomem
0xffff0000-0x10000ffff,0xeffee000-0xf000dfff,0xf000e000-0xf002dfff,0xf002e000-0xf004dfff,0xf004e000-0xf006dfff,0xf006e000-0xf008dfff,0xf008e000-0xf00adfff,0xf00ae000-0xf00cdfff,0xf00ce000-0xf00edfff,0xf00ee000-0xf010dfff,0xf010e000-0xf012dfff,0xf012e000-0xf014dfff,0xf014e000-0xf016dfff,0xf016e000-0xf018dfff,0xf018e000-0xf01adfff,0xf01ae000-0xf01cdfff,0xf01ce000-0xf01edfff,0xf01ee000-0xf020dfff,0xf020e000-0xf022dfff,0xf022e000-0xf024dfff,0xf024e000-0xf026dfff,0xf026e000-0xf028dfff,0xf028e000-0xf02adfff,0xf02ae000-0xf02cdfff,0xf02ce000-0xf02edfff,0xf02ee000-0xf030dfff,0xf030e000-0xf032dfff,0xf032e000-0xf034dfff,0xf034e000-0xf036dfff,0xf036e000-0xf038dfff,0xf038e000-0xf03adfff,0xf03ae000-0xf03cdfff,0xf03ce000-0xf03edfff
on acpi0
generic_timer0: <ARM Generic Timer> irq 4,5,6 on acpi0
Timecounter "ARM MPCore Timecounter" frequency 25000000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 25000000 Hz quality 1000
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
pmu0: <Performance Monitoring Unit> on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_syscontainer0: <System Container> on acpi0
vmbus0: <Hyper-V Vmbus> on acpi_syscontainer0
vmgenc0: <VM Generation Counter> on acpi0
acpi_ged0: <Generic Event Device> irq 3 on acpi0
acpi_ged0: Raw IRQ 35
uart0: <PrimeCell UART (PL011)> iomem 0xeffec000-0xeffecfff irq 0 on acpi0
uart0: console (115200,n,8,1)
uart1: <PrimeCell UART (PL011)> iomem 0xeffeb000-0xeffebfff irq 1 on acpi0
vmbus_res0: <Hyper-V Vmbus Resource> irq 2 on acpi0
armv8crypto0: <AES-CBC,AES-XTS,AES-GCM>
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
[nl_generic] genl_register_family: Registered family nlctrl id 16
[nl_generic] genl_register_cmds: Adding cmd GETFAMILY(3) to family nlctrl
usb_needs_explore_all: no devclass
vmbus0: irq 0x2, vector 0 end 0x2
vmbus0: the irq 18
vmbus0: version 4.0
hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
hvkvp0: <Hyper-V KVP> on vmbus0
hvshutdown0: <Hyper-V Shutdown> on vmbus0
hvtimesync0: <Hyper-V Timesync> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
hn0: Ethernet address: 00:0d:3a:06:92:c8
storvsc0: <Hyper-V SCSI> on vmbus0
hn0: link state changed to UP
storvsc1: <Hyper-V SCSI> on vmbus0
CPU 0: ARM Neoverse-N1 r3p1 affinity: 0 0
Cache Type = <64 byte D-cacheline,64 byte I-cacheline,PIPT ICache,64 byte
ERG,64 byte CWG,IDC,DIC>
Instruction Set Attributes 0 = <DP,RDM,Atomic,CRC32,SHA2,SHA1,AES+PMULL>
Trying to mount root from zfs:zroot/ROOT/default []...
Instruction Set Attributes 1 = <RCPC-8.3,DCPoP>
Instruction Set Attributes 2 = <>
Processor Features 0 = <CSV2,GIC,AdvSIMD+HP,FP+HP,EL3,EL2,EL1,EL0 32>
Processor Features 1 = <>
Memory Model Features 0 = <TGran4,TGran64,TGran16,16bit ASID,256TB PA>
Memory Model Features 1 = <PAN+ATS1E1,8bit VMID,HAF+DS>
Memory Model Features 2 = <32bit CCIDX,48bit VA,UAO>
Debug Features 0 = <DoubleLock,2 CTX BKPTs,4 Watchpoints,6 Breakpoints,PMUv3
v8.1,Debugv8>
Debug Features 1 = <>
Auxiliary Features 0 = <>
Auxiliary Features 1 = <>
AArch32 Instruction Set Attributes 5 = <RDM,CRC32,SHA2,SHA1,AES+VMULL,SEVL>
AArch32 Media and VFP Features 0 = <FPRound,FPSqrt,FPDivide,DP VFPv3+v4,SP
VFPv3+v4,AdvSIMD>
AArch32 Media and VFP Features 1 = <SIMDFMAC,FPHP Arith,SIMDHP
Arith,SIMDSP,SIMDInt,SIMDLS,FPDNaN,FPFtZ>
CPU 1: ARM Neoverse-N1 r3p1 affinity: 0 1
CPU 2: ARM Neoverse-N1 r3p1 affinity: 0 2
CPU 3: ARM Neoverse-N1 r3p1 affinity: 0 3
CPU 4: ARM Neoverse-N1 r3p1 affinity: 0 4
CPU 5: ARM Neoverse-N1 r3p1 affinity: 0 5
CPU 6: ARM Neoverse-N1 r3p1 affinity: 0 6
CPU 7: ARM Neoverse-N1 r3p1 affinity: 0 7
CPU 8: ARM Neoverse-N1 r3p1 affinity: 0 8
CPU 9: ARM Neoverse-N1 r3p1 affinity: 0 9
CPU 10: ARM Neoverse-N1 r3p1 affinity: 0 10
CPU 11: ARM Neoverse-N1 r3p1 affinity: 0 11
CPU 12: ARM Neoverse-N1 r3p1 affinity: 0 12
CPU 13: ARM Neoverse-N1 r3p1 affinity: 0 13
CPU 14: ARM Neoverse-N1 r3p1 affinity: 0 14
CPU 15: ARM Neoverse-N1 r3p1 affinity: 0 15
CPU 16: ARM Neoverse-N1 r3p1 affinity: 1 0
CPU 17: ARM Neoverse-N1 r3p1 affinity: 1 1
CPU 18: ARM Neoverse-N1 r3p1 affinity: 1 2
CPU 19: ARM Neoverse-N1 r3p1 affinity: 1 3
CPU 20: ARM Neoverse-N1 r3p1 affinity: 1 4
CPU 21: ARM Neoverse-N1 r3p1 affinity: 1 5
CPU 22: ARM Neoverse-N1 r3p1 affinity: 1 6
CPU 23: ARM Neoverse-N1 r3p1 affinity: 1 7
CPU 24: ARM Neoverse-N1 r3p1 affinity: 1 8
CPU 25: ARM Neoverse-N1 r3p1 affinity: 1 9
CPU 26: ARM Neoverse-N1 r3p1 affinity: 1 10
CPU 27: ARM Neoverse-N1 r3p1 affinity: 1 11
CPU 28: ARM Neoverse-N1 r3p1 affinity: 1 12
CPU 29: ARM Neoverse-N1 r3p1 affinity: 1 13
CPU 30: ARM Neoverse-N1 r3p1 affinity: 1 14
CPU 31: ARM Neoverse-N1 r3p1 affinity: 1 15
Release APs...done
cd0 at storvsc0 bus 0 scbus0 target 0 lun 2
cd0: <Msft Virtual DVD-ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: 300.000MB/s transfers
cd0: 0MB (314 2048 byte sectors)
da1 at storvsc0 bus 0 scbus0 target 0 lun 1
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 1228800MB (2516582400 512 byte sectors)
da0 at storvsc0 bus 0 scbus0 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30753MB (62982144 512 byte sectors)
WARNING: WITNESS option enabled, expect reduced performance.
hn0: got notify, nvs type 128
random: unblocking device.
Dual Console: Serial Primary, Video Secondary
2023-02-14T08:06:10.028832+00:00 - init 18 - - can't directly exec /etc/rc:
Permission denied
No suitable dump device was found.
Setting hostuuid: d7c2fdb2-53b9-4502-a510-9d351b7950ee.
Setting hostid: 0x62d84381.
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
async_destroy
empty_bpobj
lz4_compress
multi_vdev_crash_dump
spacemap_histogram
enabled_txg
hole_birth
extensible_dataset
embedded_data
bookmarks
filesystem_limits
large_blocks
large_dnode
sha512
skein
edonr
userobj_accounting
encryption
project_quota
device_removal
obsolete_counts
zpool_checkpoint
spacemap_v2
allocation_classes
resilver_defer
bookmark_v2
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid
zilsaxattr
head_errlog
blake3

Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.
Starting file system checks:
Invalid signature in boot block: 0000
/dev/gpt/efiesp: UNEXPECTED INCONSISTENCY; RUN fsck_msdosfs MANUALLY.
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
msdosfs: /dev/gpt/efiesp (/boot/efi)
Automatic file system check failed; help!
ERROR: ABORTING BOOT (sending SIGTERM to parent)!
2023-02-14T08:06:13.614565+00:00 - init 1 - - /bin/sh on /etc/rc terminated
abnormally, going to single user mode
Enter full pathname of shell or RETURN for /bin/sh: hvkvp0: detached
hvkvp0: <Hyper-V KVP> on vmbus0

-- 
You are receiving this mail because:
You are the assignee for the bug.