Re: intermittent bsdtar/jemalloc failures

On 10/7/21 16:52, Mark Johnston wrote:
> On Thu, Oct 07, 2021 at 04:18:28PM -0400, Michael Butler via freebsd-current wrote:
>> On 10/7/21 15:39, Konstantin Belousov wrote:
>>> On Thu, Oct 07, 2021 at 03:28:44PM -0400, Michael Butler via freebsd-current wrote:
>>>> While building a local release bundle, I sometimes get bsdtar failing (and
>>>> dumping core) as follows below. Worse, as can be seen below, it doesn't stop
>>>> the build unless I happen to notice and it yields an incomplete package.
>>>>
>>>> a usr/src/sys/netgraph/ng_checksum.h
>>>> a usr/src/sys/netgraph/ng_message.h
>>>> a usr/src/sys/netgraph/ng_echo.c
>>>> a usr/src/sys/netgraph/ng_gif.h
>>>> <jemalloc>: jemalloc_arena.c:747: Failed assertion:
>>>> "nstime_compare(&decay->epoch, &time) <= 0"
>>>> Abort trap (core dumped)
>>>> sh /usr/src/release/scripts/make-manifest.sh *.txz > MANIFEST
>>>>
>>>> What causes this? Build machine is a 2x4-core Intel box with ZFS
>>>> file-systems all around. I tried stopping NTPD temporarily but the failures
>>>> persist .. sometimes :-(
>>>>
>>>> I've seen this at different points in the archiving process so it doesn't
>>>> seem specific to building kernel.txz.
>>>
>>> What timecounter do you use? Perhaps show the whole output from
>>> sysctl kern.timecounter.
>>
>> imb@vm01:/home/imb> sysctl kern.timecounter
>> kern.timecounter.tsc_shift: 1
>> kern.timecounter.smp_tsc_adjust: 0
>> kern.timecounter.smp_tsc: 1
>> kern.timecounter.invariant_tsc: 1
>> kern.timecounter.fast_gettime: 1
>> kern.timecounter.tick: 1
>> kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC-low(1000)
>> dummy(-1000000)
>> kern.timecounter.hardware: HPET
>> kern.timecounter.alloweddeviation: 5
>> kern.timecounter.timehands_count: 2
>> kern.timecounter.stepwarnings: 0
>> kern.timecounter.tc.ACPI-fast.quality: 900
>> kern.timecounter.tc.ACPI-fast.frequency: 3579545
>> kern.timecounter.tc.ACPI-fast.counter: 16124892
>> kern.timecounter.tc.ACPI-fast.mask: 16777215
>> kern.timecounter.tc.HPET.quality: 950
>> kern.timecounter.tc.HPET.frequency: 14318180
>> kern.timecounter.tc.HPET.counter: 1883995229
>> kern.timecounter.tc.HPET.mask: 4294967295
>> kern.timecounter.tc.i8254.quality: 0
>> kern.timecounter.tc.i8254.frequency: 1193182
>> kern.timecounter.tc.i8254.counter: 57
>> kern.timecounter.tc.i8254.mask: 65535
>> kern.timecounter.tc.TSC-low.quality: 1000
>> kern.timecounter.tc.TSC-low.frequency: 1413153007
>> kern.timecounter.tc.TSC-low.counter: 2352002295
>> kern.timecounter.tc.TSC-low.mask: 4294967295
>>
>> I overrode the default selection of counter-type as NTPD drifted so
>> badly as to require stepping almost hourly :-(
> 
> Could you show output from
> 
> # kldload cpuctl
> # cpucontrol -i 0x15 /dev/cpuctl0
> # cpucontrol -i 0x16 /dev/cpuctl0
> 
> as well as a copy of the dmesg after a boot?  I am looking at a similar
> problem currently.

root@vm01:/usr/home/imb # cpucontrol -i 0x15 /dev/cpuctl0
cpuid level 0x15: 0x07280202 0x00000000 0x00000000 0x00000503
root@vm01:/usr/home/imb # cpucontrol -i 0x16 /dev/cpuctl0
cpuid level 0x16: 0x07280202 0x00000000 0x00000000 0x00000503

This is a Dell-1950 1-U box with a SAS drive-box attached ..

root@vm01:/usr/home/imb # less /var/log/dmesg.today
---<<BOOT>>---
VERBOSE_SYSINIT: DDB not enabled, symbol lookups disabled.
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #256 main-42dfad2ef1: Sat Oct  2 09:41:36 EDT 2021
 
root@vm01.auburn.protected-networks.net:/usr/obj/usr/src/amd64.amd64/sys/VM01 
amd64
FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git 
llvmorg-12.0.1-0-gfed41342a82f)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU           E5440  @ 2.83GHz (2826.31-MHz 
K8-class CPU)
   Origin="GenuineIntel"  Id=0x10676  Family=0x6  Model=0x17  Stepping=6
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   VT-x: HLT,PAUSE
   TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 65811677184 (62762 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
random: unblocking device.
Security policy loaded: MAC/ntpd (mac_ntpd)
ioapic0: MADT APIC ID 8 != hw id 0
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 3 4 1 6 2 5 7
Timecounter "TSC-low" frequency 1413155409 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
smbios0: <System Management BIOS> at iomem 0xfcdf0-0xfce0e
smbios0: Version: 2.5, BCD Revision: 2.5
acpi0: <DELL PE_SC3>
acpi0: Power Button (fixed)
Firmware Error (ACPI): Could not resolve symbol [\134_SB._OSC.CDW1], 
AE_NOT_FOUND (20210930/psargs-503)
ACPI Error: Aborting method \134_SB._OSC due to previous error 
(AE_NOT_FOUND) (20210930/psparse-689)
apei0: <ACPI Platform Error Interface> on acpi0
ipmi0: <IPMI System Interface> port 0xca8,0xcac on acpi0
ipmi0: KCS mode found at io 0xca8 on acpi
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
bce0: <QLogic NetXtreme II BCM5708 1000Base-T (B2)> mem 
0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci4
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 00:1e:c9:b2:75:6e
bce0: ASIC (0x57081020);
bce0: link state changed to DOWN
Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs 
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci2
pci5: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.3 on pci1
pci6: <PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci7: <ACPI PCI bus> on pcib7
mpt0: <LSILogic SAS/SATA Adapter> port 0xec00-0xecff mem 
0xfc6fc000-0xfc6fffff,0xfc6e0000-0xfc6effff irq 16 at device 0.0 on pci7
mpt0: MPI Version=1.5.18.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 1 Active Volume (2 Max)
mpt0: 2 Hidden Drive Members (14 Max)
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
mpt0:vol0(mpt0:0:0): pci8: <ACPI PCI bus>Settings ( Member-WCE 
Hot-Plug-Spares High-Priority-ReSync )
mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
mpt0:vol0(mpt0:0:0):  on pcib8
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 07.709.04.00-fbsd
2 Members:
       (mpt0:1:1:0): Primary Online
       (mpt0:1:9:0): Secondary Online
mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
mpt0:vol0(mpt0:0:0): Status ( Enabled )
(mpt0:vol0:0): mrsas0: <AVAGO Thunderbolt SAS Controller>Physical 
(mpt0:0:1:0), Pass-thru (mpt0:1:0:0)
(mpt0:vol0:0): Online
(mpt0:vol0:1): Physical (mpt0:0:9:0), Pass-thru (mpt0:1:1:0)
  port 0xdc00-0xdcff mem 0xfc4dc000-0xfc4dffff,0xfc480000-0xfc4bffff irq 
16 at device 0.0(mpt0:vol0:1): Online
(noperiph:mpt0:1:-1:ffffffff): rescan already queued
  on pci8
mrsas0: FW now in Ready state
mrsas0: Using MSI-X with 8 number of vectors
mrsas0: FW supports <16> MSIX vector,Online CPU 8 Current MSIX <8>
mrsas0: mrsas_init_adapter: sc->reply_q_depth 0x7e0,sc->request_alloc_sz 
0x1f78, sc->reply_alloc_sz 0x3f00,sc->io_frames_alloc_sz 0x3f100
mrsas0: max sge: 0x46, max chain frame size: 0x400, max fw cmd: 0x3ef 
sc->chain_frames_alloc_sz: 0xfbc00
mrsas0: Issuing IOC INIT command to FW.
mrsas0: IOC INIT response received from FW.
mrsas0: VD created target ID: 0x0
mrsas0: VD created target ID: 0x1
mrsas0: VD created target ID: 0x2
mrsas0: VD created target ID: 0x3
mrsas0: max_fw_cmds: 1007  max_scsi_cmds: 991
mrsas0: MSI-x interrupts setup success
mrsas0: mrsas_ocr_thread
pcib9: <PCI-PCI bridge> at device 5.0 on pci0
pci9: <PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> at device 7.0 on pci0
pci11: <PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <PCI-PCI bridge> at device 0.0 on pci12
pci13: <PCI bus> on pcib13
bce1: <QLogic NetXtreme II BCM5708 1000Base-T (B2)> mem 
0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci13
miibus1: <MII bus> on bce1
brgphy1: <BCM5708C 1000BASE-T media interface> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 00:1e:c9:b2:75:6c
bce1: ASIC (0x57081020);
bce1: link state changed to DOWN
Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs 
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 
0xbce0-0xbcff irq 21 at device 29.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 
0xbcc0-0xbcdf irq 20 at device 29.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 
0xbca0-0xbcbf irq 21 at device 29.2 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port 
0xbc80-0xbc9f irq 20 at device 29.3 on pci0
usbus3 on uhci3
usbus3: 12Mbps Full Speed USB v1.0
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xfc800000-0xfc8003ff irq 
21 at device 29.7 on pci0
usbus4: EHCI version 1.0
usbus4 on ehci0
usbus4: 480Mbps High Speed USB v2.0
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
vgapci0: <VGA-compatible display> port 0xcc00-0xccff mem 
0xd8000000-0xdfffffff,0xfc2d0000-0xfc2dffff irq 19 at device 13.0 on pci14
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
tpm0: <Trusted Platform Module> iomem 0xfed40000-0xfed44fff on acpi0
tpm: device 0x4a100000 rev 0x4e
WARNING: Device "tpm" is Giant locked and may be deleted before FreeBSD 
14.0.
ichwd0: <Intel 63XXESB watchdog timer> on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xec000-0xeffff pnpid 
ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
WARNING: Adding ifaddrs to all fibs has been turned off by default. 
Consider tuning net.add_addr_allfibs if needed
ipmi0: IPMI device rev. 0, firmware rev. 2.50, version 2.0, device 
support mask 0xdf
(probe16:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe16:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe16:mpt0:1:1:0): Retrying command, 3 more tries remain
(probe0:mpt0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
(probe0:mpt0:0:0:0): CAM status: SCSI Status Error
(probe0:mpt0:0:0:0): SCSI status: Check Condition
(probe0:mpt0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff 
(Reserved ASC/ASCQ pair)
(probe0:mpt0:0:0:0): Error 22, Unretryable error
ipmi0: Number of channels 4
ipmi0: Attached watchdog
(probe16:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00
ipmi0: Establishing power cycle handler
(probe16:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe16:mpt0:1:1:0): Retrying command, 2 more tries remain
ugen2.1: <Intel UHCI root HUB> at usbus2
ugen3.1: <Intel UHCI root HUB> at usbus3
ugen1.1: <Intel UHCI root HUB> at usbus1
ugen0.1: <Intel UHCI root HUB> at usbus0
ugen4.1: <Intel EHCI root HUB> at usbus4
(probe16:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00
uhub0(probe16:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
  on usbus0
(probe16:mpt0:1:1:0): Retrying command, 1 more tries remain
(probe16:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00
uhub1(probe16:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
  on usbus4
uhub2(probe16:mpt0:1:1:0): Retrying command, 0 more tries remain
  on usbus3
(probe16:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00
uhub3(probe16:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe16:mpt0:1:1:0): Error 5, Retries exhausted
  on usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub4 on usbus2
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
mrsas0: Disestablish mrsas intr hook
da1 at mrsas0 bus 0 scbus2 target 0 lun 0
da1: <LSI MR9286CV-8e 3.46> Fixed Direct Access SPC-3 SCSI device
da1: Serial Number 0025d5dba5b1efde25a00a2d07b00506
da1: 150.000MB/s transfers
da1: 1907200MB (3905945600 512 byte sectors)
da2 at mrsas0 bus 0 scbus2 target 1 lun 0
da2: <LSI MR9286CV-8e 3.46> Fixed Direct Access SPC-3 SCSI device
da2: Serial Number 00e667ffa5b3efde25a00a2d07b00506
da2: 150.000MB/s transfers
da2: 1907200MB (3905945600 512 byte sectors)
da3 at mrsas0 bus 0 scbus2 target 2 lun 0
da3: <LSI MR9286CV-8e 3.46> Fixed Direct Access SPC-3 SCSI device
da3: Serial Number 00b237c3fa7812e025a00a2d07b00506
da3: 150.000MB/s transfers
da3: 279488MB (572391424 512 byte sectors)
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <Dell VIRTUAL DISK 1028> Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 1906394MB (3904294912 512 byte sectors)
da4 at mrsas0 bus 0 scbus2 target 3 lun 0
da4: <LSI MR9286CV-8e 3.46> Fixed Direct Access SPC-3 SCSI device
da4: Serial Number 00bcad67fac0498a27a00a2d07b00506
da4: 150.000MB/s transfers
da4: 22887424MB (46873444352 512 byte sectors)
ses0 at mpt0 bus 0 scbus0 target 8 lun 0
ses0: <DP BACKPLANE 1.05> Fixed Enclosure Services SPC-3 SCSI device
ses0: 300.000MB/s transfers
ses0: SES Device
pass2 at mpt0 bus 1 scbus1 target 0 lun 0
pass2: <ATA Hitachi HUA72202 A3NH> Fixed Uninstalled SPC-3 SCSI device
pass2: Serial Number B8GY2W3Z
pass2: 300.000MB/s transfers
pass2: Command Queueing enabled
cd0 at ata0 bus 0 scbus4 target 0 lun 0
cd0: <HL-DT-ST CDRW/DVD GCCT10N A102> Removable CD-ROM SCSI device
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> 
IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> 
IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
Trying to mount root from zfs:zroot/ROOT/default []...
Root mount waiting for: usbus0 usbus1 usbus2 usbus3 usbus4
uhub3: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
Root mount waiting for: usbus4
Root mount waiting for: usbus4
Root mount waiting for: usbus4
uhub1: 8 ports with 8 removable, self powered
ugen4.2: <vendor 0x413c product 0xa001> at usbus4
uhub5 on uhub1
uhub5: <vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2> 
on usbus4
uhub5: MTT enabled
Root mount waiting for: usbus4
uhub5: 2 ports with 2 removable, self powered
ugen4.3: <Dell DRAC5> at usbus4
ukbd0 on uhub5
ukbd0: <Keyboard> on usbus4
kbd2 at ukbd0
ums0 on uhub5
ums0: <Mouse> on usbus4
ums0: 3 buttons and [Z] coordinates ID=0
Root mount waiting for: usbus4
ugen4.4: <vendor 0x04b4 product 0x6560> at usbus4
uhub6 on uhub1
uhub6: <vendor 0x04b4 product 0x6560, class 9/0, rev 2.00/0.0b, addr 4> 
on usbus4
uhub6: MTT enabled
ugen1.2: <NOVATEK USB Keyboard> at usbus1
ukbd1 on uhub3
ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.04, addr 2> on usbus1
kbd3 at ukbd1
Root mount waiting for: usbus4
uhub6: 4 ports with 4 removable, self powered
bridge0: Ethernet address: 58:9c:fc:10:ff:90
tap0: Ethernet address: 58:9c:fc:10:f3:53
lo0: link state changed to UP
bce0: Gigabit link up!
bce0: link state changed to UP
bce0: promiscuous mode enabled
bridge0: link state changed to UP
bce1: Gigabit link up!
bce1: link state changed to UP
nfsd: can't register svc name
tap0: link state changed to UP
[fib_algo] inet.0 (bsearch4#67) rebuild_fd_flm: switching algo to 
radix4_lockless
[fib_algo] inet.1 (bsearch4#79) rebuild_fd_flm: switching algo to 
radix4_lockless

	imb