main [and, likely, stable/14]: do not set vfs.zfs.bclone_enabled=1 with that zpool feature enabled because it still leads to panics
Date: Thu, 07 Sep 2023 18:02:16 UTC
I was requested to do a test with vfs.zfs.bclone_enabled=1 and
the bulk -a build paniced (having stored 128 *.pkg files in
.building/ first):
# more /var/crash/core.txt.3
. . .
Unread portion of the kernel message buffer:
panic: Solaris(panic): zfs: accessing past end of object 422/1108c16 (size=2560 access=2560+2560)
cpuid = 15
time = 1694103674
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0352758590
vpanic() at vpanic+0x132/frame 0xfffffe03527586c0
panic() at panic+0x43/frame 0xfffffe0352758720
vcmn_err() at vcmn_err+0xeb/frame 0xfffffe0352758850
zfs_panic_recover() at zfs_panic_recover+0x59/frame 0xfffffe03527588b0
dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x97/frame 0xfffffe0352758960
dmu_brt_clone() at dmu_brt_clone+0x61/frame 0xfffffe03527589f0
zfs_clone_range() at zfs_clone_range+0xa6a/frame 0xfffffe0352758bc0
zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0x1ae/frame 0xfffffe0352758c40
vn_copy_file_range() at vn_copy_file_range+0x11e/frame 0xfffffe0352758ce0
kern_copy_file_range() at kern_copy_file_range+0x338/frame 0xfffffe0352758db0
sys_copy_file_range() at sys_copy_file_range+0x78/frame 0xfffffe0352758e00
amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe0352758f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0352758f30
--- syscall (569, FreeBSD ELF64, copy_file_range), rip = 0x1ce4506d155a, rsp = 0x1ce44ec71e88, rbp = 0x1ce44ec72320 ---
KDB: enter: panic
__curthread () at /usr/main-src/sys/amd64/include/pcpu_aux.h:57
57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0 __curthread () at /usr/main-src/sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=textdump@entry=0)
at /usr/main-src/sys/kern/kern_shutdown.c:405
#2 0xffffffff804a442a in db_dump (dummy=<optimized out>, dummy2=<optimized out>, dummy3=<optimized out>, dummy4=<optimized out>)
at /usr/main-src/sys/ddb/db_command.c:591
#3 0xffffffff804a422d in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=true)
at /usr/main-src/sys/ddb/db_command.c:504
#4 0xffffffff804a3eed in db_command_loop ()
at /usr/main-src/sys/ddb/db_command.c:551
#5 0xffffffff804a7876 in db_trap (type=<optimized out>, code=<optimized out>)
at /usr/main-src/sys/ddb/db_main.c:268
#6 0xffffffff80bb9e57 in kdb_trap (type=type@entry=3, code=code@entry=0, tf=tf@entry=0xfffffe03527584d0) at /usr/main-src/sys/kern/subr_kdb.c:790
#7 0xffffffff8104ad3d in trap (frame=0xfffffe03527584d0)
at /usr/main-src/sys/amd64/amd64/trap.c:608
#8 <signal handler called>
#9 kdb_enter (why=<optimized out>, msg=<optimized out>)
at /usr/main-src/sys/kern/subr_kdb.c:556
#10 0xffffffff80b6aab3 in vpanic (fmt=0xffffffff82be52d6 "%s%s", ap=ap@entry=0xfffffe0352758700)
at /usr/main-src/sys/kern/kern_shutdown.c:958
#11 0xffffffff80b6a943 in panic (
fmt=0xffffffff820aa2e8 <vt_conswindow+16> "\312C$\201\377\377\377\377")
at /usr/main-src/sys/kern/kern_shutdown.c:894
#12 0xffffffff82993c5b in vcmn_err (ce=<optimized out>, fmt=0xffffffff82bfdd1f "zfs: accessing past end of object %llx/%llx (size=%u access=%llu+%llu)", adx=0xfffffe0352758890)
at /usr/main-src/sys/contrib/openzfs/module/os/freebsd/spl/spl_cmn_err.c:60
#13 0xffffffff82a84d69 in zfs_panic_recover (
fmt=0x12 <error: Cannot access memory at address 0x12>)
at /usr/main-src/sys/contrib/openzfs/module/zfs/spa_misc.c:1594
#14 0xffffffff829f8e27 in dmu_buf_hold_array_by_dnode (dn=0xfffff813dfc48978, offset=offset@entry=2560, length=length@entry=2560, read=read@entry=0, tag=0xffffffff82bd8175, numbufsp=numbufsp@entry=0xfffffe03527589bc, dbpp=0xfffffe03527589c0, flags=0)
at /usr/main-src/sys/contrib/openzfs/module/zfs/dmu.c:543
#15 0xffffffff829fc6a1 in dmu_buf_hold_array (os=<optimized out>, object=<optimized out>, read=0, numbufsp=0xfffffe03527589bc, dbpp=0xfffffe03527589c0, offset=<optimized out>, length=<optimized out>, tag=<optimized out>)
at /usr/main-src/sys/contrib/openzfs/module/zfs/dmu.c:654
#16 dmu_brt_clone (os=os@entry=0xfffff8010ae0e000, object=<optimized out>, offset=offset@entry=2560, length=length@entry=2560, tx=tx@entry=0xfffff81aaeb6e100, bps=bps@entry=0xfffffe0595931000, nbps=1, replay=0) at /usr/main-src/sys/contrib/openzfs/module/zfs/dmu.c:2301
#17 0xffffffff82b4440a in zfs_clone_range (inzp=0xfffff8100054c910, inoffp=0xfffff81910c3c7c8, outzp=0xfffff80fb3233000, outoffp=0xfffff819860a2c78, lenp=lenp@entry=0xfffffe0352758c00, cr=0xfffff80e32335200)
at /usr/main-src/sys/contrib/openzfs/module/zfs/zfs_vnops.c:1302
#18 0xffffffff829b4ece in zfs_freebsd_copy_file_range (ap=0xfffffe0352758c58)
at /usr/main-src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:6294
#19 0xffffffff80c7160e in VOP_COPY_FILE_RANGE (invp=<optimized out>, inoffp=0x40, outvp=0xfffffe03527581d0, outoffp=0xffffffff811e6eb7, lenp=0x0, flags=0, incred=0xfffff80e32335200, outcred=0x0, fsizetd=0xfffffe03586c0720) at ./vnode_if.h:2381
#20 vn_copy_file_range (invp=invp@entry=0xfffff8095e1e8000, inoffp=0x40, inoffp@entry=0xfffff81910c3c7c8, outvp=0xfffffe03527581d0, outvp@entry=0xfffff805d6107380, outoffp=0xffffffff811e6eb7, outoffp@entry=0xfffff819860a2c78, lenp=0x0, lenp@entry=0xfffffe0352758d50, flags=flags@entry=0, incred=0xfffff80e32335200, outcred=0xfffff80e32335200, fsize_td=0xfffffe03586c0720) at /usr/main-src/sys/kern/vfs_vnops.c:3085
#21 0xffffffff80c6b998 in kern_copy_file_range (
td=td@entry=0xfffffe03586c0720, infd=<optimized out>, inoffp=0xfffff81910c3c7c8, inoffp@entry=0x0, outfd=<optimized out>, outoffp=0xfffff819860a2c78, outoffp@entry=0x0, len=9223372036854775807, flags=0) at /usr/main-src/sys/kern/vfs_syscalls.c:4971
#22 0xffffffff80c6bab8 in sys_copy_file_range (td=0xfffffe03586c0720, uap=0xfffffe03586c0b20) at /usr/main-src/sys/kern/vfs_syscalls.c:5009
#23 0xffffffff8104bab9 in syscallenter (td=0xfffffe03586c0720)
at /usr/main-src/sys/amd64/amd64/../../kern/subr_syscall.c:187
#24 amd64_syscall (td=0xfffffe03586c0720, traced=0)
at /usr/main-src/sys/amd64/amd64/trap.c:1197
#25 <signal handler called>
#26 0x00001ce4506d155a in ?? ()
Backtrace stopped: Cannot access memory at address 0x1ce44ec71e88
(kgdb)
Context details follow.
Absent a openzfs-2.2 in:
ls -C1 /usr/share/zfs/compatibility.d/openzfs-2.*
/usr/share/zfs/compatibility.d/openzfs-2.0-freebsd
/usr/share/zfs/compatibility.d/openzfs-2.0-linux
/usr/share/zfs/compatibility.d/openzfs-2.1-freebsd
/usr/share/zfs/compatibility.d/openzfs-2.1-linux
I have copied:
/usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2
over to:
# ls -C1 /etc/zfs/compatibility.d/*
/etc/zfs/compatibility.d/openzfs-2.2
and used it:
# zpool get compatibility zamd64
NAME PROPERTY VALUE SOURCE
zamd64 compatibility openzfs-2.2 local
For reference:
# zpool upgrade
This system supports ZFS pool feature flags.
All pools are formatted using feature flags.
Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(7) for details.
Note that the pool 'compatibility' feature can be used to inhibit
feature upgrades.
POOL FEATURE
---------------
zamd64
redaction_list_spill
which agrees with openzfs-2.2 .
I did:
# sysctl vfs.zfs.bclone_enabled=1
vfs.zfs.bclone_enabled: 0 -> 1
I also made a snapshot: zamd64@before-bclone-test and
I then made a checkpoint. These were establshed just
after the above enable.
I then did a: zpool trim -w zamd64
The poudriere bulk command was: poudriere bulk -jmain-amd64-bulk_a -a
where main-amd64-bulk_a has nothing prebuilt. USE_TMPFS=no
is in use. No form of ALLOW_MAKE_JOBS is in use. It is a
32 builder context (32 hardware threads).
For reference:
# uname -apKU
FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #118 main-n265152-f49d6f583e9d-dirty: Mon Sep 4 14:26:56 PDT 2023 root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000
I'll note that with openzfs-2.1-freebsd compatibility I'd
previously let such a bulk -a run for about 10 hr and it
had reached 6366 port->package builds.
Prior to that I'd done shorter experiments with default
zpool features (no explicit compatibility constraint)
but vfs.zfs.bclone_enabled=0 and I'd had no problems.
(I have a separate M.2 boot media just for such experiments
and can reconstruct its content at will.)
All these have been based on the same personal
main-n265152-f49d6f583e9d-dirty system build. Unfortunately,
no appropriate snapshot of main was available to avoid my
personal context being involved for the system build used.
Similarly, the snapshot(s) of stable/14 predate:
Sun, 03 Sep 2023
. . .
git: f789381671a3 - stable/14 - zfs: merge openzfs/zfs@32949f256 (zfs-2.2-release) into stable/14
that has required fixes for other issues.
===
Mark Millard
marklmi at yahoo.com