panic: vm_fault_hold: fault on nofault entry in fusefs

[Konstantin Belousov]

On Tue, Jun 11, 2019 at 02:12:22PM -0600, Alan Somers wrote:
> Can somebody please help me to debug a fusefs problem?  I have a 100%
> reproducible panic with the above message.  Evidentially there's
> something I don't know about buf(9) and uiomove(9).  The good news is
> that the panic is sufficiently reproducible and sufficiently
> instrumented that I know exactly what's happening; I just don't know
> why.  Here's a summary of what happens.
> 
> 1) fusefs's VOP_WRITE method gets called with a buffer that spans a
> logical block boundary, but does not extend the size of the file.
> 2) It splits the write into two parts.  Each one calls getblk to
> allocate a struct buf, fills in the old data with a read, and fills
> the new data with uiomove.
> 3) After the file gets close()ed, VOP_INACTIVE calls vn_fsync_buf to
> flush dirty buffers.
> 4) VOP_STRATEGY successfully writes the first buffer and frees it with
> bufdone().
> 5) VOP_STRATEGY tries to write the second buffer, but panics during
> uiomove.  The address that caused the panic is always exactly 4KB into
> the buffer.
> 
> So what am I doing wrong?  The address that causes the panic in step 5
> was successfully accessed in step 2, so this isn't some kind of buffer
> overrun.  Does it have something to do with the fact that the read
> operation in step 2 called bufdone()?  Seems unlikely because it did
> that for both buffers, yet only the second one panics.  Or does the
> address actually fault during both VOP_WRITE and VOP_STRATEGY, but
> something low down handles the fault in the first case?  I'd be
> grateful for any help that anyone can offer.
> -Alan
> 
> P.S.
> Here's the panic's stack
> panic: vm_fault_hold: fault on nofault entry, addr: 0xfffffe0004591000
> cpuid = 1
> time = 1560283621
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0031c21f80
> vpanic() at vpanic+0x19d/frame 0xfffffe0031c21fd0
> panic() at panic+0x43/frame 0xfffffe0031c22030
> vm_fault_hold() at vm_fault_hold+0x2064/frame 0xfffffe0031c22170
> vm_fault() at vm_fault+0x60/frame 0xfffffe0031c221b0
> trap_pfault() at trap_pfault+0x188/frame 0xfffffe0031c22200
> trap() at trap+0x2b4/frame 0xfffffe0031c22310
> calltrap() at calltrap+0x8/frame 0xfffffe0031c22310
> --- trap 0xc, rip = 0xffffffff8108c9e6, rsp = 0xfffffe0031c223e0, rbp
> = 0xfffffe0031c223e0 ---
> memmove_erms() at memmove_erms+0x116/frame 0xfffffe0031c223e0
> uiomove_faultflag() at uiomove_faultflag+0x146/frame 0xfffffe0031c22420
> fuse_write_directbackend() at fuse_write_directbackend+0x1cd/frame
> 0xfffffe0031c224f0
> fuse_io_strategy() at fuse_io_strategy+0x24d/frame 0xfffffe0031c22590
> fuse_vnop_strategy() at fuse_vnop_strategy+0x2a/frame 0xfffffe0031c225a0
> VOP_STRATEGY_APV() at VOP_STRATEGY_APV+0x63/frame 0xfffffe0031c225c0
> bufstrategy() at bufstrategy+0x44/frame 0xfffffe0031c225f0
> bufwrite() at bufwrite+0x259/frame 0xfffffe0031c22640
> vn_fsync_buf() at vn_fsync_buf+0x23e/frame 0xfffffe0031c226a0
> fuse_vnop_inactive() at fuse_vnop_inactive+0x7e/frame 0xfffffe0031c226e0
> VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x63/frame 0xfffffe0031c22700
> vinactive() at vinactive+0xcd/frame 0xfffffe0031c22750
> vputx() at vputx+0x2d0/frame 0xfffffe0031c227b0
> vn_close1() at vn_close1+0x116/frame 0xfffffe0031c22820
> vn_closefile() at vn_closefile+0x4c/frame 0xfffffe0031c228a0
> _fdrop() at _fdrop+0x1a/frame 0xfffffe0031c228c0
> closef() at closef+0x1ec/frame 0xfffffe0031c22950
> closefp() at closefp+0x9c/frame 0xfffffe0031c22990
> amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe0031c22ab0
> fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0031c22ab0
> --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x8006842ba, rsp =
> 0x7fffffffe748, rbp = 0x7fffffffe760 ---
> KDB: enter: panic
Start with dumping core.  Then print out the struct buf and show it.