Target mode problems with scsi_target and firewire

[Paul Thornton]

Hi,

I've been trying to make target mode and firewire work using 8.1 release
(i386), and haven't had much success.

Using Sean Bruno's BSDCan presentation as a guide, I have a kernel with
the right bits in it, and am using scsi_target to provide a file-based
back-end (I'm not trying to make any physical SCSI discs act as the
backing store here), but the whole setup is very unreliable.

In a nutshell, and after a lot of testing on different hardware, client
machines and FreeBSD versions, it seems to come down to the fact that
the setup works for a short period of time (1s - ~30s is usual) and then
scsi_target hangs on the box being used as the target.

I've tested with FreeBSD, Windows and Mac OS as the client and I see the
same issue - sometimes I get as far as mounting the disc, and starting
to copy files; other times I can't even mount it.  Sometimes,
scsi_target hangs so early on the OS at the far end never works out that
it has gained a disc.

I'm looking for some help in how best to troubleshoot.  I'm not
sufficiently advanced with either the scsi code or the debugger to know
precisely where to look to see exactly where things are hanging.  I'll
happily try out anything things with this setup to get it working though
- it is a test after all, and not in production.

When I ktrace scsi_target, it does what seems sensible for a while until
a call to kevent never returns.  Nothing can then get rid of that hung
scsi_target short of a power cycle.  fwcontrol on both the target PC and
client still shows the other ID present, so the actual underlying link
hasn't gone away.

I've put the full output of scsi_target in debug mode, the ktrace
results, kernel config, dmesg etc relating to this here:
http://www.prt.org/temp/targ - interesting extracts of these are:

scsi_target: event -3 done
scsi_target: read ready
scsi_target: tcmd_handle atio 0x2826e740 ctio 0x2826e8c0 atioflags 0x8000
scsi_target: CTIO done freeing CTIO
scsi_target: sending ccb (0x332)
scsi_target: event -1 done
scsi_target: Working on ATIO 0x2826e5c0
scsi_target: tcmd_handle atio 0x2826e5c0 ctio 0x2826e8c0 atioflags 0x8000
scsi_target: Calling rdwr_decode
scsi_target: R/W from 0: 28 0 0 0 1 17 0 0 8 0
scsi_target: read 8 blocks @ blkno 279
scsi_target: read async 8d @ block 279
scsi_target: Starting 0x282702e0 DIR_IN @142848:4096
scsi_target: aio ready
scsi_target: tcmd_handle atio 0x2826e5c0 ctio 0x2826e8c0 atioflags 0x8000
scsi_target: sending CTIO for sync read
scsi_target: sending ccb (0x933)
scsi_target: event -3 done
(here we hang)

and the corresponding last few lines from the ktrace:

  1056 scsi_target CALL  write(0x2,0xbfbf8370,0xd)
  1056 scsi_target GIO   fd 2 wrote 13 bytes
       "event -3 done"
  1056 scsi_target RET   write 13/0xd
  1056 scsi_target CALL  write(0x2,0x281ac813,0x1)
  1056 scsi_target GIO   fd 2 wrote 1 byte
       "
       "
  1056 scsi_target RET   write 1
  1056 scsi_target CALL  kevent(0x5,0,0,0xbfbf8934,0x405,0)

My kernel configuration is GENERIC with:
device         targ
device         sbp_targ
options        VFS_AIO
options        DDB
options        KDB
options        KDB_UNATTENDED

added.
The only other firewire changes are disabling fwe, fwip, dcons and
dcons_crom.  sbp remains disabled.

I have also tried this with witness enabled; this causes a lot of
console output (as you'd expect) and when scsi_target hangs I ended up
with a panic:

--- syscall (4, FreeBSD ELF32, write), eip = 0x2819a583, esp =
0xbfbf87dc, ebp = 0xbfbf87f8 ---
Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex sbp_targ (sbp_targ) r = 0 (0xc473aaa4) locked @
cam/cam_periph.h:188
KDB: stack backtrace:
db_trace_self_wrapper(c0c58890,e6be3944,c08bb395,bc,1,...) at
db_trace_self_wrapper+0x26
kdb_backtrace(bc,1,ffffffff,c0ede344,e6be397c,...) at kdb_backtrace+0x29
_witness_debugger(c0c59f78,e6be3990,4,1,0,...) at _witness_debugger+0x25
witness_warn(5,0,c0c85213,c08bb48c,c4f35550,...) at witness_warn+0x20d
trap(e6be3a1c) at trap+0x19e
calltrap() at calltrap+0x6
--- trap 0xc, eip = 0xc0b86525, esp = 0xe6be3a5c, ebp = 0xe6be3a6c ---
fubyte(28283000,1,c0f17a60,0,0,...) at fubyte+0x1d
vmapbuf(d84e3df0,0,c0bf506b,2af,1,...) at vmapbuf+0x75
cam_periph_mapmem(c4ecf800,c52a4de0,c52a4de0,c4f38000,c4ecf800,...) at
cam_periph_mapmem+0x29e
targsendccb(c4e8e61c,c0bf6294,c4531c10,0,c5006880,...) at targsendccb+0x5e
targstart(c5006880,c4ecf800,1,c4775ac0,c4ecf800,...) at targstart+0x68
xpt_run_dev_allocq(c4766d80,c48bd808,1) at xpt_run_dev_allocq+0xdd
xpt_schedule(c5006880,1,c0bf628b,bc,0,...) at xpt_schedule+0xf4
targwrite(c5078a00,e6be3c5c,0,1b0,4,...) at targwrite+0x181
giant_write(c5078a00,e6be3c5c,0,0,0,...) at giant_write+0x60
devfs_write_f(c4f211c0,e6be3c5c,c4f0ed00,0,c4f32780,...) at
devfs_write_f+0x7f
dofilewrite(e6be3c5c,ffffffff,ffffffff,0,c4f211c0,...) at dofilewrite+0x95
kern_writev(c4f32780,4,e6be3c5c,e6be3c7c,1,...) at kern_writev+0x58
write(c4f32780,e6be3cf8,1,c0c4175f,206,...) at write+0x4f
syscall(e6be3d38) at syscall+0x200
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (4, FreeBSD ELF32, write), eip = 0x2819a583, esp =
0xbfbf87dc, ebp = 0xbfbf87f8 ---


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x28283000
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0b86525
stack pointer           = 0x28:0xe6be3a5c
frame pointer           = 0x28:0xe6be3a6c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1062 (scsi_target)
trap number             = 12
panic: page fault


Any advice on how to carry on tracking this down would be much appreciated.


Paul.