Hang on boot in isp with QLA2342 after upgrading to 6.3

[Graham Allan]

Hi,

I've been trying to figure out a problem on a system which I just
upgraded from FreeBSD 6.1 to 6.3. It's a Dell 1750 with a QLA2342
(isp2312) HBA, connected to an EMC DS-16B2 (aka Brocade 3800) switch,
and from there to a couple of AC&NC Jetstor arrays. It's all been
working fine like this for some time under 6.1.

When I boot the system under 6.3, though I get a hang like this:

isp0: Interrupting Mailbox Command (0x6f) Timeout (500000us)
isp0: PLOGI 0x031a00 failed
isp0: Interrupting Mailbox Command (0x6e) Timeout (10000000us)
isp0: Mailbox Command 'SEND SNS' failed (TIMEOUT)
isp0: isp_pdb_sync: isp_scan_fabric failed
isp0: Interrupting Mailbox Command (0x6e) Timeout (10000000us)
isp0: Mailbox Command 'SEND SNS' failed (TIMEOUT)
isp0: isp_pdb_sync: isp_scan_fabric failed
isp0: Mailbox Command 'SEND SNS' failed (COMMAND ERROR)
isp0: isp_pdb_sync: isp_scan_fabric failed

after which the system stays hung.

I do have ispfw_load="YES" set (it reports loading F/W revision
3.3.19):

isp0: <Qlogic ISP 2312 PCI FC-AL Adapter> port 0xdc00-0xdcff mem 0xfcf01000-0xfcf01fff irq 20 atdevice 4.0 on pci1
isp0: [GIANT-LOCKED]
isp0: Board Type 2312, Chip Revision 0x2, loaded F/W Revision 3.3.19
isp1: <Qlogic ISP 2312 PCI FC-AL Adapter> port 0xd800-0xd8ff mem 0xfcf00000-0xfcf00fff irq 21 atdevice 4.1 on pci1
isp1: [GIANT-LOCKED]
isp1: Board Type 2312, Chip Revision 0x2, loaded F/W Revision 3.3.19

I initially suspected a hardware problem, but I've retested with a
couple of different QLA2342 cards in a couple of different 1750
systems. I did find one or two interesting things though.

1) Reinstalling 6.1 has everything working again (as long as
ispfw_load="YES" is set).

2) if, under 6.3, I connect the HBA directly to a storage device (eg a
tape drive/loader) instead of the SAN switch, it works fine. So possibly
something related to point-to-point mode rather than fabric?

3) If I connect one port of the HBA to the tape loader, and the other
to the SAN switch, it also boots up successfully, although we get the
same errors reported as above, without the hang...

isp0: Interrupting Mailbox Command (0x6f) Timeout (500000us)
isp0: PLOGI 0x031a00 failed
isp0: Interrupting Mailbox Command (0x6e) Timeout (10000000us)
isp0: Mailbox Command 'SEND SNS' failed (TIMEOUT)
isp0: isp_pdb_sync: isp_scan_fabric failed
isp0: Interrupting Mailbox Command (0x6e) Timeout (10000000us)
isp0: Mailbox Command 'SEND SNS' failed (TIMEOUT)
isp0: isp_pdb_sync: isp_scan_fabric failed
isp0: Mailbox Command 'SEND SNS' failed (COMMAND ERROR)
isp0: isp_pdb_sync: isp_scan_fabric failed
sa0 at isp1 bus 0 target 0 lun 0
sa0: <QUANTUM DLT-S4 1F1F> Removable Sequential Access SCSI-4 device
sa0: 200.000MB/s transfers
da0 at isp0 bus 0 target 1 lun 0
da0: <JetStor FreeBSD mark R4 R001> Fixed Direct Access SCSI-5 device
da0: 200.000MB/s transfers, Tagged Queueing Enabled
da0: 953MB (1952256 512 byte sectors: 64H 32S/T 953C)
...

Finally I booted with "hint.isp.0.debug=0x1F" in /boot/device.hints,
while connected only to the switch, and... unfortunately I wasn't able
to capture the extensive output, but the final text was:

isp0: target 496 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 497 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 498 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 499 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 500 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 501 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 502 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 503 lun 0 CAM status 0xa SCSI status 0x0
isp0: target 504 lun 0 CAM status 0xa SCSI status 0x0

I will work on hooking up a serial console so I can capture the entire
output, though I'm also wondering if anyone might have some advice on
what to try next at this point?

Graham
-- 
-------------------------------------------------------------------------
Graham Allan - I.T. Manager - allan at physics.umn.edu - (612) 624-5040
School of Physics and Astronomy - University of Minnesota
-------------------------------------------------------------------------