iscsi initiator - system hang

[Dennis R. Kolpanen]

On Wed, Sep 10, 2008 at 09:32:01AM +0300, Danny Braniss wrote:
> > On a FreeBSD 7.0 system, certain commands issued against any of the
> > three mounted iscsi drives causes the system to "hang".  In this
> > context, hang means:
> > 
> >   the system continues to respond to pings
> >   sendmail stops accepting connections
> >   imapd stops accepting connections
> >   sshd stops accepting connections
> >   shell sessions already established stop accepting commands
> >   login at the system console is not possible
> > 
> > The problem has been caused, so far, by dump, restore, and pax.  These
> > commands work perfectly if they are directed against one of the
> > internal drives and not the iscsi drives.  The failures noted above do
> > not normally happen immediately after issuing one of these commands.
> > The problems seem to build over a period of minutes or tens of
> > minutes.  Note that the dump/restore/pax commands can take hours to
> > run.
> > 
> > Nothing is written to the system console or any of the log files
> > indicating a problem.  The only way to recover from the hang is by
> > means of the hardware reset button.
> > 
> > When the system was first being set up and no other users were on it,
> > shutting down all but one of the CPUs by means of sysctl and
> > "machdep.hlt_cpus" allowed restoring about 150gb to the three iscsi
> > drives.  Once the machine was placed into production, massive hardware
> > problems on an old server required this to be done immediately, this
> > trick no longer works.
> > 
> > An overview of the hardware involved:
> > 
> >   dual, quad-core Intel Xeon processors
> >   16 gb RAM FreeBSD 7.0 amd64 release
> >   NetworkAppliance FAS2020 SAN
> >   generic kernel
> > 
> > A complete dmesg output can be provided if desired.
> > 
> > By default, iscontrol creates the iscsi drives with the number of tags
> > set to one.  The performance of the iscsi drives with this default
> > setting was quite poor.  Based on a recommendation made on a mailing
> > list some time ago, /etc/iscsi.conf was changed to set the tags to
> > 128.  This had a dramatic improvement on the iscsi performance.
> > 
> > Testing on the system that was rushed into production is not really
> > possible.  However, within the next week or so, a nearly identical
> > system should become available and this one could be used for testing.
> > 
> > Any ideas on what could be wrong?  Any solutions?
> > 
> 
> maybe, but first, can you send me the output of 'sysctl net.iscsi'?
> 
> 	danny
> 

Danny,

No problem.  Here it is:

net.iscsi.driver_version: 2.0.99
net.iscsi.isid: DIB00
net.iscsi.sessions: 1
net.iscsi.0.targetname: iqn.1992-08.com.netapp:sn.135029528
net.iscsi.0.targeaddress: 10.172.172.1
net.iscsi.0.stats: recv=1288005 sent=1287999 flags=0x0000011b pdus-alloc=0 pdus-max=220 cws=64 cmd=afa32 exp=afa32 max=afa71 stat=afa32 itt=afa32
net.iscsi.0.pid: 916

Thanks.

Dennis