iSCSI stall

[peter.blok at bsd4all.org]

Hi Alexander,

I suspected re-ordering too and that’s why I changed max_tags to 1, but that didn’t work the way I expected. In the wireshark trace I still saw multiple CDB READ arriving,without a data transfer. If one of the CDBs was a SYNC CACHE everything stalled without any data transfer. When I did send a TUR, the data transfer happened and everything continued.

The CDB read had to be orgination from some ZFS (meta)operation, because bulk of the data was arroving thru zfs recv and most of the CDB were writes.

It did not stall all the time in similar sequences.

I did not have the exact target source code, nor build capability and time to debug, so I’m very glad those commits fixed it.

Peter



> On 13 Feb 2019, at 16:34, Alexander Motin <mav at freebsd.org> wrote:
> 
> Hi Peter,
> 
> If those commits helped (I did hope they improve performance in some
> configurations like this), then I guess this NAS has some problems with
> ordered tag handling in its iSCSI target, and this change workarounded
> it.  Our iSCSI initiator does not specially care about ordered requests,
> just forwards the flag straight to the target, so it should not change
> anything on our side.
> 
> On 13.02.2019 04:07, peter.blok at bsd4all.org <mailto:peter.blok at bsd4all.org> wrote:
>> Hi,
>> 
>> Did not get any response on this, but it seems the commits 344072, 344075 and 344076 have fixed the issue. My assumption is that the target doesn’t handle re-ordering with non-offset CDBs very well.
>> 
>> Normally every time when the backup (zfs recv on target) ran, it hung a couple of times and now it went thru without a glitch and without any quirks.
>> 
>> Peter
>> 
>> 
>>> On 16 Jan 2019, at 10:05, peter.blok at bsd4all.org wrote:
>>> 
>>> Hi,
>>> 
>>> I have a ReadyNAS RND4000 as an iSCSI target. The target is the only device in a ZFS pool and is receiving backups thru zfs receive. The initiator is FreeBSD 11.x (recent stable at this moment), but this probably happens with 12.0 as well.
>>> 
>>> During a zfs receive it stalls. Nothing happens anymore. When I send a TUR via camcontrol everything continues. Any other target on any other platform works fine.
>>> 
>>> When I take a network trace it looks as if it is around SYNCHRONIZE CACHE. If this takes too long in the target and before the reply is back some new READ CDBs are send, it seems to hang only showing Nop-In and Nop-out. The moment I send a TUR it continues replying with the data of the READs that were owed.
>>> 
>>> It is not a big deal, but it irritates me and I would like to create a work-around for this, assuming it is caused by the target and there is probably no way to fix it there. One test was to create a quirk to reduce the tags to 1, but that didn’t help. Although it is 1 it still seems to send multiple CDB concurrently.
>>> 
>>> In the past I have attempted to port FreeBSD to the ReadyNAS, but got stuck on fan control. I wasn’t able to control the fan and it made too much noise. I was able to get the network up and running.
>>> 
>>> Any tips?
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> freebsd-scsi at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
>>> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"
>> 
> 
> -- 
> Alexander Motin