iSCSI disconnects dilema

[Scott Long]

Danny Braniss wrote:
>> I forget, does iSCSI have a concept of a flush_cache command, or the
>> equivalent of what parallel SCSI does with ordered tags?
> 
> not realy - or I can't find it. iSCSI is mainly and envelope for
> scsi commands, so whatever the CAM does, it will pass it on. 
> There are some managemenet commands, so the target can tell the initiator
> that it's going down for example (and what should the driver
> do in such a case in freebsd?)
> 

If the periph is open (i.e. mounted), I'd just ignore this and have the
stack go through a normal retry timeout cycle to see if the device comes
back.  If it's closed, then I'd remove the periph.  Knowing if it's
opened or closed is likely hard to do from the iSCSI driver, which is
one reason why iSCSI knowledge needs to eventually be moved upwards in
CAM.

>>                                                           If so, then
>> that's how your app or OS knows that the transaction got committed to
>> stable storage.  It's been long assumed in the external storage world
>> that you are at the mercy of the external storage cache, so the problem
>> that Danny is referring to is nothing new.  The real question is how
>> to implement the equivalent mechanism that iSCSI provides in a way that
>> the OS/app can make use of it.  For example, CAM issues an ordered tag
>> periodically to flush the disk cache to stable storage. 
> nice, (or wishful thinking :-), the scsi part of iSCSI is/can be 
> software/virtual.
> 

If the target device returns a successful completion from a command, the
initiator must assume that it's not lying.  You could do a flush/sync
cache command after every I/O, but then you'd have a completely
unacceptable level of performance.  But again, this is not a new problem
specific to iSCSI.  It's long been a design consideration of external
storage, and is why external storage 1) carries a high price tag to
accompany good engineering and testing, and 2) comes with some form of
battery backup, to prevent data loss in case of power loss.

>>                                                         Most storage
>> drivers, including CAM, will issue some sort of a flush_cache command to
>> the controller and media during system shutdown.
> 
> this took me a long time to fix! the userland program got killed at shutdown,
> the link was lost, and so there was no way to flush buffers, fixed by calling
> fget(...) too.
> 
> I guess I can summarize: (and use the 3 monkey law :-)
> 	1- assume the target is 'well behaved' and will flush cache.
> 	2- there is - currently - no way to tell the OS that not all
> 	   seems to be as expected.
> 	3- keep quiet and hope for the best.
> danny
> 
> 

So you had a scenario where a program was doing I/O right up to system
(initiator) shutdown, and some of those I/O's got lost in the process?
I guess I don't understand why the OS didn't flush all outstanding I/O
buffers after terminating the program and before finishing the shutdown.
Maybe you are doing something illegal in your driver, or maybe you need
to implement a kernel shutdown hook that will allow you to block the
shutdown until everything is flushed.

Scott