Adding write_same16 to CAM Target Layer

[Kenneth D. Merry]

On Sat, Mar 29, 2014 at 05:57:38 +0530, bharat singh wrote:
> I am using block back end with block size 512bytes. Yes as per t10 doc
> initiator will send 512bytes of data to be filled over 1MB.
> 
> Block size is 512 and no of blocks is 2048 in my case.

Hmm.  If datalen is actually 1048576, and the initiator is sending the same
amount of data, there shouldn't be a problem.

But that doesn't agree with what you said above, or with the spec.  The
spec does say that it will transfer one block.  If blocksize = 512 and 
num_blocks = 2048, you're asking for 1MB.  

It looks like the initiator may be sending 1MB.  But I'm not sure why
you're getting an overrun.  If anything, I'd expect an underrun.

> >From CTL layer can I reallocate the data buffers.

Yes, but that will slow things down to free and allocate the buffers again.

In order to scale WRITE SAME, you'd probably want to allocate say a 1MB
buffer in the block backend.  Transfer a 512 byte block from the initiator,
and then copy it to the rest of the blocks in the 1MB and issue the write.

Or, change the file dispatch path (ctl_be_block_dispatch_file()) to have a
long S/G list of segments in a uio.  For the device path
(ctl_be_block_dispatch_dev()) that would be a more difficult thing to
implement.  There isn't currently a way that I know of to attach a
scatter/gather list to a struct bio.  (But there should be!)

> This command is needed for initiators hosted on ESXi.

It'll be nice to support ESXi.  Are you able to contribute your changes
back to FreeBSD once you've got them working?

Ken

>  On 29-Mar-2014 12:23 AM, "Kenneth D. Merry" <ken at freebsd.org> wrote:
> 
> > On Wed, Mar 26, 2014 at 11:44:32 +0530, bharat singh wrote:
> > > Hello,
> > >
> > > I am trying to add write_same16 (block zero) support to my FC stack.
> > > This i am trying to achieve by adding a hook in CTL layer, reading
> > > start_LBA and no of blocks to fill with 0.
> > >
> > > struct ctl_scsiio {
> > > >-------struct ctl_io_hdr io_hdr;>------/* common to all I/O types */
> > > >-------uint32_t   ext_sg_entries;>-----/* 0 = no S/G list, > 0 = num
> > > entries */
> > > >-------uint8_t>   *ext_data_ptr;>------/* data buffer or S/G list */
> > > >-------uint32_t   ext_data_len;>-------/* Data transfer length */
> > > >-------uint32_t   ext_data_filled;>----/* Amount of data filled so far
> > */
> > > >-------uint32_t   kern_sg_entries;>----/* 0 = no S/G list, > 0 = num
> > > entries */
> > > >-------uint32_t   rem_sg_entries;>-----/* 0 = no S/G list, > 0 = num
> > > entries */
> > > >-------uint8_t    *kern_data_ptr;>-----/* data buffer or S/G list */
> > > >-------uint32_t   kern_data_len;>------/* Length of this S/G
> > list/buffer */
> > > >-------uint32_t   kern_total_len;>-----/* Total length of this
> > transaction
> > > */
> > > >-------uint32_t   kern_data_resid;>----/* Length left to transfer after
> > > this*/
> > > >-------uint32_t   kern_rel_offset;>----/* Byte Offset of this transfer
> > */
> > > >-------struct     scsi_sense_data sense_data;>-/* sense data */
> > > >-------uint8_t>   sense_len;>-->-------/* Returned sense length */
> > > >-------uint8_t>   scsi_status;>>-------/* SCSI status byte */
> > > >-------uint8_t>   sense_residual;>-----/* sense residual length */
> > > >-------uint32_t   residual;>--->-------/* data residual length */
> > > >-------uint32_t   tag_num;>---->-------/* tag number */
> > > >-------ctl_tag_type tag_type;>->-------/* simple, ordered, head of
> > > queue,etc.*/
> > > >-------uint8_t    cdb_len;>---->-------/* CDB length */
> > > >-------uint8_t>   cdb[CTL_MAX_CDBLEN];>/* CDB */
> > > >-------int>----   (*be_move_done)(union ctl_io *io); /* called by fe */
> > > >-------int        (*io_cont)(union ctl_io *io); /* to continue
> > processing
> > > */
> > > >-------uint32_t   ctl_lun_masking_disabled; /* Disable lun mask checks
> > for
> > > ctladm calls */
> > > };
> > >
> > > For write_same16 initiators will send a block of 0s with start_LBA and no
> > > of blocks. Assuming that I am trying to:
> > > datalen = blocksize * num_blocks;
> > > uint8_t * databuf = malloc(datalen);
> > > bzero(databuf, 0, datalen);
> > > ctsio->kern_data_ptr = dataptr;
> > > ctsio->kern_data_len = datalen;
> > > lbalen.lba = lba;
> > > lbalen.len = num_blocks;
> > > memcpy(ctsio->io_hdr.ctl_private[CTL_PRIV_LBA_LEN].bytes, &lbalen,
> > > sizeof(lbalen));
> > > ret = lun->backend->data_submit((union ctl_io *)ctsio);
> > >
> > > but it's failing in ISP layer with data overflow. My initiator is sending
> > > 2048 blocks of size 512B each and a payload of 512 bytes in data-out
> > buffer.
> > > isp0: isp_target_start_ctio: [0x120154] data overflow by 1048064 bytes
> > > (1:3:0:1): WRITE SAME(16). CDB: 93 00 00 00 00 00 00 1f e8 00 00 00 08 00
> > > 00 00
> > >
> > > Have any one done changes to the CTL read/write path, am I missing
> > > something regarding buffer population.
> > > Thanks for the help in advance.
> >
> > What back end are you using?  The block backend, or are you writing your
> > own?
> >
> > In the case of the block backend, it is expecting to do its own memory
> > allocation and then call ctl_datamove() to trigger the DMA.
> >
> > In this case, you're doing a malloc, but if you're going to the block
> > backend the data buffer will get overwritten.
> >
> > As for why you're getting an overrun, the amount of the overrun is 1048064
> > bytes.  That is 512 less than 1048576.  Is your blocksize and number of
> > blocks set correctly?  It looks like datalen is probably 512, but your
> > initiator is sending 1MB.
> >
> > Ken
> > --
> > Kenneth Merry
> > ken at FreeBSD.ORG
> >

-- 
Kenneth Merry
ken at FreeBSD.ORG