[RFC] CAM pass(4) patch for NVMe

[Warner Losh]

On Tue, Apr 4, 2017 at 11:21 AM, Chuck Tuffli <chuck at tuffli.net> wrote:
> On Tue, Apr 4, 2017 at 7:54 AM, Warner Losh <imp at bsdimp.com> wrote:
>> On Tue, Apr 4, 2017 at 8:30 AM, Chuck Tuffli <chuck at tuffli.net> wrote:
>>> Hi
>>>
>>> I posted a patch on Phabricator[1] which adds IO pass-through support
>>> for NVMe and would like some feedback on it. The bulk of the change is
>>> pretty straight forward, but there was one part I'm not sure is the
>>> best approach. The driver needs to know what type of command the
>>> application is submitting in order to place it on the correct type of
>>> NVMe queue. The patch uses bit 5 (i.e. 0x10) in the xflags variable of
>>> the ccb header for this purpose. xflags was convenient to use, but
>>> since other drivers don't use it, my guess is this is "the wrong way".
>>> If it is OK to use xflags for this, cool, otherwise I'm open to
>>> suggestions.
>>
>> Yea, one of the comments I posted to review... At the very least, it
>> needs nvmecontrol uses different devices to distinguish between the
>> different queues. I find this somewhat unsatisfying, but it works.
>> nda, however, doesn't publish a transport level or sim level device
>> that one could easily latch on to. The issue becomes one of security:
>> If I can open nda0/pass0, can I control namespace operations for the
>> drive? Download new firmware? etc. However, you can do all these
>> things with other classes of devices today, so maybe that's not the
>> biggest issue.
>>
>>> As background, NVMe has two categories of commands: administrative and
>>> NVM (sometimes referred to as "IO"). I don't think it is possible for
>>> the driver to guess the command category based on the cmd bytes as:
>>>  - the command opcode values alias. E.g. Flush (NVM) and Delete
>>> Submission queue (Admin) use opcode 0x0
>>>  - both categories have commands which use a non-zero Namespace ID
>>> (like a LUN). so filtering on NSID wouldn't work
>>
>> In theory, you could use 0xffff as the NSID, since that's not a valid NSID.
>
> Hmm, I think any value greater than zero is valid according to the
> spec (note that 0xffffffff has a special meaning). Certainly, devices
> won't support NSID values above their published Identify Controller.NN
> (Number of Namespaces) value, but hijacking the NSID value to specify
> the NVMe queue seems problematic as there are Admin commands which
> require a valid NSID.

Fair Enough. I'd thought 0xffff was the magic number :). However, you
raise a good point.

Grep tells me all the xflags are actually unused. So we could use it,
but after chatting with Scott Long, I'm not sure that we should.

However, I think Jim's idea of having a separate command for commands
for the I/O queue and commands for the admin queue might be the better
part of valor here. I'd initially read Jim's mail as use #defines for
the xflags values, but that's not at all what he was saying.

The code change would be a bit bigger, but not by a lot. It's super
easy to add new XPT_ function code.

>> The current support of name spaces is a bit weak in nvme/nvd too. And
>> it's even weaker in nda since I didn't have a multi-namespace drive
>> until lately.
>>
>> So what are your goals with this work? I'm aware of another effort to
>> improve the name space support in nvmecontrol which isn't going
>> through nda. I'm also working on getting the nvmecontrol functionality
>> moved over to camcontrol.
>
> I'm working on a tool to gather SMART information and have been using
> CAM pass for ATA and SCIS. Having pass support for NVMe too would
> simplify the code and seemed like any easy addition seeing as how
> you'd already done all the heavy lifting.

Cool. I'd like to get all the functionality in nvmecontrol into
camcontrol, but haven't had a decent notion of how to do that for this
new transport that's so different than ATA and SCSI. Though we have
camcontrol identify and camcontrol inquiry for ATA and SCSI
respectively. There's no way to specify the protocol to send down for
things that could be generic, or base what we do based on the protocol
field of the passX device we wind up opening.

You might want to look at nvmecontrol. It already prints out all the
standard SMART log pages for NVMe as well as all the vendor-specific
ones I could find w/o NDA (and a few I did under NDA that aren't in
the tree due to vendor hesitance at approving their release). I had
hoped to use the same code for camcontrol when I got it working there,
but haven't worked to that point yet. I haven't done the SMART error
reporting that NVMe can do yet.

Warner