From nobody Wed Feb 09 01:46:53 2022 X-Original-To: scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6E53119B2FC5 for ; Wed, 9 Feb 2022 01:46:55 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JtjRM2SbKz3qB9; Wed, 9 Feb 2022 01:46:55 +0000 (UTC) (envelope-from jhb@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644371215; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8rZf1dXiJrRzMa0GuDltgTWaTm+mnbWxmpIpLr5IVlw=; b=igOOSU994RF0HWG/em6SASibm4SyH8zBUIIdS98zrYxErQhOgHnPgLSLWL3IvqpNZBo0hg ZGPJ99bAtdxfPZawLEVQlUBBNOzvfDxvnRTdx4KRTebPr/XniQxv5bqSbIxhHS2WBwHklg T3uteF2RXmvyqYhimxfah4uOlVgNm/IfKcWPSRkR25PYAE3XhAUCdXU4aIU3aBZlJePHYx xx2LWW24PjPQpVe7Xca8a5IQSoZ5K0y4+PKlA1sxtgOXThfwi1bMSLT124770mFki4U94q BTBvk1hKf6PXbWe3RbJr2DSY2IVmrrRdJLKhdChOEGK9ABqmJssmqBl3eldm4Q== Received: from [10.0.1.4] (ralph.baldwin.cx [66.234.199.215]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id 8B2BFE81D; Wed, 9 Feb 2022 01:46:54 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Message-ID: <12e16b21-429e-b0e0-1412-6eea61b31e16@FreeBSD.org> Date: Tue, 8 Feb 2022 17:46:53 -0800 List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Content-Language: en-US To: Ken Merry , Alexander Motin Cc: Scott Long , =?UTF-8?Q?Edward_Tomasz_Napiera=c5=82a?= , "scsi@freebsd.org" References: <694eabe2-e796-ebdc-b3f1-eff8f8fc1b24@FreeBSD.org> <2cf0c467-6bb6-55c0-586d-54ffec559c78@FreeBSD.org> <737FE056-D58D-40FB-A374-16DD5C0E99CF@freebsd.org> From: John Baldwin Subject: Re: NVMeoF and ctl In-Reply-To: <737FE056-D58D-40FB-A374-16DD5C0E99CF@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644371215; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8rZf1dXiJrRzMa0GuDltgTWaTm+mnbWxmpIpLr5IVlw=; b=buF6Ku3eL+JhAbiXOCRyBMGShItvhqsTDEWFgOUg5Xb4G5WzTIT6KUqK9z+v6xKFozHauH 7MqELO+JbZD8McbYAylBsZ68KWf2WfAv7YtB8hUFR02YcTGp9TOCKZCVTfQCxsXCc2IbOt LFK9dUsxlXg/l2mJvGmo9HYpRZcWIKUKLUa0gS6bWeB05VZHAZ+YloSkB4p8htWWw/09vN j3ylDmiWBEbsVdgeuIPHv2kABFZyQtZE551GursgIhiSB6Ty+HiN/CogkW95on2G+LwjWS ZpLRsabmj1ouhudg/S2DW9bhhDE1DHwPiW8ABx2fxo/rrHL55uLz7L3rtuxyWg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644371215; a=rsa-sha256; cv=none; b=C9hNmSstOTnI6xDDIXBJxyZYalvpt0yxNbKNvdR7FPAv90IQCQAxLuq0eVtkxhug31I3MO S5N5jASmHhhEQpo3g9u9NSZh0iPPChYXl7A0VV6Uod/X7RpSxrxT3p1jBpR+HYXXuzBvu+ AmlATZGFh2UkdIeJZm7Mh8PQf9UrAJQdvPesdiRjKlp5S/10xTiWFcZ2WeuOFl+Mt2YRTc 15O6lWwRWIERW0B8M7A8zTY4PC+Z4K+dIg9ZTP7S55Yy4qhCdOKmH1FX0XLkbA+RiX+2hl S3Yq+Vs9P1ZKDGSLnfNeUCAfAWMFEezMJgY1fwRKmYvdqD6/0sLfJVQDP6c9fg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N Replying to all of the preceding: I think the only thing I thought might be reusable is the backends and perhaps letting ctld be able to export a given backend via multiple protocols (which is only really a feature over a completely separate stack for pseudo backends like ramdisks, physical devices and zvols could potentially just be shared between daemons). For offload drivers like Chelsio I suspect we will end up with separate drivers (e.g. cxgben or some such) rather than trying to reuse the same icl interface for both iSCSI and NVMeoF. On 2/8/22 8:37 AM, Ken Merry wrote: > CTL is very SCSI specific, but of course when I wrote it in 2003, it was the Copan Target Layer and ran on Linux, CAM only supported SCSI, and I only had vague hopes of getting CTL into FreeBSD one day. > > Scott and Alexander have some good points. > > A few thoughts: > > 1. Most if not all HBAs that support NVMeoF will also support SCSI. (Chelsio, Qlogic, Emulex, and Mellanox support both). Whatever we do (refactored multi-protocol CTL or separate stacks), we’ll want to allow users to run NVMeoF and SCSI target and initiator at the same time. If you go the separate target stack route, you can of course have separate peripheral driver code to connect to CAM. (I’m assuming you would still want to go through CAM…) > > 2. From a user standpoint, it might be nice to have a single configuration and management interface…but that could potentially make the thing more unwieldy. I guess whatever we do, we’ll want it to be well thought out. > > 3. It would be nice to have functionality like CTL that allows an internally-visible NVMe target implementation. We’ve got some NVMe device emulation in Bhyve, but this would be more generic, and could be used to provide storage to Bhyve VMs, or useful to test new NVMe initiator code without extra hardware or cranking up a VM. > > 4. As Alexander pointed out, NVMe’s ordering requirements are not as complex as SCSI. See sys/cam/ctl/ctl_ser_table.c and the OOA queue for an illustration of the SCSI complexity. NVMe also allows for multiple queues, and namespaces (which I suppose are like multiple SCSI LUNs). Performance, mainly low latency, will probably be a primary design goal. A separate stack might make that easier, although if you did it through CTL, you would split SCSI and NVMe off in the peripheral driver code (scsi_ctl.c) and the two codepaths probably wouldn’t come back together until you got to the block or Ramdisk backend. > > I don’t think it must be done one way or the other. There are some tradeoffs. > > I’m glad you’re getting paid to work on it, NVMe target is a feature we need in FreeBSD, and I’m sure you’ll do a good job with it. :) > > Ken > — > Ken Merry > ken@FreeBSD.ORG > > > >> On Feb 7, 2022, at 21:48, Alexander Motin wrote: >> >> I feel that would we subtract SCSI out of CTL, there would not much left, aside of some very basic interfaces. And those may appear benefitting from taking different approaches with NVMe's multiple queues, more relaxed request ordering semantics, etc. Into recent NVMe specifications they've pumped in many things to bet on par with SCSI, but I am not sure it is similar enough to not turn common code into a huge mess. Though I haven't looked what Linux did in that front and how good idea it was there. >> >> On 07.02.2022 19:31, Scott Long wrote: >>> CTL stands for “CAM Target Layer”, but yes, it’s a Periph and it’s deeply tied to SCSI protocol, even if it’s mostly transport agnostic. I guess the answer to your question depends on the scope of your contract. It would be ideal to refactor CTL into protocol-specific sub-modules, but that might take a significant amount of work, and might not be all that satisfying at the end. I’d probably just copy CTL into a new, independent module, start replacing SCSI protocol idioms with NVMe ones, and along the way look for low-hanging fruit that can be refactored into a common library. >>> Scott >>>> On Feb 7, 2022, at 5:24 PM, John Baldwin wrote: >>>> >>>> One of the things I will be working on in the near future is NVMe over fabrics >>>> support, and specifically over TCP as Chelsio NICs include NVMe offload support >>>> (I think PDU encap/decap similar to the cxgbei driver for iSCSI offload). >>>> >>>> A question I have about this is if it makes sense for NVMeoF target support to >>>> make use of ctl? From what I can see, the code in ctl today seems to be >>>> very SCSI specific including both in the kernel and in the userland ctld >>>> unlike the Linux target code which appears to support both NVMeoF and iSCSI >>>> in its ctld equivalent. Is the intention for there to be a cleaner separation >>>> here, and if so do you have any thoughts on what the design would be like? >>>> Or should NVMeoF just be its own thing separate from ctl and ctld? >>>> >>>> -- >>>> John Baldwin >>>> >> >> -- >> Alexander Motin > -- John Baldwin