From nobody Fri May 03 00:15:50 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVrtb06Xlz5KK1t; Fri, 3 May 2024 00:15:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VVrtZ5nTjz4QXx; Fri, 3 May 2024 00:15:50 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1714695350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZuTK/oSY9J8UnB1EMlf0ssTjl6kizw7wX37McxaCSVg=; b=V6wKlKArdNNb3IWZeEJLgsEU1YEic2EW2x8WJnk+rN4KFNG/BxhJHvHvx77eTx8eCZLlzF SWtzIZkU9MwgjaG4n8u/58NcguAR4Bm1wAq4URyv2oySWdWiIP5VpWLNrOLwY0qrQWgglF s51dHAEj8r0SI7h1m0AaHapbEMtYnx6YKh0vao+E+F5notgOnaORcG8H/QOYMkqwM8L74M 6yibyPXJGhvqfjVj1mGRDrDdNRP0eU2PJVhzZAOockeEWu9J5HQWVf2rUE3+hKMaY8LqoC 0ieOMpme20vwdv35VCCk58PJUiim759OoDAeechPg55zESufnFYT18At6wirKQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1714695350; a=rsa-sha256; cv=none; b=i3B/XDXEDaQ2KVzKfoNAtNg/BGG4cR9Yv1vZCuobQO/VNZ0FvMNw27VK4HQh5K6BRQSH3E 36BAUzM+IGS7sDm2jRCNvByUsWxESW56G4kjyo8NCAOp+scqll5gPjFZoxjRg5wWxtEAkO Jl9J1rCpcMxg/czlZojj7VzuUqJH934GCkbvJwnq4nDsRHlm/b0AX+U3/UFxlrLyWc71kc jdcOQZicrr8h6Fh1/8fUvOsR4HEKsXl5K0OV5/ID5wQ8bTql+VNlffN+mAzV1rzIqY82/O cw+QQ62JZjP6nbNDTZdQrOnX28WJk+kxUYyGgbsuiAXlCoU5kLV1m+ljurwEDA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1714695350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZuTK/oSY9J8UnB1EMlf0ssTjl6kizw7wX37McxaCSVg=; b=E5EjXzBlQ4WDOhox+N6ZUy7Sjp2qKVnMCcrk8+iOp2AvHcez8dkdQBnDbnI1lS6pqjcKyo m6eKmsjr2cAtw53xuNk6JiuRAiyIn7AdytklfwF1GqQ2hcI82MXNknxHz6gRd0R4kIdXre BUUMzevNs8lI8PdExK31elK1fUp+EtvLVBrfuLpxFJ1KqVkyCpCup82uQUM2m+iWiTPTYu it8pyXedjSKl8sV7D43iSrBDK947diYJp26QvWVpAmiC7wAoa4roQR7SwaFwRGgH2+0XVX s5EmgxfgqZmVdaNDaLK3tByI8WuRKummLhYnonllSCRhj//V0rSVvEi/CxbQLg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VVrtZ5LsfzNgW; Fri, 3 May 2024 00:15:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 4430Focn079269; Fri, 3 May 2024 00:15:50 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 4430FotP079266; Fri, 3 May 2024 00:15:50 GMT (envelope-from git) Date: Fri, 3 May 2024 00:15:50 GMT Message-Id: <202405030015.4430FotP079266@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: John Baldwin Subject: git: 1058c12197ab - main - nvmecontrol: New commands to support Fabrics hosts List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhb X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1058c12197aba80d0777e3484f350436fca55fd7 Auto-Submitted: auto-generated The branch main has been updated by jhb: URL: https://cgit.FreeBSD.org/src/commit/?id=1058c12197aba80d0777e3484f350436fca55fd7 commit 1058c12197aba80d0777e3484f350436fca55fd7 Author: John Baldwin AuthorDate: 2024-05-02 23:30:10 +0000 Commit: John Baldwin CommitDate: 2024-05-02 23:30:10 +0000 nvmecontrol: New commands to support Fabrics hosts - discover: Connects to a remote Discovery controller, fetches its Discovery Log Page, and enumerates the remote controllers described in the log page. The -v option can be used to display the Identify Controller data structure for the Discovery controller. This is only really useful for debugging. - connect: Connects to a remote I/O controller and establishes an association of an admin queue and a single I/O queue. The association is handed off to the in-kernel host to create a new nvmeX device. - connect-all: Connects to a Discovery controller and attempts to create an association with each I/O controller enumerated in the Discovery controller's Discovery Log Page. - reconnect: Establishes a new association with a remote I/O controller for an existing nvmeX device. This can be used to restore access to a remote I/O controller after the loss of a prior association due to a transport error, controller reboot, etc. - disconnect: Deletes one or more nvmeX devices after detaching its namespaces and terminating any active associations. The devices to delete can be identified by either a nvmeX device name or the NQN of the remote controller. - disconnect-all: Deletes all active associations with remote controllers. Reviewed by: imp Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D44715 --- sbin/nvmecontrol/Makefile | 8 +- sbin/nvmecontrol/connect.c | 283 ++++++++++++++++++++++ sbin/nvmecontrol/disconnect.c | 82 +++++++ sbin/nvmecontrol/discover.c | 300 ++++++++++++++++++++++++ sbin/nvmecontrol/fabrics.c | 520 +++++++++++++++++++++++++++++++++++++++++ sbin/nvmecontrol/fabrics.h | 41 ++++ sbin/nvmecontrol/nvmecontrol.8 | 165 ++++++++++++- sbin/nvmecontrol/reconnect.c | 167 +++++++++++++ 8 files changed, 1563 insertions(+), 3 deletions(-) diff --git a/sbin/nvmecontrol/Makefile b/sbin/nvmecontrol/Makefile index f534093b1332..81674475ba1f 100644 --- a/sbin/nvmecontrol/Makefile +++ b/sbin/nvmecontrol/Makefile @@ -3,7 +3,11 @@ PACKAGE=nvme-tools PROG= nvmecontrol SRCS+= comnd.c +SRCS+= connect.c SRCS+= devlist.c +SRCS+= disconnect.c +SRCS+= discover.c +SRCS+= fabrics.c SRCS+= firmware.c SRCS+= format.c SRCS+= identify.c @@ -17,13 +21,15 @@ SRCS+= nvmecontrol.c SRCS+= passthru.c SRCS+= perftest.c SRCS+= power.c +SRCS+= reconnect.c SRCS+= reset.c SRCS+= resv.c SRCS+= sanitize.c SRCS+= selftest.c +CFLAGS+= -I${SRCTOP}/lib/libnvmf MAN= nvmecontrol.8 LDFLAGS+= -rdynamic -LIBADD+= util +LIBADD+= nvmf util SUBDIR= modules HAS_TESTS= SUBDIR.${MK_TESTS}+= tests diff --git a/sbin/nvmecontrol/connect.c b/sbin/nvmecontrol/connect.c new file mode 100644 index 000000000000..afb78725a3c7 --- /dev/null +++ b/sbin/nvmecontrol/connect.c @@ -0,0 +1,283 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2023-2024 Chelsio Communications, Inc. + * Written by: John Baldwin + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "comnd.h" +#include "fabrics.h" + +/* + * Settings that are currently hardcoded but could be exposed to the + * user via additional command line options: + * + * - ADMIN queue entries + * - MaxR2T + */ + +static struct options { + const char *transport; + const char *address; + const char *cntlid; + const char *subnqn; + const char *hostnqn; + uint32_t kato; + uint16_t num_io_queues; + uint16_t queue_size; + bool data_digests; + bool flow_control; + bool header_digests; +} opt = { + .transport = "tcp", + .address = NULL, + .cntlid = "dynamic", + .subnqn = NULL, + .hostnqn = NULL, + .kato = NVMF_KATO_DEFAULT / 1000, + .num_io_queues = 1, + .queue_size = 0, + .data_digests = false, + .flow_control = false, + .header_digests = false, +}; + +static void +tcp_association_params(struct nvmf_association_params *params) +{ + params->tcp.pda = 0; + params->tcp.header_digests = opt.header_digests; + params->tcp.data_digests = opt.data_digests; + /* XXX */ + params->tcp.maxr2t = 1; +} + +static int +connect_nvm_controller(enum nvmf_trtype trtype, int adrfam, const char *address, + const char *port, uint16_t cntlid, const char *subnqn) +{ + struct nvme_controller_data cdata; + struct nvmf_association_params aparams; + struct nvmf_qpair *admin, **io; + int error; + + memset(&aparams, 0, sizeof(aparams)); + aparams.sq_flow_control = opt.flow_control; + switch (trtype) { + case NVMF_TRTYPE_TCP: + tcp_association_params(&aparams); + break; + default: + warnx("Unsupported transport %s", nvmf_transport_type(trtype)); + return (EX_UNAVAILABLE); + } + + io = calloc(opt.num_io_queues, sizeof(*io)); + error = connect_nvm_queues(&aparams, trtype, adrfam, address, port, + cntlid, subnqn, opt.hostnqn, opt.kato, &admin, io, + opt.num_io_queues, opt.queue_size, &cdata); + if (error != 0) + return (error); + + error = nvmf_handoff_host(admin, opt.num_io_queues, io, &cdata); + if (error != 0) { + warnc(error, "Failed to handoff queues to kernel"); + return (EX_IOERR); + } + free(io); + return (0); +} + +static void +connect_discovery_entry(struct nvme_discovery_log_entry *entry) +{ + int adrfam; + + switch (entry->trtype) { + case NVMF_TRTYPE_TCP: + switch (entry->adrfam) { + case NVMF_ADRFAM_IPV4: + adrfam = AF_INET; + break; + case NVMF_ADRFAM_IPV6: + adrfam = AF_INET6; + break; + default: + warnx("Skipping unsupported address family for %s", + entry->subnqn); + return; + } + switch (entry->tsas.tcp.sectype) { + case NVME_TCP_SECURITY_NONE: + break; + default: + warnx("Skipping unsupported TCP security type for %s", + entry->subnqn); + return; + } + break; + default: + warnx("Skipping unsupported transport %s for %s", + nvmf_transport_type(entry->trtype), entry->subnqn); + return; + } + + /* + * XXX: Track portids and avoid duplicate connections for a + * given (subnqn,portid)? + */ + + /* XXX: Should this make use of entry->aqsz in some way? */ + connect_nvm_controller(entry->trtype, adrfam, entry->traddr, + entry->trsvcid, entry->cntlid, entry->subnqn); +} + +static void +connect_discovery_log_page(struct nvmf_qpair *qp) +{ + struct nvme_discovery_log *log; + int error; + + error = nvmf_host_fetch_discovery_log_page(qp, &log); + if (error != 0) + errc(EX_IOERR, error, "Failed to fetch discovery log page"); + + for (u_int i = 0; i < log->numrec; i++) + connect_discovery_entry(&log->entries[i]); + free(log); +} + +static void +discover_controllers(enum nvmf_trtype trtype, const char *address, + const char *port) +{ + struct nvmf_qpair *qp; + + qp = connect_discovery_adminq(trtype, address, port, opt.hostnqn); + + connect_discovery_log_page(qp); + + nvmf_free_qpair(qp); +} + +static void +connect_fn(const struct cmd *f, int argc, char *argv[]) +{ + enum nvmf_trtype trtype; + const char *address, *port; + char *tofree; + u_long cntlid; + int error; + + if (arg_parse(argc, argv, f)) + return; + + if (opt.num_io_queues <= 0) + errx(EX_USAGE, "Invalid number of I/O queues"); + + if (strcasecmp(opt.transport, "tcp") == 0) { + trtype = NVMF_TRTYPE_TCP; + } else + errx(EX_USAGE, "Unsupported or invalid transport"); + + nvmf_parse_address(opt.address, &address, &port, &tofree); + if (port == NULL) + errx(EX_USAGE, "Explicit port required"); + + cntlid = nvmf_parse_cntlid(opt.cntlid); + + error = connect_nvm_controller(trtype, AF_UNSPEC, address, port, cntlid, + opt.subnqn); + if (error != 0) + exit(error); + + free(tofree); +} + +static void +connect_all_fn(const struct cmd *f, int argc, char *argv[]) +{ + enum nvmf_trtype trtype; + const char *address, *port; + char *tofree; + + if (arg_parse(argc, argv, f)) + return; + + if (opt.num_io_queues <= 0) + errx(EX_USAGE, "Invalid number of I/O queues"); + + if (strcasecmp(opt.transport, "tcp") == 0) { + trtype = NVMF_TRTYPE_TCP; + } else + errx(EX_USAGE, "Unsupported or invalid transport"); + + nvmf_parse_address(opt.address, &address, &port, &tofree); + discover_controllers(trtype, address, port); + + free(tofree); +} + +static const struct opts connect_opts[] = { +#define OPT(l, s, t, opt, addr, desc) { l, s, t, &opt.addr, desc } + OPT("transport", 't', arg_string, opt, transport, + "Transport type"), + OPT("cntlid", 'c', arg_string, opt, cntlid, + "Controller ID"), + OPT("nr-io-queues", 'i', arg_uint16, opt, num_io_queues, + "Number of I/O queues"), + OPT("queue-size", 'Q', arg_uint16, opt, queue_size, + "Number of entries in each I/O queue"), + OPT("keep-alive-tmo", 'k', arg_uint32, opt, kato, + "Keep Alive timeout (in seconds)"), + OPT("hostnqn", 'q', arg_string, opt, hostnqn, + "Host NQN"), + OPT("flow_control", 'F', arg_none, opt, flow_control, + "Request SQ flow control"), + OPT("hdr_digests", 'g', arg_none, opt, header_digests, + "Enable TCP PDU header digests"), + OPT("data_digests", 'G', arg_none, opt, data_digests, + "Enable TCP PDU data digests"), + { NULL, 0, arg_none, NULL, NULL } +}; +#undef OPT + +static const struct args connect_args[] = { + { arg_string, &opt.address, "address" }, + { arg_string, &opt.subnqn, "SubNQN" }, + { arg_none, NULL, NULL }, +}; + +static const struct args connect_all_args[] = { + { arg_string, &opt.address, "address" }, + { arg_none, NULL, NULL }, +}; + +static struct cmd connect_cmd = { + .name = "connect", + .fn = connect_fn, + .descr = "Connect to a fabrics controller", + .ctx_size = sizeof(opt), + .opts = connect_opts, + .args = connect_args, +}; + +static struct cmd connect_all_cmd = { + .name = "connect-all", + .fn = connect_all_fn, + .descr = "Discover and connect to fabrics controllers", + .ctx_size = sizeof(opt), + .opts = connect_opts, + .args = connect_all_args, +}; + +CMD_COMMAND(connect_cmd); +CMD_COMMAND(connect_all_cmd); diff --git a/sbin/nvmecontrol/disconnect.c b/sbin/nvmecontrol/disconnect.c new file mode 100644 index 000000000000..b1b6af6271e8 --- /dev/null +++ b/sbin/nvmecontrol/disconnect.c @@ -0,0 +1,82 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2023-2024 Chelsio Communications, Inc. + * Written by: John Baldwin + */ + +#include +#include +#include +#include +#include + +#include "nvmecontrol.h" + +static struct options { + const char *dev; +} opt = { + .dev = NULL +}; + +static const struct args args[] = { + { arg_string, &opt.dev, "controller-id|namespace-id|SubNQN" }, + { arg_none, NULL, NULL }, +}; + +static void +disconnect(const struct cmd *f, int argc, char *argv[]) +{ + int error, fd; + char *path; + + if (arg_parse(argc, argv, f)) + return; + if (nvmf_nqn_valid(opt.dev)) { + error = nvmf_disconnect_host(opt.dev); + if (error != 0) + errc(EX_IOERR, error, "failed to disconnect from %s", + opt.dev); + } else { + open_dev(opt.dev, &fd, 1, 1); + get_nsid(fd, &path, NULL); + close(fd); + + error = nvmf_disconnect_host(path); + if (error != 0) + errc(EX_IOERR, error, "failed to disconnect from %s", + path); + } + + exit(0); +} + +static void +disconnect_all(const struct cmd *f __unused, int argc __unused, + char *argv[] __unused) +{ + int error; + + error = nvmf_disconnect_all(); + if (error != 0) + errc(EX_IOERR, error, + "failed to disconnect from remote controllers"); + + exit(0); +} + +static struct cmd disconnect_cmd = { + .name = "disconnect", + .fn = disconnect, + .descr = "Disconnect from a fabrics controller", + .args = args, +}; + +static struct cmd disconnect_all_cmd = { + .name = "disconnect-all", + .fn = disconnect_all, + .descr = "Disconnect from all fabrics controllers", +}; + +CMD_COMMAND(disconnect_cmd); +CMD_COMMAND(disconnect_all_cmd); diff --git a/sbin/nvmecontrol/discover.c b/sbin/nvmecontrol/discover.c new file mode 100644 index 000000000000..c782ebeb7452 --- /dev/null +++ b/sbin/nvmecontrol/discover.c @@ -0,0 +1,300 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2023-2024 Chelsio Communications, Inc. + * Written by: John Baldwin + */ + +#include +#include +#include +#include +#include +#include + +#include "comnd.h" +#include "fabrics.h" +#include "nvmecontrol_ext.h" + +static struct options { + const char *transport; + const char *address; + const char *hostnqn; + bool verbose; +} opt = { + .transport = "tcp", + .address = NULL, + .hostnqn = NULL, + .verbose = false, +}; + +static void +identify_controller(struct nvmf_qpair *qp) +{ + struct nvme_controller_data cdata; + int error; + + error = nvmf_host_identify_controller(qp, &cdata); + if (error != 0) + errc(EX_IOERR, error, "Failed to fetch controller data"); + nvme_print_controller(&cdata); +} + +static const char * +nvmf_address_family(uint8_t adrfam) +{ + static char buf[8]; + + switch (adrfam) { + case NVMF_ADRFAM_IPV4: + return ("AF_INET"); + case NVMF_ADRFAM_IPV6: + return ("AF_INET6"); + case NVMF_ADRFAM_IB: + return ("InfiniBand"); + case NVMF_ADRFAM_FC: + return ("Fibre Channel"); + case NVMF_ADRFAM_INTRA_HOST: + return ("Intra-host"); + default: + snprintf(buf, sizeof(buf), "0x%02x\n", adrfam); + return (buf); + } +} + +static const char * +nvmf_subsystem_type(uint8_t subtype) +{ + static char buf[8]; + + switch (subtype) { + case NVMF_SUBTYPE_DISCOVERY: + return ("Discovery"); + case NVMF_SUBTYPE_NVME: + return ("NVMe"); + default: + snprintf(buf, sizeof(buf), "0x%02x\n", subtype); + return (buf); + } +} + +static const char * +nvmf_secure_channel(uint8_t treq) +{ + switch (treq & 0x03) { + case NVMF_TREQ_SECURE_CHANNEL_NOT_SPECIFIED: + return ("Not specified"); + case NVMF_TREQ_SECURE_CHANNEL_REQUIRED: + return ("Required"); + case NVMF_TREQ_SECURE_CHANNEL_NOT_REQUIRED: + return ("Not required"); + default: + return ("0x03"); + } +} + +static const char * +nvmf_controller_id(uint16_t cntlid) +{ + static char buf[8]; + + switch (cntlid) { + case NVMF_CNTLID_DYNAMIC: + return ("Dynamic"); + case NVMF_CNTLID_STATIC_ANY: + return ("Static"); + default: + snprintf(buf, sizeof(buf), "%u", cntlid); + return (buf); + } +} + +static const char * +nvmf_rdma_service_type(uint8_t qptype) +{ + static char buf[8]; + + switch (qptype) { + case NVMF_RDMA_QPTYPE_RELIABLE_CONNECTED: + return ("Reliable connected"); + case NVMF_RDMA_QPTYPE_RELIABLE_DATAGRAM: + return ("Reliable datagram"); + default: + snprintf(buf, sizeof(buf), "0x%02x\n", qptype); + return (buf); + } +} + +static const char * +nvmf_rdma_provider_type(uint8_t prtype) +{ + static char buf[8]; + + switch (prtype) { + case NVMF_RDMA_PRTYPE_NONE: + return ("None"); + case NVMF_RDMA_PRTYPE_IB: + return ("InfiniBand"); + case NVMF_RDMA_PRTYPE_ROCE: + return ("RoCE (v1)"); + case NVMF_RDMA_PRTYPE_ROCE2: + return ("RoCE (v2)"); + case NVMF_RDMA_PRTYPE_IWARP: + return ("iWARP"); + default: + snprintf(buf, sizeof(buf), "0x%02x\n", prtype); + return (buf); + } +} + +static const char * +nvmf_rdma_cms(uint8_t cms) +{ + static char buf[8]; + + switch (cms) { + case NVMF_RDMA_CMS_RDMA_CM: + return ("RDMA_IP_CM"); + default: + snprintf(buf, sizeof(buf), "0x%02x\n", cms); + return (buf); + } +} + +static const char * +nvmf_tcp_security_type(uint8_t sectype) +{ + static char buf[8]; + + switch (sectype) { + case NVME_TCP_SECURITY_NONE: + return ("None"); + case NVME_TCP_SECURITY_TLS_1_2: + return ("TLS 1.2"); + case NVME_TCP_SECURITY_TLS_1_3: + return ("TLS 1.3"); + default: + snprintf(buf, sizeof(buf), "0x%02x\n", sectype); + return (buf); + } +} + +static void +print_discovery_entry(u_int i, struct nvme_discovery_log_entry *entry) +{ + printf("Entry %02d\n", i + 1); + printf("========\n"); + printf(" Transport type: %s\n", + nvmf_transport_type(entry->trtype)); + printf(" Address family: %s\n", + nvmf_address_family(entry->adrfam)); + printf(" Subsystem type: %s\n", + nvmf_subsystem_type(entry->subtype)); + printf(" SQ flow control: %s\n", + (entry->treq & (1 << 2)) == 0 ? "required" : "optional"); + printf(" Secure Channel: %s\n", nvmf_secure_channel(entry->treq)); + printf(" Port ID: %u\n", entry->portid); + printf(" Controller ID: %s\n", + nvmf_controller_id(entry->cntlid)); + printf(" Max Admin SQ Size: %u\n", entry->aqsz); + printf(" Sub NQN: %s\n", entry->subnqn); + printf(" Transport address: %s\n", entry->traddr); + printf(" Service identifier: %s\n", entry->trsvcid); + switch (entry->trtype) { + case NVMF_TRTYPE_RDMA: + printf(" RDMA Service Type: %s\n", + nvmf_rdma_service_type(entry->tsas.rdma.rdma_qptype)); + printf(" RDMA Provider Type: %s\n", + nvmf_rdma_provider_type(entry->tsas.rdma.rdma_prtype)); + printf(" RDMA CMS: %s\n", + nvmf_rdma_cms(entry->tsas.rdma.rdma_cms)); + printf(" Partition key: %u\n", + entry->tsas.rdma.rdma_pkey); + break; + case NVMF_TRTYPE_TCP: + printf(" Security Type: %s\n", + nvmf_tcp_security_type(entry->tsas.tcp.sectype)); + break; + } +} + +static void +dump_discovery_log_page(struct nvmf_qpair *qp) +{ + struct nvme_discovery_log *log; + int error; + + error = nvmf_host_fetch_discovery_log_page(qp, &log); + if (error != 0) + errc(EX_IOERR, error, "Failed to fetch discovery log page"); + + printf("Discovery\n"); + printf("=========\n"); + if (log->numrec == 0) { + printf("No entries found\n"); + } else { + for (u_int i = 0; i < log->numrec; i++) + print_discovery_entry(i, &log->entries[i]); + } + free(log); +} + +static void +discover(const struct cmd *f, int argc, char *argv[]) +{ + enum nvmf_trtype trtype; + struct nvmf_qpair *qp; + const char *address, *port; + char *tofree; + + if (arg_parse(argc, argv, f)) + return; + + if (strcasecmp(opt.transport, "tcp") == 0) { + trtype = NVMF_TRTYPE_TCP; + } else + errx(EX_USAGE, "Unsupported or invalid transport"); + + nvmf_parse_address(opt.address, &address, &port, &tofree); + qp = connect_discovery_adminq(trtype, address, port, opt.hostnqn); + free(tofree); + + /* Use Identify to fetch controller data */ + if (opt.verbose) { + identify_controller(qp); + printf("\n"); + } + + /* Fetch Log pages */ + dump_discovery_log_page(qp); + + nvmf_free_qpair(qp); +} + +static const struct opts discover_opts[] = { +#define OPT(l, s, t, opt, addr, desc) { l, s, t, &opt.addr, desc } + OPT("transport", 't', arg_string, opt, transport, + "Transport type"), + OPT("hostnqn", 'q', arg_string, opt, hostnqn, + "Host NQN"), + OPT("verbose", 'v', arg_none, opt, verbose, + "Display the discovery controller's controller data"), + { NULL, 0, arg_none, NULL, NULL } +}; +#undef OPT + +static const struct args discover_args[] = { + { arg_string, &opt.address, "address" }, + { arg_none, NULL, NULL }, +}; + +static struct cmd discover_cmd = { + .name = "discover", + .fn = discover, + .descr = "List discovery log pages from a fabrics controller", + .ctx_size = sizeof(opt), + .opts = discover_opts, + .args = discover_args, +}; + +CMD_COMMAND(discover_cmd); diff --git a/sbin/nvmecontrol/fabrics.c b/sbin/nvmecontrol/fabrics.c new file mode 100644 index 000000000000..6470e4062b39 --- /dev/null +++ b/sbin/nvmecontrol/fabrics.c @@ -0,0 +1,520 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2023-2024 Chelsio Communications, Inc. + * Written by: John Baldwin + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "fabrics.h" + +/* + * Subroutines shared by several Fabrics commands. + */ +static char nqn[NVMF_NQN_MAX_LEN]; +static uint8_t hostid[16]; +static bool hostid_initted = false; + +static bool +init_hostid(void) +{ + int error; + + if (hostid_initted) + return (true); + + error = nvmf_hostid_from_hostuuid(hostid); + if (error != 0) { + warnc(error, "Failed to generate hostid"); + return (false); + } + error = nvmf_nqn_from_hostuuid(nqn); + if (error != 0) { + warnc(error, "Failed to generate host NQN"); + return (false); + } + + hostid_initted = true; + return (true); +} + +void +nvmf_parse_address(const char *in_address, const char **address, + const char **port, char **tofree) +{ + char *cp; + + /* + * Accepts the following address formats: + * + * [IPv6 address]:port + * IPv4 address:port + * hostname:port + * [IPv6 address] + * IPv6 address + * IPv4 address + * hostname + */ + if (in_address[0] == '[') { + /* IPv6 address in square brackets. */ + cp = strchr(in_address + 1, ']'); + if (cp == NULL || cp == in_address + 1) + errx(EX_USAGE, "Invalid address %s", in_address); + *tofree = strndup(in_address + 1, cp - (in_address + 1)); + *address = *tofree; + + /* Skip over ']' */ + cp++; + switch (*cp) { + case '\0': + *port = NULL; + return; + case ':': + if (cp[1] != '\0') { + *port = cp + 1; + return; + } + /* FALLTHROUGH */ + default: + errx(EX_USAGE, "Invalid address %s", in_address); + } + } + + /* Look for the first colon. */ + cp = strchr(in_address, ':'); + if (cp == NULL) { + *address = in_address; + *port = NULL; + *tofree = NULL; + return; + } + + /* If there is another colon, assume this is an IPv6 address. */ + if (strchr(cp + 1, ':') != NULL) { + *address = in_address; + *port = NULL; + *tofree = NULL; + return; + } + + /* Both strings on either side of the colon must be non-empty. */ + if (cp == in_address || cp[1] == '\0') + errx(EX_USAGE, "Invalid address %s", in_address); + + *tofree = strndup(in_address, cp - in_address); + *address = *tofree; + + /* Skip over ':' */ + *port = cp + 1; +} + +uint16_t +nvmf_parse_cntlid(const char *cntlid) +{ + u_long value; + + if (strcasecmp(cntlid, "dynamic") == 0) + return (NVMF_CNTLID_DYNAMIC); + else if (strcasecmp(cntlid, "static") == 0) + return (NVMF_CNTLID_STATIC_ANY); + else { + value = strtoul(cntlid, NULL, 0); + + if (value > NVMF_CNTLID_STATIC_MAX) + errx(EX_USAGE, "Invalid controller ID"); + + return (value); + } +} + +bool +tcp_qpair_params(struct nvmf_qpair_params *params, int adrfam, + const char *address, const char *port) +{ + struct addrinfo hints, *ai, *list; + int error, s; + + memset(&hints, 0, sizeof(hints)); + hints.ai_family = adrfam; + hints.ai_protocol = IPPROTO_TCP; + error = getaddrinfo(address, port, &hints, &list); + if (error != 0) { + warnx("%s", gai_strerror(error)); + return (false); + } + + for (ai = list; ai != NULL; ai = ai->ai_next) { + s = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (s == -1) + continue; + + if (connect(s, ai->ai_addr, ai->ai_addrlen) != 0) { + close(s); + continue; + } + + params->tcp.fd = s; + freeaddrinfo(list); + return (true); + } + warn("Failed to connect to controller at %s:%s", address, port); + return (false); +} + +static void +tcp_discovery_association_params(struct nvmf_association_params *params) +{ + params->tcp.pda = 0; + params->tcp.header_digests = false; + params->tcp.data_digests = false; + params->tcp.maxr2t = 1; +} + +struct nvmf_qpair * +connect_discovery_adminq(enum nvmf_trtype trtype, const char *address, + const char *port, const char *hostnqn) +{ + struct nvmf_association_params aparams; + struct nvmf_qpair_params qparams; + struct nvmf_association *na; + struct nvmf_qpair *qp; + uint64_t cap, cc, csts; + int error, timo; + + memset(&aparams, 0, sizeof(aparams)); + aparams.sq_flow_control = false; + switch (trtype) { + case NVMF_TRTYPE_TCP: + /* 7.4.9.3 Default port for discovery */ + if (port == NULL) + port = "8009"; + tcp_discovery_association_params(&aparams); + break; + default: + errx(EX_UNAVAILABLE, "Unsupported transport %s", + nvmf_transport_type(trtype)); + } + + if (!init_hostid()) *** 723 LINES SKIPPED ***