From nobody Tue Feb 08 15:14:08 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5BC9719C3CA6; Tue, 8 Feb 2022 15:14:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JtRPG38Llz4dcP; Tue, 8 Feb 2022 15:14:09 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333251; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+o4g6W0BDBx+ialGc64pXsWMWCu/ZsG8AUYgYo5CQMo=; b=g0cbsQIp70mLMH3yif6N/iE2i6WNhoqRDpUV1AISaKvEwHg1I9UfZK3LswanMR1F7PxRXk hU9WkQ1n5rKpLEUoFjWiXaQUZsTw4K8ojmHDRXGjHJcA2l3yipOoglxwEekYNYx9GtGBeb 8MqdxWP4fCEGM86JYpMJDvxXyol7BGkDZ57cfv5JVxS3KWNMG+3a4rP4yNDfyIIqH8kfLl xLkqATHEHCw+QrRySg1oo/QVIC2nEUi0RuSSG8Dl0wjZ/MSQfZaRM1krzRCcrded5U0q/i seALcJ7SVw+/nefHBaEQkOwXbtii9FR7c2kG3I5kX2GR8W37MaO9ENJUe4k2Ag== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A47AE1BA74; Tue, 8 Feb 2022 15:14:08 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 218FE87R020890; Tue, 8 Feb 2022 15:14:08 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 218FE8xR020889; Tue, 8 Feb 2022 15:14:08 GMT (envelope-from git) Date: Tue, 8 Feb 2022 15:14:08 GMT Message-Id: <202202081514.218FE8xR020889@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: 26b0e75d64d0 - stable/13 - mlx5: Implement flow steering helper functions for TCP sockets. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 26b0e75d64d02505f8d1eb7d1f9750e3e37e9f91 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1644333251; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+o4g6W0BDBx+ialGc64pXsWMWCu/ZsG8AUYgYo5CQMo=; b=K4iui2z2yU7+GwkJO3KyxQcoSiDBvuuDNkXeGZOxDMogMr05sQ5q246NTcOmbv9NuoaBho J++n4zLbvqio39zt3FqDAu1miNn9L2qBma6MMbfXvPqJbwQjxT4IpFrpQVMFio85IebnCV Wj2czEb266qwIpfyKq4SoNDMbET5BD7n15qYbv99OKw1CXjTxDU5l52698P9CwErALqDzm Vc5++d5UC0eU09n6H5jpMs1yJEhtO4650uncYhc7zWNMO/AHWB8qCfkFxkMjLnx3dcCUC3 d1q0RQ6gcIny919YuvCeSQeSL7+FhaKQTFu/0jqeuTNPuxRVwyDbc2PvMnZ4bg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1644333251; a=rsa-sha256; cv=none; b=UgNFESYiR0jubo/t0cyqifAwHYKE5gEYBU7M5e8vpfz7j/DShWT18Dun6rJXEk8n+PLmDU ZzmMdPFuLeBXw+BJz/GZBUNZ4IIk9Mzka74X5QRyNL21ZPycwXbhYeCTcrFOPII6B7IBlf sJO0k3lCegT/xxwINs/qTLDGcinmFc1vP3RZkefwr37giN0jCpD7ecvfSc6MK4UR/c9KeC 5fXpfw2onnFRtErxKiVgXs7VMnK9jvaUKpXpKnDDqEeM112drjEJN4pOfqQ7952cZ8VsUG 11MXWXiiJc7yZI++6/hjzJCJurliJp/bxVPtR7Qqw+x+LWdc4t4rLboLQ6gujA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=26b0e75d64d02505f8d1eb7d1f9750e3e37e9f91 commit 26b0e75d64d02505f8d1eb7d1f9750e3e37e9f91 Author: Hans Petter Selasky AuthorDate: 2022-02-08 15:08:53 +0000 Commit: Hans Petter Selasky CommitDate: 2022-02-08 15:08:53 +0000 mlx5: Implement flow steering helper functions for TCP sockets. This change adds convenience functions to setup a flow steering rule based on a TCP socket. The helper function gets all the address information from the socket and returns a steering rule, to be used with HW TLS RX offload. Sponsored by: NVIDIA Networking (cherry picked from commit 2c0ade806aa7b450dc4f4c53b5345050eb6dcb4b) --- sys/conf/files | 2 + sys/dev/mlx5/device.h | 6 + sys/dev/mlx5/mlx5_core/fs_tcp.h | 41 +++ sys/dev/mlx5/mlx5_core/mlx5_fs_tcp.c | 404 ++++++++++++++++++++++++++++++ sys/dev/mlx5/mlx5_en/en.h | 15 ++ sys/dev/mlx5/mlx5_en/mlx5_en_flow_table.c | 57 +++-- sys/dev/mlx5/mlx5_ifc.h | 6 +- sys/modules/mlx5/Makefile | 1 + 8 files changed, 512 insertions(+), 20 deletions(-) diff --git a/sys/conf/files b/sys/conf/files index d37d73bce98d..9b0342287b22 100644 --- a/sys/conf/files +++ b/sys/conf/files @@ -4887,6 +4887,8 @@ dev/mlx5/mlx5_core/mlx5_eswitch.c optional mlx5 pci \ compile-with "${OFED_C}" dev/mlx5/mlx5_core/mlx5_fs_cmd.c optional mlx5 pci \ compile-with "${OFED_C}" +dev/mlx5/mlx5_core/mlx5_fs_tcp.c optional mlx5 pci \ + compile-with "${OFED_C}" dev/mlx5/mlx5_core/mlx5_fs_tree.c optional mlx5 pci \ compile-with "${OFED_C}" dev/mlx5/mlx5_core/mlx5_fw.c optional mlx5 pci \ diff --git a/sys/dev/mlx5/device.h b/sys/dev/mlx5/device.h index c35ef6f2ff0c..cec2da7e684b 100644 --- a/sys/dev/mlx5/device.h +++ b/sys/dev/mlx5/device.h @@ -1022,6 +1022,12 @@ enum mlx5_mcam_feature_groups { #define MLX5_CAP_FLOWTABLE_MAX(mdev, cap) \ MLX5_GET(flow_table_nic_cap, mdev->hca_caps_max[MLX5_CAP_FLOW_TABLE], cap) +#define MLX5_CAP_FLOWTABLE_NIC_RX(mdev, cap) \ + MLX5_CAP_FLOWTABLE(mdev, flow_table_properties_nic_receive.cap) + +#define MLX5_CAP_FLOWTABLE_NIC_RX_MAX(mdev, cap) \ + MLX5_CAP_FLOWTABLE_MAX(mdev, flow_table_properties_nic_receive.cap) + #define MLX5_CAP_ESW_FLOWTABLE(mdev, cap) \ MLX5_GET(flow_table_eswitch_cap, \ mdev->hca_caps_cur[MLX5_CAP_ESWITCH_FLOW_TABLE], cap) diff --git a/sys/dev/mlx5/mlx5_core/fs_tcp.h b/sys/dev/mlx5/mlx5_core/fs_tcp.h new file mode 100644 index 000000000000..fa11ad9c4cb5 --- /dev/null +++ b/sys/dev/mlx5/mlx5_core/fs_tcp.h @@ -0,0 +1,41 @@ +/*- + * Copyright (c) 2020-2021, Mellanox Technologies, Ltd. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS `AS IS' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef __MLX5E_ACCEL_FS_TCP_H__ +#define __MLX5E_ACCEL_FS_TCP_H__ + +struct inpcb; +struct mlx5_flow_rule; +struct mlx5e_priv; + +int mlx5e_accel_fs_tcp_create(struct mlx5e_priv *); +void mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *); +struct mlx5_flow_rule * +mlx5e_accel_fs_add_inpcb(struct mlx5e_priv *, + struct inpcb *, uint32_t tirn, uint32_t flow_tag, uint16_t vlan_id); +#define MLX5E_ACCEL_FS_ADD_INPCB_NO_VLAN 0xFFFF +void mlx5e_accel_fs_del_inpcb(struct mlx5_flow_rule *); + +#endif /* __MLX5E_ACCEL_FS_TCP_H__ */ diff --git a/sys/dev/mlx5/mlx5_core/mlx5_fs_tcp.c b/sys/dev/mlx5/mlx5_core/mlx5_fs_tcp.c new file mode 100644 index 000000000000..543c7a4ef502 --- /dev/null +++ b/sys/dev/mlx5/mlx5_core/mlx5_fs_tcp.c @@ -0,0 +1,404 @@ +/*- + * Copyright (c) 2020-2021, Mellanox Technologies, Ltd. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS `AS IS' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include "opt_inet.h" +#include "opt_inet6.h" + +#include + +#include +#include +#include + +#include + +#include + +#if defined(INET) || defined(INET6) +static void +accel_fs_tcp_set_ipv4_flow(struct mlx5_flow_spec *spec, struct inpcb *inp) +{ + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ip_protocol); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.ip_protocol, IPPROTO_TCP); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ip_version); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.ip_version, 4); + memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value, + outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4), + &inp->inp_faddr, 4); + memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value, + outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4), + &inp->inp_laddr, 4); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, + outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, + outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4); +} +#endif + +#ifdef INET6 +static void +accel_fs_tcp_set_ipv6_flow(struct mlx5_flow_spec *spec, struct inpcb *inp) +{ + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ip_protocol); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.ip_protocol, IPPROTO_TCP); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.ip_version); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.ip_version, 6); + memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value, + outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6), + &inp->in6p_faddr, 16); + memcpy(MLX5_ADDR_OF(fte_match_param, spec->match_value, + outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6), + &inp->in6p_laddr, 16); + memset(MLX5_ADDR_OF(fte_match_param, spec->match_criteria, + outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6), + 0xff, 16); + memset(MLX5_ADDR_OF(fte_match_param, spec->match_criteria, + outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6), + 0xff, 16); +} +#endif + +void +mlx5e_accel_fs_del_inpcb(struct mlx5_flow_rule *rule) +{ + mlx5_del_flow_rule(rule); +} + +struct mlx5_flow_rule * +mlx5e_accel_fs_add_inpcb(struct mlx5e_priv *priv, + struct inpcb *inp, uint32_t tirn, uint32_t flow_tag, + uint16_t vlan_id) +{ + struct mlx5_flow_destination dest = {}; + struct mlx5e_flow_table *ft = NULL; + struct mlx5e_accel_fs_tcp *fs_tcp; + struct mlx5_flow_rule *flow; + struct mlx5_flow_spec *spec; + + spec = kvzalloc(sizeof(*spec), GFP_KERNEL); + if (!spec) + return (ERR_PTR(-ENOMEM)); + + fs_tcp = &priv->fts.accel_tcp; + + spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS; + + INP_RLOCK(inp); + /* Set VLAN ID to match, if any. */ + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.cvlan_tag); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, outer_headers.first_vid); + if (vlan_id != MLX5E_ACCEL_FS_ADD_INPCB_NO_VLAN) { + MLX5_SET_TO_ONES(fte_match_param, spec->match_value, outer_headers.cvlan_tag); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.first_vid, vlan_id); + } + + /* Set TCP port numbers. */ + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, + outer_headers.tcp_dport); + MLX5_SET_TO_ONES(fte_match_param, spec->match_criteria, + outer_headers.tcp_sport); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_dport, + ntohs(inp->inp_lport)); + MLX5_SET(fte_match_param, spec->match_value, outer_headers.tcp_sport, + ntohs(inp->inp_fport)); + + /* Set IP addresses. */ + switch (INP_SOCKAF(inp->inp_socket)) { +#ifdef INET + case AF_INET: + accel_fs_tcp_set_ipv4_flow(spec, inp); + ft = &fs_tcp->tables[MLX5E_ACCEL_FS_IPV4_TCP]; + break; +#endif +#ifdef INET6 + case AF_INET6: + if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0 && + IN6_IS_ADDR_V4MAPPED(&inp->in6p_faddr)) { + accel_fs_tcp_set_ipv4_flow(spec, inp); + ft = &fs_tcp->tables[MLX5E_ACCEL_FS_IPV4_TCP]; + } else { + accel_fs_tcp_set_ipv6_flow(spec, inp); + ft = &fs_tcp->tables[MLX5E_ACCEL_FS_IPV6_TCP]; + } + break; +#endif + default: + break; + } + INP_RUNLOCK(inp); + + if (!ft) { + flow = ERR_PTR(-EINVAL); + goto out; + } + + dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR; + dest.tir_num = tirn; + + flow = mlx5_add_flow_rule(ft->t, spec->match_criteria_enable, + spec->match_criteria, + spec->match_value, + MLX5_FLOW_CONTEXT_ACTION_FWD_DEST, + flow_tag, + &dest); +out: + kvfree(spec); + return (flow); +} + +static int +accel_fs_tcp_add_default_rule(struct mlx5e_priv *priv, int type) +{ + static u32 match_criteria[MLX5_ST_SZ_DW(fte_match_param)]; + static u32 match_value[MLX5_ST_SZ_DW(fte_match_param)]; + struct mlx5_flow_destination dest = {}; + struct mlx5e_accel_fs_tcp *fs_tcp; + struct mlx5_flow_rule *rule; + + fs_tcp = &priv->fts.accel_tcp; + + dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE; + + /* + * Traffic not matched by flow table rules should be forwarded + * to the next flow table in order to not be dropped by the + * default action. Refer to the diagram in + * mlx5_en_flow_table.c for more information about the order + * of flow tables. + */ + dest.ft = (type == MLX5E_ACCEL_FS_TCP_NUM_TYPES - 1) ? + priv->fts.vlan.t : fs_tcp->tables[type + 1].t; + + rule = mlx5_add_flow_rule(fs_tcp->tables[type].t, 0, match_criteria, match_value, + MLX5_FLOW_CONTEXT_ACTION_FWD_DEST, MLX5_FS_DEFAULT_FLOW_TAG, &dest); + if (IS_ERR(rule)) + return (PTR_ERR(rule)); + + fs_tcp->default_rules[type] = rule; + return (0); +} + +#define MLX5E_ACCEL_FS_TCP_NUM_GROUPS (2) +#define MLX5E_ACCEL_FS_TCP_GROUP1_SIZE (BIT(16) - 1) +#define MLX5E_ACCEL_FS_TCP_GROUP2_SIZE (BIT(0)) +#define MLX5E_ACCEL_FS_TCP_TABLE_SIZE (MLX5E_ACCEL_FS_TCP_GROUP1_SIZE +\ + MLX5E_ACCEL_FS_TCP_GROUP2_SIZE) +static int +accel_fs_tcp_create_groups(struct mlx5e_flow_table *ft, int type) +{ + int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in); + void *outer_headers_c; + int ix = 0; + u32 *in; + int err; + u8 *mc; + + ft->g = kcalloc(MLX5E_ACCEL_FS_TCP_NUM_GROUPS, sizeof(*ft->g), GFP_KERNEL); + in = kvzalloc(inlen, GFP_KERNEL); + if (!in || !ft->g) { + kfree(ft->g); + kvfree(in); + return (-ENOMEM); + } + + mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria); + outer_headers_c = MLX5_ADDR_OF(fte_match_param, mc, outer_headers); + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, ip_protocol); + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, ip_version); + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, cvlan_tag); + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, first_vid); + + switch (type) { + case MLX5E_ACCEL_FS_IPV4_TCP: + case MLX5E_ACCEL_FS_IPV6_TCP: + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, tcp_dport); + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, tcp_sport); + break; + default: + err = -EINVAL; + goto out; + } + + switch (type) { + case MLX5E_ACCEL_FS_IPV4_TCP: + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, + src_ipv4_src_ipv6.ipv4_layout.ipv4); + MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, + dst_ipv4_dst_ipv6.ipv4_layout.ipv4); + break; + case MLX5E_ACCEL_FS_IPV6_TCP: + memset(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c, + src_ipv4_src_ipv6.ipv6_layout.ipv6), + 0xff, 16); + memset(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c, + dst_ipv4_dst_ipv6.ipv6_layout.ipv6), + 0xff, 16); + break; + default: + err = -EINVAL; + goto out; + } + + MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS); + MLX5_SET_CFG(in, start_flow_index, ix); + ix += MLX5E_ACCEL_FS_TCP_GROUP1_SIZE; + MLX5_SET_CFG(in, end_flow_index, ix - 1); + ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in); + if (IS_ERR(ft->g[ft->num_groups])) + goto err; + ft->num_groups++; + + /* Default Flow Group */ + memset(in, 0, inlen); + MLX5_SET_CFG(in, start_flow_index, ix); + ix += MLX5E_ACCEL_FS_TCP_GROUP2_SIZE; + MLX5_SET_CFG(in, end_flow_index, ix - 1); + ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in); + if (IS_ERR(ft->g[ft->num_groups])) + goto err; + ft->num_groups++; + + kvfree(in); + return (0); + +err: + err = PTR_ERR(ft->g[ft->num_groups]); + ft->g[ft->num_groups] = NULL; +out: + kvfree(in); + + return (err); +} + +static void +accel_fs_tcp_destroy_groups(struct mlx5e_flow_table *ft) +{ + int i; + + for (i = ft->num_groups - 1; i >= 0; i--) { + if (!IS_ERR_OR_NULL(ft->g[i])) + mlx5_destroy_flow_group(ft->g[i]); + ft->g[i] = NULL; + } + ft->num_groups = 0; +} + +static int +accel_fs_tcp_create_table(struct mlx5e_priv *priv, int type) +{ + struct mlx5e_flow_table *ft = &priv->fts.accel_tcp.tables[type]; + int err; + + ft->num_groups = 0; + ft->t = mlx5_create_flow_table(priv->fts.accel_tcp.ns, 0, "tcp", + MLX5E_ACCEL_FS_TCP_TABLE_SIZE); + if (IS_ERR(ft->t)) { + err = PTR_ERR(ft->t); + ft->t = NULL; + return (err); + } + + err = accel_fs_tcp_create_groups(ft, type); + if (err) + goto err_destroy_flow_table; + + return (0); + +err_destroy_flow_table: + mlx5_destroy_flow_table(ft->t); + ft->t = NULL; + return (err); +} + +static void +accel_fs_tcp_destroy_table(struct mlx5e_priv *priv, int i) +{ + struct mlx5e_accel_fs_tcp *fs_tcp; + struct mlx5e_flow_table *ft; + + fs_tcp = &priv->fts.accel_tcp; + ft = fs_tcp->tables + i; + + mlx5_del_flow_rule(fs_tcp->default_rules[i]); + + accel_fs_tcp_destroy_groups(ft); + kfree(ft->g); + ft->g = NULL; + mlx5_destroy_flow_table(ft->t); + ft->t = NULL; +} + +void +mlx5e_accel_fs_tcp_destroy(struct mlx5e_priv *priv) +{ + int i; + + if (!MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, ft_field_support.outer_ip_version)) + return; + + for (i = 0; i < MLX5E_ACCEL_FS_TCP_NUM_TYPES; i++) + accel_fs_tcp_destroy_table(priv, i); +} + +int +mlx5e_accel_fs_tcp_create(struct mlx5e_priv *priv) +{ + int i, err; + + if (!MLX5_CAP_FLOWTABLE_NIC_RX(priv->mdev, ft_field_support.outer_ip_version)) + return (0); + + /* Setup namespace pointer. */ + priv->fts.accel_tcp.ns = mlx5_get_flow_namespace( + priv->mdev, MLX5_FLOW_NAMESPACE_OFFLOADS); + + /* + * Create flow tables first, because the priority level is + * assigned at allocation time. + */ + for (i = 0; i != MLX5E_ACCEL_FS_TCP_NUM_TYPES; i++) { + err = accel_fs_tcp_create_table(priv, i); + if (err) + goto err_destroy_tables; + } + + /* Create default rules last. */ + for (i = 0; i != MLX5E_ACCEL_FS_TCP_NUM_TYPES; i++) { + err = accel_fs_tcp_add_default_rule(priv, i); + if (err) + goto err_destroy_rules; + } + return (0); + +err_destroy_rules: + while (i--) + mlx5_del_flow_rule(priv->fts.accel_tcp.default_rules[i]); + i = MLX5E_ACCEL_FS_TCP_NUM_TYPES; + +err_destroy_tables: + while (i--) + accel_fs_tcp_destroy_table(priv, i); + return (err); +} diff --git a/sys/dev/mlx5/mlx5_en/en.h b/sys/dev/mlx5/mlx5_en/en.h index a04975f6f339..8b74bdc08b77 100644 --- a/sys/dev/mlx5/mlx5_en/en.h +++ b/sys/dev/mlx5/mlx5_en/en.h @@ -72,6 +72,8 @@ #include #include +#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v) + #define MLX5E_MAX_PRIORITY 8 #define MLX5E_MAX_FEC_10X_25X 4 @@ -1015,6 +1017,18 @@ struct mlx5e_flow_table { struct mlx5_flow_group **g; }; +enum accel_fs_tcp_type { + MLX5E_ACCEL_FS_IPV4_TCP, + MLX5E_ACCEL_FS_IPV6_TCP, + MLX5E_ACCEL_FS_TCP_NUM_TYPES, +}; + +struct mlx5e_accel_fs_tcp { + struct mlx5_flow_namespace *ns; + struct mlx5e_flow_table tables[MLX5E_ACCEL_FS_TCP_NUM_TYPES]; + struct mlx5_flow_rule *default_rules[MLX5E_ACCEL_FS_TCP_NUM_TYPES]; +}; + struct mlx5e_flow_tables { struct mlx5_flow_namespace *ns; struct mlx5e_flow_table vlan; @@ -1024,6 +1038,7 @@ struct mlx5e_flow_tables { struct mlx5e_flow_table main_vxlan; struct mlx5_flow_rule *main_vxlan_rule[MLX5E_NUM_TT]; struct mlx5e_flow_table inner_rss; + struct mlx5e_accel_fs_tcp accel_tcp; }; struct mlx5e_xmit_args { diff --git a/sys/dev/mlx5/mlx5_en/mlx5_en_flow_table.c b/sys/dev/mlx5/mlx5_en/mlx5_en_flow_table.c index 10ff5a3e7417..5d52415381aa 100644 --- a/sys/dev/mlx5/mlx5_en/mlx5_en_flow_table.c +++ b/sys/dev/mlx5/mlx5_en/mlx5_en_flow_table.c @@ -33,28 +33,44 @@ #include #include #include +#include /* * The flow tables with rules define the packet processing on receive. - * Currently, the following structure is set up to handle different offloads - * like VLAN decapsulation, packet classification, RSS hashing, VxLAN checksum - * offloading: + * Currently the following structure is set up to handle different + * offloads like TLS RX offload, VLAN decapsulation, packet + * classification, RSS hashing, VxLAN checksum offloading: * - * - * +=========+ +=========+ +=================+ - * |VLAN ft: | |VxLAN | |VxLAN Main | + * +=========+ +=========+ +=================+ + * |TCP/IPv4 | |TCP/IPv4 | |TCP/IPv4 Match | + * |Flowtable|------>| |----->|Outer Proto Match|=====> TLS TIR n + * | | |Catch-all|\ | | + * +=========+ +=========+| +=================+ + * | + * +------------------------+ + * V + * +=========+ +=========+ +=================+ + * |TCP/IPv6 | |TCP/IPv6 | |TCP/IPv6 Match | + * |Flowtable|------>| |----->|Outer Proto Match|=====> TLS TIR n + * | | |Catch-all|\ | | + * +=========+ +=========+| +=================+ + * | + * +------------------------+ + * V + * +=========+ +=========+ +=================+ + * |VLAN ft: | |VxLAN | |VxLAN Main | * |CTAG/STAG|------>| VNI|----->|Inner Proto Match|=====> Inner TIR n - * |VID/noVID|/ |Catch-all|\ | | - * +=========+ +=========+| +=================+ - * | - * | - * | - * v - * +=================+ - * |Main | - * |Outer Proto Match|=====> TIR n - * | | - * +=================+ + * |VID/noVID|/ |Catch-all|\ | | + * +=========+ +=========+| +=================+ + * | + * | + * | + * v + * +=================+ + * |Main | + * |Outer Proto Match|=====> TIR n + * | | + * +=================+ * * The path through flow rules directs each packet into an appropriate TIR, * according to the: @@ -2292,8 +2308,14 @@ mlx5e_open_flow_tables(struct mlx5e_priv *priv) if (err) goto err_destroy_inner_rss_flow_table; + err = mlx5e_accel_fs_tcp_create(priv); + if (err) + goto err_del_vxlan_catchall_rule; + return (0); +err_del_vxlan_catchall_rule: + mlx5e_del_vxlan_catchall_rule(priv); err_destroy_inner_rss_flow_table: mlx5e_destroy_inner_rss_flow_table(priv); err_destroy_main_vxlan_flow_table: @@ -2311,6 +2333,7 @@ err_destroy_vlan_flow_table: void mlx5e_close_flow_tables(struct mlx5e_priv *priv) { + mlx5e_accel_fs_tcp_destroy(priv); mlx5e_del_vxlan_catchall_rule(priv); mlx5e_destroy_inner_rss_flow_table(priv); mlx5e_destroy_main_vxlan_flow_table(priv); diff --git a/sys/dev/mlx5/mlx5_ifc.h b/sys/dev/mlx5/mlx5_ifc.h index dc288e350d05..bea12c63a3a4 100644 --- a/sys/dev/mlx5/mlx5_ifc.h +++ b/sys/dev/mlx5/mlx5_ifc.h @@ -296,7 +296,7 @@ struct mlx5_ifc_flow_table_fields_supported_bits { u8 outer_dmac[0x1]; u8 outer_smac[0x1]; u8 outer_ether_type[0x1]; - u8 reserved_0[0x1]; + u8 outer_ip_version[0x1]; u8 outer_first_prio[0x1]; u8 outer_first_cfi[0x1]; u8 outer_first_vid[0x1]; @@ -329,7 +329,7 @@ struct mlx5_ifc_flow_table_fields_supported_bits { u8 inner_dmac[0x1]; u8 inner_smac[0x1]; u8 inner_ether_type[0x1]; - u8 reserved_3[0x1]; + u8 inner_ip_version[0x1]; u8 inner_first_prio[0x1]; u8 inner_first_cfi[0x1]; u8 inner_first_vid[0x1]; @@ -519,7 +519,7 @@ struct mlx5_ifc_fte_match_set_lyr_2_4_bits { u8 cvlan_tag[0x1]; u8 svlan_tag[0x1]; u8 frag[0x1]; - u8 reserved_1[0x4]; + u8 ip_version[0x4]; u8 tcp_flags[0x9]; u8 tcp_sport[0x10]; diff --git a/sys/modules/mlx5/Makefile b/sys/modules/mlx5/Makefile index 87203b635bf7..fcd6197e9488 100644 --- a/sys/modules/mlx5/Makefile +++ b/sys/modules/mlx5/Makefile @@ -12,6 +12,7 @@ mlx5_diagnostics.c \ mlx5_eq.c \ mlx5_eswitch.c \ mlx5_fs_cmd.c \ +mlx5_fs_tcp.c \ mlx5_fs_tree.c \ mlx5_fw.c \ mlx5_fwdump.c \