From nobody Fri Jul 19 14:26:43 2024 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQX6t4gQ8z5RkXK; Fri, 19 Jul 2024 14:26:46 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQX6t45t7z4m6c; Fri, 19 Jul 2024 14:26:46 +0000 (UTC) (envelope-from kp@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721399206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H34F2qk4nK8QSnhm6GjAldkPLoJTEpaJy/aCMyugH8M=; b=Gg8hAXMaKyZPraB2x8cF5hTPa5BRraWS8L6P+Bad7f0T8jERH219UoSPJ/Mr17iXA9H8cq 1mJp7B3S19Qq/SZaCvaxtAVoyCFfsA4d+CE0ViIdtdkOc02o8cd46U12x2b0n7pv+lO9n+ GYzYRnPr46O6FEg8iBKlzxip1rdGOBXJGELpkxhTJ2nc3tZB7MnSJL/369YgJHa8XEfiWK ohCrwZ2E6KUYFAE4rpUnpG5p8kkzCUH9Ihx8YSu+7XD1O8Abxfjf5Ztj/gK5v4EUZj2ihP tLu1wFv2H4+yrIrP7xf9h83KyNYnvJsPW27EJuSt+sAEtdqBBWGrrumus+u1YA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1721399206; a=rsa-sha256; cv=none; b=WwpYkmmA97lGG3qw+Ox77UbmucutmBVG0aIv4RWYON++i7PTREp12zs+RsmMhHegRmwXX3 /3mj6j0h+NQJTQVE+sjGyrdqZWlnBu0SkFWMvf1U/ppYdmviD3O9PvKaok/Wun7g9ahaRH L+Na/KM/nfIL3g8F+KOEvB10O2bVUIg3b9WU2/HY31yQm+JngY2tOgyQdhDhDv5Gl22594 NZu5VUOpqDWSK50k/IV1y+yFuAU4cgKjJS35XjvpNX+pFyNE0K4cFqJN+vFRClBx6r/2H4 6kvK0CaMwzGEkFFdahXes0igdyiuXmEx20aC3TTVF4WZ0/AkSz1mG0leNxM8CQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1721399206; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H34F2qk4nK8QSnhm6GjAldkPLoJTEpaJy/aCMyugH8M=; b=U/u6Owt9G1De+VAMWyv+2FcfsUDOel6icaKbdfbMgsRVIDmi/wVRA+gXoDSVud6Gwka1pG E/sQPeHD09fJPXGyp5AVWfB9CVwt6b/u+Pij5Q1NedAMQYVJOyqyOSlJxW6x/idd7o6vQk kWGJ/yWYbWYVMamOYqU/K6LYc5k5r2mr/3gNXXdjssTcWB8/YjPe7F3MCGlly0+bwz+Jfb O2HvFcOdRufDbTHZqPD4NVUZA/pRJ6WJSpnCH+tmArH9ciysJmsudv3rjSEl2F8csvFlza aqFgP2KKYRRdkpIN6SbRsZmchRg5IsJw7nmzhq1kSlDejfViiAmquTE0tUpFWQ== Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mx1.codepro.be", Issuer "R3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 4WQX6t1ty4z1Lf5; Fri, 19 Jul 2024 14:26:46 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 24EB054BCD; Fri, 19 Jul 2024 16:26:44 +0200 (CEST) From: Kristof Provost To: Konstantin Belousov Cc: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org Subject: Re: git: ef2a572bf6bd - main - ipsec_offload: kernel infrastructure Date: Fri, 19 Jul 2024 16:26:43 +0200 X-Mailer: MailMate (1.14r5937) Message-ID: In-Reply-To: <202407121125.46CBP8eo093121@gitrepo.freebsd.org> References: <202407121125.46CBP8eo093121@gitrepo.freebsd.org> List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_MailMate_AF135622-7B0D-4427-95DE-D24615AC4C5A_=" --=_MailMate_AF135622-7B0D-4427-95DE-D24615AC4C5A_= Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable On 12 Jul 2024, at 13:25, Konstantin Belousov wrote: > The branch main has been updated by kib: > > URL: = > https://cgit.FreeBSD.org/src/commit/?id=3Def2a572bf6bdcac97ef29ce631d2f= 50f938e1ec8 > > commit ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8 > Author: Konstantin Belousov > AuthorDate: 2021-08-22 19:38:04 +0000 > Commit: Konstantin Belousov > CommitDate: 2024-07-12 04:27:58 +0000 > > ipsec_offload: kernel infrastructure > > Inline IPSEC offload moves almost whole IPSEC processing from the > CPU/MCU and possibly crypto accelerator, to the network card. > > The transmitted packet content is not touched by CPU during TX > operations, kernel only does the required policy and security > association lookups to find out that given flow is offloaded, and = > then > packet is transmitted as plain text to the card. For driver = > convenience, > a metadata is attached to the packet identifying SA which must = > process > the packet. Card does encryption of the payload, padding, = > calculates > authentication, and does the reformat according to the policy. > > Similarly, on receive, card does the decapsulation, decryption, = > and > authentification. Kernel receives the identifier of SA that was > used to process the packet, together with the plain-text packet. > > Overall, payload octets are only read or written by card DMA = > engine, > removing a lot of memory subsystem overhead, and saving CPU time = > because > IPSEC algos calculations are avoided. > > If driver declares support for inline IPSEC offload (with the > IFCAP2_IPSEC_OFFLOAD capability set and registering method table = > struct > if_ipsec_accel_methods), kernel offers the SPD and SAD to driver. > Driver decides which policies and SAs can be offloaded based on > hardware capacity, and acks/nacks each SA for given interface to > kernel. Kernel needs to keep this information to make a decision = > to > skip software processing on TX, and to assume processing already = > done > on RX. This shadow SPD/SAD database of offloads is rooted from > policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and = > SAs > (struct secasvar accel_ipfs, struct ifp_handle_sav). > > Some extensions to the PF_KEY socket allow to limit interfaces for > which given SP/SA could be offloaded (proposed for offload). = > Also, > additional statistics extensions allow to observe = > allocation/octet/use > counters for specific SA. > > Since SPs and SAs are typically instantiated in non-sleepable = > context, > while offloading them into card is expected to require costly = > async > manipulations of the card state, calls to the driver for offload = > and > termination are executed in the threaded taskqueue. It also = > solves > the issue of allocating resources needed for the offload database. > Neither ipf_handle_sp nor ipf_handle_sav do not add reference to = > the > owning SP/SA, the offload must be terminated before last reference = > is > dropped. ipsec_accel only adds transient references to ensure = > safe > pointer ownership by taskqueue. > > Maintaining the SA counters for hardware-accelerated packets is = > the > duty of the driver. The helper = > ipsec_accel_drv_sa_lifetime_update() > is provided to hide accel infrastructure from drivers which would = > use > expected callout to query hardware periodically for updates. > > Reviewed by: rscheff (transport, stack integration), np > Sponsored by: NVIDIA networking > Differential revision: https://reviews.freebsd.org/D44219 > --- > sys/conf/files | 2 + > sys/conf/options | 1 + > sys/modules/ipsec/Makefile | 5 +- > sys/netipsec/ipsec.c | 17 + > sys/netipsec/ipsec.h | 11 + > sys/netipsec/ipsec_input.c | 11 + > sys/netipsec/ipsec_offload.c | 1061 = > ++++++++++++++++++++++++++++++++++++++++++ > sys/netipsec/ipsec_offload.h | 191 ++++++++ > sys/netipsec/ipsec_output.c | 15 + > sys/netipsec/ipsec_pcb.c | 38 +- > sys/netipsec/key.c | 270 ++++++++++- > sys/netipsec/key.h | 6 + > sys/netipsec/key_debug.c | 5 + > sys/netipsec/keydb.h | 14 + > 14 files changed, 1628 insertions(+), 19 deletions(-) > I=E2=80=99m seeing messages like `ipsec_accel_sa_install_newkey: spi 0x10= 01 = flags 0x40 seq 0` running the test suite now. Are those stray debug printfs? Best regards, Kristof --=_MailMate_AF135622-7B0D-4427-95DE-D24615AC4C5A_= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On 12 Jul 2024, at 13:25, Konstantin Belousov wrote:

The branch main has been updated by= kib:

URL: https://cgit.FreeBSD.org/src/co= mmit/?id=3Def2a572bf6bdcac97ef29ce631d2f50f938e1ec8

commit ef2a572bf6bdcac97ef29ce631d2f50f938e1ec8
Author: Konstantin Belousov <kib@FreeBSD.org>
AuthorDate: 2021-08-22 19:38:04 +0000
Commit: Konstantin Belousov <kib@FreeBSD.org>
CommitDate: 2024-07-12 04:27:58 +0000

ipsec_offload: kernel infrastructure

Inline IPSEC offload moves almost whole IPSEC process= ing from the
CPU/MCU and possibly crypto accelerator, to the network card.

The transmitted packet content is not touched by CPU = during TX
operations, kernel only does the required policy and security
association lookups to find out that given flow is offloaded, and the= n
packet is transmitted as plain text to the card. For driver convenien= ce,
a metadata is attached to the packet identifying SA which must proces= s
the packet. Card does encryption of the payload, padding, calculates
authentication, and does the reformat according to the policy.

Similarly, on receive, card does the decapsulation, d= ecryption, and
authentification. Kernel receives the identifier of SA that was
used to process the packet, together with the plain-text packet.

Overall, payload octets are only read or written by c= ard DMA engine,
removing a lot of memory subsystem overhead, and saving CPU time beca= use
IPSEC algos calculations are avoided.

If driver declares support for inline IPSEC offload (= with the
IFCAP2_IPSEC_OFFLOAD capability set and registering method table stru= ct
if_ipsec_accel_methods), kernel offers the SPD and SAD to driver.
Driver decides which policies and SAs can be offloaded based on
hardware capacity, and acks/nacks each SA for given interface to
kernel. Kernel needs to keep this information to make a decision to
skip software processing on TX, and to assume processing already done=
on RX. This shadow SPD/SAD database of offloads is rooted from
policies (struct secpolicy accel_ifps, struct ifp_handle_sp) and SAs
(struct secasvar accel_ipfs, struct ifp_handle_sav).

Some extensions to the PF_KEY socket allow to limit i= nterfaces for
which given SP/SA could be offloaded (proposed for offload). Also,
additional statistics extensions allow to observe allocation/octet/us= e
counters for specific SA.

Since SPs and SAs are typically instantiated in non-s= leepable context,
while offloading them into card is expected to require costly async
manipulations of the card state, calls to the driver for offload and
termination are executed in the threaded taskqueue. It also solves
the issue of allocating resources needed for the offload database.
Neither ipf_handle_sp nor ipf_handle_sav do not add reference to the
owning SP/SA, the offload must be terminated before last reference is=
dropped. ipsec_accel only adds transient references to ensure safe
pointer ownership by taskqueue.

Maintaining the SA counters for hardware-accelerated = packets is the
duty of the driver. The helper ipsec_accel_drv_sa_lifetime_update()
is provided to hide accel infrastructure from drivers which would use=
expected callout to query hardware periodically for updates.

Reviewed by: rscheff (transport, stack integration= ), np
Sponsored by: NVIDIA networking
Differential revision: https://reviews.freebsd.org/D44219
---
sys/conf/files | 2 +
sys/conf/options | 1 +
sys/modules/ipsec/Makefile | 5 +-
sys/netipsec/ipsec.c | 17 +
sys/netipsec/ipsec.h | 11 +
sys/netipsec/ipsec_input.c | 11 +
sys/netipsec/ipsec_offload.c | 1061 ++++++++++++++++++++++++++++++++++++= ++++++
sys/netipsec/ipsec_offload.h | 191 ++++++++
sys/netipsec/ipsec_output.c | 15 +
sys/netipsec/ipsec_pcb.c | 38 +-
sys/netipsec/key.c | 270 ++++++++++-
sys/netipsec/key.h | 6 +
sys/netipsec/key_debug.c | 5 +
sys/netipsec/keydb.h | 14 +
14 files changed, 1628 insertions(+), 19 deletions(-)


I=E2=80=99m seeing messages like ipsec_accel_sa_install_newkey: spi = 0x1001 flags 0x40 seq 0 running the test suite now.
Are those stray debug printfs?

Best regards,
Kristof

--=_MailMate_AF135622-7B0D-4427-95DE-D24615AC4C5A_=--