svn commit: r359474 - head/sys/kern

[Andrew Gallatin]

Author: gallatin
Date: Mon Mar 30 23:29:53 2020
New Revision: 359474
URL: https://svnweb.freebsd.org/changeset/base/359474

Log:
  KTLS: Coalesce adjacent TLS trailers & headers to improve PCIe bus efficiency
  
  KTLS uses the embedded header and trailer fields of unmapped
  mbufs. This can lead to "silly" buffer lengths, where we have an
  mbuf chain that will create a scatter/gather lists with a
  regular pattern of 13 bytes followed by 16 bytes between each
  adjacent TLS record.
  
  For software ktls we typically wind up with a pattern where we
  have several TLS records encrypted, and made ready at once. When
  these records are made ready, we can coalesce these silly buffers
  in sbready_compress by copying 13b TLS header of the next record
  into the 16b TLS trailer of the current record. After doing so,
  we now have a small 29 byte chunk between each TLS record.
  
  This marginally increases PCIe bus efficiency. We've seen an
  almost 1Gb/s increase in peak throughput on Broadwell based Xeons
  running a 100% software TLS workload with Mellanox ConnectX-4
  NICs.
  
  Note that this change is ifdef'ed for KTLS, as KTLS is currently
  the only user of the hdr/trailer feature of unmapped mbufs, and
  peeking into them is expensive, since the ext_pgs struct lives in
  separately allocated memory, and may be cold in cache.
  
  This optimization is not applicable to HW ("NIC") TLS, as that
  depends on having the entire TLS record described by a single
  unmapped mbuf, so we cannot shift parts of the record between
  mbufs for HW TLS.
  
  Reviewed by:	jhb, hselasky, scottl
  Sponsored by:	Netflix
  Differential Revision:	https://reviews.freebsd.org/D24204

Modified:
  head/sys/kern/uipc_sockbuf.c

Modified: head/sys/kern/uipc_sockbuf.c
==============================================================================
--- head/sys/kern/uipc_sockbuf.c	Mon Mar 30 22:13:32 2020	(r359473)
+++ head/sys/kern/uipc_sockbuf.c	Mon Mar 30 23:29:53 2020	(r359474)
@@ -112,7 +112,42 @@ sbready_compress(struct sockbuf *sb, struct mbuf *m0, 
 
 	for (m = m0; m != end; m = m->m_next) {
 		MPASS((m->m_flags & M_NOTREADY) == 0);
+		/*
+		 * NB: In sbcompress(), 'n' is the last mbuf in the
+		 * socket buffer and 'm' is the new mbuf being copied
+		 * into the trailing space of 'n'.  Here, the roles
+		 * are reversed and 'n' is the next mbuf after 'm'
+		 * that is being copied into the trailing space of
+		 * 'm'.
+		 */
+		n = m->m_next;
+#ifdef KERN_TLS
+		/* Try to coalesce adjacent ktls mbuf hdr/trailers. */
+		if ((n != NULL) && (n != end) && (m->m_flags & M_EOR) == 0 &&
+		    (m->m_flags & M_NOMAP) &&
+		    (n->m_flags & M_NOMAP) &&
+		    !mbuf_has_tls_session(m) &&
+		    !mbuf_has_tls_session(n)) {
+			struct mbuf_ext_pgs *mpgs, *npgs;
+			int hdr_len, trail_len;
 
+			mpgs = m->m_ext.ext_pgs;
+			npgs = n->m_ext.ext_pgs;
+			hdr_len = npgs->hdr_len;
+			trail_len = mpgs->trail_len;
+			if (trail_len != 0 && hdr_len != 0 &&
+			    trail_len + hdr_len <= MBUF_PEXT_TRAIL_LEN) {
+				/* copy n's header to m's trailer */
+				memcpy(&mpgs->trail[trail_len], npgs->hdr,
+				    hdr_len);
+				mpgs->trail_len += hdr_len;
+				m->m_len += hdr_len;
+				npgs->hdr_len = 0;
+				n->m_len -= hdr_len;
+			}
+		}
+#endif
+
 		/* Compress small unmapped mbufs into plain mbufs. */
 		if ((m->m_flags & M_NOMAP) && m->m_len <= MLEN &&
 		    !mbuf_has_tls_session(m)) {
@@ -124,15 +159,6 @@ sbready_compress(struct sockbuf *sb, struct mbuf *m0, 
 			}
 		}
 
-		/*
-		 * NB: In sbcompress(), 'n' is the last mbuf in the
-		 * socket buffer and 'm' is the new mbuf being copied
-		 * into the trailing space of 'n'.  Here, the roles
-		 * are reversed and 'n' is the next mbuf after 'm'
-		 * that is being copied into the trailing space of
-		 * 'm'.
-		 */
-		n = m->m_next;
 		while ((n != NULL) && (n != end) && (m->m_flags & M_EOR) == 0 &&
 		    M_WRITABLE(m) &&
 		    (m->m_flags & M_NOMAP) == 0 &&