From nobody Sat Sep 28 21:52:14 2024 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XGLf737kFz5XtfN; Sat, 28 Sep 2024 21:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XGLf713qkz44nQ; Sat, 28 Sep 2024 21:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1727560335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=lEK0vnUZSALcac3zPYOhq8eVQEMX5ggo7/yzo3jOhHo=; b=at8l2ytETIl/V+aeRYJDcjo1qb1khq8v5hNCrDmcgh0Vxhs3nku4ZkpWVDiyQ2pTMEWNeG klWtifewvcRNhDhUEsEL8xrLrgdzBaBnXQPRzivcaHHlrtBtSf8C+s2Qo5CBFJikBEPDAV JGsaKmNQEiByh9ioBZJuv71FHUM39Xq/Jp54xg6kkRXzOTwPofEMWm8yRk9yX7eVxjFXpv myJnIvTaMdYOzFHXvr459l0Wm0SYdHUIdSsRwslhedMNR7TG0dvO3rl20Sd5AOPC5I2uEa fwV1Ke3Z084JsazAzPdH23zCP/9XrTDSzyApbo6zEARw5B9uERMlm4FcHyhUUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1727560335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=lEK0vnUZSALcac3zPYOhq8eVQEMX5ggo7/yzo3jOhHo=; b=hozK1C6JqSRqruGiqk81tRdga0ci0Ms4eGYa+reZvLAxumU6vqaf07MAduiWIoGei1qjcU OIaJFckV2KOiVqMCM8kKFXe0Buxwv2iwca8Pq3n7ndmF9cgpEb1uSZc4D6E4LoBl4P+VdG XueljWccCL1GzAVZPh829X4ZHDD2hjeus4OlRT6DV5qXLVFCC4Dnkwn/kadqPFvodG6cR1 rVUcH4egqTJwugEcphbhR7/hZzV0iIywc+2U3FZXyv+QGj3dFF20zynx5u/mGnTcO4UPxN b6IZX9NYrYvDLp8XBF5wRUpMzEH9IyjShBbolfXICngXgs2m0i8M68S4ROLlHA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1727560335; a=rsa-sha256; cv=none; b=Ol0t+QmxB5NFJU9ACqx+SkjlOznYNYZ3h7boeufh4bCDw1aBOKhziL9ohSHwpRScYx6MMF rhhDOil/o3nadSQ/DDi/3EzAx2y+8CnTBU99LZnT8+eEbk3TDvK85EwzXjKKG+WcS6Qugx 0ADD1H6cq0uFnSgt8jW+Bmqgd4MbaFQdlL/jzlMlIk2NezjnjoUsvAymJ6hDwSgZenbCWv uIPpNSIeguep9fUNey1nFIgTqjkTfwiHVI5gWpQcWqBO1Bl8aAbXkHXbDNbTsEyt7PIwIT 6CDMrN36zfREGxrA0ZiEPnxSAxiUELGVUhYuU94xw4TBJPAiRRpndQYWDwmYnw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XGLf70YYnzjLJ; Sat, 28 Sep 2024 21:52:15 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 48SLqEW6051598; Sat, 28 Sep 2024 21:52:14 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 48SLqEds051595; Sat, 28 Sep 2024 21:52:14 GMT (envelope-from git) Date: Sat, 28 Sep 2024 21:52:14 GMT Message-Id: <202409282152.48SLqEds051595@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: "Bjoern A. Zeeb" Subject: git: d2587a699e37 - stable/13 - malloc(9): extend contigmalloc(9) by a "slab cookie" List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: bz X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: d2587a699e37868909b279b109bb0a379c077c5b Auto-Submitted: auto-generated The branch stable/13 has been updated by bz: URL: https://cgit.FreeBSD.org/src/commit/?id=d2587a699e37868909b279b109bb0a379c077c5b commit d2587a699e37868909b279b109bb0a379c077c5b Author: Bjoern A. Zeeb AuthorDate: 2024-06-30 19:00:44 +0000 Commit: Bjoern A. Zeeb CommitDate: 2024-09-28 18:39:34 +0000 malloc(9): extend contigmalloc(9) by a "slab cookie" Extend kern_malloc.c internals to also cover contigmalloc(9) by a "slab cookie" and not just malloc/malloc_large. This allows us to call free(9) even on contigmalloc(9) addresses and deprecate contigfree(9). Update the contigmalloc(9) man page accordingly. The way this is done (free(9) working for contigmalloc) will hide the UMA/VM bits from a consumer which may otherwise need to know whether the original allocation was by malloc or contigmalloc by looking at the cookie (likely via an accessor function). This simplifies the implementation of consumers of mixed environments a lot. This is preliminary work to allow LinuxKPI to be adjusted to better play by the rules Linux puts out for various allocations. Most of this was described/explained to me by jhb. One may observe that realloc(9) is currently unchanged (and contrary to [contig]malloc/[contig]free an implementation may need access the "slab cookie" information given it will likely be implementation dependent which allocation type to use if size changes beyond the usable size of the initial allocation). Described by: jhb Sponsored by: The FreeBSD Foundation Reviewed by: markj, kib Differential Revision: https://reviews.freebsd.org/D45812 (cherry picked from commit 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b) --- share/man/man9/contigmalloc.9 | 9 ++- sys/kern/kern_malloc.c | 128 +++++++++++++++++++++++++++++++++--------- 2 files changed, 108 insertions(+), 29 deletions(-) diff --git a/share/man/man9/contigmalloc.9 b/share/man/man9/contigmalloc.9 index f52cb7bde0ed..9e7086885870 100644 --- a/share/man/man9/contigmalloc.9 +++ b/share/man/man9/contigmalloc.9 @@ -23,7 +23,7 @@ .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" -.Dd October 30, 2018 +.Dd July 22, 2024 .Dt CONTIGMALLOC 9 .Os .Sh NAME @@ -113,6 +113,13 @@ function deallocates memory allocated by a previous call to .Fn contigmalloc or .Fn contigmalloc_domainset . +Its use is deprecated in favor of +.Xr free 9 +which no longer requires the caller to know the +.Fa size +and also accepts +.Dv NULL +as an address. .Sh IMPLEMENTATION NOTES The .Fn contigmalloc diff --git a/sys/kern/kern_malloc.c b/sys/kern/kern_malloc.c index 0720c36740ee..1a00e184c83f 100644 --- a/sys/kern/kern_malloc.c +++ b/sys/kern/kern_malloc.c @@ -117,6 +117,16 @@ dtrace_malloc_probe_func_t __read_mostly dtrace_malloc_probe; #define DEBUG_REDZONE_ARG #endif +typedef enum { + SLAB_COOKIE_SLAB_PTR = 0x0, + SLAB_COOKIE_MALLOC_LARGE = 0x1, + SLAB_COOKIE_CONTIG_MALLOC = 0x2, +} slab_cookie_t; +#define SLAB_COOKIE_MASK 0x3 +#define SLAB_COOKIE_SHIFT 2 +#define GET_SLAB_COOKIE(_slab) \ + ((slab_cookie_t)(uintptr_t)(_slab) & SLAB_COOKIE_MASK) + /* * When realloc() is called, if the new size is sufficiently smaller than * the old size, realloc() will allocate a new, smaller block to avoid @@ -452,6 +462,21 @@ malloc_type_freed(struct malloc_type *mtp, unsigned long size) * If M_NOWAIT is set, this routine will not block and return NULL if * the allocation fails. */ +#define IS_CONTIG_MALLOC(_slab) \ + (GET_SLAB_COOKIE(_slab) == SLAB_COOKIE_CONTIG_MALLOC) +#define CONTIG_MALLOC_SLAB(_size) \ + ((void *)(((_size) << SLAB_COOKIE_SHIFT) | SLAB_COOKIE_CONTIG_MALLOC)) +static inline size_t +contigmalloc_size(uma_slab_t slab) +{ + uintptr_t va; + + KASSERT(IS_CONTIG_MALLOC(slab), + ("%s: called on non-contigmalloc allocation: %p", __func__, slab)); + va = (uintptr_t)slab; + return (va >> SLAB_COOKIE_SHIFT); +} + void * contigmalloc(unsigned long size, struct malloc_type *type, int flags, vm_paddr_t low, vm_paddr_t high, unsigned long alignment, @@ -461,8 +486,11 @@ contigmalloc(unsigned long size, struct malloc_type *type, int flags, ret = (void *)kmem_alloc_contig(size, flags, low, high, alignment, boundary, VM_MEMATTR_DEFAULT); - if (ret != NULL) + if (ret != NULL) { + /* Use low bits unused for slab pointers. */ + vsetzoneslab((uintptr_t)ret, NULL, CONTIG_MALLOC_SLAB(size)); malloc_type_allocated(type, round_page(size)); + } return (ret); } @@ -475,25 +503,28 @@ contigmalloc_domainset(unsigned long size, struct malloc_type *type, ret = (void *)kmem_alloc_contig_domainset(ds, size, flags, low, high, alignment, boundary, VM_MEMATTR_DEFAULT); - if (ret != NULL) + if (ret != NULL) { + /* Use low bits unused for slab pointers. */ + vsetzoneslab((uintptr_t)ret, NULL, CONTIG_MALLOC_SLAB(size)); malloc_type_allocated(type, round_page(size)); + } return (ret); } /* - * contigfree: + * contigfree (deprecated). * * Free a block of memory allocated by contigmalloc. * * This routine may not block. */ void -contigfree(void *addr, unsigned long size, struct malloc_type *type) +contigfree(void *addr, unsigned long size __unused, struct malloc_type *type) { - - kmem_free((vm_offset_t)addr, size); - malloc_type_freed(type, round_page(size)); + free(addr, type); } +#undef IS_CONTIG_MALLOC +#undef CONTIG_MALLOC_SLAB #ifdef MALLOC_DEBUG static int @@ -563,22 +594,19 @@ malloc_dbg(caddr_t *vap, size_t *sizep, struct malloc_type *mtp, /* * Handle large allocations and frees by using kmem_malloc directly. */ -static inline bool -malloc_large_slab(uma_slab_t slab) -{ - uintptr_t va; - - va = (uintptr_t)slab; - return ((va & 1) != 0); -} - +#define IS_MALLOC_LARGE(_slab) \ + (GET_SLAB_COOKIE(_slab) == SLAB_COOKIE_MALLOC_LARGE) +#define MALLOC_LARGE_SLAB(_size) \ + ((void *)(((_size) << SLAB_COOKIE_SHIFT) | SLAB_COOKIE_MALLOC_LARGE)) static inline size_t malloc_large_size(uma_slab_t slab) { uintptr_t va; va = (uintptr_t)slab; - return (va >> 1); + KASSERT(IS_MALLOC_LARGE(slab), + ("%s: called on non-malloc_large allocation: %p", __func__, slab)); + return (va >> SLAB_COOKIE_SHIFT); } static caddr_t __noinline @@ -592,8 +620,8 @@ malloc_large(size_t *size, struct malloc_type *mtp, struct domainset *policy, sz = roundup(*size, PAGE_SIZE); kva = kmem_malloc_domainset(policy, sz, flags); if (kva != 0) { - /* The low bit is unused for slab pointers. */ - vsetzoneslab(kva, NULL, (void *)((sz << 1) | 1)); + /* Use low bits unused for slab pointers. */ + vsetzoneslab((uintptr_t)kva, NULL, MALLOC_LARGE_SLAB(sz)); uma_total_inc(sz); *size = sz; } @@ -618,6 +646,8 @@ free_large(void *addr, size_t size) kmem_free((vm_offset_t)addr, size); uma_total_dec(size); } +#undef IS_MALLOC_LARGE +#undef MALLOC_LARGE_SLAB /* * malloc: @@ -908,18 +938,30 @@ free(void *addr, struct malloc_type *mtp) vtozoneslab((vm_offset_t)addr & (~UMA_SLAB_MASK), &zone, &slab); if (slab == NULL) - panic("free: address %p(%p) has not been allocated.\n", + panic("free: address %p(%p) has not been allocated", addr, (void *)((u_long)addr & (~UMA_SLAB_MASK))); - if (__predict_true(!malloc_large_slab(slab))) { + switch (GET_SLAB_COOKIE(slab)) { + case __predict_true(SLAB_COOKIE_SLAB_PTR): size = zone->uz_size; #if defined(INVARIANTS) && !defined(KASAN) free_save_type(addr, mtp, size); #endif uma_zfree_arg(zone, addr, slab); - } else { + break; + case SLAB_COOKIE_MALLOC_LARGE: size = malloc_large_size(slab); free_large(addr, size); + break; + case SLAB_COOKIE_CONTIG_MALLOC: + size = contigmalloc_size(slab); + kmem_free((vm_offset_t)addr, size); + size = round_page(size); + break; + default: + panic("%s: addr %p slab %p with unknown cookie %d", __func__, + addr, slab, GET_SLAB_COOKIE(slab)); + /* NOTREACHED */ } malloc_type_freed(mtp, size); } @@ -951,7 +993,8 @@ zfree(void *addr, struct malloc_type *mtp) panic("free: address %p(%p) has not been allocated.\n", addr, (void *)((u_long)addr & (~UMA_SLAB_MASK))); - if (__predict_true(!malloc_large_slab(slab))) { + switch (GET_SLAB_COOKIE(slab)) { + case __predict_true(SLAB_COOKIE_SLAB_PTR): size = zone->uz_size; #if defined(INVARIANTS) && !defined(KASAN) free_save_type(addr, mtp, size); @@ -959,11 +1002,22 @@ zfree(void *addr, struct malloc_type *mtp) kasan_mark(addr, size, size, 0); explicit_bzero(addr, size); uma_zfree_arg(zone, addr, slab); - } else { + break; + case SLAB_COOKIE_MALLOC_LARGE: size = malloc_large_size(slab); kasan_mark(addr, size, size, 0); explicit_bzero(addr, size); free_large(addr, size); + break; + case SLAB_COOKIE_CONTIG_MALLOC: + size = round_page(contigmalloc_size(slab)); + explicit_bzero(addr, size); + kmem_free((vm_offset_t)addr, size); + break; + default: + panic("%s: addr %p slab %p with unknown cookie %d", __func__, + addr, slab, GET_SLAB_COOKIE(slab)); + /* NOTREACHED */ } malloc_type_freed(mtp, size); } @@ -1010,10 +1064,20 @@ realloc(void *addr, size_t size, struct malloc_type *mtp, int flags) ("realloc: address %p out of range", (void *)addr)); /* Get the size of the original block */ - if (!malloc_large_slab(slab)) + switch (GET_SLAB_COOKIE(slab)) { + case __predict_true(SLAB_COOKIE_SLAB_PTR): alloc = zone->uz_size; - else + break; + case SLAB_COOKIE_MALLOC_LARGE: alloc = malloc_large_size(slab); + break; + default: +#ifdef INVARIANTS + panic("%s: called for addr %p of unsupported allocation type; " + "slab %p cookie %d", __func__, addr, slab, GET_SLAB_COOKIE(slab)); +#endif + return (NULL); + } /* Reuse the original block if appropriate */ if (size <= alloc && @@ -1095,10 +1159,18 @@ malloc_usable_size(const void *addr) panic("malloc_usable_size: address %p(%p) is not allocated.\n", addr, (void *)((u_long)addr & (~UMA_SLAB_MASK))); - if (!malloc_large_slab(slab)) + switch (GET_SLAB_COOKIE(slab)) { + case __predict_true(SLAB_COOKIE_SLAB_PTR): size = zone->uz_size; - else + break; + case SLAB_COOKIE_MALLOC_LARGE: size = malloc_large_size(slab); + break; + default: + __assert_unreachable(); + size = 0; + break; + } #endif return (size); }