From nobody Fri Nov 22 14:05:13 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xvxgt4VcTz5dwb5; Fri, 22 Nov 2024 14:05:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xvxgt2lLQz4xJm; Fri, 22 Nov 2024 14:05:14 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1732284314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ATWFr7jgaKSUq2XfigXyvZHUJWuCNnajfL+PeMg2l3A=; b=B7J5vLhl0306yTQQLNzKIdA4wzJAINpjaBiE4E36BL59bWc2Nirq0/xYI5AnFAMPYV50Za O5941lfXOXZ0h/i+IpFXhZqnfczn2kVAPwTEGh/JifuyVwxFKNnl1ZIYXS3hNTrlDc7q5F eHi0NcArw0TbnXRsZvP2phWZY73SCcFLrSWu47KCidg/cF+0nwna+BRe6LWuPLM88hqlLu 05Q1Xkgos/GaKkD/RtAqxUONQP6g2ruYjw2if+WarhjU276+42fd0jOedr9gVP6iY1rPop WQZFrhEvJlqE4XkmUuoEVzV3Ivyd9KaHEOGj8nbOUay1/Lo+NAarYSbua0RK3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1732284314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ATWFr7jgaKSUq2XfigXyvZHUJWuCNnajfL+PeMg2l3A=; b=hQ1QDmu+nxu3Ev51k3MN1qaOYzFe8ouPgSTtYDhtQFPe1fxdq/B9RpuaMZSJXoDkdq3DX/ b3+SRE1pqfSciAAUmCK+HFqsLNHJKL6zpNM8UWI3T8uFKpimpa/tXCHYjBRifS8xuckEA4 r32Wkt3Z57iTcI0IQ9wVQnCCHdhNbYS3qiPUMpEilCnQcg5tR28rD1XHKGpB1te89AcHX1 0ZEjj5eJn07HMo728zr0aii3pm2VSSji16U2qGHqSYwOgXGVMMlAHdlXewjzgcgU8J1HJw rXZEdu4yEhHwhi+w+9DQ0CniRyt7bxvmbKDH2OuMEqU0+SU2yxtxirh9NOYHgw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1732284314; a=rsa-sha256; cv=none; b=a10og8J9L4bUTiIwV1yvbks/KwZzuWjVuHNhE+hxQneebVadRlRcDPdiPmGaRAsriiz04W 0t9FAfHS3P5c4UYtmQXuOgWQQ8BX67C22EM+OqMtxAPS5MG6JRftCTbfjrfBAGF1s+m0A5 3dlAkCZ9V2jtuWnYrABtavorIoCU0i0Jc1FshP8AshAuLn1EaLTA50ZWgiEKCZJ7oYxDWe CDOnQr+J0FMiySo9+dBtWzG7j+XuVwiU8lqo6Bjo05Povof65h+nY21y2UV7AIwmIHBMQ/ MG3b64muTsbPQ7mDzt5Whqwi9l71kr2d737Jcg6NRicjSO+dYHh1eF2VtCLAJw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Xvxgt05VMzS4X; Fri, 22 Nov 2024 14:05:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 4AME5Dae065270; Fri, 22 Nov 2024 14:05:13 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 4AME5Ddk065267; Fri, 22 Nov 2024 14:05:13 GMT (envelope-from git) Date: Fri, 22 Nov 2024 14:05:13 GMT Message-Id: <202411221405.4AME5Ddk065267@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 5d12db2dafec - main - dtrace: Avoid excessive pcpu allocations List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 5d12db2dafece9f6a0453c4a45c4abed6b1e15ec Auto-Submitted: auto-generated The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=5d12db2dafece9f6a0453c4a45c4abed6b1e15ec commit 5d12db2dafece9f6a0453c4a45c4abed6b1e15ec Author: Mark Johnston AuthorDate: 2024-11-22 13:55:20 +0000 Commit: Mark Johnston CommitDate: 2024-11-22 14:04:12 +0000 dtrace: Avoid excessive pcpu allocations We were previously allocating MAXCPU structures for several purposes, but this is generally unnecessary and is quite excessive, especially after MAXCPU was bumped to 1024 on amd64 and arm64. We already are careful to allocate only as many per-CPU tracing buffers as are needed; extend this to other allocations. For example, in a 2-vCPU VM, the size of a consumer state structure drops from 64KB to 128B. The size of the per-consumer `dts_buffer` and `dts_aggbuffer` arrays shrink similarly. Ditto for pre-allocations of local and global D variable storage space. MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D47667 --- .../contrib/opensolaris/uts/common/dtrace/dtrace.c | 66 ++++++++++------------ sys/cddl/dev/dtrace/dtrace_ioctl.c | 2 +- sys/cddl/dev/dtrace/dtrace_load.c | 4 +- 3 files changed, 33 insertions(+), 39 deletions(-) diff --git a/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c b/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c index d3e40e96e897..5f635cc088d7 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c +++ b/sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c @@ -741,7 +741,7 @@ dtrace_canstore_statvar(uint64_t addr, size_t sz, size_t *remain, return (0); maxglobalsize = dtrace_statvar_maxsize + sizeof (uint64_t); - maxlocalsize = maxglobalsize * NCPU; + maxlocalsize = maxglobalsize * (mp_maxid + 1); for (i = 0; i < nsvars; i++) { dtrace_statvar_t *svar = svars[i]; @@ -1744,7 +1744,7 @@ dtrace_dynvar_clean(dtrace_dstate_t *dstate) dtrace_dynvar_t **rinsep; int i, j, work = 0; - for (i = 0; i < NCPU; i++) { + CPU_FOREACH(i) { dcpu = &dstate->dtds_percpu[i]; rinsep = &dcpu->dtdsc_rinsing; @@ -1782,7 +1782,7 @@ dtrace_dynvar_clean(dtrace_dstate_t *dstate) * rinsing list -- and then we borrow this CPU to * rinse our dirty list. */ - for (j = 0; j < NCPU; j++) { + CPU_FOREACH(j) { dtrace_dstate_percpu_t *rinser; rinser = &dstate->dtds_percpu[j]; @@ -1800,7 +1800,7 @@ dtrace_dynvar_clean(dtrace_dstate_t *dstate) break; } - if (j == NCPU) { + if (j > mp_maxid) { /* * We were unable to find another CPU that * could accept this dirty list -- we are @@ -1841,7 +1841,7 @@ dtrace_dynvar_clean(dtrace_dstate_t *dstate) dtrace_sync(); - for (i = 0; i < NCPU; i++) { + CPU_FOREACH(i) { dcpu = &dstate->dtds_percpu[i]; if (dcpu->dtdsc_rinsing == NULL) @@ -2190,7 +2190,7 @@ retry: case DTRACE_DSTATE_CLEAN: { void *sp = &dstate->dtds_state; - if (++cpu >= NCPU) + if (++cpu > mp_maxid) cpu = 0; if (dcpu->dtdsc_dirty != NULL && @@ -6504,7 +6504,7 @@ dtrace_dif_emulate(dtrace_difo_t *difo, dtrace_mstate_t *mstate, size_t lim; sz += sizeof (uint64_t); - ASSERT(svar->dtsv_size == NCPU * sz); + ASSERT(svar->dtsv_size == (mp_maxid + 1) * sz); a += curcpu * sz; if (*(uint8_t *)a == UINT8_MAX) { @@ -6521,7 +6521,8 @@ dtrace_dif_emulate(dtrace_difo_t *difo, dtrace_mstate_t *mstate, break; } - ASSERT(svar->dtsv_size == NCPU * sizeof (uint64_t)); + ASSERT(svar->dtsv_size == + (mp_maxid + 1) * sizeof (uint64_t)); tmp = (uint64_t *)(uintptr_t)svar->dtsv_data; regs[rd] = tmp[curcpu]; break; @@ -6544,7 +6545,7 @@ dtrace_dif_emulate(dtrace_difo_t *difo, dtrace_mstate_t *mstate, size_t lim; sz += sizeof (uint64_t); - ASSERT(svar->dtsv_size == NCPU * sz); + ASSERT(svar->dtsv_size == (mp_maxid + 1) * sz); a += curcpu * sz; if (regs[rd] == 0) { @@ -6565,7 +6566,8 @@ dtrace_dif_emulate(dtrace_difo_t *difo, dtrace_mstate_t *mstate, break; } - ASSERT(svar->dtsv_size == NCPU * sizeof (uint64_t)); + ASSERT(svar->dtsv_size == + (mp_maxid + 1) * sizeof (uint64_t)); tmp = (uint64_t *)(uintptr_t)svar->dtsv_data; tmp[curcpu] = regs[rd]; break; @@ -10678,10 +10680,11 @@ dtrace_difo_init(dtrace_difo_t *dp, dtrace_vstate_t *vstate) svarp = &vstate->dtvs_locals; if (v->dtdv_type.dtdt_flags & DIF_TF_BYREF) - dsize = NCPU * (v->dtdv_type.dtdt_size + + dsize = (mp_maxid + 1) * + (v->dtdv_type.dtdt_size + sizeof (uint64_t)); else - dsize = NCPU * sizeof (uint64_t); + dsize = (mp_maxid + 1) * sizeof (uint64_t); break; @@ -12603,7 +12606,7 @@ dtrace_buffer_consumed(dtrace_buffer_t *bufs, hrtime_t when) { int i; - for (i = 0; i < NCPU; i++) { + CPU_FOREACH(i) { dtrace_buffer_t *buf = &bufs[i]; if (buf->dtb_size == 0) @@ -12627,7 +12630,7 @@ dtrace_buffer_free(dtrace_buffer_t *bufs) { int i; - for (i = 0; i < NCPU; i++) { + CPU_FOREACH(i) { dtrace_buffer_t *buf = &bufs[i]; if (buf->dtb_tomax == NULL) { @@ -14379,7 +14382,8 @@ dtrace_dstate_init(dtrace_dstate_t *dstate, size_t size) dstate->dtds_size = size; dstate->dtds_base = base; dstate->dtds_percpu = kmem_cache_alloc(dtrace_state_cache, KM_SLEEP); - bzero(dstate->dtds_percpu, NCPU * sizeof (dtrace_dstate_percpu_t)); + bzero(dstate->dtds_percpu, + (mp_maxid + 1) * sizeof (dtrace_dstate_percpu_t)); hashsize = size / (dstate->dtds_chunksize + sizeof (dtrace_dynhash_t)); @@ -14413,14 +14417,10 @@ dtrace_dstate_init(dtrace_dstate_t *dstate, size_t size) VERIFY((uintptr_t)start < limit); VERIFY((uintptr_t)start >= (uintptr_t)base); - maxper = (limit - (uintptr_t)start) / NCPU; + maxper = (limit - (uintptr_t)start) / (mp_maxid + 1); maxper = (maxper / dstate->dtds_chunksize) * dstate->dtds_chunksize; -#ifndef illumos CPU_FOREACH(i) { -#else - for (i = 0; i < NCPU; i++) { -#endif dstate->dtds_percpu[i].dtdsc_free = dvar = start; /* @@ -14430,7 +14430,7 @@ dtrace_dstate_init(dtrace_dstate_t *dstate, size_t size) * whatever is left over. In either case, we set the limit to * be the limit of the dynamic variable space. */ - if (maxper == 0 || i == NCPU - 1) { + if (maxper == 0 || i == mp_maxid) { limit = (uintptr_t)base + size; start = NULL; } else { @@ -14603,7 +14603,7 @@ dtrace_state_create(struct cdev *dev, struct ucred *cred __unused) char c[30]; dtrace_state_t *state; dtrace_optval_t *opt; - int bufsize = NCPU * sizeof (dtrace_buffer_t), i; + int bufsize = (mp_maxid + 1) * sizeof (dtrace_buffer_t), i; int cpu_it; ASSERT(MUTEX_HELD(&dtrace_lock)); @@ -14651,12 +14651,6 @@ dtrace_state_create(struct cdev *dev, struct ucred *cred __unused) state->dts_dev = dev; #endif - /* - * We allocate NCPU buffers. On the one hand, this can be quite - * a bit of memory per instance (nearly 36K on a Starcat). On the - * other hand, it saves an additional memory reference in the probe - * path. - */ state->dts_buffer = kmem_zalloc(bufsize, KM_SLEEP); state->dts_aggbuffer = kmem_zalloc(bufsize, KM_SLEEP); @@ -14666,12 +14660,12 @@ dtrace_state_create(struct cdev *dev, struct ucred *cred __unused) * assumed to be seeded at this point (if from Fortuna seed file). */ arc4random_buf(&state->dts_rstate[0], 2 * sizeof(uint64_t)); - for (cpu_it = 1; cpu_it < NCPU; cpu_it++) { + for (cpu_it = 1; cpu_it <= mp_maxid; cpu_it++) { /* * Each CPU is assigned a 2^64 period, non-overlapping * subsequence. */ - dtrace_xoroshiro128_plus_jump(state->dts_rstate[cpu_it-1], + dtrace_xoroshiro128_plus_jump(state->dts_rstate[cpu_it - 1], state->dts_rstate[cpu_it]); } @@ -14970,7 +14964,7 @@ dtrace_state_go(dtrace_state_t *state, processorid_t *cpu) cyc_handler_t hdlr; cyc_time_t when; #endif - int rval = 0, i, bufsize = NCPU * sizeof (dtrace_buffer_t); + int rval = 0, i, bufsize = (mp_maxid + 1) * sizeof (dtrace_buffer_t); dtrace_icookie_t cookie; mutex_enter(&cpu_lock); @@ -15224,10 +15218,10 @@ dtrace_state_go(dtrace_state_t *state, processorid_t *cpu) * We enable anonymous tracing before APs are started, so we must * activate buffers using the current CPU. */ - if (state == dtrace_anon.dta_state) - for (int i = 0; i < NCPU; i++) + if (state == dtrace_anon.dta_state) { + CPU_FOREACH(i) dtrace_buffer_activate_cpu(state, i); - else + } else dtrace_xcall(DTRACE_CPUALL, (dtrace_xcall_t)dtrace_buffer_activate, state); #else @@ -15410,7 +15404,7 @@ dtrace_state_destroy(dtrace_state_t *state) #ifdef illumos minor_t minor = getminor(state->dts_dev); #endif - int i, bufsize = NCPU * sizeof (dtrace_buffer_t); + int i, bufsize = (mp_maxid + 1) * sizeof (dtrace_buffer_t); dtrace_speculation_t *spec = state->dts_speculations; int nspec = state->dts_nspeculations; uint32_t match; @@ -15724,7 +15718,7 @@ dtrace_helper_trace(dtrace_helper_action_t *helper, if ((svar = vstate->dtvs_locals[i]) == NULL) continue; - ASSERT(svar->dtsv_size >= NCPU * sizeof (uint64_t)); + ASSERT(svar->dtsv_size >= (mp_maxid + 1) * sizeof (uint64_t)); ent->dtht_locals[i] = ((uint64_t *)(uintptr_t)svar->dtsv_data)[curcpu]; } diff --git a/sys/cddl/dev/dtrace/dtrace_ioctl.c b/sys/cddl/dev/dtrace/dtrace_ioctl.c index be7fff170166..01d51197fde8 100644 --- a/sys/cddl/dev/dtrace/dtrace_ioctl.c +++ b/sys/cddl/dev/dtrace/dtrace_ioctl.c @@ -229,7 +229,7 @@ dtrace_ioctl(struct cdev *dev, u_long cmd, caddr_t addr, "DTRACEIOC_AGGSNAP":"DTRACEIOC_BUFSNAP", curcpu, desc.dtbd_cpu); - if (desc.dtbd_cpu >= MAXCPU || CPU_ABSENT(desc.dtbd_cpu)) + if (desc.dtbd_cpu > mp_maxid || CPU_ABSENT(desc.dtbd_cpu)) return (ENOENT); mutex_enter(&dtrace_lock); diff --git a/sys/cddl/dev/dtrace/dtrace_load.c b/sys/cddl/dev/dtrace/dtrace_load.c index ef2d29e143f8..115e6d0a8ed7 100644 --- a/sys/cddl/dev/dtrace/dtrace_load.c +++ b/sys/cddl/dev/dtrace/dtrace_load.c @@ -100,8 +100,8 @@ dtrace_load(void *dummy) mutex_enter(&dtrace_lock); dtrace_state_cache = kmem_cache_create("dtrace_state_cache", - sizeof (dtrace_dstate_percpu_t) * NCPU, DTRACE_STATE_ALIGN, - NULL, NULL, NULL, NULL, NULL, 0); + sizeof (dtrace_dstate_percpu_t) * (mp_maxid + 1), + DTRACE_STATE_ALIGN, NULL, NULL, NULL, NULL, NULL, 0); ASSERT(MUTEX_HELD(&cpu_lock)); dtrace_bymod = dtrace_hash_create(offsetof(dtrace_probe_t, dtpr_mod),