From nobody Sat Apr 20 15:07:03 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VMFH14Gdvz5Hy0q for ; Sat, 20 Apr 2024 15:07:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VMFH10svxz4lwd; Sat, 20 Apr 2024 15:07:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qk1-x729.google.com with SMTP id af79cd13be357-78efd1a0022so217503485a.3; Sat, 20 Apr 2024 08:07:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713625627; x=1714230427; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=db0YGeBPkiMNHJsxuXjwEeNKFs/oxXobQwmGg2Saq/Y=; b=YORk3nOkF43so0E8xgeiVjODXvfPvHyybCopQOUtheuTDMdGgkIDt6op+HHm4U7zS1 5RUwZq0W0crQtLzrnX9TKOPYl3wWCwetT0LIsQL7mapPOjikaPTA/vj9zQpjt/NoCZZ8 rj5UNxGSDhE8ezRVefBjz+92MMJtWR3OW5gx1YdsUwE2v1s00KjApfuLtvZNYUsQJ45W kmkTvrmYF4SkjfvBtCfkdVH8rma6Ap66kRKhrzhgnLAKJOt1UOAg1tQKGGo3fHaTm0ur 4p6UEjP78JVT06xPpInyoejTNy38yftis1af3PUY8Xzuhhe846pKJwP2ThuE62+94tnJ NcTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713625627; x=1714230427; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=db0YGeBPkiMNHJsxuXjwEeNKFs/oxXobQwmGg2Saq/Y=; b=dF4q5ndObAS3bL9cHLgCKHDcssPSPHeCr7eV5AN9czPVx9JETVkNiE8yAlTEbXB5NH dTGPf+6GvA05dJw168ARpJn/CaVhXNIsyIuvSoKhGEMNF/hPCUl9rEIH3ZSJkwbR9p/L XCHmVBuEPQPAYa0HQDGYJI5aH1uIQoGPohPdFhJVaNOYfoX62tsPm2Cg2e8/L7g4gwVX +BPRpVFIpxasq7N/SWgv8quHpJPnKDKDJHMVx+IwdtxEYIVDsw40W5JGAqRpUS5Sqb8l qhxNIBgX6IRUgj0RiyJSy1uUrgLMMI7cTpcBbD9j4Cg8jCVzOqjFzNimXRlvElwG6oGq 5Saw== X-Gm-Message-State: AOJu0YwlxqVDunHFyQFUlsC1rjxx50Olk5By/n9qLqrqKcL0H3R59USm GvEWpdQNfyYmnNWZJMrHqsvV2tHD60vxxWyKl78ipVjkZL/bzliSJrYwgg== X-Google-Smtp-Source: AGHT+IE/HyZE0F2IOM3FmdOkj5NnLh7gGf8NpymhdyeQ1CDH8kxAXNQFWdY9NJ4zCmw9vX6HQ3X25A== X-Received: by 2002:a05:620a:29c2:b0:790:675c:5f4b with SMTP id s2-20020a05620a29c200b00790675c5f4bmr1245984qkp.49.1713625627034; Sat, 20 Apr 2024 08:07:07 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id u5-20020a05620a022500b0078f13d368f3sm2116256qkm.95.2024.04.20.08.07.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 20 Apr 2024 08:07:06 -0700 (PDT) Date: Sat, 20 Apr 2024 11:07:03 -0400 From: Mark Johnston To: Alan Somers Cc: FreeBSD Hackers Subject: Re: Stressing malloc(9) Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4VMFH10svxz4lwd On Fri, Apr 19, 2024 at 04:23:51PM -0600, Alan Somers wrote: > TLDR; > How can I create a workload that causes malloc(9)'s performance to plummet? > > Background: > I recently witnessed a performance problem on a production server. > Overall throughput dropped by over 30x. dtrace showed that 60% of the > CPU time was dominated by lock_delay as called by three functions: > printf (via ctl_worker_thread), g_eli_alloc_data, and > g_eli_write_done. One thing those three have in common is that they > all use malloc(9). Fixing the problem was as simple as telling CTL to > stop printing so many warnings, by tuning > kern.cam.ctl.time_io_secs=100000. > > But even with CTL quieted, dtrace still reports ~6% of the CPU cycles > in lock_delay via g_eli_alloc_data. So I believe that malloc is > limiting geli's performance. I would like to try replacing it with > uma(9). What is the size of the allocations that g_eli_alloc_data() is doing? malloc() is a pretty thin layer over UMA for allocations <= 64KB. Larger allocations are handled by a different path (malloc_large()) which goes directly to the kmem_* allocator functions. Those functions are very expensive: they're serialized by global locks and need to update the pmap (and perform TLB shootdowns when memory is freed). They're not meant to be used at a high rate. My first guess would be that your production workload was hitting this path, and your benchmarks are not. If you have stack traces or lock names from DTrace, that would help validate this theory, in which case using UMA to cache buffers would be a reasonable solution. > But on a non-production server, none of my benchmark workloads causes > g_eli_alloc_data to break a sweat. I can't get its CPU consumption to > rise higher than 0.5%. And that's using the smallest sector size and > block size that I can. > > So my question is: does anybody have a program that can really stress > malloc(9)? I'd like to run it in parallel with my geli benchmarks to > see how much it interferes. > > -Alan >