From nobody Fri Dec 24 13:08:08 2021 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A08FC1913195; Fri, 24 Dec 2021 13:08:11 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JL6n73d0Mz3kjN; Fri, 24 Dec 2021 13:08:11 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 0285F27152; Fri, 24 Dec 2021 13:08:10 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: <02d9ca2a-2fd7-942b-f411-be007ef88327@FreeBSD.org> Date: Fri, 24 Dec 2021 15:08:08 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 Content-Language: en-US To: FreeBSD Current , FreeBSD Hackers From: Andriy Gapon Subject: schedgraph.d experience, per-CPU buffers, pipes Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1640351291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ukPiNy1UwYaPjiemcRS5XSKrwtDEydPuax1eH3ax+E4=; b=n5bKNtGfvCOCPNBZDkntZURk9X9pwCpjTtocm/t9x/2YGoqjTIFTqcIAj6FKflM6X8xwWt gD6pN+66wGCVPW/SM/LeuhB7W6VrK/jYLv0Sv/NkzQGN3FgjYskMxMvtDpfWIdXol1KARt vBAm9FjimV9PgBysNqeHONGY6FcKQRe+jSMcavU3Y90QRxZCfI/1ofc4fKZUZLrko1U6/H 5hQBX09Zyoy3yGz8Ys+86o982rLguA4/lkTaJ0PLZo3Ga/lz47FFvYfXfMJ2wgyXlRssjH PTbX4NoelMiIYw5qfXld0Y7GphMxfxSL7MnSEkhFR2k62mjor7KRVB6GFuDtTA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1640351291; a=rsa-sha256; cv=none; b=hk7rDkwr3MrN3Oe59bi4t0eJMuNAQtHj2Eqb97+Lca2OWTqN8jzMkQdXKaZLOLAlcNIqJq TrVhU+pbwzSeNirZ1ygaRaYfB59EpNhc5CRQQ43BazyXZ3Z9iaHVwAoXusEkUr3I8VVXFz 882YEezSdA5Rf2vkiqWSYrBME3uLIHqhuBFaDSbYmkJfoFPL1W2/aQGVokQdYYY4LF1bju uz9ZzgIhoKypPencyHYzu/b4C5rAD1IOkrft+/zHXCHsfJHU41P0kXycwzmIbvE6hEO3Bu F6RTzcBX5r0/3mJKL7l6XbibYdAleHG0djocV/QeLXPob3mbxpvxhSQPCGMMJA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N I would like to share some experience or maybe rather a warning about using DTrace for tracing scheduling events. Unlike KTR which has a global circular buffer, DTrace with bufpolicy=ring uses per-CPU circular buffers. So, if there is an asymmetry in processor load, the buffers will fill and wrap-around at different speeds. In the end, they might have approximately equal numbers of events but those may cover very different time intervals. So, some additional post-processing is required to find the latest event among first ones of each per-CPU buffer. Any traces from before that would have information gaps ("missing" processors) and would be very confusing. Also, I noticed that processes passing a lot of data through pipes produce a lot of scheduling events as they seem to get blocked and unlocked every few microseconds (on a modern performant system with the default pipe sizing configuration). That contributes to a quick wrap-around of circular buffers. -- Andriy Gapon