From nobody Mon Mar 27 13:28:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PlYY65vdFz41X8w for ; Mon, 27 Mar 2023 13:28:26 +0000 (UTC) (envelope-from mandree@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PlYY65Mcqz3sTJ; Mon, 27 Mar 2023 13:28:26 +0000 (UTC) (envelope-from mandree@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679923706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/IF62W68WqABKseHo5QXfRri9Qv+g6BJwJOXukT2mY=; b=eFBS0pp36dwLf1EzPxLz3cmhiu/UotU2cw6YnKzgVdAK48jrfFsDeBWhsfJz7lUlo9hrAU E3vnnYG8KxT58ZoZB1oCjgzf2+RmI8K0K0F8WIIo0jevqPub8fbgF3DoTiEKucugnvg1xo yjAosdPQLudK7pSVGZ5GeZukjZzfEoIwb5XrBy7mDf2a3Q4/hqOriHgZNw5dCPtnQ6fwzw ozLtgg1P9o+v6uRCb7SF7+CaYB3m+lFZQksp29pdslvg78UoElMLGn+ceq5a7r3sj7A8SZ LKsWmCKOIypyFU6xKDSxWrp1y/nI15/sqFUb8xKGV2s/AkONy6JwBb8ONJr17w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679923706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h/IF62W68WqABKseHo5QXfRri9Qv+g6BJwJOXukT2mY=; b=qIhdaGrAnmsCS5wy19xML+0NfDU0xp36m3uZofqF+d4JxIudpZG4XjaTQNSs7/Kr032QJx a4VAaYLHO6VnPymPAyg2T81dc+5qn3YhJFFwQ3V3VIvPCA2e5HdaIddYCKpdV3DB3IIVxG rZYOLWjSNHuEYpd4wJ/KmJxjRjUuINAsIOiZ3+AVutqi4+30+sqct6f+gs/z5RXh1csJMH cPYHN2VeRQvnLYg3s/UC5jzQ5l+g6BwMEEMOvTi/z/tR7/OtvmRRkNvodrIY9tVCtuAJSY nxagozseIY60WyWS8susdidu8Pc4xjuV5zo8sUdheBf14xLeB7BmaZ7IsDIxYg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679923706; a=rsa-sha256; cv=none; b=sC/7ojmIBFMnjRbu0jmHu3x6mOvP/8SxALx6QNgN4QmeAZ/DvdXIWkdyFyHdlZ3CowU97U L2XLGV53bvmpjPYq8Vxyx4n1yFhb07lTAQ5WRpsxomBF9XD+35dKLeXTxg0gae1HJVWDHm jKgfh1NhXq/lEvi1l8LqtXwQULbJoEnqhNptzc2aE/cxhL2Th5vVq5m9O6iwV9VnxXp3i3 rQ6q159foNHyg9i65DarxoU/F3skZ9hoQsAqqhNk3RldNwJy/ybNrm8ZJxs/HuzhOjPrpj MMPa09YdOs3DvO8BCit62rjGvNW0THQZyvcOqCiEgF8shqAjbpNDmUysmqmBxA== Received: from mandree.no-ip.org (pd9e073a3.dip0.t-ipconnect.de [217.224.115.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PlYY63RXLz1Q0X; Mon, 27 Mar 2023 13:28:26 +0000 (UTC) (envelope-from mandree@freebsd.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id 78B2ADC47FE; Mon, 27 Mar 2023 15:28:24 +0200 (CEST) Message-ID: <1d4d589c-eafe-6219-2e6e-884d49b7ff57@FreeBSD.org> Date: Mon, 27 Mar 2023 15:28:24 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 From: Matthias Andree Content-Language: en-US To: Mateusz Guzik , sgk@troutmask.apl.washington.edu Cc: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> Organization: FreeBSD.org Subject: Re: Periodic rant about SCHED_ULE In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Am 27.03.23 um 14:13 schrieb Mateusz Guzik: > > I repeat the setup: 8 cores, 8 processes doing cpu-bound stuff while > niced to 20 vs make -j buildkernel > > I had a little more look here, slapped in some hacks as a POC and got > an improvement from 67 minutes above to 21. > > Hacks are: > 1. limit hog timeslice to 1 tick so that is more eager to bail > 2. always preempt if pri < cpri > > So far I can confidently state the general problem: ULE penalizes > non-cpu hogs for blocking (even if it is not their fault, so to speak) > and that bumps their prio past preemption threshold, at which point > they can't preempt said hogs (despite hogs having a higher priority). > At the same time hogs use their full time slices, while non-hogs get > off cpu very early and have to wait a long time to get back on, at > least in part due to inability to preempt said hogs. > > As I mentioned elsewhere in the thread, interactivity scoring takes > "voluntary off cpu time" into account. As literally anything but > getting preempted counts as "voluntary sleep", workers get shafted for > going off cpu while waiting on locks in the kernel. > > If I/O needs to happen and the thread waits for the result, it most > likely does it early in its timeslice and once it's all ready it waits > for background hogs to get off cpu -- it can't preempt them. > > All that said: > 1. "interactivity scoring" (see sched_interact_score) > > I don't know if it makes any sense to begin with. Even if it does, it > counts stuff it should not by not differentiating between deliberately > going off cpu (e.g., actual sleep) vs just waiting for a file being > read. Imagine firefox reading a file from disk and being considered > less interactive for it. > > I don't have a solution for this problem. I *suspect* the way to go > would be to explicitly mark xorg/wayland/whatever as "interactive" and > have it inherited by its offspring. At the same time it should not > follow to stuff spawned in terminals. Not claiming this is perfect, > but it does eliminate the guessing game. > > Even so, 4BSD does not have any mechanism of the sort and reportedly > remains usable on a desktop just by providing some degree of fairness. > > Given that, I suspect the short term solution would whack said scoring > altogether and focus on fairness (see below). > > 2. fairness > > As explained above doing any offcpu-time inducing work instantly > shafts threads versus cpu hogs, even if said hogs are niced way above > them. > > Here I *suspect* position to add in the runqueue should be related to > how much slice was left when the thread went off cpu, while making > sure that hogs get to run eventually. Not that I have a nice way of > implementing this -- maybe a separate queue for known hogs and picking > them every n turns or similar. > There is some analogy - not sure if we can learn something from it. TCP (the network's Transmission Control Protocol) has the big overspanning item "fairness between streams", and possibly we need to put this into focus. Apparently we have now established that threads that do not use up their quantum are not scheduled *sooner* which seems to harm both interactivity AND total processing time (because, say, they may want to run a spell checker after receiving a key-press key-release sequence or send off an e-mail after a click on the send button). What I currently fail to understand is why we have not gone to improving SCHED_ULE yet. This seems to be the first time I've paid attention to some details. Have I missed similar discussion in the past? Or have they stalled soonish? -- Matthias Andree FreeBSD ports committer