From nobody Tue Oct 08 08:39:35 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XN8Zv614yz5Z9bH; Tue, 08 Oct 2024 08:39:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XN8Zv5TfMz4Fcr; Tue, 8 Oct 2024 08:39:35 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728376775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=99FjkK3BOW4eWejYgi53WEQYKOtr3IxqaKvCm/g5uKg=; b=grHwYfOgaNzIvftpdvxwAhA6BhEgyxgSvWLbN9ifY98Vx2qItzClG4wFJCBFv00KZZtJas /UtaBnjtZhbBDC82emijMGpftsoIxo4dTW462TZcmaewIWr3y4IplVe56tqgqflB2cz8Bb phoF29kA2VyBrxkKC9GhtLckWPlTS/yIX14XeNOzmLGxpVd/mpZIDN+Tooy4GkSy7DCXx8 vr4vOX2ZHMVA5xdxzkpSFPcjNTn0EouYzB2VL3Se9e7IcKcrAoAVViTYwaoQuTDqKsf81g j5yGuFtOQjExVay5k+sFRnCnUYRl9kdsOuUiJdGb1Ji6nFVKfiKmlye0nWwDow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1728376775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=99FjkK3BOW4eWejYgi53WEQYKOtr3IxqaKvCm/g5uKg=; b=XyHAlazOaajA9EfOXdm071aYaNxJ+tQrqIGOtXM4BXFAD91l+JpuBmLObmnlWjCxn4VHPn vSoqvV83Y/J5Vww8iPXvTlX5RwYIP1c4XX0quGWP72mikH12WG3f38Ny4ggBjNgHpf0RYe CU5SdZBbvQc5IrD1QImBAHx/nXnGPlbWes4NKMf5yU4v46AcY+y1dTheCO1Tkq/L1yd671 j9HBA+Mk5wATSnaegpFKSg5no/Z/qWTwtkSHmg/cl23BMGLYVyvaiUdmT92x4snQFgDc9A uY2CD3RvB3d85ZOA6e8Fy88uiPqfmLEKIXp6OkIlkoIrCh/uZeD3YQv67FWhgQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1728376775; a=rsa-sha256; cv=none; b=G5pW2wHISKCMW16CigvpsbnmsJkaRPCwZgYY1c9n2oV6BMtHS1TYWrr0er7+SgdVl1iijZ O8bK7kZLWxYjpJaQkLo0aa8Lk+gDH/1ZyvDMMCOJzcvMLWFBnq4LIJlFG9a6ue69T/y3rj 97nD5tCtjpMA45dAtNn9Xl+mDMTujoAA79fRkNcTxsoN1B17PugDvdBTtf1m0JIl42vWP9 2lwbwXu2+4DRD5dJXcROzvJdaufKEZrv/DHIPoeur1cp+SBHyJCvGYg1py0jV+LJfUrGtP u3Iz3RXHSkZRH09N6d1shZVNYzEKoM/F3CFOwf2T+kMZMI7rY5/VT5W2u4kcdA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4XN8Zv54tYzNfj; Tue, 8 Oct 2024 08:39:35 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 4988dZ8o004752; Tue, 8 Oct 2024 08:39:35 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 4988dZxt004749; Tue, 8 Oct 2024 08:39:35 GMT (envelope-from git) Date: Tue, 8 Oct 2024 08:39:35 GMT Message-Id: <202410080839.4988dZxt004749@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Dimitry Andric Subject: git: 789f0f34e4ca - stable/14 - Merge commit b84d773fd004 from llvm git (by Fangrui Song): List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: dim X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 789f0f34e4caa813168bfceeb6cdc5f147953be2 Auto-Submitted: auto-generated The branch stable/14 has been updated by dim: URL: https://cgit.FreeBSD.org/src/commit/?id=789f0f34e4caa813168bfceeb6cdc5f147953be2 commit 789f0f34e4caa813168bfceeb6cdc5f147953be2 Author: Dimitry Andric AuthorDate: 2024-09-22 17:08:47 +0000 Commit: Dimitry Andric CommitDate: 2024-10-08 08:37:14 +0000 Merge commit b84d773fd004 from llvm git (by Fangrui Song): [Parallel] Revert sequential task changes https://reviews.llvm.org/D148728 introduced `bool Sequential` to unify `execute` and the old `spawn` without argument. However, sequential tasks might be executed by any worker thread (non-deterministic), leading to non-determinism output for ld.lld -z nocombreloc (see https://reviews.llvm.org/D133003). In addition, the extra member variables have overhead. This sequential task has only been used for lld parallel relocation scanning. This patch restores the behavior before https://reviews.llvm.org/D148728 . Fix #105958 Pull Request: https://github.com/llvm/llvm-project/pull/109084 This fixes the non-reproducibility we had noticed when linking our EFI loaders, and for which we committed a workaround in f5ce3f4ef562. MFC after: 3 days (cherry picked from commit 54521a2ff93ae06c95c31f79f89dc23c9b51c20b) --- contrib/llvm-project/lld/ELF/Relocations.cpp | 56 ++++++++++++++-------- .../llvm/include/llvm/Support/Parallel.h | 4 +- contrib/llvm-project/llvm/lib/Support/Parallel.cpp | 49 +++++-------------- 3 files changed, 48 insertions(+), 61 deletions(-) diff --git a/contrib/llvm-project/lld/ELF/Relocations.cpp b/contrib/llvm-project/lld/ELF/Relocations.cpp index 92a1b9baaca3..8cb01b63d2cc 100644 --- a/contrib/llvm-project/lld/ELF/Relocations.cpp +++ b/contrib/llvm-project/lld/ELF/Relocations.cpp @@ -1584,30 +1584,44 @@ template void elf::scanRelocations() { bool serial = !config->zCombreloc || config->emachine == EM_MIPS || config->emachine == EM_PPC64; parallel::TaskGroup tg; - for (ELFFileBase *f : ctx.objectFiles) { - auto fn = [f]() { + auto outerFn = [&]() { + for (ELFFileBase *f : ctx.objectFiles) { + auto fn = [f]() { + RelocationScanner scanner; + for (InputSectionBase *s : f->getSections()) { + if (s && s->kind() == SectionBase::Regular && s->isLive() && + (s->flags & SHF_ALLOC) && + !(s->type == SHT_ARM_EXIDX && config->emachine == EM_ARM)) + scanner.template scanSection(*s); + } + }; + if (serial) + fn(); + else + tg.spawn(fn); + } + auto scanEH = [] { RelocationScanner scanner; - for (InputSectionBase *s : f->getSections()) { - if (s && s->kind() == SectionBase::Regular && s->isLive() && - (s->flags & SHF_ALLOC) && - !(s->type == SHT_ARM_EXIDX && config->emachine == EM_ARM)) - scanner.template scanSection(*s); + for (Partition &part : partitions) { + for (EhInputSection *sec : part.ehFrame->sections) + scanner.template scanSection(*sec); + if (part.armExidx && part.armExidx->isLive()) + for (InputSection *sec : part.armExidx->exidxSections) + if (sec->isLive()) + scanner.template scanSection(*sec); } }; - tg.spawn(fn, serial); - } - - tg.spawn([] { - RelocationScanner scanner; - for (Partition &part : partitions) { - for (EhInputSection *sec : part.ehFrame->sections) - scanner.template scanSection(*sec); - if (part.armExidx && part.armExidx->isLive()) - for (InputSection *sec : part.armExidx->exidxSections) - if (sec->isLive()) - scanner.template scanSection(*sec); - } - }); + if (serial) + scanEH(); + else + tg.spawn(scanEH); + }; + // If `serial` is true, call `spawn` to ensure that `scanner` runs in a thread + // with valid getThreadIndex(). + if (serial) + tg.spawn(outerFn); + else + outerFn(); } static bool handleNonPreemptibleIfunc(Symbol &sym, uint16_t flags) { diff --git a/contrib/llvm-project/llvm/include/llvm/Support/Parallel.h b/contrib/llvm-project/llvm/include/llvm/Support/Parallel.h index 8170da98f15a..c34619ab7d96 100644 --- a/contrib/llvm-project/llvm/include/llvm/Support/Parallel.h +++ b/contrib/llvm-project/llvm/include/llvm/Support/Parallel.h @@ -97,9 +97,7 @@ public: // Spawn a task, but does not wait for it to finish. // Tasks marked with \p Sequential will be executed // exactly in the order which they were spawned. - // Note: Sequential tasks may be executed on different - // threads, but strictly in sequential order. - void spawn(std::function f, bool Sequential = false); + void spawn(std::function f); void sync() const { L.sync(); } diff --git a/contrib/llvm-project/llvm/lib/Support/Parallel.cpp b/contrib/llvm-project/llvm/lib/Support/Parallel.cpp index 9b14b05b5211..26da702969b7 100644 --- a/contrib/llvm-project/llvm/lib/Support/Parallel.cpp +++ b/contrib/llvm-project/llvm/lib/Support/Parallel.cpp @@ -12,7 +12,6 @@ #include "llvm/Support/Threading.h" #include -#include #include #include #include @@ -39,7 +38,7 @@ namespace { class Executor { public: virtual ~Executor() = default; - virtual void add(std::function func, bool Sequential = false) = 0; + virtual void add(std::function func) = 0; virtual size_t getThreadCount() const = 0; static Executor *getDefaultExecutor(); @@ -98,13 +97,10 @@ public: static void call(void *Ptr) { ((ThreadPoolExecutor *)Ptr)->stop(); } }; - void add(std::function F, bool Sequential = false) override { + void add(std::function F) override { { std::lock_guard Lock(Mutex); - if (Sequential) - WorkQueueSequential.emplace_front(std::move(F)); - else - WorkQueue.emplace_back(std::move(F)); + WorkStack.push_back(std::move(F)); } Cond.notify_one(); } @@ -112,42 +108,23 @@ public: size_t getThreadCount() const override { return ThreadCount; } private: - bool hasSequentialTasks() const { - return !WorkQueueSequential.empty() && !SequentialQueueIsLocked; - } - - bool hasGeneralTasks() const { return !WorkQueue.empty(); } - void work(ThreadPoolStrategy S, unsigned ThreadID) { threadIndex = ThreadID; S.apply_thread_strategy(ThreadID); while (true) { std::unique_lock Lock(Mutex); - Cond.wait(Lock, [&] { - return Stop || hasGeneralTasks() || hasSequentialTasks(); - }); + Cond.wait(Lock, [&] { return Stop || !WorkStack.empty(); }); if (Stop) break; - bool Sequential = hasSequentialTasks(); - if (Sequential) - SequentialQueueIsLocked = true; - else - assert(hasGeneralTasks()); - - auto &Queue = Sequential ? WorkQueueSequential : WorkQueue; - auto Task = std::move(Queue.back()); - Queue.pop_back(); + auto Task = std::move(WorkStack.back()); + WorkStack.pop_back(); Lock.unlock(); Task(); - if (Sequential) - SequentialQueueIsLocked = false; } } std::atomic Stop{false}; - std::atomic SequentialQueueIsLocked{false}; - std::deque> WorkQueue; - std::deque> WorkQueueSequential; + std::vector> WorkStack; std::mutex Mutex; std::condition_variable Cond; std::promise ThreadsCreated; @@ -205,16 +182,14 @@ TaskGroup::~TaskGroup() { L.sync(); } -void TaskGroup::spawn(std::function F, bool Sequential) { +void TaskGroup::spawn(std::function F) { #if LLVM_ENABLE_THREADS if (Parallel) { L.inc(); - detail::Executor::getDefaultExecutor()->add( - [&, F = std::move(F)] { - F(); - L.dec(); - }, - Sequential); + detail::Executor::getDefaultExecutor()->add([&, F = std::move(F)] { + F(); + L.dec(); + }); return; } #endif