From nobody Mon Jul 15 17:59:42 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WN92j4gTCz5RNWv; Mon, 15 Jul 2024 17:59:57 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WN92j2rMLz4KYq; Mon, 15 Jul 2024 17:59:57 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-a77ec5d3b0dso581569666b.0; Mon, 15 Jul 2024 10:59:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721066396; x=1721671196; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=2GRnwiHQMUEEXdUbRZ6GSckdat1ZukdaXXDsjKRGehY=; b=CM0NkkQEA01Jko9PHZn8cfu2LTpDRMcGy3zicXjukVFNq/nOXUNjLI8Xwk97MVrhSl yO+RaHex6/rBZxuJhofBmzdKdk7eNnigJDoTrOW2qIqtxGJQ3Ui0A5nk3T8ltY503Zhk 2rfcHS78r6w+ZC+JJfJRLI5OOh5/ZW8/+RjXQHTghoLy2KRVtltvR0WBlpi9HUHJFu7g fqYZAl2aIiQMHWqxC242+FkwxNLES2i8ep/CRAzFQCvQpj0mDAzgJEn8hPee1dS3zPuO Pbs2vO7pq2xF8PbBseMFybL6bTHquUtkmZbFDsDzOo0t5HpT7oSphQHD6bmXF1iCW2aI bJDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721066396; x=1721671196; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2GRnwiHQMUEEXdUbRZ6GSckdat1ZukdaXXDsjKRGehY=; b=DXBHa7G8i9XEoosep2afNU3oIWpNVwJ9I/KO+q215JmaHDdZwVfLh+1i7K/LrSLl/p Fzaf4GnkmuGcTj3a0vHTM1oXTj+r2B2ZFbTTX+wyntUDVIJ9A5mwCQn+qtKnVHbOaxaZ VSznogkVOIs07BE4wisWRumKWsrhaReW1s4ho7L1c5HM8mfhSyuP3gnVir18/TAisdjW dbdmaUoO2dSETdS81Ei3ccgZu91/HuYjtlU3Pa7bCwlhEnUc1eltz5LZURnSDTeZuuCE Nhqq2tXa1VptS80XUAIWnDXQjhHc26P01Nz8ummg74dcIqjNyVGffONJeUh5/VrnNYip 4Ejg== X-Forwarded-Encrypted: i=1; AJvYcCUCaoLnngoRNey19kkFGCX0AzgRB4o0t9AL1AFJWcjHCFvJWNDFKBoNq7Elnb1bGRhdStMbEJhnl8DZ+ZLvi0S58kL+LaZU+cXBIjPR3hKEjrhZh5A3iB9YVUpX8P7Ktnx5EsrHXzJhua2ZAHuQFKz9NNne5E0uXGViWJyI56EEYaDwrABq1KNr5Fu3jGY= X-Gm-Message-State: AOJu0YwZTUK5FdMvonNYJpyNml9bFzyskqySLEhRWpEPzIBxyxULCR19 d2aedS0pyOldrfIVvi9H1ofROWN8rMEXShFc34FX7yZNBjXwfvhhlYf5Jw3jEIBzUkNSIdRlTQo oqahdQunSiKcV/GQLmvXoD1BIDMvoHzfLSpM= X-Google-Smtp-Source: AGHT+IEtn0Eq+xGfJrUcd5FYZbJ7sEI8HGUUtkw0CCzYkxWyRnhOY72GznVYRhDu0tpU37RyMMKvQWaRgGoGMnWSOHI= X-Received: by 2002:a17:906:d69:b0:a75:3c2d:cd90 with SMTP id a640c23a62f3a-a79e6afae3dmr34639766b.65.1721066395563; Mon, 15 Jul 2024 10:59:55 -0700 (PDT) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 References: <202407111107.46BB7uSb007077@gitrepo.freebsd.org> <81cfe7ad-cbee-4122-abef-e47ce2b34f05@FreeBSD.org> In-Reply-To: <81cfe7ad-cbee-4122-abef-e47ce2b34f05@FreeBSD.org> From: Mateusz Guzik Date: Mon, 15 Jul 2024 19:59:42 +0200 Message-ID: Subject: Re: git: 87ee63bac69d - main - locks: add a runtime check for missing turnstile To: John Baldwin Cc: Mateusz Guzik , src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4WN92j2rMLz4KYq On Mon, Jul 15, 2024 at 6:22=E2=80=AFPM John Baldwin wrot= e: > > On 7/11/24 07:07, Mateusz Guzik wrote: > > The branch main has been updated by mjg: > > > > URL: https://cgit.FreeBSD.org/src/commit/?id=3D87ee63bac69dc49291f55590= b8baa57cad6c7d85 > > > > commit 87ee63bac69dc49291f55590b8baa57cad6c7d85 > > Author: Mateusz Guzik > > AuthorDate: 2024-07-11 00:17:27 +0000 > > Commit: Mateusz Guzik > > CommitDate: 2024-07-11 11:06:52 +0000 > > > > locks: add a runtime check for missing turnstile > > > > There are sometimes bugs which result in the unlock fast path fail= ing, > > which in turns causes a not-helpful crash report when dereferencin= g a > > NULL turnstile. Help debugging such cases by pointing out what hap= pened > > along with some debug. > > > > Sponsored by: Rubicon Communications, LLC ("Netgate") > > --- > > sys/kern/kern_mutex.c | 4 +++- > > sys/kern/kern_rwlock.c | 16 ++++++++++++---- > > 2 files changed, 15 insertions(+), 5 deletions(-) > > > > diff --git a/sys/kern/kern_mutex.c b/sys/kern/kern_mutex.c > > index 90361b23c09a..0fa624cc4bb1 100644 > > --- a/sys/kern/kern_mutex.c > > +++ b/sys/kern/kern_mutex.c > > @@ -1053,7 +1053,9 @@ __mtx_unlock_sleep(volatile uintptr_t *c, uintptr= _t v) > > turnstile_chain_lock(&m->lock_object); > > _mtx_release_lock_quick(m); > > ts =3D turnstile_lookup(&m->lock_object); > > - MPASS(ts !=3D NULL); > > + if (__predict_false(ts =3D=3D NULL)) { > > + panic("got NULL turnstile on mutex %p v %zx", m, v); > > + } > > Hmm, this is just an expanded KASSERT() but always on rather than conditi= onal on INVARIANTS? > > Do you have examples of the type of bugs that cause this? (Is it unlocki= ng a freed mutex > or the like?) We generally hide all these types of checks under INVARIAN= TS rather than > shipping them in release kernels. > Use-after-free, overflow, underflow, bitflip or what have you all can fail the fast path. Once that happens and the kernel crashes with a null pointer deref, here is a crash at netgate which prodded this: calltrap() at calltrap+0x8/frame 0xfffffe0106720920 --- trap 0xc, rip =3D 0xffffffff80d5ab70, rsp =3D 0xfffffe01067209f0, rbp =3D 0xfffffe0106720a00 --- turnstile_broadcast() at turnstile_broadcast+0x40/frame 0xfffffe0106720a00 __rw_wunlock_hard() at __rw_wunlock_hard+0x9e/frame 0xfffffe0106720a30 nd6_resolve_slow() at nd6_resolve_slow+0x2d7/frame 0xfffffe0106720aa0 nd6_resolve() at nd6_resolve+0x125/frame 0xfffffe0106720b10 ether_output() at ether_output+0x4e7/frame 0xfffffe0106720ba0 ip_output_send() at ip_output_send+0xdc/frame 0xfffffe0106720be0 ip_output() at ip_output+0x1295/frame 0xfffffe0106720ce0 ip_forward() at ip_forward+0x3c2/frame 0xfffffe0106720d90 ip_input() at ip_input+0x705/frame 0xfffffe0106720df0 swi_net() at swi_net+0x138/frame 0xfffffe0106720e60 ithread_loop() at ithread_loop+0x257/frame 0xfffffe0106720ef0 fork_exit() at fork_exit+0x7f/frame 0xfffffe0106720f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0106720f30 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- Neither the register dump nor anything in the backtrace indicate what happe= ned. Since the kernel is going down anyway, one may as well get some debug from = it. --=20 Mateusz Guzik