From nobody Thu May 30 05:53:35 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Vqb664pcvz5Ly04 for ; Thu, 30 May 2024 05:53:50 +0000 (UTC) (envelope-from nkumarababu@gmail.com) Received: from mail-ot1-x336.google.com (mail-ot1-x336.google.com [IPv6:2607:f8b0:4864:20::336]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Vqb655G6Sz4mhw for ; Thu, 30 May 2024 05:53:49 +0000 (UTC) (envelope-from nkumarababu@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=Kq6yMqdW; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of nkumarababu@gmail.com designates 2607:f8b0:4864:20::336 as permitted sender) smtp.mailfrom=nkumarababu@gmail.com Received: by mail-ot1-x336.google.com with SMTP id 46e09a7af769-6f72b8db7deso291900a34.2 for ; Wed, 29 May 2024 22:53:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717048428; x=1717653228; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=Lsyabcg+bqs07PI+x6OyA+pg/1sNC16q3splZu/7MjI=; b=Kq6yMqdWIKY9U0sSKcfHWpRouSlMVLLHwvhIMxebgv8ymcFGICOEbj969P7tY64lXX /5jIPqOwtyl4OxHlFORvZ1ndG9lcI+eej6fEPRPU8lDgFm/yvQcearbS9h7xKJiOcSp4 mj+3d+cFSmUFPPIZhPfQRG5hbSOP60grUPQAGvL5L1T+4oJCeF191Oa0hBZ0g3ZZzQwj U90wkQs2w03Fv6EjiEkYD73MRC0jtMPBDEkttT1KprSBrSxicyqb/tC25T0rIHmAl+wg azGdbM70qOL0WmpXAPxikMw7MBdOqjkMfkv9L/SQu74FKkflrmhyNTcNYFx4NiHQGekf halA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717048428; x=1717653228; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Lsyabcg+bqs07PI+x6OyA+pg/1sNC16q3splZu/7MjI=; b=G72rr38q67q0lXac5D7hilVBhgy9aegcGtIqI63vXl2HewrhLrTzK/sSUuWd0gd7zm MpGTGf0S0wbQwEXfwF6XnwTO2s4pFALlJTS64Lly1qf2uKTd/LHmQQMCosf6/cpxw0WK ieQNZy5POjOc3PCjkXXbxEgMKhkNQq42+dbwK3d6nsxFAD3m7ITldn+FjVyDa+JL3k3R XcXqUyCjMATYMGVeU7u3DyK2x9WpTC2Lq6jtpf8HJUhMNct0wJIErCgJx+V4fhwvRLdy ZViUVxcbVIInVY1ov5fV6HflcmHmVqC/10C+3PrQdG73xakJ9UzZuaoJ9v1vzhq2isVI 9abQ== X-Gm-Message-State: AOJu0YxX1sP3I7zgWhxGH4A+2AQlM5BnKlwZoL6ca1N57UXc7njBqlON 4uqX28S81h2v/9IWQfe5foqtPa+p6h/16+JVw08TMbp5bff4n70L2bxuAliHKL/9wdh8YqC8+Xq I/P7AaMn8IzwE9uUY6Ww4Ck5+He59mnvwTG0= X-Google-Smtp-Source: AGHT+IFQLu/hjNzhffHCIfM9IklhX2JnlFWSc6SWaKMJgqaofW4siYygT5yeIV7qaiVsaUZH1lb+/Co4c8UmobOhJbQ= X-Received: by 2002:a05:6870:8921:b0:24c:ae57:b4b4 with SMTP id 586e51a60fabf-25060bda4c5mr915752fac.24.1717048427668; Wed, 29 May 2024 22:53:47 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 From: Kumara Babu Date: Wed, 29 May 2024 22:53:35 -0700 Message-ID: Subject: Upperlimit for bwait() To: freebsd-hackers@freebsd.org Content-Type: multipart/mixed; boundary="00000000000098c36f0619a57dd5" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.99)[-0.989]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; HAS_ATTACHMENT(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::336:from] X-Rspamd-Queue-Id: 4Vqb655G6Sz4mhw --00000000000098c36f0619a57dd5 Content-Type: multipart/alternative; boundary="00000000000098c36d0619a57dd3" --00000000000098c36d0619a57dd3 Content-Type: text/plain; charset="UTF-8" Hello, There have been a few incidents reported on Juniper devices with FreeBSD, where buffer IO operations sleep for more than 30 mins. Theoretically, this can happen due to faulty hardware or in virtual platforms due to faulty connection between guest and host, filesystem corruption, too many buffer IO operations, and/or host not responding due to various reasons. When that happens, as this buffer IO writes hold a lock before going to sleep, the threads waiting for that lock would starve for so long. There is no upper limit for this bwait() as of now. If that wait goes beyond 30 mins for a sleeping thread OR 15 mins for a thread blocked on turnstile, deadlkres crashes the kernel assuming a possible deadlock. We perhaps could gracefully handle such lengthy buffer IO operations by adding a timeout in bwait() - like say 10 minutes. If the buffer IO is not completed in a few mins, it probably would not complete forever and/or would be slowing down the entire system. So it is better to stop such faulty IO operations. For now, since we had seen these instances only with BIO operations, I have a patch to set this value only from bufwait(). Please find the patch attached. I am not very sure if 10 mins is a good upper limit for all the scenarios for bwait(). If it is, then we could just change msleep() in bwait() to set a 10 mins upper limit by default. Please let me know if this approach works for all the usecases - If not, is there a better alternative ? And is 10 mins okay for a timeout ? Thanks and Regards, Kumara --00000000000098c36d0619a57dd3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello,

There have been a few incidents reported on Juniper devic= es with FreeBSD, where buffer IO operations sleep for more than 30 mins. Th= eoretically, this can happen due to faulty hardware or in virtual platforms= due to faulty connection between guest and host, filesystem corruption, to= o many buffer IO operations, and/or host not responding due to various reas= ons. When that happens, as this buffer IO writes hold a lock before going t= o sleep, the threads waiting for that lock would starve for so long. There = is no upper limit for this bwait() as of now. If that wait goes beyond 30 m= ins for a sleeping thread OR 15 mins for a thread blocked on turnstile, dea= dlkres crashes the kernel assuming a possible deadlock.

We perhaps could grac= efully handle such lengthy buffer IO operations by adding a timeout in bwai= t() - like say 10 minutes. If the buffer IO is not completed in a few mins,= it probably would not complete forever and/or would be slowing down the en= tire system. So it is better to stop such faulty IO operations.

For now, since we= had seen these instances only with BIO operations, I have a patch to set t= his value only from bufwait(). Please find the patch attached. I am not ver= y sure if 10 mins is a good upper limit for all the scenarios for bwait(). = If it is, then we could just change msleep() in bwait() to set a 10 mins up= per limit by default.=C2=A0

Pl= ease let me know if this approach works for all the usecases - If not, is t= here a better alternative ?=C2=A0 And is 10 mins okay for a timeout ?

Thanks and Regards,

Kumara

--00000000000098c36d0619a57dd3-- --00000000000098c36f0619a57dd5 Content-Type: application/octet-stream; name="bwait_timeout.patch" Content-Disposition: attachment; filename="bwait_timeout.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lwsua22p0 ZGlmZiAtLWdpdCBhL3N5cy9rZXJuL3Zmc19iaW8uYyBiL3N5cy9rZXJuL3Zmc19iaW8uYwppbmRl eCBiNTQ2NmZiMmNkNTMuLjc4OGFiMmUxYjdmMSAxMDA2NDQKLS0tIGEvc3lzL2tlcm4vdmZzX2Jp by5jCisrKyBiL3N5cy9rZXJuL3Zmc19iaW8uYwpAQCAtNzUsNiArNzUsNyBAQAogI2luY2x1ZGUg PHN5cy9zbXAuaD4KICNpbmNsdWRlIDxzeXMvc3lzY3RsLmg+CiAjaW5jbHVkZSA8c3lzL3N5c2Nh bGxzdWJyLmg+CisjaW5jbHVkZSA8c3lzL3N5c2xvZy5oPgogI2luY2x1ZGUgPHN5cy92bWVtLmg+ CiAjaW5jbHVkZSA8c3lzL3ZtbWV0ZXIuaD4KICNpbmNsdWRlIDxzeXMvdm5vZGUuaD4KQEAgLTM5 MCw2ICszOTEsOSBAQCBzdGF0aWMgaW50IGJkaXJ0eXdhaXQ7CiAvKiBNYXhpbXVtIG51bWJlciBv ZiBidWZmZXIgZG9tYWlucy4gKi8KICNkZWZpbmUJQlVGX0RPTUFJTlMJOAogCisvKiBUaW1lb3V0 IGZvciBidWZmZXIgSS9POiAxMCBtaW5zICovCisjZGVmaW5lIEJUSU1FT1VUICAgICAgIDYwMCAq IGh6CisKIHN0cnVjdCBidWZkb21haW5zZXQgYmRsb2RpcnR5OwkJLyogRG9tYWlucyA+IGxvZGly dHkgKi8KIHN0cnVjdCBidWZkb21haW5zZXQgYmRoaWRpcnR5OwkJLyogRG9tYWlucyA+IGhpZGly dHkgKi8KIApAQCAtNDUzNyw5ICs0NTQxLDkgQEAgaW50CiBidWZ3YWl0KHN0cnVjdCBidWYgKmJw KQogewogCWlmIChicC0+Yl9pb2NtZCA9PSBCSU9fUkVBRCkKLQkJYndhaXQoYnAsIFBSSUJJTywg ImJpb3JkIik7CisJCWJ3YWl0KGJwLCBQUklCSU8sICJiaW9yZCIsIEJUSU1FT1VUKTsKIAllbHNl Ci0JCWJ3YWl0KGJwLCBQUklCSU8sICJiaW93ciIpOworCQlid2FpdChicCwgUFJJQklPLCAiYmlv d3IiLCBCVElNRU9VVCk7CiAJaWYgKGJwLT5iX2ZsYWdzICYgQl9FSU5UUikgewogCQlicC0+Yl9m bGFncyAmPSB+Ql9FSU5UUjsKIAkJcmV0dXJuIChFSU5UUik7CkBAIC01MTIxLDE0ICs1MTI1LDIy IEBAIGJkb25lKHN0cnVjdCBidWYgKmJwKQogfQogCiB2b2lkCi1id2FpdChzdHJ1Y3QgYnVmICpi cCwgdV9jaGFyIHByaSwgY29uc3QgY2hhciAqd2NoYW4pCitid2FpdChzdHJ1Y3QgYnVmICpicCwg dV9jaGFyIHByaSwgY29uc3QgY2hhciAqd2NoYW4sIGludCB0aW1vKQogewogCXN0cnVjdCBtdHgg Km10eHA7CisJaW50IHJldDsKIAogCW10eHAgPSBtdHhfcG9vbF9maW5kKG10eHBvb2xfc2xlZXAs IGJwKTsKIAltdHhfbG9jayhtdHhwKTsKLQl3aGlsZSAoKGJwLT5iX2ZsYWdzICYgQl9ET05FKSA9 PSAwKQotCQltc2xlZXAoYnAsIG10eHAsIHByaSwgd2NoYW4sIDApOworCXdoaWxlICgoYnAtPmJf ZmxhZ3MgJiBCX0RPTkUpID09IDApIHsKKwkJcmV0ID0gbXNsZWVwKGJwLCBtdHhwLCBwcmksIHdj aGFuLCB0aW1vKTsKKwkJaWYgKHJldCA9PSBFV09VTERCTE9DSykgeworCQkJbG9nIChMT0dfRVJS LCAiJXM6IFdhaXRlZCB0b28gbG9uZyglZCkgZm9yIGEgYnVmZmVyIElPIHRvIGNvbXBsZXRlXG4i LCBfX2Z1bmNfXywgdGltbyk7CisJCQlicC0+Yl9lcnJvciA9IEVUSU1FRE9VVDsKKwkJCWJwLT5i X2ZsYWdzIHw9IEJJT19FUlJPUjsKKwkJCWJyZWFrOworCQl9CisJfQogCW10eF91bmxvY2sobXR4 cCk7CiB9CiAKZGlmZiAtLWdpdCBhL3N5cy9zeXMvYnVmLmggYi9zeXMvc3lzL2J1Zi5oCmluZGV4 IDcwZmIyODEyYzNiYS4uZWY0Yzk2NTYwYTU3IDEwMDY0NAotLS0gYS9zeXMvc3lzL2J1Zi5oCisr KyBiL3N5cy9zeXMvYnVmLmgKQEAgLTYwMyw3ICs2MDMsNyBAQCB2b2lkCXBicmVsYm8oc3RydWN0 IGJ1ZiAqKTsKIHZvaWQJcGJyZWx2cChzdHJ1Y3QgYnVmICopOwogaW50CWFsbG9jYnVmKHN0cnVj dCBidWYgKmJwLCBpbnQgc2l6ZSk7CiB2b2lkCXJlYXNzaWduYnVmKHN0cnVjdCBidWYgKik7Ci12 b2lkCWJ3YWl0KHN0cnVjdCBidWYgKiwgdV9jaGFyLCBjb25zdCBjaGFyICopOwordm9pZAlid2Fp dChzdHJ1Y3QgYnVmICosIHVfY2hhciwgY29uc3QgY2hhciAqLCBpbnQpOwogdm9pZAliZG9uZShz dHJ1Y3QgYnVmICopOwogCiB0eXBlZGVmIGRhZGRyX3QgKHZiZ19nZXRfbGJsa25vX3QpKHN0cnVj dCB2bm9kZSAqLCB2bV9vb2Zmc2V0X3QpOwpkaWZmIC0tZ2l0IGEvc3lzL3Vmcy9mZnMvZmZzX3Jh d3JlYWQuYyBiL3N5cy91ZnMvZmZzL2Zmc19yYXdyZWFkLmMKaW5kZXggM2E0MTVkNzY2MzAzLi5k YjNiNmM2ZjM2YmYgMTAwNjQ0Ci0tLSBhL3N5cy91ZnMvZmZzL2Zmc19yYXdyZWFkLmMKKysrIGIv c3lzL3Vmcy9mZnMvZmZzX3Jhd3JlYWQuYwpAQCAtMzE0LDcgKzMxNCw3IEBAIGZmc19yYXdyZWFk X21haW4oc3RydWN0IHZub2RlICp2cCwKIAkJCX0KIAkJfQogCQkKLQkJYndhaXQoYnAsIFBSSUJJ TywgInJhd3JkIik7CisJCWJ3YWl0KGJwLCBQUklCSU8sICJyYXdyZCIsIDApOwogCQl2dW5tYXBi dWYoYnApOwogCQkKIAkJaW9sZW4gPSBicC0+Yl9iY291bnQgLSBicC0+Yl9yZXNpZDsKQEAgLTM4 MSw3ICszODEsNyBAQCBmZnNfcmF3cmVhZF9tYWluKHN0cnVjdCB2bm9kZSAqdnAsCiAJCXVtYV96 ZnJlZShmZnNyYXdfcGJ1Zl96b25lLCBicCk7CiAJfQogCWlmIChuYnAgIT0gTlVMTCkgewkJCS8q IFJ1biBkb3duIHJlYWRhaGVhZCBidWZmZXIgKi8KLQkJYndhaXQobmJwLCBQUklCSU8sICJyYXdy ZCIpOworCQlid2FpdChuYnAsIFBSSUJJTywgInJhd3JkIiwgMCk7CiAJCXZ1bm1hcGJ1ZihuYnAp OwogCQlwYnJlbHZwKG5icCk7CiAJCXVtYV96ZnJlZShmZnNyYXdfcGJ1Zl96b25lLCBuYnApOwpk aWZmIC0tZ2l0IGEvc3lzL3ZtL3N3YXBfcGFnZXIuYyBiL3N5cy92bS9zd2FwX3BhZ2VyLmMKaW5k ZXggZWUyMzZjN2YzOTg4Li5lOTEyYWEwODVmYzcgMTAwNjQ0Ci0tLSBhL3N5cy92bS9zd2FwX3Bh Z2VyLmMKKysrIGIvc3lzL3ZtL3N3YXBfcGFnZXIuYwpAQCAtMTU5NSw3ICsxNTk1LDcgQEAgc3dh cF9wYWdlcl9wdXRwYWdlcyh2bV9vYmplY3RfdCBvYmplY3QsIHZtX3BhZ2VfdCAqbWEsIGludCBj b3VudCwKIAkJLyoKIAkJICogV2FpdCBmb3IgdGhlIHN5bmMgSS9PIHRvIGNvbXBsZXRlLgogCQkg Ki8KLQkJYndhaXQoYnAsIFBWTSwgInN3d3J0Iik7CisJCWJ3YWl0KGJwLCBQVk0sICJzd3dydCIs IDApOwogCiAJCS8qCiAJCSAqIE5vdyB0aGF0IHdlIGFyZSB0aHJvdWdoIHdpdGggdGhlIGJwLCB3 ZSBjYW4gY2FsbCB0aGUKZGlmZiAtLWdpdCBhL3N5cy92bS92bm9kZV9wYWdlci5jIGIvc3lzL3Zt L3Zub2RlX3BhZ2VyLmMKaW5kZXggZDMyZmVjODQ1MDQzLi5hNmZjZTZiMzQ1ZWQgMTAwNjQ0Ci0t LSBhL3N5cy92bS92bm9kZV9wYWdlci5jCisrKyBiL3N5cy92bS92bm9kZV9wYWdlci5jCkBAIC03 MDcsNyArNzA3LDcgQEAgdm5vZGVfcGFnZXJfaW5wdXRfc21sZnModm1fb2JqZWN0X3Qgb2JqZWN0 LCB2bV9wYWdlX3QgbSkKIAkJCWJwLT5iX2lvb2Zmc2V0ID0gZGJ0b2IoYnAtPmJfYmxrbm8pOwog CQkJYnN0cmF0ZWd5KGJwKTsKIAotCQkJYndhaXQoYnAsIFBWTSwgInZuc3JkIik7CisJCQlid2Fp dChicCwgUFZNLCAidm5zcmQiLCAwKTsKIAogCQkJaWYgKChicC0+Yl9pb2ZsYWdzICYgQklPX0VS Uk9SKSAhPSAwKSB7CiAJCQkJS0FTU0VSVChicC0+Yl9lcnJvciAhPSAwLApAQCAtMTE2OCw3ICsx MTY4LDcgQEAgdm5vZGVfcGFnZXJfZ2VuZXJpY19nZXRwYWdlcyhzdHJ1Y3Qgdm5vZGUgKnZwLCB2 bV9wYWdlX3QgKm0sIGludCBjb3VudCwKIAl9IGVsc2UgewogCQlicC0+Yl9pb2RvbmUgPSBiZG9u ZTsKIAkJYnN0cmF0ZWd5KGJwKTsKLQkJYndhaXQoYnAsIFBWTSwgInZucmVhZCIpOworCQlid2Fp dChicCwgUFZNLCAidm5yZWFkIiwgMCk7CiAJCWVycm9yID0gdm5vZGVfcGFnZXJfZ2VuZXJpY19n ZXRwYWdlc19kb25lKGJwKTsKIAkJZm9yIChpID0gMDsgaSA8IGJwLT5iX25wYWdlczsgaSsrKQog CQkJYnAtPmJfcGFnZXNbaV0gPSBOVUxMOwo= --00000000000098c36f0619a57dd5--