From nobody Sat Oct 21 14:29:37 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SCP3j5SWmz4y5F7; Sat, 21 Oct 2023 14:29:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SCP3j3M1Wz4G2h; Sat, 21 Oct 2023 14:29:37 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697898577; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WUDsoqn3Xs1dDcAIz9juvgkkwI2F2aG0d9Oux1P20fU=; b=l6mTcmY8zc9mZyJJVfhk9Jgju9EEvnVdjVesI6JHea1CodhMApaiogq83YBXm2ksvG4Lu0 gjShOPqrHtEYylxkADTw16W2U5Hstv0aKF1fpxzGvRP239I87IZ5u3NNYjf121sBo5t9im YhaYNPcPfdUN6Q9fKDpiNCZGxdXoGc6LPcOg2FxwsEWA3LiH/RXxXFhD+eh1FW7QZIknV+ pXnmwh8wDeVzsu6awLrRbD8vNkddX1lvegl5ykNHt0MGkFMdXymdGDredWxi/b+E6YumCj jQd40x3BCtX3bm9ZOZPiHKsdDVPQoH6xKMsDo7kkJ6vqa0xrJU3bopFRRRx7jw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1697898577; a=rsa-sha256; cv=none; b=gdFncFkEB59ztYQGpDSsqdWlypnzH6n8fwedQHigW4jXyppUT4l2RCuVW2A1JEx3CWz9uR u9d7mPPHrZJ7Uw7lJsItpYIPZrn6/cJqIYivASEmAe+FcNQopL79BvXPiQnOZ687KNmMnI Ki7QLrZQ3BRQi2hL/H6vGK2b3RywFGBLvr+mU7EbXhXPXJsWstq6aL7tesiGL/tI/1NGEG dWHf4LAcCHDl0RcBrRNcTGMhvbBo4o0C0q2nTyiCCO9wUGljmQJSy0u+bMp9XPgk5cOhDS dTy3+hYANqo309+/ILh9Q+oIj7vmb0RFMadTWb66AWCuld5FwY3hceyXxpG2OA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697898577; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=WUDsoqn3Xs1dDcAIz9juvgkkwI2F2aG0d9Oux1P20fU=; b=k97K2msf8TTVtGOLgaDpKWEUf2qRweRazNCYgeCzS+6LJP1GKBHQMIGWLe2dB6igAtSLWf PS22LGAVBW7grr2hIahVrCVvmI4hEYLwmB4TFHLBgSRHiYP7VEK+9vLpwt3vjq5aI/c5MO gMvHKEwXf0d7IITeV/HeJIwUmoP1uRBU8Z89Zfu3wwh8yQLgBaDiqaaN8tojd+pGtMlqAw RercgvtqEYLpN4raOgDVxY3E6nNNQ70BAWg0Vp6Czzq+zwHTrkz6xz2AjdISlM0izJGBYA 9bfghi2KkfquHbDYsY9DiKpXNE1+94e2RRcNFl5TZDTWm9mhDdLL3izIQHqRMw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SCP3j2BLhz4Dg; Sat, 21 Oct 2023 14:29:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 39LETbud071727; Sat, 21 Oct 2023 14:29:37 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 39LETbrH071724; Sat, 21 Oct 2023 14:29:37 GMT (envelope-from git) Date: Sat, 21 Oct 2023 14:29:37 GMT Message-Id: <202310211429.39LETbrH071724@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Christos Margiolis Subject: git: 72a8e373f2d1 - stable/14 - tty/teken: fix UTF8 sequence validation logic List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: christos X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 72a8e373f2d1886f27d1f724aa9e8e55e08809eb Auto-Submitted: auto-generated The branch stable/14 has been updated by christos: URL: https://cgit.FreeBSD.org/src/commit/?id=72a8e373f2d1886f27d1f724aa9e8e55e08809eb commit 72a8e373f2d1886f27d1f724aa9e8e55e08809eb Author: Bojan Novković AuthorDate: 2023-10-13 05:14:36 +0000 Commit: Christos Margiolis CommitDate: 2023-10-21 14:28:35 +0000 tty/teken: fix UTF8 sequence validation logic This patch fixes UTF-8 sequence validation logic in teken_utf8_bytes_to_codepoint() and fixes fallback behaviour in ttydisc_rubchar() when an invalid UTF8 sequence is encountered. The code previously used __bitcount() to extract sequence length information from the leading byte. However, this assumption breaks for certain code points that have additional bits set in the first half of the leading byte (e.g. Cyrillic characters). This lead to incorrect behaviour when deleting those characters using backspaces. The code now checks the number of consecutive set bits in the leading byte starting from the MSB, as per RFC 3629. Reviewed by: christos MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D42147 (cherry picked from commit 2fed1c579c52d63b72fc08ffcc652ba0183f9254) --- sys/kern/tty_ttydisc.c | 12 +++++++----- sys/teken/teken_wcwidth.h | 23 ++++++++++++++++++++--- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/sys/kern/tty_ttydisc.c b/sys/kern/tty_ttydisc.c index eae7162e31c0..c46579cefbb1 100644 --- a/sys/kern/tty_ttydisc.c +++ b/sys/kern/tty_ttydisc.c @@ -844,23 +844,25 @@ ttydisc_rubchar(struct tty *tp) */ ttyinq_write(&tp->t_inq, bytes, UTF8_STACKBUF, 0); + ttyinq_unputchar(&tp->t_inq); } else { /* Find codepoint and width. */ codepoint = teken_utf8_bytes_to_codepoint(bytes, nb); - if (codepoint != - TEKEN_UTF8_INVALID_CODEPOINT) { - cwidth = teken_wcwidth( - codepoint); - } else { + if (codepoint == + TEKEN_UTF8_INVALID_CODEPOINT || + (cwidth = teken_wcwidth( + codepoint)) == -1) { /* * Place all bytes back into the * inq and fall back to * default behaviour. */ + cwidth = 1; ttyinq_write(&tp->t_inq, bytes, nb, 0); + ttyinq_unputchar(&tp->t_inq); } } tp->t_column -= cwidth; diff --git a/sys/teken/teken_wcwidth.h b/sys/teken/teken_wcwidth.h index bc40c23a6eb7..14af2c6f9530 100644 --- a/sys/teken/teken_wcwidth.h +++ b/sys/teken/teken_wcwidth.h @@ -128,15 +128,32 @@ static inline teken_char_t teken_utf8_bytes_to_codepoint(uint8_t bytes[4], int nbytes) { - /* Check for malformed characters. */ - if (__bitcount(bytes[0] & 0xf0) != nbytes) + /* + * Check for malformed characters by comparing 'nbytes' + * to the byte length of the character. + * + * The table in section 3 of RFC 3629 defines 4 different + * values indicating the length of a UTF-8 byte sequence. + * + * 0xxxxxxx -> 1 byte + * 110xxxxx -> 2 bytes + * 1110xxxx -> 3 bytes + * 11110xxx -> 4 bytes + * + * The length is determined by the higher-order bits in + * the leading octet (except in the first case, where an MSB + * of 0 means a byte length of 1). Here we flip the 4 upper + * bits and count the leading zeros using __builtin_clz() + * to determine the number of bytes. + */ + if (__builtin_clz(~(bytes[0] & 0xf0) << 24) != nbytes) return (TEKEN_UTF8_INVALID_CODEPOINT); switch (nbytes) { case 1: return (bytes[0] & 0x7f); case 2: - return (bytes[0] & 0xf) << 6 | (bytes[1] & 0x3f); + return (bytes[0] & 0x1f) << 6 | (bytes[1] & 0x3f); case 3: return (bytes[0] & 0xf) << 12 | (bytes[1] & 0x3f) << 6 | (bytes[2] & 0x3f); case 4: