From nobody Fri Nov 03 07:15:29 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMBpy26Wmz50F0y for ; Fri, 3 Nov 2023 07:15:38 +0000 (UTC) (envelope-from garyj@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMBpx4cH6z3HYf; Fri, 3 Nov 2023 07:15:37 +0000 (UTC) (envelope-from garyj@gmx.de) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1698995735; x=1699600535; i=garyj@gmx.de; bh=FWT1WiH6Q+DNgITFvetTuif6aBwqYT1/DDjo0euntrM=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References: Reply-To; b=sf5nLb1Qyd7P4Uh80AgrvdAVJ9HvT/fBXVtHGq+7pBeMcz88qfhltt4yHNgH5aQP aWpdZqt6I/lKbMsVIfK+HR19MhY6QNd6+HfYpM7LW2Xg2konKEKWCi6DoDNl87rdC eVdDDXxT288Lp8LdIGzX+KPhuiwQ6MNDbvL10UbiyEr6z24eY784lC1DBx0Fouts4 pRva2eSHZzHph/JsG4xqtX9FZKJ4DedoUPT2bCDiOpD2pr6AHZKbcKDr8RJXUvq4F IyAkLAiTzY1LCd9NPh9CPsXlTpkpvTOZlJPn5NWKcHManMyXwVVNfxtgss5UFfOKm cKUbyUGKqHXA/4ZRNg== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from ernst.home ([217.226.57.134]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mirna-1rb9QB0QHE-00eu2i; Fri, 03 Nov 2023 08:15:35 +0100 Date: Fri, 3 Nov 2023 07:15:29 +0000 From: Gary Jennejohn To: Christos Margiolis Cc: freebsd-arch@freebsd.org, bojan.novkovic@fer.hr, imp@freebsd.org Subject: Re: HEADS UP: IUTF8 to be enabled by default Message-ID: <20231103081529.016be29d@ernst.home> In-Reply-To: References: Reply-To: garyj@gmx.de X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:GlfeU2caXTLBJpUleA+fWNL/sgjZlJXQWX3rKlj0viaVCPrk+9e hquDMWP1d6qSRXdXZhh06V88pIuLBYrCG3MYabf67BGv62BnoHL1uS7FTsm8UXQ02trhzGR +a7i6bytdwYCBP3QuWMnUlvmM0hsoE4iEuDXAzmidwnqYsfe6zWLgP+ftZQbtnnFzjfQ0mR 6x2tab4G9SroT59Ws2ofw== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:wo+O13zraLg=;pVE1JprvRxqGB6hXmmolOTEReEG LgSVIZX2YgkKrAI6exErv/ro6h5V+PYK/GA6/oNmiNqJr0NgZD+t550H4UF2hsDOOnZIKbyCb 7vfaSeBW5Hmi6ddLaholhZ1hLC8//SnFezwQKzoptkQVnM+vPneOx+X0Qa5pTd0gVROHGWqpW VnbP94AxBMAZLsevYN/NUsysGbgMR5EprgQZAxecQaHA/WXQ0uNxT9voqHWbvT/ttUoClWtNu TA01/LvPKk/R9TnZqRALQ9T6pQ1BstTm6jgop1O7XcKPVXpfIQV/MYtOwJLs2emfEeFY1JWBb FQR2xr/Qb1twKvrgjehnG+2seWX4iyd+b5R5alscciYloDtjlUSohCUQrT/dRhxJn3OIwFrhE I7yX2XtWxgWtAGGmizLYWHQJJFpgvJDQ1Lqn0gCwW6VDew+y6gFVmyi5ILcI1B24r2jdjBCjV 7P7+iaaZPeU1QA+8hyDwzxEb5rGfV92qQAG/6mcXxPeg2QJ/4uKJA4Jn9+F/IytyLiNVRagsT /aFpTjIiv6fmEwa/zgMS3atNzvUUAGsjpUgIUHJ5IiZEIKsPv08vk+1MxH80tQueYEssPmX1m a/+o4ZzUrhfyZ2xcqpqHK5VJxW7rWYahDS0rdjA1g7rb9Hf8g5HsV9e7q1R2YIZmP29sdmBex 2GrAe+9CNZSz2Av1BY9myeHUWup9K7d7oLK9Jj07oOahEHCAmodHDHQkjZfF/BPKV3aRcvPTE 0IGMIv6L5/8obwBNN6VbqRMa87RR02nMmZaaUqk4cvBVDJkbR+WbWukI/uju094ee6T6iV5T0 uPovRO46Bz3OOmdQ7lmcVjMMxvXv7J8JFB1k1JIUoHTTqcndcqWf482vsyYKFncx7OYwD7si4 Ei9yy2P0v56sM+EbyQkrRm1IqgDHt3ybYBBnteKcrlCfkPbtEIkzJdLl8X97V2kd+v2T1qBF7 SIPDQkaQIZjIRGmssVd2kSZLJ54= X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE] X-Rspamd-Queue-Id: 4SMBpx4cH6z3HYf On Thu, 2 Nov 2023 21:43:32 +0200 Christos Margiolis wrote: > Hello again and sorry for the poorly worded previous email, > > To give a bit more context, during EuroBSDCon 2023, me and Bojan > Novkovi? started working on a patch to fix backspacing of UTF-8 > characters in the tty driver. What was happening is if you typed a >1 > byte UTF-8 character and then backspaced it, the driver would actually > delete only 1 byte from the character, instead of all its bytes, which > ended up leaving garbage in the buffer since the character wasn't fully > deleted. To test this, run cat(1), type a UTF-8 character (e.g =E9, =E8,= =E0, > non-latin characters, etc), press backspace only once, and look at the > output: > > $ cat > ?? > ?? > > Bojan then implemented a new IUTF8 flag for stty [1], which enables > proper handling for UTF-8 backspacing in the tty driver [2]. > > In the Phabricator review of the tty(4) patch [3], I proposed the idea > of having the IUTF8 flag enabled by default. imp@ mentioned that since > the default locale is UTF-8, having the flag set by default shouldn't be > a problem. > > Two possible solutions I have thought of: > > 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h. > 2. Add a check in tty_init_termios() whether the current locale is > UTF-8 (how?), and enable it there. > Use getenv("LANG") and check whether UTF-8 is part of the string? My LANG is set to C.UTF-8, for example. > What do you think? Could this change cause any side-effects we haven't > thought about? > > Christos > > [1] https://cgit.freebsd.org/src/commit/?id=3D128f63cedc14ae21b35f74e11e= 2fe1a5659c58e8 > [2] https://cgit.freebsd.org/src/commit/?id=3D9e589b0938579f3f4d89fa5c05= 1f845bf754184d > [3] https://reviews.freebsd.org/D42067 > =2D- Gary Jennejohn