Re: HEADS UP: IUTF8 to be enabled by default

On Fri, Nov 3, 2023, 1:15 AM Gary Jennejohn <garyj@gmx.de> wrote:

> On Thu, 2 Nov 2023 21:43:32 +0200
> Christos Margiolis <christos@freebsd.org> wrote:
>
> > Hello again and sorry for the poorly worded previous email,
> >
> > To give a bit more context, during EuroBSDCon 2023, me and Bojan
> > Novkovi? started working on a patch to fix backspacing of UTF-8
> > characters in the tty driver. What was happening is if you typed a >1
> > byte UTF-8 character and then backspaced it, the driver would actually
> > delete only 1 byte from the character, instead of all its bytes, which
> > ended up leaving garbage in the buffer since the character wasn't fully
> > deleted. To test this, run cat(1), type a UTF-8 character (e.g é, è, à,
> > non-latin characters, etc), press backspace only once, and look at the
> > output:
> >
> > $ cat
> > ??<backspace>
> > ??
> >
> > Bojan then implemented a new IUTF8 flag for stty [1], which enables
> > proper handling for UTF-8 backspacing in the tty driver [2].
> >
> > In the Phabricator review of the tty(4) patch [3], I proposed the idea
> > of having the IUTF8 flag enabled by default. imp@ mentioned that since
> > the default locale is UTF-8, having the flag set by default shouldn't be
> > a problem.
> >
> > Two possible solutions I have thought of:
> >
> > 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h.
> > 2. Add a check in tty_init_termios() whether the current locale is
> >    UTF-8 (how?), and enable it there.
> >
>
> Use getenv("LANG") and check whether UTF-8 is part of the string?
>

This string is set too late for the default. Also, drivers don't have
access to process data.

Warner

My LANG is set to C.UTF-8, for example.
>
> > What do you think? Could this change cause any side-effects we haven't
> > thought about?
> >
> > Christos
> >
> > [1]
> https://cgit.freebsd.org/src/commit/?id=128f63cedc14ae21b35f74e11e2fe1a5659c58e8
> > [2]
> https://cgit.freebsd.org/src/commit/?id=9e589b0938579f3f4d89fa5c051f845bf754184d
> > [3] https://reviews.freebsd.org/D42067
> >
>
>
> --
> Gary Jennejohn
>