svn commit: r362333 - in head: contrib/flex contrib/flex/src usr.bin/lex usr.bin/lex/lib

[Mark Johnston]

On Sat, Feb 13, 2021 at 02:39:56AM -0500, Jung-uk Kim wrote:
> On 21. 2. 13., Jung-uk Kim wrote:
> > On 21. 2. 12., Mark Johnston wrote:
> >> On Thu, Jun 18, 2020 at 06:09:16PM +0000, Jung-uk Kim wrote:
> >>> Author: jkim
> >>> Date: Thu Jun 18 18:09:16 2020
> >>> New Revision: 362333
> >>> URL: https://svnweb.freebsd.org/changeset/base/362333
> >>>
> >>> Log:
> >>>   MFV:	r362286
> >>>   
> >>>   Merge flex 2.6.4.
> >>
> >> This seems to have introduced a regression: input() now returns 0
> >> instead of EOF to indicate that the end of input was reached.  This has
> >> been reported in a few places:
> >>
> >> https://github.com/westes/flex/issues/448
> >> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911415
> >>
> >> It breaks the scanner used by libdtrace, and as a result dtrace is
> >> unable to resolve some probe argument types:
> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253440
> >>
> >> I have a small patch to libdtrace which works around this.  The upstream
> >> commit that introduced the change is fairly old at this point, and
> >> upstream seems somewhat unresponsive, so I'm not sure it'll get
> >> reverted.  Some other scanners in the tree look like they'd be affected,
> >> like crypto/heimdal/lib/asn1/lex.l or
> >> sys/contrib/dev/acpica/compiler/aslsupport.l, so I'm wondering if we
> >> should revert this particular change for 13.0.  Certainly we should fix
> >> the DTrace bug one way or another for 13.0.
> > 
> > I think it does not make much sense and we should revert the upstream
> > change.  Does the attached patch work for you?

The original patch fixes it, the updated one does not since libdtrace
sets -l in LFLAGS. :)
Removing it causes a build failure but it is trivial to resolve.  I am
running some tests now to see if anything regressed.

> Now I believe the author wanted Lex compatibility.  Maybe we should do
> the attached patch instead?

Is it common to use the base system's flex for ports and out-of-tree
code generally?  I have no real idea.  I'd be worried about configure
scripts that use the flex version to determine whether or not to apply a
workaround for this change.  I also note that Debian does not seem to
have patched this.

Looking through consumers in the tree (all *.l files), the following
scanners could be affected:
- libdtrace
- contrib/com_err/lex.l
- contrib/ipfilter/iplang/iplang_l.l
  - does not appear to be built
- crypto/heimdal/lib/asn1/lex.l
- crypto/heimdal/lib/com_err/lex.l
- crypto/heimdal/lib/hx509/sel-lex.l
- crypto/heimdal/lib/sl/slc-lex.l
- sys/contrib/dev/acpica/compiler/aslsupport.l
- sys/contrib/dev/acpica/compiler/prparser.l

It would not be too difficult to patch them, and I think in most cases
the problem is not very severe and is only triggered by invalid input.
For instance, asn1_compile enters an infinite loop if the input file
contains an unterminated comment.

So for 13.0 I am kind of inclined to just patch consumers.  If you think
we should revert the change in flex, I'm ok with that too and will
remove the use of -l in libdtrace.

> diff --git a/contrib/flex/src/flex.skl b/contrib/flex/src/flex.skl
> index 242645f53245..c23b944ea473 100644
> --- a/contrib/flex/src/flex.skl
> +++ b/contrib/flex/src/flex.skl
> @@ -1863,7 +1863,11 @@ m4_ifdef( [[M4_YY_USE_LINENO]],
>  				case EOB_ACT_END_OF_FILE:
>  					{
>  					if ( yywrap( M4_YY_CALL_ONLY_ARG ) )
> +#ifdef YY_FLEX_LEX_COMPAT
>  						return 0;
> +#else
> +						return EOF;
> +#endif
>  
>  					if ( ! YY_G(yy_did_buffer_switch_on_eof) )
>  						YY_NEW_FILE;