From nobody Mon Jan 17 01:20:27 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 53A14196B5B2 for ; Mon, 17 Jan 2022 01:20:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4JcYxj49nvz3Pkc for ; Mon, 17 Jan 2022 01:20:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1642382434; bh=wLJmiDRFSpmupXhYK+qfk7jaFGCneD76TALnctiQo5c=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=JrffxhNMIbiUM3fyPzVsC20pK95f/dv4q/v5t3XL/ztWgtSxt9yHYIm6mbuzvOX5pyzQK9aSbiFOBKneg8QmG6KPSdg1ukI9B1m79nAi2In5GCOLxwqYRq8EivvisKUCzK+IR71AAGGM8TXeUr5VwDbGHiId/WSlLkXs3fW497xcpBQEjRIjKWJpfAD2S9LM27MbB7Yr1MWnlZOD0nn9kH0x/4C+7ufJ+gVxgFEB3fBXwIFhuSG067CB408LFMzBoXoDcF3PVaPMHdenzAlyas8fZaDJjTkvpoPs/vXgFCWlEGcnOVTxG63ArUkUrDDXGd5lnSSurkHrpTU++feIqw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1642382434; bh=W6Qm/R1T7HuyyLGfLyx8MKyiOg56eclczNa1Io5ZXDE=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=ph3vfoeZWoR0PjCgrHNFGPcl/0gzsI+GgLZMtbUafd4ZPf27yrOVDiq0aXQ08YK50N7PxeLm1cIUqln7rhsxUhaU1EquztAgaCqv0MZk4LT/O6snm9MdTKfWdMWHqTyCPVA+8DZH95i2ZlFrHHe5x05OZ2URLTupQCcfpnOWA6Th4P3zDHio9JcakNiSa3TCbCmA3mC8tpsZ4St/1zkHXQruT6na0JeXF1Cf62P07eUnBbpzfDsab63Zss/je6yPXeRp2FPTJxgTJm1b9tOS6ZEr7TdFt8FBnIUjDSj4duls8vNaocKzp82CxfDmJLuom61WQP1ZujIzTwDks1OMnA== X-YMail-OSG: PNVYpE4VM1nr43AN7GFDPlu9Fh9ipOsFY4WjgNvJM_NNGsgrf1avcw2T1lMQeYA 4kI0I6Nj9R6UXeg8ibdE.DFcnMhsSxbWZitcE6LIEyP__8qB_jRNbuLEpdKgdsL_xZc3ODz7SgQE 4shIIgjPqssfapkMlUBEtrP7Dz_puKrDMKeWU4lKR4RFqL5J3Y9QS51Xm.IjmaRdH465pgn91bTq J2VwQ4E9spFghRu4ezmW1MfiKbPsXSi09bHDkAcdQuu_UUXhkq3gGdi87LAh21rIEKqSq3RUWk6C FOTk7YlwHoaTM59TCYJSMF4kRAJIiEFKTeeflFfFCZ1ujEN6IzVC7NORDgktdC4IqSAnRr5Kuw8y Wxkk1lAHy4b35zdX4ddmzy6tm3CbeQCczqXmPTLbm9Ukcvef9I5L5dcXHN38dhqLZGzPzFIHWPdX kFBSHxkulM5On5kzZdIxoiOJLIDWy8Co85FJ7WQMlZdPyOqQVTurmu3SkRP6P8DxVlCbEo8ydEbE 2JtMYcppVfMreNh7W7FwhiWdad9uLF8UjjT3SJxh_pfO6TF8d8kJlQhBihhQ3cNQ959wBiE7I44X bTq05v59NMAdlEkOI0bXFYGKewO7Jw4HshWoXbiPAnMEoTYr7NPbZEomtzD78w.sjukd4kuYeXRP WXdoWNS2gvzuEFmxaNTdau97Ljmh8jtmdqTYiGVQaLcbDL7EnMfxgSq7aMbt7zlkbbiMn0JunCKE 7g9wJ8F_0fbRinkHnolE9K5PG9N1Do0dDHkfyWhyWU6AGECHmxtSQ9MKQu_o87k_KIy0V1RtJ.XE a20gSD.H7ER05wXr4g1gmXoC7wYkLO1TNiVsHweLLDz.U0FomnBqa8W1X.bIp9U13AlC3u97mqxM UUV44e.LvxmC2pCV9GTuuGokrjG3hoICGeWIx5oMgTAI9Fc8en1.AqNYSnj2IWpFvAivEhpw4..B 1LBpl1erJ9dtEM5i_WFJQJ6swTasnaFl828t3zvnivnQ_vj.7kVuoyl26oU1Tlv6hXWLPFUYJXzI x29Af.jh7D0l0EtbT3wjbBAFetsZrc_wHfAEs5VZQUgZP2F__E0NNAFgSZ26DjlITIOghPLElaKk LHyBuG9PFZu_8T5v8flW7HBek.uwVjbDrLyVbX0k6ziNztGxIt6gMvJRK_uIscwyKGKmscAnGW2. 4cvwAIsS0HP1QIBTKyz32.7JJa0KJo9xprfsYW5ravBR_oB9J0cunoLQJWyPD36n5W73fXwMEUMk WT.7QLqL061wXAemOwoFNKB8Xn0Wt9MVkLygFUhQrXpL4z8R_GXPgfkijUiV3LySrM3xKatZpx4M V3X_0_gxCe5Uc_e.vmtFy7h85wWCctO5Y.o9AKGrj3RDSPJQ1lK63aez7O4ciMtVDm3a4YdYHgf3 Nc3e4m5a_5nYOKjIN9m1cWmxhvKxzUiPV2waUL..Wtt4tCPc5dA3WDyV6ewZg_GDtT9QnVw22zVi RaZ7o95FFlnF07OolOTS6r6DoUKmtGcOg2wWGdY3duO4OjezF4mDWWx.Ng9z1morX5M_8UMtYdOX TdQOf_5NRALR52eCpO8Toon4nlGJJTg_jkZ6LlrBQnse7pgAbk1Ax685sp4lrRng8IfjgyLQyNcF IRRjpYH_er6qr1yk0DwT2ivoHQe2A1QcknX3eUVjZWBq3PUpr6VH1ZOaLpDdbamu0xpMky8mdztt 8zdKJMSukm68RJlJb8vnygZsYXrrZ0nTxU_y0btjOZoU9VmMIsCgF4EiD1X7x1S2FH3qwfskZIm7 f9CnjaEw5LjoEHprLtqehQTKCCpOKSenIdMrDeTgF4tPsswz5zSOmWIY1RZmWvMCUW_zo454IFhk 1gE23esPQqnVOcGObkjGHJyWvysYEuVByDNBIMWy2Tek0knRHykO_y9O2xS7SmV95TRVgh0Jpd4Z C2ObNT48irfwV9G0TTFsvL0KqkxsSfen_H9nT4Tm44vjSAnAnaIw5cTwauBLfn7389PCT4CO3FJ2 wIn8zGein5xLhIna3_g3_gW4KlmjwLhfrMndeV7XCUBA7Fjmz4lyhNayfQJ_M9Ias_Nff808qOAJ QyJAPxB2DG4wnI3RvPcpC5ByXCttyoWbEuK_Nsx4qYYLp2UPu4CS5YWwlrAd04IQzbcVqmf3qYsO MAXH0QAelFvyQDAj3wUEskfQKoYOewwyel5zx6KiH71hPEC7t7n11Vrik51OlrwgaibWEPCV7hci DFpy2glx0uVhnN3r7gdSZx5ClaAnDozUfEciox8PAiPv6fqDUQ7K0Nj.5tkC4fPSfjeGuE2_hR9O zYVNH1dAu8UJU1Cc- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Mon, 17 Jan 2022 01:20:34 +0000 Received: by kubenode517.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID d3f01c2ba491c35560569b7f7fe3c767; Mon, 17 Jan 2022 01:20:28 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: WITH_ASAN= vs. vfork: ASAN can not deal with vfork in a reliable manor, so what to do to avoid vfork fairly generally, including for kyua? From: Mark Millard In-Reply-To: Date: Sun, 16 Jan 2022 17:20:27 -0800 Cc: Dimitry Andric Content-Transfer-Encoding: quoted-printable Message-Id: References: To: freebsd-current X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4JcYxj49nvz3Pkc X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=JrffxhNM; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-2.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; MLMMJ_DEST(0.00)[freebsd-current]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jan-15, at 15:25, Mark Millard wrote: > ASAN is documented to not be able to deal reliably with vfork use > (inaccurate results, result mixing interpretations of the old > and new contexts, and so on). [There may be other routines > sufficiently analogous to vfork to have the same sorts of issues. > I'll write only of vfork explicitly.] >=20 > It turns out that using SH_DISABLE_VFORK=3D with kyua runs > helps avoid a bunch of junk failure ASAN reports that involve > /bin/sh using vfork. So I've been recently using the likes of: >=20 > env SH_DISABLE_VFORK=3D ASAN_OPTIONS=3Ddetect_container_overflow=3D0 \ > kyua test -k /usr/tests/Kyuafile >=20 > in my kyua runs. But that need not cover all the vfork use in > the kyua runs --or in general system operation. >=20 >=20 >=20 > There is more that could be done. For example . . . >=20 > contrib/llvm-project/compiler-rt/lib/asan/asan_interceptors.h > currently has: >=20 > #if SANITIZER_LINUX && = \ > (defined(__arm__) || defined(__aarch64__) || defined(__i386__) || \ > defined(__x86_64__) || SANITIZER_RISCV64) > # define ASAN_INTERCEPT_VFORK 1 > #else > # define ASAN_INTERCEPT_VFORK 0 > #endif >=20 > The "# define ASAN_INTERCEPT_VFORK 1" causes interception to > use fork for the vfork implementation, instead of an actual > vfork. >=20 > It looks to me like asan_interceptors.h for FreeBSD should > also be using: >=20 > # define ASAN_INTERCEPT_VFORK 1 >=20 > but it currently is not. Well, looking around there does not seem to be a FreeBSD-supporting implementation to go with use of "# define ASAN_INTERCEPT_VFORK 1". So there would be more to it than just causing the #define to happen. Thus, there does not seem to be a quick way for me to see what would result in the kyua run from more avoiding of vfork use (beyond what use of SH_DISABLE_VFORK=3D did). > There is even the possibility that a WITH_ASAN=3D buildworld could > build a world that uses fork behavior for vfork and never does an > actual vfork (by behavior). Basically restricting itself to things > that fit with ASAN use. >=20 >=20 >=20 > I tracked this issue down via a report in a kyua run that > referenced: >=20 > QUOTE > ERROR: AddressSanitizer: stack-buffer-underflow on address = 0x7fffffffa580 at pc 0x000801e0f8ed bp 0x7fffffff9bf0 sp 0x7fffffff9be8 > WRITE of size 8 at 0x7fffffffa580 thread T0 > . . . > Address 0x7fffffffa580 is located in stack of thread T0 at offset 0 in = frame > #0 0x7fffffffa5ef () >=20 > This frame has 1 object(s): > [32, 288) 'buf' (line 180) > HINT: this may be a false positive if your program uses some custom = stack unwind mechanism, swapcontext or vfork > (longjmp and C++ exceptions *are* supported) > END QUOTE >=20 > The "'buf' (line 180)" turned out to be from the declaration in > bin/sh/exec.c 's tryexec : >=20 > static void > tryexec(char *cmd, char **argv, char **envp) > { > int e, in; > ssize_t n; > char buf[256]; >=20 > execve(cmd, argv, envp); > e =3D errno; > if (e =3D=3D ENOEXEC) { > INTOFF; > in =3D open(cmd, O_RDONLY | O_NONBLOCK); > if (in !=3D -1) { > n =3D pread(in, buf, sizeof buf, 0); > close(in); > if (n > 0 && isbinary(buf, n)) { > errno =3D ENOEXEC; > return; > } > } > *argv =3D cmd; > *--argv =3D __DECONST(char *, _PATH_BSHELL); > execve(_PATH_BSHELL, argv, envp); > } > errno =3D e; > } >=20 > So I could finally see/validate that an example > stack-buffer-* report was tied to vfork handling > (analyzing other code related to the tryexec use). >=20 > The report happens well after tryexec did its > execve. The information about buf is just old, > no longer accurate information, leading to a > false report: >=20 >=20 > =3D=3D87961=3D=3DERROR: AddressSanitizer: stack-buffer-underflow on = address 0x7fffffffa580 at pc 0x000801e0f8ed bp 0x7fffffff9bf0 sp = 0x7fffffff9be8 > WRITE of size 8 at 0x7fffffffa580 thread T0 > #0 0x801e0f8ec in bintime2timespec = /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/tmp/us= r/include/sys/time.h:285:14 > #1 0x801e0f8ec in __vdso_clock_gettime = /usr/main-src/lib/libc/sys/__vdso_gettimeofday.c:195:2 > #2 0x801e0e1a0 in clock_gettime = /usr/main-src/lib/libc/sys/clock_gettime.c:48:11 > #3 0x10d54da in clock_gettime = /usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/saniti= zer_common_interceptors.inc:2189:13 > #4 0x11234f5 in __sanitizer::MonotonicNanoTime() = /usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/saniti= zer_linux_libcdep.cpp:860:3 > #5 0x10ba02c in = __sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSp= aceView> >::PopulateFreeArray(__sanitizer::AllocatorStats*, unsigned = long, __sanitizer::SizeClassAllocator > 64<__asan::AP64<__sanitizer::LocalAddressSpaceView> >::RegionInfo*, = unsigned long) = /usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/saniti= zer_allocator_primary64.h:790:45 > #6 0x10b9c4b in = __sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSp= aceView> >::GetFromAllocator(__sanitizer::AllocatorStats*, unsigned = long, unsigned int*, unsigned long) /u > = sr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitize= r_allocator_primary64.h:220:11 > #7 0x10b9955 in = __sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocato= r64<__asan::AP64<__sanitizer::LocalAddressSpaceView> > = >::Refill(__sanitizer::SizeClassAllocator64LocalCac > = he<__sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddres= sSpaceView> > >::PerClass*, = __sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSp= aceView> >*, unsigned lo > ng) = /usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/saniti= zer_allocator_local_cache.h:103:9 > #8 0x10b9615 in = __sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocato= r64<__asan::AP64<__sanitizer::LocalAddressSpaceView> > = >::Allocate(__sanitizer::SizeClassAllocator64<__asa > n::AP64<__sanitizer::LocalAddressSpaceView> >*, unsigned long) = /usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/saniti= zer_allocator_local_cache.h:39:11 > #9 0x10b9511 in = __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__asan::A= P64<__sanitizer::LocalAddressSpaceView> >, = __sanitizer::LargeMmapAllocatorPtrArrayDynamic>::Allocate(__san > = itizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocator64<_= _asan::AP64<__sanitizer::LocalAddressSpaceView> > >*, unsigned long, = unsigned long) /usr/main-src/contrib/llvm-project/compile > r-rt/lib/sanitizer_common/sanitizer_allocator_combined.h:69:20 > #10 0x10b6086 in __asan::Allocator::Allocate(unsigned long, = unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, = bool) /usr/main-src/contrib/llvm-project/compiler-rt/lib/asan/asan_a > llocator.cpp:537:29 > #11 0x10b4818 in __asan::asan_malloc(unsigned long, = __sanitizer::BufferedStackTrace*) = /usr/main-src/contrib/llvm-project/compiler-rt/lib/asan/asan_allocator.cpp= :980:34 > #12 0x110be9b in malloc = /usr/main-src/contrib/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.= cpp:130:10 > #13 0x117aca3 in ckmalloc /usr/main-src/bin/sh/memalloc.c:71:6 > #14 0x115b1b1 in reprocess /usr/main-src/bin/sh/expand.c:912:9 > #15 0x1155163 in evalvar /usr/main-src/bin/sh/expand.c:784:3 > #16 0x1155163 in argstr /usr/main-src/bin/sh/expand.c:319:8 > #17 0x1151178 in expandarg /usr/main-src/bin/sh/expand.c:241:2 > #18 0x11427c8 in evalcommand /usr/main-src/bin/sh/eval.c:857:4 > #19 0x114705b in evalbackcmd /usr/main-src/bin/sh/eval.c:681:4 > #20 0x1151bfc in expbackq /usr/main-src/bin/sh/expand.c:476:2 > #21 0x1151bfc in argstr /usr/main-src/bin/sh/expand.c:323:4 > #22 0x1151178 in expandarg /usr/main-src/bin/sh/expand.c:241:2 > #23 0x11427c8 in evalcommand /usr/main-src/bin/sh/eval.c:857:4 > #24 0x113eeb7 in evaltree /usr/main-src/bin/sh/eval.c:289:4 > #25 0x113eb88 in evalstring /usr/main-src/bin/sh/eval.c > #26 0x113e9f9 in evalcmd /usr/main-src/bin/sh/eval.c:139:17 > #27 0x1145289 in evalcommand /usr/main-src/bin/sh/eval.c:1107:16 > #28 0x113eeb7 in evaltree /usr/main-src/bin/sh/eval.c:289:4 > #29 0x113f86b in evaltree /usr/main-src/bin/sh/eval.c:212:4 > #30 0x1144d89 in evalcommand /usr/main-src/bin/sh/eval.c:1053:3 > #31 0x113eeb7 in evaltree /usr/main-src/bin/sh/eval.c:289:4 > #32 0x117a316 in cmdloop /usr/main-src/bin/sh/main.c:228:4 > #33 0x1179788 in main /usr/main-src/bin/sh/main.c:175:3 >=20 > Address 0x7fffffffa580 is located in stack of thread T0 at offset 0 in = frame > #0 0x7fffffffa5ef () >=20 > This frame has 1 object(s): > [32, 288) 'buf' (line 180) > HINT: this may be a false positive if your program uses some custom = stack unwind mechanism, swapcontext or vfork > (longjmp and C++ exceptions *are* supported) > SUMMARY: AddressSanitizer: stack-buffer-underflow = /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/tmp/us= r/include/sys/time.h:285:14 in bintime2timespec > Shadow bytes around the buggy address: > 0x4ffffffff460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > =3D>0x4ffffffff4b0:[f1]f1 f1 f1 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff4d0: 00 00 00 00 f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00 > 0x4ffffffff4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x4ffffffff4f0: f1 f1 f1 f1 00 f2 f2 f2 00 f3 f3 f3 00 00 00 00 > 0x4ffffffff500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07=20 > Heap left redzone: fa > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Container overflow: fc > Array cookie: ac > Intra object redzone: bb > ASan internal: fe > Left alloca redzone: ca > Right alloca redzone: cb > =3D=3D87961=3D=3DABORTING =3D=3D=3D Mark Millard marklmi at yahoo.com