From nobody Sat Mar 26 15:56:21 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 754231A46A80 for ; Sat, 26 Mar 2022 15:56:42 +0000 (UTC) (envelope-from royger@gmail.com) Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KQk953l8Zz3vR2 for ; Sat, 26 Mar 2022 15:56:41 +0000 (UTC) (envelope-from royger@gmail.com) Received: by mail-wr1-x42d.google.com with SMTP id h23so14498306wrb.8 for ; Sat, 26 Mar 2022 08:56:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fXQNLhXYbwRZ9WO7T+BhVV9+0Up77RvoHt23kV18qJk=; b=pOBjalGLCgWTAe0KlJjHb9CxHhGNPoF67b7cFFHqNzpMu7n8EEspBHSt86y8uoivYN e0GMin6guK/9hPp2Qi/fS2klpd32Qq+pgtXukBR2AyfdWDy2vJaMUR+3T91Lnz9OEFVZ UBJrFzVU8A+tpP5QKsR6y8aF91SyZOmFBXPTiBc7HJAulWmgOosYeJ1xuTfUVvahqCCR 8OSpcvtCpg0ZM5qj9d6Z/CP0FRBA0HT/jZ61Hh5SDuC7wirqZbI9vQ3itsDtXQRVFBWy qKjkxTCL2HfsvIlL+F2Re0bgw49lOEdkHyXAN6XwiGRIc7TPL85M0iFZ3+wpOAeCH/lE ak0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fXQNLhXYbwRZ9WO7T+BhVV9+0Up77RvoHt23kV18qJk=; b=f/GPiX2HV/I3jYXyAxaFsIWiCG32U0xUFQvPVwrccOIc6d4318ndly4rXtUaRYGV7t zXZEgbsfWiPGhCF1Cy1hbgYV4+j9hvI9BpvYTl4YN6g4O2Ayt52h+I1qFuiiABi3vWVl yxB/ZIpkaEK/QbP2t3cIkiyVnZPiXv741j3EPVloWh3FqX7fU6Z22R1fa31+MlZsZkK7 DkZL/FUKYojwbDoziojk3WVECfR4Z+1RKpLR69ThcP7PnjVoogfs3CQcCrPRmzp5XW1G n/xUUQ33B14m85c9YAx2ulUJd7Q2ckEaOlukMje5Xb2rq/GXNwGf6GNb838u+rNeFmqY SjSQ== X-Gm-Message-State: AOAM5334A1NcEsw5x4NPF5S5b3BLLrqq175KfmUvjnSpBRJ/w+xj3TSQ SfgAxO6MhYj76azdxnPOXnzO6R80jriTtz+wmac= X-Google-Smtp-Source: ABdhPJxiFe+1lJNWil3Cq5CXwiktl5VzBVEi6d2/PgXW/oZTTQ9CYIZ62RK5Y+AlK0MyR/IHqrzN6ks2Bl//LVV81bg= X-Received: by 2002:adf:ed50:0:b0:203:da73:e0fd with SMTP id u16-20020adfed50000000b00203da73e0fdmr14169616wro.516.1648310193971; Sat, 26 Mar 2022 08:56:33 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 References: <088c8222-063a-1db5-da83-a5a0168d66c6@gmail.com> <639f7ce0-8a07-884c-c1cf-8257b9f3d9e8@gmail.com> <4da2302b-0745-ea1d-c868-5a8a5fc66b18@gmail.com> In-Reply-To: From: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Date: Sat, 26 Mar 2022 16:56:21 +0100 Message-ID: Subject: Re: ZFS + FreeBSD XEN dom0 panic To: Ze Dupsys Cc: freebsd-xen@freebsd.org, Brian Buhrow Content-Type: multipart/mixed; boundary="00000000000097864305db211f9b" X-Rspamd-Queue-Id: 4KQk953l8Zz3vR2 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=pOBjalGL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of royger@gmail.com designates 2a00:1450:4864:20::42d as permitted sender) smtp.mailfrom=royger@gmail.com X-Spamd-Result: default: False [-1.35 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_ATTACHMENT(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; R_MIXED_CHARSET(0.56)[subject]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FREEFALL_USER(0.00)[royger]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_SPAM_SHORT(1.00)[0.999]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; MIME_UNKNOWN(0.10)[application/x-patch]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::42d:from]; MLMMJ_DEST(0.00)[freebsd-xen]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --00000000000097864305db211f9b Content-Type: multipart/alternative; boundary="00000000000097864105db211f99" --00000000000097864105db211f99 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable El ds., 26 de mar=C3=A7 2022, 15:39, Roger Pau Monn=C3=A9 va escriure: > On Sat, Mar 26, 2022 at 02:08:06PM +0200, Ze Dupsys wrote: > > On 2022.03.26. 11:11, Roger Pau Monn=C3=A9 wrote: > > > > > > Hm, do you think you could upload (or attach) your > > > /usr/lib/debug/boot/kernel/kernel.debug and provide an updated panic > > > trace using that same exact kernel? > > > > Yes, it is just too big for email attachment. > > Uploaded at: https://files.fm/f/mp3v3qa22 > > > > This time i starved Dom0 of RAM(2G) to speed panic up. Panic trace it t= he > > same. > > > > Trace: > > Fatal trap 12: page fault while in kernel mode > > cpuid =3D 2; apic id =3D 04 > > fault virtual address =3D 0x22710028 > > fault code =3D supervisor read data, page not present > > instruction pointer =3D 0x20:0xffffffff80c6a2b2 > > stack pointer =3D 0x28:0xfffffe009e486b30 > > frame pointer =3D 0x28:0xfffffe009e486b30 > > code segment =3D base 0x0, limit 0xfffff, type 0x1b > > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > > current process =3D 3995 (devmatch) > > trap number =3D 12 > > panic: page fault > > cpuid =3D 2 > > time =3D 1648293768 > > KDB: stack backtrace: > > #0 0xffffffff80c7c285 at kdb_backtrace+0x65 > > #1 0xffffffff80c2e2e1 at vpanic+0x181 > > #2 0xffffffff80c2e153 at panic+0x43 > > #3 0xffffffff810c8b97 at trap+0xba7 > > #4 0xffffffff810c8bef at trap+0xbff > > #5 0xffffffff810c8243 at trap+0x253 > > #6 0xffffffff810a0848 at calltrap+0x8 > > #7 0xffffffff80c86ed1 at rman_is_region_manager+0x241 > > #8 0xffffffff80c3eb41 at sbuf_new_for_sysctl+0x101 > > #9 0xffffffff80c3df8c at kernel_sysctl+0x3ec > > #10 0xffffffff80c3e603 at userland_sysctl+0x173 > > #11 0xffffffff80c3e44f at sys___sysctl+0x5f > > #12 0xffffffff810c949c at amd64_syscall+0x10c > > #13 0xffffffff810a115b at Xfast_syscall+0xfb > > Uptime: 10m19s > > It's weird, because here you get a page fault, but there are also > traces with: > > general protection fault while in kernel mode > cpuid =3D 3; a(d8) Scan for VGA option rom > pic id =3D 06 > instruction pointer =3D 0x20:0xffffffff810c5d64 > stack pointer =3D 0x28:0xfffffe00a20fe990 > frame pointer =3D 0x28:0xfffffe00a20fe990 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 8998 (devmatch) > trap number =3D 9 > panic: general protection fault > cpuid =3D 3 > time =3D 1647416577 > KDB: stack backtrace: > #0 0xffffffff80c7ca05 at kdb_backtrace+0x65 > #1 0xffffffff80c2ea11 at vpanic+0x181 > #2 0xffffffff80c2e883 at panic+0x43 > #3 0xffffffff810c9b97 at trap+0xba7 > #4 0xffffffff810c907b at trap+0x8b > #5 0xffffffff810a0dc8 at calltrap+0x8 > #6 0xffffffff80c83067 at kvprintf+0x1007 > #7 0xffffffff80c83df9 at snprintf+0x59 > #8 0xffffffff80c8768b at rman_is_region_manager+0x27b > #9 0xffffffff80c3f271 at sbuf_new_for_sysctl+0x101 > #10 0xffffffff80c3e6bc at kernel_sysctl+0x3ec > #11 0xffffffff80c3ed33 at userland_sysctl+0x173 > #12 0xffffffff80c3eb7f at sys___sysctl+0x5f > #13 0xffffffff810ca49c at amd64_syscall+0x10c > #14 0xffffffff810a16db at Xfast_syscall+0xfb > > That show a general protection fault instead of a page fault. > > I've built an hypervisor with debug enabled for you, it's at: > > https://people.freebsd.org/~royger/xen-debug > > This is the same as the one in ports, just build with debug=3Dy. If you > can place it in /boot/ and change your xen_kernel to: > > xen_kernel=3D"/boot/xen-debug" > > It might provide some additional info. > > I've also noticed it seems to always be 'devmatch' the process that > triggers the panic. > > > > > cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs > addr2line > > -e /usr/lib/debug/boot/kernel/kernel.debug > > /usr/src/sys/kern/subr_kdb.c:443 > > /usr/src/sys/kern/kern_shutdown.c:0 > > /usr/src/sys/kern/kern_shutdown.c:844 > > /usr/src/sys/amd64/amd64/trap.c:944 > > /usr/src/sys/amd64/amd64/trap.c:0 > > /usr/src/sys/amd64/amd64/trap.c:0 > > /usr/src/sys/amd64/amd64/exception.S:292 > > /usr/src/sys/kern/subr_rman.c:0 > > I've been able to get a better trace with gdb and your debug symbols, > and this is: > > (gdb) info line *0xffffffff80c6a2b2 > Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address > 0xffffffff80c6a2b2 > and ends at 0xffffffff80c6a2b6 . > (gdb) info line *0xffffffff80c86ed1 > Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address > 0xffffffff80c86ecc > and ends at 0xffffffff80c86ed5 . > > The page fault happens exactly at: > > https://cgit.freebsd.org/src/tree/sys/kern/subr_bus.c?h=3Dstable/13#n1386 > > Which is called from > > https://cgit.freebsd.org/src/tree/sys/kern/subr_rman.c?h=3Dstable/13#n105= 2 > > I'm trying to figure out how the device could be removed or > disconnected from the rman. I will try to create a patch to catch the > device that leaves rman regions when destroyed/removed. Replying from my phone so the format will likely be mangled. I think I've found at least one issue with blkback leaking resources on destroy if the ring was not connected. Could you give the following patch a try? I've just build tested it, so can't guarantee it will work. Thanks, Roger. --00000000000097864105db211f99 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
El ds., 26 de mar=C3=A7 2022, 15:39, Roger Pau Monn=C3=A9 <= roger.pau@citrix.com> va escriure:
On Sat, Mar 26, 2022 at 02:08:06PM +0200, Ze Dupsys wro= te:
> On 2022.03.26. 11:11, Roger Pau Monn=C3=A9 wrote:
> >
> > Hm, do you think you could upload (or attach) your
> > /usr/lib/debug/boot/kernel/kernel.debug and provide an updated pa= nic
> > trace using that same exact kernel?
>
> Yes, it is just too big for email attachment.
> Uploaded at: https://files.fm/f/mp= 3v3qa22
>
> This time i starved Dom0 of RAM(2G) to speed panic up. Panic trace it = the
> same.
>
> Trace:
> Fatal trap 12: page fault while in kernel mode
> cpuid =3D 2; apic id =3D 04
> fault virtual address =3D 0x22710028
> fault code=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D supervisor rea= d data, page not present
> instruction pointer=C2=A0 =C2=A0=3D 0x20:0xffffffff80c6a2b2
> stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe009e48= 6b30
> frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe009e48= 6b30
> code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base 0x0, limit 0xf= ffff, type 0x1b
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0=3D DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags=C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, IO= PL =3D 0
> current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =3D 3995 (devmatch)
> trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 12
> panic: page fault
> cpuid =3D 2
> time =3D 1648293768
> KDB: stack backtrace:
> #0 0xffffffff80c7c285 at kdb_backtrace+0x65
> #1 0xffffffff80c2e2e1 at vpanic+0x181
> #2 0xffffffff80c2e153 at panic+0x43
> #3 0xffffffff810c8b97 at trap+0xba7
> #4 0xffffffff810c8bef at trap+0xbff
> #5 0xffffffff810c8243 at trap+0x253
> #6 0xffffffff810a0848 at calltrap+0x8
> #7 0xffffffff80c86ed1 at rman_is_region_manager+0x241
> #8 0xffffffff80c3eb41 at sbuf_new_for_sysctl+0x101
> #9 0xffffffff80c3df8c at kernel_sysctl+0x3ec
> #10 0xffffffff80c3e603 at userland_sysctl+0x173
> #11 0xffffffff80c3e44f at sys___sysctl+0x5f
> #12 0xffffffff810c949c at amd64_syscall+0x10c
> #13 0xffffffff810a115b at Xfast_syscall+0xfb
> Uptime: 10m19s

It's weird, because here you get a page fault, but there are also
traces with:

general protection fault while in kernel mode
cpuid =3D 3; a(d8) Scan for VGA option rom
pic id =3D 06
instruction pointer=C2=A0 =C2=A0 =C2=A0=3D 0x20:0xffffffff810c5d64
stack pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00a2= 0fe990
frame pointer=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 0x28:0xfffffe00a2= 0fe990
code segment=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D base 0x0, limit 0= xfffff, type 0x1b
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =3D DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags=C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D interrupt enabled, resume, = IOPL =3D 0
current process=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 8998 (devmatch)
trap number=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D 9
panic: general protection fault
cpuid =3D 3
time =3D 1647416577
KDB: stack backtrace:
#0 0xffffffff80c7ca05 at kdb_backtrace+0x65
#1 0xffffffff80c2ea11 at vpanic+0x181
#2 0xffffffff80c2e883 at panic+0x43
#3 0xffffffff810c9b97 at trap+0xba7
#4 0xffffffff810c907b at trap+0x8b
#5 0xffffffff810a0dc8 at calltrap+0x8
#6 0xffffffff80c83067 at kvprintf+0x1007
#7 0xffffffff80c83df9 at snprintf+0x59
#8 0xffffffff80c8768b at rman_is_region_manager+0x27b
#9 0xffffffff80c3f271 at sbuf_new_for_sysctl+0x101
#10 0xffffffff80c3e6bc at kernel_sysctl+0x3ec
#11 0xffffffff80c3ed33 at userland_sysctl+0x173
#12 0xffffffff80c3eb7f at sys___sysctl+0x5f
#13 0xffffffff810ca49c at amd64_syscall+0x10c
#14 0xffffffff810a16db at Xfast_syscall+0xfb

That show a general protection fault instead of a page fault.

I've built an hypervisor with debug enabled for you, it's at:

https://people.freebsd.= org/~royger/xen-debug

This is the same as the one in ports, just build with debug=3Dy. If you
can place it in /boot/ and change your xen_kernel to:

xen_kernel=3D"/boot/xen-debug"

It might provide some additional info.

I've also noticed it seems to always be 'devmatch' the process = that
triggers the panic.

>
> cat /tmp/panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//= 9; | xargs addr2line
> -e /usr/lib/debug/boot/kernel/kernel.debug
> /usr/src/sys/kern/subr_kdb.c:443
> /usr/src/sys/kern/kern_shutdown.c:0
> /usr/src/sys/kern/kern_shutdown.c:844
> /usr/src/sys/amd64/amd64/trap.c:944
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/trap.c:0
> /usr/src/sys/amd64/amd64/exception.S:292
> /usr/src/sys/kern/subr_rman.c:0

I've been able to get a better trace with gdb and your debug symbols, and this is:

(gdb) info line *0xffffffff80c6a2b2
Line 1386 of "/usr/src/sys/kern/subr_bus.c" starts at address 0xf= fffffff80c6a2b2 <device_get_name+18>
=C2=A0 =C2=A0and ends at 0xffffffff80c6a2b6 <device_get_name+22>.
(gdb) info line *0xffffffff80c86ed1
Line 1052 of "/usr/src/sys/kern/subr_rman.c" starts at address 0x= ffffffff80c86ecc <sysctl_rman+540>
=C2=A0 =C2=A0and ends at 0xffffffff80c86ed5 <sysctl_rman+549>.

The page fault happens exactly at:

https://cgit.freebsd.org/src/tree/sys/kern/subr_bus.c?h=3Dstable/13#n= 1386

Which is called from

https://cgit.freebsd.org/src/tree/sys/kern/subr_rman.c?h=3Dstable/13= #n1052

I'm trying to figure out how the device could be removed or
disconnected from the rman. I will try to create a patch to catch the
device that leaves rman regions when destroyed/removed.
<= /div>

Replying from my phone s= o the format will likely be mangled.=C2=A0

I think I've found at least one issue with blkback l= eaking resources on destroy if the ring was not connected. Could you give t= he following patch a try? I've just build tested it, so can't guara= ntee it will work.=C2=A0

Thanks, Roger.=C2=A0
--00000000000097864105db211f99-- --00000000000097864305db211f9b Content-Type: application/x-patch; name="blkback.patch" Content-Disposition: attachment; filename="blkback.patch" Content-Transfer-Encoding: base64 Content-ID: <17fc6eb5f8e17e280a21> X-Attachment-Id: 17fc6eb5f8e17e280a21 ZGlmZiAtLWdpdCBhL3N5cy9kZXYveGVuL2Jsa2JhY2svYmxrYmFjay5jIGIvc3lzL2Rldi94ZW4v YmxrYmFjay9ibGtiYWNrLmMKaW5kZXggMzM0MTQyOTViZjUuLjY2NGY1MmE3NGU3IDEwMDY0NAot LS0gYS9zeXMvZGV2L3hlbi9ibGtiYWNrL2Jsa2JhY2suYworKysgYi9zeXMvZGV2L3hlbi9ibGti YWNrL2Jsa2JhY2suYwpAQCAtMjc4MSw5ICsyNzgxLDYgQEAgeGJiX2Rpc2Nvbm5lY3Qoc3RydWN0 IHhiYl9zb2Z0YyAqeGJiKQogCiAJRFBSSU5URigiXG4iKTsKIAotCWlmICgoeGJiLT5mbGFncyAm IFhCQkZfUklOR19DT05ORUNURUQpID09IDApCi0JCXJldHVybiAoMCk7Ci0KIAltdHhfdW5sb2Nr KCZ4YmItPmxvY2spOwogCXhlbl9pbnRyX3VuYmluZCgmeGJiLT54ZW5faW50cl9oYW5kbGUpOwog CXRhc2txdWV1ZV9kcmFpbih4YmItPmlvX3Rhc2txdWV1ZSwgJnhiYi0+aW9fdGFzayk7IAo= --00000000000097864305db211f9b--