From nobody Thu Apr 14 10:06:50 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5DDA21B3E7F2 for ; Thu, 14 Apr 2022 10:06:55 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KfFVk1y1Dz3jMs for ; Thu, 14 Apr 2022 10:06:54 +0000 (UTC) (envelope-from zedupsys@gmail.com) Received: by mail-lf1-x132.google.com with SMTP id p10so8164914lfa.12 for ; Thu, 14 Apr 2022 03:06:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=reply-to:to:cc:from:subject:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=17ErmlYZWm67AosXkFx8xcbRBU0XQ89bgQt2HvSAZRI=; b=PY+rBUWiysG75qkvBPelC186iHD/2AwVFQFJel1nCGdPylliIen7dAMM661ta4mY2g h9E7EAc0BIt1A2i4nQHVBxDMpvwxJxR0i0thF6GYgGvp52zxSR3bm7JoJV4JM3oP7NB1 sa5HyFmfXHUfOC5kHrKpjyr0RRdjOmrVN4jfoet226cs9oRe6ou99qO2gEUI+6qS6n9z Zl76sbMxOnXCN7VOQs8FDWRMsAyjkp9p9nNelku1w+fdTNCLRtrjW1oLhafjRQJ0YVea 6n5UVpJm1gNphTDbMmZ9wKj7oXwYMtxeraOtlbKOHjWPuxJl5VMgMMXJZfRdIMNao38Q EqVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:to:cc:from:subject:message-id:date :user-agent:mime-version:content-language:content-transfer-encoding; bh=17ErmlYZWm67AosXkFx8xcbRBU0XQ89bgQt2HvSAZRI=; b=iwQjSsCZnkATjwpt7BV58Me8NZxw2boG/RUr50jMyXsXf+2aGnIOSsRIMlOaYz2i6G i7mSbruTK5pLCtS2bnrY4tW9hddDiThecgSe4Y7pM0A7pewAdAAI6/CctFaoPejrXYgR zgpENLWh+nwnQHG89EJE/qqPjnCpEG7BFTrG79lrXPLUg5BuC3mfoo7xTOcVe69tT9q+ TIUuo6cuAKFvXQ6f1I+Ap4XUy+85zkd0OndFdabqcU7I7JRmSR6q3qJtm7J/WfG37x7P K8eG6WHByzWcWeOQNvd8tGeh8Tp9VTSpfb6PyE/v/jpWtvNZsvTmlzMXrnL7PolKwrZM ETYA== X-Gm-Message-State: AOAM530QuNk6tiu+yg5JDH1/IR0XNWTClO59lj/NdUKMCGvTNtWkSNIp Nb156Rdyr0F4OJcOoFQ5x6mhmRsDN2Y= X-Google-Smtp-Source: ABdhPJwUfsD17emEklh0pZdr751q5MhjQzoDt0piKRWjmDgPaCgEStnp9vdQssJwwke7nooyFHjUag== X-Received: by 2002:ac2:4c51:0:b0:44a:34b8:fd72 with SMTP id o17-20020ac24c51000000b0044a34b8fd72mr1433208lfk.360.1649930812791; Thu, 14 Apr 2022 03:06:52 -0700 (PDT) Received: from [10.3.0.1] ([213.110.65.3]) by smtp.googlemail.com with ESMTPSA id f3-20020a056512322300b0046bbce218f2sm190184lfe.79.2022.04.14.03.06.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 14 Apr 2022 03:06:52 -0700 (PDT) Reply-To: zedupsys@gmail.com To: freebsd-fs@freebsd.org Cc: roger.pau@citrix.com From: Ze Dupsys Subject: ZFS, kernel panic due to unconditional NULL de-reference for (v)db Message-ID: <741ca49e-9ebd-be92-1389-4ab2227e6cb7@gmail.com> Date: Thu, 14 Apr 2022 13:06:50 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4KfFVk1y1Dz3jMs X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=PY+rBUWi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zedupsys@gmail.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=zedupsys@gmail.com X-Spamd-Result: default: False [-2.58 / 15.00]; HAS_REPLYTO(0.00)[zedupsys@gmail.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.96)[-0.959]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_REPLYTO(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.38)[0.377]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::132:from]; MLMMJ_DEST(0.00)[freebsd-fs]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello everyone, In RELEASE-13.0 source, /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:4456, dbuf_write_children_ready(zio_t *zio, arc_buf_t *buf, void *vdb) .. dmu_buf_impl_t *db = vdb; .. ASSERT3U(db->db_level, >, 0); .. for (i = 0, bp = db->db.db_data; i < 1ULL << epbs; i++, bp++) { If vdb == NULL, this function panics. And this is what kgdb backtrace shows. #9 0xffffffff821dc99d in dbuf_write_children_ready (zio=, buf=, vdb=0x0) at /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:4642 We do not know the internals of ZFS and kgdb backtrace is somewhat imprecise, thus the question is, in which scenarios call to dbuf_write_children_ready could have vdb pointer set to NULL? Any hints, ideas? FWIW, more often than not this happens when machine is powered down, so maybe some data structure is "half-freed". The call into ZFS code happens through (*dev_data->csw->d_strategy)(bios[bio_idx]), at the moment we suspect that there might be something wrong with data provided to d_strategy, but have no clue what could cause vdb to be NULL in the code above. Panic info below. Thanks. Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0x68 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff821dc99d stack pointer = 0x28:0xfffffe00c6b497d0 frame pointer = 0x28:0xfffffe00c6b49870 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (xbbd26 taskq) trap number = 12 panic: page fault cpuid = 3 time = 1649915274 KDB: stack backtrace: #0 0xffffffff80c57385 at kdb_backtrace+0x65 #1 0xffffffff80c09d61 at vpanic+0x181 #2 0xffffffff80c09bd3 at panic+0x43 #3 0xffffffff8108b187 at trap+0xbc7 #4 0xffffffff8108b1df at trap+0xc1f #5 0xffffffff8108a83d at trap+0x27d #6 0xffffffff81061818 at calltrap+0x8 #7 0xffffffff821c035a at dmu_read+0x2a #8 0xffffffff8218da3a at zvol_geom_bio_strategy+0x2aa #9 0xffffffff80a7f074 at xbd_instance_create+0xa3d4 #10 0xffffffff80a7b00a at xbd_instance_create+0x636a #11 0xffffffff80c6b021 at taskqueue_run+0x2a1 #12 0xffffffff80c6c33c at taskqueue_thread_loop+0xac #13 0xffffffff80bc7c9e at fork_exit+0x7e #14 0xffffffff8106289e at fork_trampoline+0xe cat panic.log| sed -Ee 's/^#[0-9]* //' -e 's/ .*//' | xargs addr2line -e /usr/lib/debug/boot/kernel/kernel.debug /usr/src/sys/kern/subr_bus.c:2410 /usr/src/sys/kern/kern_racct.c:632 /usr/src/sys/kern/kern_racct.c:617 /usr/src/sys/dev/isci/isci_sysctl.c:92 /usr/src/sys/dev/isci/isci_sysctl.c:0 /usr/src/sys/dev/isci/isci_oem_parameters.c:130 /usr/src/sys/dev/hyperv/input/hv_kbd.c:540 ??:0 ??:0 /usr/src/sys/dev/xen/blkback/blkback.c:3083 /usr/src/sys/xen/xenbus/xenbusvar.h:96 /usr/src/sys/kern/subr_kobj.c:145 /usr/src/sys/kern/subr_module.c:255 /usr/src/sys/kern/kern_event.c:0 /usr/src/sys/dev/hyperv/pcib/vmbus_pcib.c:1158 (kgdb) backtrace #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c09956 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff80c09dd0 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff80c09bd3 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff8108b187 in trap_fatal (frame=0xfffffe00c6b49710, eva=104) at /usr/src/sys/amd64/amd64/trap.c:915 #6 0xffffffff8108b1df in trap_pfault (frame=frame@entry=0xfffffe00c6b49710, usermode=false, signo=, signo@entry=0x0, ucode=, ucode@entry=0x0) at /usr/src/sys/amd64/amd64/trap.c:732 #7 0xffffffff8108a83d in trap (frame=0xfffffe00c6b49710) at /usr/src/sys/amd64/amd64/trap.c:398 #8 #9 0xffffffff821dc99d in dbuf_write_children_ready (zio=, buf=, vdb=0x0) at /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:4642 #10 0xffffffff821c035a in arc_evict_impl (state=, spa=, bytes=, type=) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4377 #11 arc_evict_meta_balanced (meta_used=) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4443 #12 arc_evict_meta (meta_used=) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4533 #13 arc_evict () at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4627 #14 arc_evict_cb (arg=, zthr=) at /usr/src/sys/contrib/openzfs/module/zfs/arc.c:4938 #15 0xffffffff8218da3a in zfs_deleteextattr (ap=0x1430f6000) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:5592 #16 0xffffffff80a7f074 in xbb_dispatch_dev (xbb=0xfffff8011a6ff800, reqlist=, operation=, bio_flags=0) at /usr/src/sys/dev/xen/blkback/blkback.c:2207 #17 0xffffffff80a7b00a in xbb_dispatch_io (xbb=0xfffff8011a6ff800, reqlist=) at /usr/src/sys/dev/xen/blkback/blkback.c:1767 #18 xbb_run_queue (context=0xfffff8011a6ff800, pending=) at /usr/src/sys/dev/xen/blkback/blkback.c:1987 #19 0xffffffff80c6b021 in taskqueue_run_locked (queue=queue@entry=0xfffff8011a9f1e00) at /usr/src/sys/kern/subr_taskqueue.c:476 #20 0xffffffff80c6c33c in taskqueue_thread_loop (arg=, arg@entry=0xfffff8011a6ff800) at /usr/src/sys/kern/subr_taskqueue.c:793 #21 0xffffffff80bc7c9e in fork_exit (callout=0xffffffff80c6c290 , arg=0xfffff8011a6ff800, frame=0xfffffe00c6b49c00) at /usr/src/sys/kern/kern_fork.c:1069 #22