From nobody Sun Nov 12 16:51:40 2023 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SSzBq4phLz50H7p for ; Sun, 12 Nov 2023 16:52:51 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SSzBp6nPhz4mcS for ; Sun, 12 Nov 2023 16:52:50 +0000 (UTC) (envelope-from mavbsd@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=WZ7KPEqj; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::112d as permitted sender) smtp.mailfrom=mavbsd@gmail.com; dmarc=none Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-5a84204e7aeso42830197b3.0 for ; Sun, 12 Nov 2023 08:52:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699807970; x=1700412770; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=eQpKBmFIcakM2IGrHJcFnJcPmjrbVDdv3xdprJ/BGsU=; b=WZ7KPEqjmf7tx+DbCM9gN6LA9OFD3vlt8Dgiqe1Hja1PJfU9nOpT6sdM0W8V/FrP5H OiZ3xat02VtBtlJclH37j6hmAV+oN7VUPduBf9cjuwRUMsuKXJGqutXT2tdNq5VOyCL1 lfnCHLyr4XmPMNm/A6oBS55ZFeMsjscS4RjQ/D+wCW8kAnp7zJNAAjclbAP7hlCHkcKd REWPJb73eMCYp1So/I0XeSR+MOm5iKf32Cv9AHG7jh4fJ79S5qMCIcA7UVfURLh+Z/I/ 4+WjvkZNM82Y/H5Sfcf5kdDY/MXl4JhiIdAw3Ei39BTtndjDog06q1+WanV+mXuP28rF WFJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699807970; x=1700412770; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eQpKBmFIcakM2IGrHJcFnJcPmjrbVDdv3xdprJ/BGsU=; b=efbvSf5NqpqQ7ft2w29iNyNMk2oQGQ/+XoWGhjOpFXqY8hSAGDhGnSZD27Y7DL7I+X QIzDnE8z6SduCttfPzcTwG4fz8VQD40+9jKSKiQkcbfDCr5VSM7QVfPpehU+fxliu80W F7kgPxJNQ+AKOjbJnYTO7NqLxCr43MGBTDFDM04813E0CCDGp0WzkNtFCeVrHL/HGaV7 1w9J1M8uy8LRPijU8MBbpWddUPFL0H7Kzo8nMk68HDgWM5qO75gWl0ZoFpUGSqEDgdgj WS99zSfYKj9CMTMpDFlyDs3RxZ6E5jr/KrpLWCcFjsPjmxIQ0YWTNvP77QRdGacH4K6E 9K6A== X-Gm-Message-State: AOJu0Yz28u+yXy6wGQn6yr6CMHDJJg1WRlauIUs8WMG1HTylcka8zRxo KPwvfNbDWiQ/lCwQMiPnIvm7e6Jp8kE= X-Google-Smtp-Source: AGHT+IHRToxzUZHBfq1pJ7bm2vSH6NoSnQFM+yQWqhE8PfNmEt5BBfOX56uWoTP08lL1tacYijVnpA== X-Received: by 2002:a0d:cc53:0:b0:5af:b0ca:6950 with SMTP id o80-20020a0dcc53000000b005afb0ca6950mr4817399ywd.42.1699807969545; Sun, 12 Nov 2023 08:52:49 -0800 (PST) Received: from [10.230.45.5] ([38.32.73.2]) by smtp.gmail.com with ESMTPSA id j10-20020a81920a000000b005a8eadbadbesm1284671ywg.19.2023.11.12.08.52.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 12 Nov 2023 08:52:48 -0800 (PST) Message-ID: Date: Sun, 12 Nov 2023 11:51:40 -0500 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: crash zfs_clone_range() Content-Language: en-US To: Ronald Klop , current@freebsd.org References: <349700057.3452.1699611152405@localhost> From: Alexander Motin In-Reply-To: <349700057.3452.1699611152405@localhost> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_NA(0.00)[freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::112d:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MLMMJ_DEST(0.00)[current@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SSzBp6nPhz4mcS X-Spamd-Bar: --- Hi Ronald, As I can see, the clone request to ZFS came through nullfs, and it crashed immediately on enter. I've never been a VFS layer expert, but to me it may be a nullfs problem, not zfs. Is there chance you was (un-)mounting something when this happened? On 10.11.2023 05:12, Ronald Klop wrote: > Hi, > > Had this crash today on RPI4/15-CURRENT. > > FreeBSD rpi4 15.0-CURRENT FreeBSD 15.0-CURRENT #19 > main-b0203aaa46-dirty: Sat Nov  4 11:48:33 CET 2023 > ronald@rpi4:/home/ronald/dev/freebsd/obj/home/ronald/dev/freebsd/src/arm64.aarch64/sys/GENERIC-NODEBUG arm64 > > $ sysctl -a | grep bclon > vfs.zfs.bclone_enabled: 1 > > I started a jail with poudriere to build a package. The jail uses null > mounts over ZFS. > > [root]# cu -s 115200 -l /dev/cuaU0 > Connected > > db> bt > Tracing pid 95213 tid 100438 td 0xffff0000e1e97900 > db_trace_self() at db_trace_self > db_stack_trace() at db_stack_trace+0x120 > db_command() at db_command+0x2e4 > db_command_loop() at db_command_loop+0x58 > db_trap() at db_trap+0x100 > kdb_trap() at kdb_trap+0x334 > handle_el1h_sync() at handle_el1h_sync+0x18 > --- exception, esr 0xf2000000 > kdb_enter() at kdb_enter+0x48 > vpanic() at vpanic+0x1dc > panic() at panic+0x48 > data_abort() at data_abort+0x2fc > handle_el1h_sync() at handle_el1h_sync+0x18 > --- exception, esr 0x96000004 > rms_rlock() at rms_rlock+0x1c > zfs_clone_range() at zfs_clone_range+0x68 > zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0x19c > null_bypass() at null_bypass+0x118 > vn_copy_file_range() at vn_copy_file_range+0x18c > kern_copy_file_range() at kern_copy_file_range+0x36c > sys_copy_file_range() at sys_copy_file_range+0x8c > do_el0_sync() at do_el0_sync+0x634 > handle_el0_sync() at handle_el0_sync+0x48 > --- exception, esr 0x56000000 > > > Oh.. While typing this I rebooted the machine and it happened again. I > didn't start anything in particular although the machine runs some jails. > > x0: 0x00000000000000e0 >   x1: 0xffffa00090317a48 >   x2: 0xffffa000f79d4f00 >   x3: 0xffffa000c61a44a8 >   x4: 0xffff0000deefe460 ($d.2 + 0xdd776560) >   x5: 0xffffa001250e4c00 >   x6: 0xffff0000e54025b5 ($d.5 + 0xc) >   x7: 0x000000000000030a >   x8: 0xffff0000e1559000 ($d.2 + 0xdfdd1100) >   x9: 0x0000000000000001 >  x10: 0x0000000000000000 >  x11: 0x0000000000000001 >  x12: 0x0000000000000002 >  x13: 0x0000000000000000 >  x14: 0x0000000000000001 >  x15: 0x0000000000000000 >  x16: 0xffff0000016dce88 (__stop_set_modmetadata_set + 0x1310) >  x17: 0xffff0000004e0d44 (rms_rlock + 0x0) >  x18: 0xffff0000deefe280 ($d.2 + 0xdd776380) >  x19: 0x0000000000000000 >  x20: 0xffff0000deefe460 ($d.2 + 0xdd776560) >  x21: 0x7fffffffffffffff >  x22: 0xffffa00090317a48 >  x23: 0xffffa000f79d4f00 >  x24: 0xffffa001067ef910 >  x25: 0x00000000000000e0 >  x26: 0xffffa000158a8000 >  x27: 0x0000000000000000 >  x28: 0xffffa000158a8000 >  x29: 0xffff0000deefe280 ($d.2 + 0xdd776380) >   sp: 0xffff0000deefe280 >   lr: 0xffff000001623564 (zfs_clone_range + 0x6c) >  elr: 0xffff0000004e0d60 (rms_rlock + 0x1c) > spsr: 0x00000000a0000045 >  far: 0x0000000000000108 >  esr: 0x0000000096000004 > panic: data abort in critical section or under mutex > cpuid = 1 > time = 1699610885 > KDB: stack backtrace: > db_trace_self() at db_trace_self > db_trace_self_wrapper() at db_trace_self_wrapper+0x38 > vpanic() at vpanic+0x1a0 > panic() at panic+0x48 > data_abort() at data_abort+0x2fc > handle_el1h_sync() at handle_el1h_sync+0x18 > --- exception, esr 0x96000004 > rms_rlock() at rms_rlock+0x1c > zfs_clone_range() at zfs_clone_range+0x68 > zfs_freebsd_copy_file_range() at zfs_freebsd_copy_file_range+0x19c > null_bypass() at null_bypass+0x118 > vn_copy_file_range() at vn_copy_file_range+0x18c > kern_copy_file_range() at kern_copy_file_range+0x36c > sys_copy_file_range() at sys_copy_file_range+0x8c > do_el0_sync() at do_el0_sync+0x634 > handle_el0_sync() at handle_el0_sync+0x48 > --- exception, esr 0x56000000 > KDB: enter: panic > [ thread pid 3792 tid 100394 ] > Stopped at      kdb_enter+0x48: str     xzr, [x19, #768] > db> > > I'll keep the debugger open for a while. Can I type something for > additional info? > > Regards, > Ronald. -- Alexander Motin